Apache Mesos - Logging

Logging

Mesos handles the logs of each Mesos component differently depending on the degree of control Mesos has over the source code of the component.

Roughly, these categories are:

Internal - Master and Agent.
Containers - Executors and Tasks.
External - Components launched outside of Mesos, like Frameworks and ZooKeeper. These are expected to implement their own logging solution.

Internal

The Mesos Master and Agent use the Google’s logging library. For information regarding the command-line options used to configure this library, see the configuration documentation. Google logging options that are not explicitly mentioned there can be configured via environment variables.

Both Master and Agent also expose a /logging/toggle HTTP endpoint which temporarily toggles verbose logging:

POST <ip:port>/logging/toggle?level=[1|2|3]&duration=VALUE

The effect is analogous to setting the GLOG_v environment variable prior to starting the Master/Agent, except the logging level will revert to the original level after the given duration.

Containers

For background, see the containerizer documentation.

Mesos does not assume any structured logging for entities running inside containers. Instead, Mesos will store the stdout and stderr of containers into plain files (“stdout” and “stderr”) located inside the sandbox.

In some cases, the default Container logger behavior of Mesos is not ideal:

Logging may not be standardized across containers.
Logs are not easily aggregated.
Log file sizes are not managed. Given enough time, the “stdout” and “stderr” files can fill up the Agent’s disk.

`ContainerLogger` Module

The ContainerLogger module was introduced in Mesos 0.27.0 and aims to address the shortcomings of the default logging behavior for containers. The module can be used to change how Mesos redirects the stdout and stderr of containers.

The interface for a ContainerLogger can be found here.

Mesos comes with two ContainerLogger modules:

The SandboxContainerLogger implements the existing logging behavior as a ContainerLogger. This is the default behavior.
The LogrotateContainerLogger addresses the problem of unbounded log file sizes.

`LogrotateContainerLogger`

The LogrotateContainerLogger constrains the total size of a container’s stdout and stderr files. The module does this by rotating log files based on the parameters to the module. When a log file reaches its specified maximum size, it is renamed by appending a .N to the end of the filename, where N increments each rotation. Older log files are deleted when the specified maximum number of files is reached.

Invoking the module

The LogrotateContainerLogger can be loaded by specifying the library liblogrotate_container_logger.so in the --modules flag when starting the Agent and by setting the --container_logger Agent flag to org_apache_mesos_LogrotateContainerLogger.

Module parameters

Key	Explanation
`max_stdout_size`/`max_stderr_size`	Maximum size, in bytes, of a single stdout/stderr log file. When the size is reached, the file will be rotated. Defaults to 10 MB. Minimum size of 1 (memory) page, usually around 4 KB.
`logrotate_stdout_options`/ `logrotate_stderr_options`	Additional config options to pass into `logrotate` for stdout. This string will be inserted into a `logrotate` configuration file. i.e. For “stdout”: /path/to/stdout { [logrotate_stdout_options] size [max_stdout_size] } NOTE: The `size` option will be overridden by this module.
`environment_variable_prefix`	Prefix for environment variables meant to modify the behavior of the logrotate logger for the specific container being launched. The logger will look for four prefixed environment variables in the container’s `CommandInfo`’s `Environment`: `MAX_STDOUT_SIZE` `LOGROTATE_STDOUT_OPTIONS` `MAX_STDERR_SIZE` `LOGROTATE_STDERR_OPTIONS` If present, these variables will overwrite the global values set via module parameters. Defaults to `CONTAINER_LOGGER_`.
`launcher_dir`	Directory path of Mesos binaries. The `LogrotateContainerLogger` will find the `mesos-logrotate-logger` binary under this directory. Defaults to `/usr/local/libexec/mesos`.
`logrotate_path`	If specified, the `LogrotateContainerLogger` will use the specified `logrotate` instead of the system’s `logrotate`. If `logrotate` is not found, then the module will exit with an error.

How it works

Every time a container starts up, the LogrotateContainerLogger starts up companion subprocesses of the mesos-logrotate-logger binary.
The module instructs Mesos to redirect the container’s stdout/stderr to the mesos-logrotate-logger.
As the container outputs to stdout/stderr, mesos-logrotate-logger will pipe the output into the “stdout”/“stderr” files. As the files grow, mesos-logrotate-logger will call logrotate to keep the files strictly under the configured maximum size.
When the container exits, mesos-logrotate-logger will finish logging before exiting as well.

The LogrotateContainerLogger is designed to be resilient across Agent failover. If the Agent process dies, any instances of mesos-logrotate-logger will continue to run.

Writing a Custom `ContainerLogger`

For basics on module writing, see the modules documentation.

There are several caveats to consider when designing a new ContainerLogger:

Logging by the ContainerLogger should be resilient to Agent failover. If the Agent process dies (which includes the ContainerLogger module), logging should continue. This is usually achieved by using subprocesses.
When containers shut down, the ContainerLogger is not explicitly notified. Instead, encountering EOF in the container’s stdout/stderr signifies that the container has exited. This provides a stronger guarantee that the ContainerLogger has seen all the logs before exiting itself.
The ContainerLogger should not assume that containers have been launched with any specific ContainerLogger. The Agent may be restarted with a different ContainerLogger.
Each containerizer running on an Agent uses its own instance of the ContainerLogger. This means more than one ContainerLogger may be running in a single Agent. However, each Agent will only run a single type of ContainerLogger.