This document serves as a guide for users who wish to upgrade an existing Mesos cluster. Some versions require particular upgrade techniques when upgrading a running cluster. Some upgrades will have incompatible changes.
This section provides an overview of the changes for each version (in particular when upgrading from the next lower version). For more details please check the respective sections below.
We categorize the changes as follows:
A New feature/behavior
C Changed feature/behavior
D Deprecated feature/behavior
R Removed feature/behavior| Version | Mesos Core | Flags | Framework API | Module API | Endpoints |
|---|---|---|---|---|---|
| 1.7.x |
|
|
|
|
|
| 1.6.x |
|
|
|
|
|
| 1.5.x |
|
|
|
|
|
| 1.4.x |
|
|
|
|
|
| 1.3.x |
|
|
|
|
|
| 1.2.x |
|
|
|
|
|
| 1.1.x |
|
||||
| 1.0.x |
|
|
|
|
|
| 0.28.x |
|
|
|||
| 0.27.x |
|
|
|
||
| 0.26.x |
|
|
|||
| 0.25.x |
|
|
linux/devices isolator has been added. This isolator automatically populates containers with devices that have been whitelisted with the --allowed_devices agent flag.cgroups/all has been added to the agent flag --isolation. This allows cgroups isolator to automatically load all the local enabled cgroups subsystems. If this option is specified in the agent flag --isolation along with other cgroups related options (e.g., cgroups/cpu), those options will be just ignored./sys/fs/cgroup to containers with image launched by Mesos containerizer.HOST_PATH, SANDBOX_PATH, IMAGE, SECRET, and DOCKER_VOLUME volumes were always mounted for container in read-write mode, i.e., the Volume.mode field was not honored. Now we will mount these volumes based on the Volume.mode field so framework can choose to mount the volume for the container in either read-write mode or read-only mode.CREATE_VOLUME, CREATE_BLOCK, DESTROY_VOLUME and DESTROY_BLOCK operations:
CREATE_DISK to create a MOUNT or BLOCK disk resource from a RAW disk resource. The CreateMountDisk and CreateBlockDisk ACLs control which principals are allowed to create MOUNT or BLOCK disks for which roles.DESTROY_DISK to reclaim a MOUNT or BLOCK disk resource back to a RAW disk resource. The DestroyMountDisk and DestroyBlockDisk ACLs control which principals are allowed to reclaim MOUNT or BLOCK disks for which roles.ViewResourceProvider ACL has been introduced to control which principals are allowed to call the GET_RESOURCE_PROVIDERS agent API.--enforce_container_ports flag has been added to toggle whether the network/ports isolator should enforce TCP ports usage limits.--gc_non_executor_container_sandboxes agent flag has been added to garbage collect the sandboxes of nested containers, which includes the tasks groups launched by the default executor. We recommend enabling the flag if you have frameworks that launch multiple task groups on the same default executor instance.ContainerLogger module interface has been changed. The prepare() method now takes ContainerID and ContainerConfig instead.Isolator::recover() has been updated to take an std::vector instead of std::list of container states.--fetcher_stall_timeout has been added. This flag specifies the amount of time for the container image and artifact fetchers to wait before aborting a stalled download (i.e., the speed keeps below one byte per second). NOTE: This flag only applies when downloading data from the net and does not apply to HDFS.disk_profile_adaptor.hpp for interface changes.--xfs_kill_containers has been added. By setting this flag, the disk/xfs isolator will now kill containers that exceed the disk limit.TASK_STARTING status update for every task they’ve successfully received and are about to start. The possibility of any executor sending this update has been documented since the beginning of Mesos, but prior to this version the built-in executors did not actually send it. This means that all schedulers using one of the built-in executors must be upgraded to expect TASK_STARTING updates before upgrading Mesos itself.limitation, was added to the TaskStatus message. This field is a TaskResourceLimitation message that describes the resources that caused a task to fail with a resource limitation reason.network/ports isolator has been added. The isolator supports the following new agent flags:
--container_ports_watch_interval specifies the interval at which the isolator reconciles port assignments.--check_agent_port_range_only excludes ports outside the agent’s range from port reconciliation.--executor_secret_key has been deprecated. Operators should use --jwt_secret_key instead.Resource.disk.source.path.root and Resource.disk.source.mount.root can now be set to relative paths to an agent’s work directory. The containerizers will interpret the paths based on the --work_dir flag on an agent.GET_CONTAINERS has been updated to support listing nested or standalone containers. One can specify the following fields in the request:
show_nested: Whether to show nested containers.show_standalone: Whether to show standalone containers.--reconfiguration_policy has been added. By setting the value of this flag to additive, operators can allow the agent to be restarted with increased resources without requiring the agent ID to be changed. Note that if this feature is used, the master version is required to be >= 1.5 as well.catchup() method has been added to the replicated log reader API. The method allows to catch-up positions missing in the local non-leading replica to allow safe eventually consistent reads from it. Note about backwards compatibility: In order for the feature to work correctly in presence of log truncations all log replicas need to be updated.mesos-agent kernel supports ambient capabilities (Linux 4.3 or later), the capabilities specified in the LinuxInfo.effective_capabilities message will be made ambient in the container task.LinuxInfo.bounding_capabilities message. Operators can specify the default bounding capabilities using the agent --bounding_capabilities flag. This flag also specifies the maximum bounding set that a framework is allowed to specify.LinuxInfo.capabilities field has been deprecated in favor of LinuxInfo.effective_capabilities.--effective_capabilities flag has been added to specify the default effective capability set for tasks.--bounding_capabilities flag has been added to specify the default bounding capability set for tasks.--allowed-capabilities flag has been deprecated in favor of --effective_capabilities.Allocator::updateSlave was change. While previously the passed value denoted a new amount of oversubscribed (revocable) resources on the agent, it now denotes the new amount of total resources on the agent. This requires custom allocator implementations to update their interpretation of the passed value.--no-enforce_container_disk_quota option to efficiently measure disk resource usage without enforcing any usage limits.Resources class in the internal Mesos C++ library changed its behavior to only support post-RESERVATION_REFINEMENT format. If a framework is using this internal utility, it is likely to break if the RESERVATION_REFINEMENT capability is not enabled.set_quotas and remove_quotas has been removed from the local authorizer. Before upgrading the Mesos binaries, consolidate the ACLs used under set_quotas and remove_quotes under their replacement ACL update_quotas. After consolidation of the ACLs, the binaries could be safely replaced.shutdown_frameworks has been removed from the local authorizer. Before upgrading the Mesos binaries, replace all instances of the ACL shutdown_frameworks with the newer ACL teardown_frameworks. After updating the ACLs, the binaries can be safely replaced.FrameworkInfo.role field in favor of FrameworkInfo.roles and the MULTI_ROLE capability. Frameworks using the new field can continue to use a single role.role field in the master and agent endpoints is deprecated in favor of roles. Any tooling parsing endpoint information and relying on the role field needs to be updated before multi-role frameworks can be safely run in the cluster.MULTI_ROLE framework capability. Also, the interface has changed.authenticate_http_executors and executor_secret_key: Used to enable required HTTP executor authentication and set the key file used for generation and authentication of HTTP executor tokens. Note that enabling these flags after upgrade is disruptive to HTTP executors that were launched before the upgrade. For more information on the recommended upgrade procedure when enabling these flags, see the authentication documentation.In order to upgrade a running cluster:
ContainerLogger’s prepare() method. The method now takes an additional argument for the user the logger should run a subprocess as. Please see MESOS-5856 for more information.allocator.hpp for interface changes.authorizer.proto for module interface changes, and acls.proto for corresponding LocalAuthorizer ACL changes.authorizer.proto for interface changes.recovered_frameworks in v1 GetFrameworks call. Now it will be empty.orphan_executors in v1 GetExecutors call. Now it will be empty.orphan_tasks in v1 GetTasks call. Now it will be empty.In order to upgrade a running cluster:
ContainerLogger’s recover() method. The ContainerLogger had an incomplete interface for a stateful implementation. This removes the incomplete parts to avoid adding tech debt in the containerizer. Please see MESOS-6371 for more information.offeredResources argument to the Allocator::updateAllocation() method. It is used to indicate the resources that the operations passed to updateAllocation() are applied to. MESOS-4431 (particularly /r/45961/) has more details on the motivation.SSL_ are used to control libprocess SSL support. However, it was found that those environment variables may collide with some libraries or programs (e.g., openssl, curl). From Mesos 1.0, SSL_* environment variables are deprecated in favor of the corresponding LIBPROCESS_SSL_* variables.ContainerConfig (see include/mesos/slave/isolator.proto):
required ExecutorInfo executorInfo = 1;optional TaskInfo taskInfo = 2;--executor_environment_variables flag on the agent to explicitly specify what environment variables the executors will get. Mesos generated environment variables (i.e., $MESOS_, $LIBPROCESS_) will not be affected. If $PATH is not specified for an executor, a default value /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin will be used.allocator/event_queue_dispatches is now deprecated. The new name is allocator/mesos/event_queue_dispatches to better support metrics for alternative allocator implementations.--docker_stop_timeout agent flag is deprecated.slave subcommand for mesos-cli.work_dir command-line flag. This flag is now required; the agent will exit immediately if it is not provided.registry_strict command-line flag. If this flag is specified, the master will exit immediately. Note that this flag was previously marked as experimental and not recommended for production use.200 OK to 202 Accepted:
/reserve/unreserve/create-volumes/destroy-volumesoutput_file field to CommandInfo.URI in Scheduler API and v1 Scheduler HTTP API.SetQuota and RemoveQuota ACLs have been deprecated. To replace these, a new ACL UpdateQuota have been introduced. In addition, a new ACL GetQuota have been added; these control which principals are allowed to query quota information for which roles. These changes affect the --acls flag for the local authorizer in the following ways:
update_quotas ACL cannot be used in combination with either the set_quotas or remove_quotas ACL. The local authorizer will produce an error in such a case;set_quotas or remove_quotas ACLs, the operator should first upgrade the Mesos binaries. At this point, the deprecated ACLs will still be enforced. After the upgrade has been verified, the operator should replace deprecated values for set_quotas and remove_quotas with equivalent values for update_quotas;get_quotas ACL after the upgrade to control which principals are allowed to query quota information.ObjectApprover. An ObjectApprover can be used to synchronously check authorizations for a given object and is hence useful when authorizing a large number of objects and/or large objects (which need to be copied using request-based authorization). NOTE: This is a breaking change for authorizer modules./state master endpoint: an operator can now authorize users to only see a subset of the running frameworks, tasks, or executors.subject and object fields in the authorization::Request protobuf message have been changed to be optional. If these fields are not set, the request should only be allowed for ACLs with ANY semantics. NOTE: This is a semantic change for authorizer modules.Allocator has been moved to be consistent with other packages.--authenticate_http flag has been deprecated in favor of --authenticate_http_readwrite. Setting --authenticate_http_readwrite will now enable authentication for all endpoints which previously had authentication support. These happen to be the endpoints which allow modification of the cluster state, or “read-write” endpoints. Note that /logging/toggle, /profiler/start, /profiler/stop, /maintenance/schedule, /machine/up, and /machine/down previously did not have authentication support, but in 1.0 if either --authenticate_http or --authenticate_http_readwrite is set, those endpoints will now require authentication. A new flag has also been introduced, --authenticate_http_readonly, which enables authentication for endpoints which support authentication and do not allow modification of the state of the cluster, like /state or /flags.Mesos 1.0 introduces authorization support for several HTTP endpoints. Note that some of these endpoints are used by the web UI, and thus using the web UI in a cluster with authorization enabled will require that ACLs be set appropriately. Please refer to the authorization documentation for details.
/files/debug/logging/toggle/metrics/snapshot/slave(id)/containers/slave(id)/monitor/statisticsIf the defined ACLs used permissive: false, the listed HTTP endpoints will stop working unless ACLs for the get_endpoints actions are defined.
In order to upgrade a running cluster:
ReserveResources and CreateVolume ACLs have been changed to roles. In both cases, principals can now be authorized to perform these operations for particular roles. This means that by default, a framework or operator can reserve resources/create volumes for any role. To restrict this behavior, ACLs can be added to the master which authorize principals to reserve resources/create volumes for specified roles only. Previously, frameworks could only reserve resources for their own role; this behavior can be preserved by configuring the ReserveResources ACLs such that the framework’s principal is only authorized to reserve for the framework’s role. NOTE This renders existing ReserveResources and CreateVolume ACL definitions obsolete; if you are authorizing these operations, your ACL definitions should be updated.In order to upgrade a running cluster:
--roles flag). In Mesos 0.27, if --roles is omitted, any role name can be used; controlling which principals are allowed to register as which roles should be done using ACLs. The role whitelist functionality is still supported but is deprecated.executorLost callback in the Scheduler interface will now be called whenever the agent detects termination of a custom executor. This callback was never called in previous versions, so please make sure any framework schedulers can now safely handle this callback. Note that this callback may not be reliably delivered.prepare interface has been changed slightly. Instead of keeping adding parameters to the prepare interface, we decide to use a protobuf (ContainerConfig). Also, we renamed ContainerPrepareInfo to ContainerLaunchInfo to better capture the purpose of this struct. See MESOS-4240 and MESOS-4282 for more information. If you are an isolator module writer, you will have to adjust your isolator module according to the new interface and re-compile with 0.27.ACLs.shutdown_frameworks has been deprecated in favor of the new ACLs.teardown_frameworks. This affects the --acls master flag for the local authorizer.
Reserved resources are now accounted for in the DRF role sorter. Previously unaccounted reservations will influence the weighted DRF sorter. If role weights were explicitly set, they may need to be adjusted in order to account for the reserved resources in the cluster.
In order to upgrade a running cluster:
The names of some TaskStatus::Reason enums have been changed. But the tag numbers remain unchanged, so it is backwards compatible. Frameworks using the new version might need to do some compile time adjustments:
Credential protobuf has been changed. Credential field secret is now a string, it used to be bytes. This will affect framework developers and language bindings ought to update their generated protobuf with the new version. This fixes JSON based credentials file support./state endpoints on master and agent will no longer include data fields as part of the JSON models for ExecutorInfo and TaskInfo out of consideration for memory scalability (see MESOS-3794 and this email thread).
data field was originally found via frameworks[*].executors[*].data.data field was originally found via executors[*].tasks[*].data.NetworkInfo protobuf has been changed. The fields protocol and ip_address are now deprecated. The new field ip_addresses subsumes the information provided by them.In order to upgrade a running cluster:
The following endpoints will be deprecated in favor of new endpoints. Both versions will be available in 0.25 but the deprecated endpoints will be removed in a subsequent release.
For master endpoints:
For agent endpoints:
For both master and agent:
In order to upgrade a running cluster:
Support for live upgrading a driver based scheduler to HTTP based (experimental) scheduler has been added.
Master now publishes its information in ZooKeeper in JSON (instead of protobuf). Make sure schedulers are linked against >= 0.23.0 libmesos before upgrading the master.
In order to upgrade a running cluster:
The ‘stats.json’ endpoints for masters and agents have been removed. Please use the ‘metrics/snapshot’ endpoints instead.
The ‘/master/shutdown’ endpoint is deprecated in favor of the new ‘/master/teardown’ endpoint.
In order to enable decorator modules to remove metadata (environment variables or labels), we changed the meaning of the return value for decorator hooks in Mesos 0.23.0. Please refer to the modules documentation for more details.
Agent ping timeouts are now configurable on the master via --slave_ping_timeout and --max_slave_ping_timeouts. Agents should be upgraded to 0.23.x before changing these flags.
A new scheduler driver API, acceptOffers, has been introduced. This is a more general version of the launchTasks API, which allows the scheduler to accept an offer and specify a list of operations (Offer.Operation) to perform using the resources in the offer. Currently, the supported operations include LAUNCH (launching tasks), RESERVE (making dynamic reservations), UNRESERVE (releasing dynamic reservations), CREATE (creating persistent volumes) and DESTROY (releasing persistent volumes). Similar to the launchTasks API, any unused resources will be considered declined, and the specified filters will be applied on all unused resources.
The Resource protobuf has been extended to include more metadata for supporting persistence (DiskInfo), dynamic reservations (ReservationInfo) and oversubscription (RevocableInfo). You must not combine two Resource objects if they have different metadata.
In order to upgrade a running cluster:
Agent checkpoint flag has been removed as it will be enabled for all agents. Frameworks must still enable checkpointing during registration to take advantage of checkpointing their tasks.
The stats.json endpoints for masters and agents have been deprecated. Please refer to the metrics/snapshot endpoint.
The C++/Java/Python scheduler bindings have been updated. In particular, the driver can be constructed with an additional argument that specifies whether to use implicit driver acknowledgements. In statusUpdate, the TaskStatus now includes a UUID to make explicit acknowledgements possible.
The Authentication API has changed slightly in this release to support additional authentication mechanisms. The change from ‘string’ to ‘bytes’ for AuthenticationStartMessage.data has no impact on C++ or the over-the-wire representation, so it only impacts pure language bindings for languages like Java and Python that use different types for UTF-8 strings vs. byte arrays.
message AuthenticationStartMessage { required string mechanism = 1; optional bytes data = 2; }
All Mesos arguments can now be passed using file:// to read them out of a file (either an absolute or relative path). The –credentials, –whitelist, and any flags that expect JSON backed arguments (such as –modules) behave as before, although support for just passing an absolute path for any JSON flags rather than file:// has been deprecated and will produce a warning (and the absolute path behavior will be removed in a future release).
In order to upgrade a running cluster:
MesosSchedulerDriverImpl with Credentials, your code must be updated to pass the implicitAcknowledgements argument before Credentials. You may run a 0.21.0 Python scheduler against a 0.22.0 master, and vice versa.In order to upgrade a running cluster:
The Mesos API has been changed slightly in this release. The CommandInfo has been changed (see below), which makes launching a command more flexible. The ‘value’ field has been changed from required to optional. However, it will not cause any issue during the upgrade (since the existing schedulers always set this field).
message CommandInfo {
...
// There are two ways to specify the command:
// 1) If 'shell == true', the command will be launched via shell
// (i.e., /bin/sh -c 'value'). The 'value' specified will be
// treated as the shell command. The 'arguments' will be ignored.
// 2) If 'shell == false', the command will be launched by passing
// arguments to an executable. The 'value' specified will be
// treated as the filename of the executable. The 'arguments'
// will be treated as the arguments to the executable. This is
// similar to how POSIX exec families launch processes (i.e.,
// execlp(value, arguments(0), arguments(1), ...)).
optional bool shell = 6 [default = true];
optional string value = 3;
repeated string arguments = 7;
...
}The Python bindings are also changing in this release. There are now sub-modules which allow you to use either the interfaces and/or the native driver.
import mesos.native for the native driversimport mesos.interface for the stub implementations and protobufsTo ensure a smooth upgrade, we recommend to upgrade your python framework and executor first. You will be able to either import using the new configuration or the old. Replace the existing imports with something like the following:
try: from mesos.native import MesosExecutorDriver, MesosSchedulerDriver from mesos.interface import Executor, Scheduler from mesos.interface import mesos_pb2 except ImportError: from mesos import Executor, MesosExecutorDriver, MesosSchedulerDriver, Scheduler import mesos_pb2
If you’re using a pure language binding, please ensure that it sends status update acknowledgements through the master before upgrading.
In order to upgrade a running cluster:
There are new required flags on the master (--work_dir and --quorum) to support the Registrar feature, which adds replicated state on the masters.
No required upgrade ordering across components.
In order to upgrade a running cluster:
In order to upgrade a running cluster:
In order to upgrade a running cluster:
In order to upgrade a running cluster:
reconcileTasks driver method.MesosSchedulerDriver constructor that takes Credential to authenticate.In order to upgrade a running cluster:
NOTE: After the restart unauthenticated frameworks will not be allowed to register.
In order to upgrade a running cluster:
In order to upgrade a running cluster:
In order to upgrade a running cluster: