While networking plays a key role in data center infrastructure, it is – for now – beyond the scope of Mesos to try to address the concerns of networking setup, topology and performance. However, Mesos can ease integrations with existing networking solutions and enable features, like IP per container, task-granular task isolation and service discovery. More often than not, it will be challenging to provide a one-size-fits-all networking solution. The requirements and available solutions will vary across all cloud-only, on-premise, and hybrid deployments.
One of the primary goals for the networking support in Mesos was to have a pluggable mechanism to allow users to enable custom networking solution as needed. As a result, several extensions were added to Mesos components in version 0.25.0 to enable networking support. Further, all the extensions are opt-in to allow older frameworks and applications without networking support to coexist with the newer ones.
The rest of this document describes the overall architecture of all the involved components, configuration steps for enabling IP-per-container, and required framework changes.
A key observation is that the networking support is enabled via a Mesos module and thus the Mesos master and agents are completely oblivious of it. It is completely up to the networking module to provide the desired support. Next, the IP requests are provided on a best effort manner. Thus, the framework should be willing to handle ignored (in cases where the module(s) are not present) or declined (the IPs can’t be assigned due to various reasons) requests.
To maximize backwards-compatibility with existing frameworks, schedulers must opt-in to network isolation per-container. Schedulers opt in to network isolation using new data structures in the TaskInfo message.
Isolator interfaceA Mesos framework uses the TaskInfo message to requests IPs for each container being launched. (The request is ignored if the Mesos cluster doesn’t have support for IP-per-container.)
Mesos Master processes TaskInfos and forwards them to the Agent for launching tasks.
The cleanup module gets notified if there is an Agent-lost event.
The cleanup module communicates with the IPAM client to release all IP address(es) associated with the lost Agent. The IPAM may have a grace period before the address(es) are recycled.
The network isolator module is not part of standard Mesos distribution. However, there is an example implementation at https://github.com/mesosphere/net-modules.
Once the network isolation module has been built into a shared dynamic library, we can load it into Mesos Agent (see modules documentation on instructions for building and loading a module).
A new NetworkInfo message has been introduced:
message NetworkInfo {
enum Protocol {
IPv4 = 1;
IPv6 = 2;
}
message IPAddress {
optional Protocol protocol = 1;
optional string ip_address = 2;
}
repeated IPAddress ip_addresses = 5;
optional string name = 6;
optional Protocol protocol = 1 [deprecated = true]; // Since 0.26.0
optional string ip_address = 2 [deprecated = true]; // Since 0.26.0
repeated string groups = 3;
optional Labels labels = 4;
};When requesting an IP address from the IPAM, one needs to set the protocol field to IPv4 or IPv6. Setting ip_address to a valid IP address allows the framework to specify a static IP address for the container (if supported by the NIM). This is helpful in situations where a task must be bound to a particular IP address even as it is killed and restarted on a different node.
Setting name to a valid network name allows the framework to specify a network for the container to join, it is up to the network isolator to decide how to interpret this field, e.g., network/cni isolator will interpret it as the name of a CNI network.
Frameworks wanting to enable IP per container, need to provide NetworkInfo message in TaskInfo. Here are a few examples:
A request for one address of unspecified protocol version using the default command executor
TaskInfo {
...
command: ...,
container: ContainerInfo {
network_infos: [
NetworkInfo {
ip_addresses: [
IPAddress {
protocol: None;
ip_address: None;
}
]
groups: [];
labels: None;
}
]
}
}A request for one IPv4 and one IPv6 address, in two groups using the default command executor
TaskInfo {
...
command: ...,
container: ContainerInfo {
network_infos: [
NetworkInfo {
ip_addresses: [
IPAddress {
protocol: IPv4;
ip_address: None;
},
IPAddress {
protocol: IPv6;
ip_address: None;
}
]
groups: ["dev", "test"];
labels: None;
}
]
}
}A request for two network interfaces, each with one IP address, each in a different network group using the default command executor
TaskInfo {
...
command: ...,
container: ContainerInfo {
network_infos: [
NetworkInfo {
ip_addresses: [
IPAddress {
protocol: None;
ip_address: None;
}
]
groups: ["foo"];
labels: None;
},
NetworkInfo {
ip_addresses: [
IPAddress {
protocol: None;
ip_address: None;
}
]
groups: ["bar"];
labels: None;
},
]
}
}A request for a specific IP address using a custom executor
TaskInfo {
...
executor: ExecutorInfo {
...,
container: ContainerInfo {
network_infos: [
NetworkInfo {
ip_addresses: [
IPAddress {
protocol: None;
ip_address: "10.1.2.3";
}
]
groups: [];
labels: None;
}
]
}
}
}A request for joining a specific network using the default command executor
TaskInfo {
...
command: ...,
container: ContainerInfo {
network_infos: [
NetworkInfo {
name: "network1";
}
]
}
}NOTE: The Mesos Containerizer will reject any CommandInfo that has a ContainerInfo. For this reason, when opting in to network isolation when using the Mesos Containerizer, set TaskInfo.ContainerInfo.NetworkInfo.
The NetworkInfo message allows frameworks to request IP address(es) to be assigned at task launch time on the Mesos agent. After opting in to network isolation for a given executor’s container in this way, frameworks will need to know what address(es) were ultimately assigned in order to perform health checks, or any other out-of-band communication.
This is accomplished by adding a new field to the TaskStatus message.
message ContainerStatus {
repeated NetworkInfo network_infos;
}
message TaskStatus {
...
optional ContainerStatus container;
...
};Further, the container IP addresses are also exposed via Master’s state endpoint. The JSON output from Master’s state endpoint contains a list of task statuses. If a task’s container was started with it’s own IP address, the assigned IP address will be exposed as part of the TASK_RUNNING status.
NOTE: Since per-container address(es) are strictly opt-in from the framework, the framework may ignore the IP address(es) provided in StatusUpdate if it didn’t set NetworkInfo in the first place.
A network isolator module implements the Isolator interface provided by Mesos. The module is loaded as a dynamic shared library in to the Mesos Agent and gets hooked up in the container launch sequence. A network isolator may communicate with external IPAM and network virtualizer tools to fulfill framework requirements.
In terms of the Isolator API, there are three key callbacks that a network isolator module should implement:
Isolator::prepare() provides the module with a chance to decide whether or not the enable network isolation for the given task container. If the network isolation is to be enabled, the Isolator::prepare call would inform the Agent to create a private network namespace for the coordinator. It is this interface, that will also generate an IP address (statically or with the help of an external IPAM agent) for the container.
Isolator::isolate() provide the module with the opportunity to isolate the container after it has been created but before the executor is launched inside the container. This would involve creating virtual ethernet adapter for the container and assigning it an IP address. The module can also use help of an external network virtualizer/isolator for setting up network for the container.
Isolator::cleanup() is called when the container terminates. This allows the module to perform any cleanups such as recovering resources and releasing IP addresses as needed.