» Consul Connect Envoy

Command: consul connect envoy

The connect Envoy command is used to generate a bootstrap configuration for Envoy proxy for use with Consul Connect.

The default behaviour is to generate the necessary bootstrap configuration for Envoy based on the environment variables and options provided and by taking to the local Consul agent. It execs an external Envoy binary with that configuration leaving the Envoy process running in the foreground. An error is returned on operating systems other than linux or macOS since Envoy does not build for other platforms currently.

If the -bootstrap option is specified, the bootstrap config is generated in the same way and then printed to stdout. This allows it to be redirected to a file and used with envoy -c bootstrap.json. This works on all operating systems allowing configuration to be generated on a host that Envoy doesn't build on but then used in a virtualized environment that can run Envoy.

» Usage

Usage: consul connect envoy [options] [-- pass-through options]

» API Options

The standard API options are used to connect to the local agent to discover the proxy configuration needed.

  • -grpc-addr=<addr> - Address of the Consul agent with grpc port. This can be an IP address or DNS address, but it must include the port. This can also be specified via the CONSUL_GRPC_ADDR environment variable. In Consul 1.3 and later, the default value is http://127.0.0.1:8502, and https can optionally be used instead. The scheme can also be set to HTTPS by setting the environment variable CONSUL_HTTP_SSL=true. This may be a unix domain socket using unix:///path/to/socket if the agent is configured to listen that way.

  • -ca-file=<value> - Path to a CA file to use for TLS when communicating with Consul. This can also be specified via the CONSUL_CACERT environment variable.

  • -ca-path=<value> - Path to a directory of CA certificates to use for TLS when communicating with Consul. This can also be specified via the CONSUL_CAPATH environment variable.

  • -client-cert=<value> - Path to a client cert file to use for TLS when verify_incoming is enabled. This can also be specified via the CONSUL_CLIENT_CERT environment variable.

  • -client-key=<value> - Path to a client key file to use for TLS when verify_incoming is enabled. This can also be specified via the CONSUL_CLIENT_KEY environment variable.

  • -http-addr=<addr> - Address of the Consul agent with the port. This can be an IP address or DNS address, but it must include the port. This can also be specified via the CONSUL_HTTP_ADDR environment variable. In Consul 0.8 and later, the default value is http://127.0.0.1:8500, and https can optionally be used instead. The scheme can also be set to HTTPS by setting the environment variable CONSUL_HTTP_SSL=true. This may be a unix domain socket using unix:///path/to/socket if the agent is configured to listen that way.

  • -tls-server-name=<value> - The server name to use as the SNI host when connecting via TLS. This can also be specified via the CONSUL_TLS_SERVER_NAME environment variable.

  • -token=<value> - ACL token to use in the request. This can also be specified via the CONSUL_HTTP_TOKEN environment variable. If unspecified, the query will default to the token of the Consul agent at the HTTP address.

  • -token-file=<value> - File containing the ACL token to use in the request instead of one specified via the -token argument or CONSUL_HTTP_TOKEN environment variable. This can also be specified via the CONSUL_HTTP_TOKEN_FILE environment variable.

» Envoy Options

  • -sidecar-for - The ID (not name if they differ) of the service instance this proxy will represent. The target service doesn't need to exist on the local agent yet but a sidecar proxy registration with proxy.destination_service_id equal to the passed value must be present. If multiple proxy registrations targeting the same local service instance are present the command will error and -proxy-id should be used instead.

  • -proxy-id - The proxy service ID on the local agent. This must already be present on the local agent.

  • -envoy-binary - The full path to a specific Envoy binary to exec. By default the current $PATH is searched for envoy.

  • -admin-bind - The host:port to bind Envoy's admin HTTP API. Default is localhost:19000. Envoy requires that this be enabled. The host part must be resolvable DNS name or IP address.

  • -bootstrap - If present, the command will simply output the generated bootstrap config to stdout in JSON protobuf form. This can be directed to a file and used to start Envoy with envoy -c bootstrap.json.

  • -- [pass-through options] - Any options given after a double dash are passed directly through to the envoy invocation. See Envoy's documentation for more details. The command always specifies --config-file and --v2-config-only and by default passes --disable-hot-restart see hot restart.

» Examples

Assume a local service instance is registratered on the local agent with a sidecar proxy (using the sidecar service registration helper) as below.

service {
  name = "web"
  port = 8080
  connect { sidecar_service {} }
}

The sidecar Envoy process can be started with.

$ consul connect envoy -sidecar-for web

This example assumes that the correct environment variables are used to set the local agent connection information and ACL token, or that the agent is using all-default configuration.

To pass additional arguments directly to Envoy, for example output logging level, you can use:

$ consul connect envoy -sidecar-for web -- -l debug

To run multiple different proxy instances on the same host, you will need to use -admin-bind on all but one to ensure they don't attempt to bind to the same port as in the following example.

$ consul connect envoy -sidecar-for db -admin-bind localhost:19001

» Exec Security Details

The command needs to pass the bootstrap config through to Envoy. Envoy currently only supports passing this as a file path or passing a whole string on the command line with --config-yaml. Since the bootstrap needs to contain the ACL token to authorize the proxy, this secret needs careful handling.

Passing a secret via command option is unacceptable as on many unix systems these are readable to any user on the host for example via /proc or via a setuid process like ps.

Creating a temporary file is more secure in that it can only be read by the current user but risks leaving secret material on disk for an unbounded length of time and in a location that is opaque to the operator.

To work around these issues, the command currently creates a temporary file and immediately unlinks it so it can't be read by any other process that doesn't already have the file descriptor. It then writes the bootstrap JSON, and unsets the CLOEXEC bit on the file handle so that it remains available to the Envoy process after exec. Finally it execs Envoy with --config-file /dev/fd/X where X is the the file descriptor number of the temp file.

This ensures that Envoy can read the file without any other normal user process being able to (assuming they don't have privileged access to /proc). Once the Envoy process stops, there is no longer any reference to the file to clean up.

» Envoy Hot Restart

Envoy supports hot restart which requires simple external coordination. By default, this command will add --disable-hot-restart when it runs Envoy.

The reason for this default behavior is to make it easy to test and run local demonstrations with multiple Envoy instances outside of cgroups or network namespaces.

To use hot restart, Envoy needs to be started with either the --restart-epoch option. If this command detects that option in the pass-through flags it will not add --disable-hot-restart allowing hot restart to work normally.

The only difference to note over running Envoy directly is that --restart-epoch must be explicitly set to 0 for the initial launch of the Envoy instance to avoid disabling hot restart entirely. The official hot-restarter.py always sets this option so should work as recommended.