chef-client Security¶

[edit on GitHub]

All communication with the Chef server must be authenticated using the Chef server API, which is a REST API that allows requests to be made to the Chef server. Only authenticated requests will be authorized. Most of the time, and especially when using knife, the chef-client, or the Chef server web interface, the use of the Chef server API is transparent. In some cases, the use of the Chef server API requires more detail, such as when making the request in Ruby code, with a knife plugin, or when using cURL.

Authentication¶

The authentication process ensures the Chef server responds only to requests made by trusted users. Public key encryption is used by the Chef server. When a node and/or a workstation is configured to run the chef-client, both public and private keys are created. The public key is stored on the Chef server, while the private key is returned to the user for safe keeping. (The private key is a .pem file located in the .chef directory or in /etc/chef.)

Both the chef-client and knife use the Chef server API when communicating with the Chef server. The chef-validator uses the Chef server API, but only during the first chef-client run on a node.

Each request to the Chef server from those executables sign a special group of HTTP headers with the private key. The Chef server then uses the public key to verify the headers and verify the contents.

chef-validator¶

Every request made by the chef-client to the Chef server must be an authenticated request using the Chef server API and a private key. When the chef-client makes a request to the Chef server, the chef-client authenticates each request using a private key located in /etc/chef/client.pem.

During a chef-client Run¶

As part of every chef-client run, the chef-client authenticates to the Chef server using an RSA private key and the Chef server API.

authentication_protocol_version¶

The authentication_protocol_version option in the client.rb file is used to determine the authentication protocol that communicates with Chef server. For example, specify protocol version 1.3 to enable support for SHA-256 algorithms:

knife[:authentication_protocol_version] = '1.3'

Note that authentication protocol 1.3 is only supported on Chef server versions 12.4.0 and above.

SSL Certificates¶

Warning

The following information does not apply to hosted Chef server 12, only to on-premises Chef server 12.

Chef server 12 enables SSL verification by default for all requests made to the server, such as those made by knife and the chef-client. The certificate that is generated during the installation of the Chef server is self-signed, which means the certificate is not signed by a trusted certificate authority (CA) that ships with the chef-client. The certificate generated by the Chef server must be downloaded to any machine from which knife and/or the chef-client will make requests to the Chef server.

For example, without downloading the SSL certificate, the following knife command:

$ knife client list

responds with an error similar to:

ERROR: SSL Validation failure connecting to host: chef-server.example.com ...
ERROR: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 ...

This is by design and will occur until a verifiable certificate is added to the machine from which the request is sent.

Changes Prior to Chef 12¶

The following changes were made during certain chef-client release prior to the chef-client 12 release:

In the chef-client 11.8 release, the verify_api_cert setting was added to the client.rb file with a default value of false.
In the chef-client 11.12 release, the local_key_generation setting was added to the client.rb file.

The ssl_verify_mode continued to default to :verify_none, but now returned a warning: SSL validation of HTTPS requests is disabled..., followed by steps for how to configure SSL certificate validation for the chef-client.

Two knife commands—knife ssl check and knife ssl fetch were added.

A new directory in the chef-repo—/.chef/trusted_certs—was added.

These new settings and tools enabled users who wanted to use stronger SSL settings to generate the private/public key pair from the chef-client, verify HTTPS requests, verify SSL certificates, and pull the SSL certificate from the Chef server down to the /.chef/trusted_certs directory.
In the chef-client 12 release, the default value for local_key_generation was changed to true and the default value for ssl_verify_mode was changed to :verify_peer.

Starting with chef-client 12, SSL certificate validation is enabled by default and the knife ssl fetch is a necessary part of the setup process for every workstation.

`/.chef/trusted_certs`¶

The /.chef/trusted_certs directory stores trusted SSL certificates used to access the Chef server:

On each workstation, this directory is the location into which SSL certificates are placed after they are downloaded from the Chef server using the knife ssl fetch subcommand
On every node, this directory is the location into which SSL certificates are placed when a node has been bootstrapped with the chef-client from a workstation

SSL_CERT_FILE¶

Use the SSL_CERT_FILE environment variable to specify the location for the SSL certificate authority (CA) bundle that is used by the chef-client.

A value for SSL_CERT_FILE is not set by default. Unless updated, the locations in which Chef will look for SSL certificates are:

chef-client: /opt/chef/embedded/ssl/certs/cacert.pem
Chef development kit: /opt/chefdk/embedded/ssl/certs/cacert.pem

Keeping the default behavior is recommended. To use a custom CA bundle, update the environment variable to specify the path to the custom CA bundle. If (for some reason) SSL certificate verification stops working, ensure the correct value is specified for SSL_CERT_FILE.

client.rb Settings¶

Use following client.rb settings to manage SSL certificate preferences:

Setting	Description
`local_key_generation`	Whether the Chef server or chef-client generates the private/public key pair. When `true`, the chef-client generates the key pair, and then sends the public key to the Chef server. Default value: `true`.
`ssl_ca_file`	The file in which the OpenSSL key is saved. This setting is generated automatically by the chef-client and most users do not need to modify it.
`ssl_ca_path`	The path to where the OpenSSL key is located. This setting is generated automatically by the chef-client and most users do not need to modify it.
`ssl_client_cert`	The OpenSSL X.509 certificate used for mutual certificate validation. This setting is only necessary when mutual certificate validation is configured on the Chef server. Default value: `nil`.
`ssl_client_key`	The OpenSSL X.509 key used for mutual certificate validation. This setting is only necessary when mutual certificate validation is configured on the Chef server. Default value: `nil`.
`ssl_verify_mode`	Set the verify mode for HTTPS requests. Use `:verify_none` to do no validation of SSL certificates. Use `:verify_peer` to do validation of all SSL certificates, including the Chef server connections, S3 connections, and any HTTPS remote_file resource URLs used in the chef-client run. This is the recommended setting. Depending on how OpenSSL is configured, the `ssl_ca_path` may need to be specified. Default value: `:verify_peer`.
`verify_api_cert`	Verify the SSL certificate on the Chef server. When `true`, the chef-client always verifies the SSL certificate. When `false`, the chef-client uses the value of `ssl_verify_mode` to determine if the SSL certificate requires verification. Default value: `false`.

Knife Subcommands¶

The chef-client includes two knife commands for managing SSL certificates:

Use knife ssl check to troubleshoot SSL certificate issues
Use knife ssl fetch to pull down a certificate from the Chef server to the /.chef/trusted_certs directory on the workstation.

After the workstation has the correct SSL certificate, bootstrap operations from that workstation will use the certificate in the /.chef/trusted_certs directory during the bootstrap operation.

knife ssl check¶

Run the knife ssl check subcommand to verify the state of the SSL certificate, and then use the reponse to help troubleshoot issues that may be present.

Verified

If the SSL certificate can be verified, the response to

$ knife ssl check

is similar to:

Connecting to host chef-server.example.com:443
Successfully verified certificates from 'chef-server.example.com'

Unverified

If the SSL certificate cannot be verified, the response to

$ knife ssl check

is similar to:

Connecting to host chef-server.example.com:443
ERROR: The SSL certificate of chef-server.example.com could not be verified
Certificate issuer data:
  /C=US/ST=WA/L=S/O=Corp/OU=Ops/CN=chef-server.example.com/emailAddress=you@example.com

Configuration Info:

OpenSSL Configuration:
* Version: OpenSSL 1.0.1j 15 Oct 2014
* Certificate file: /opt/chefdk/embedded/ssl/cert.pem
* Certificate directory: /opt/chefdk/embedded/ssl/certs
Chef SSL Configuration:
* ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/Users/grantmc/Downloads/chef-repo/.chef/trusted_certs"

TO FIX THIS ERROR:

If the server you are connecting to uses a self-signed certificate,
you must configure chef to trust that certificate.

By default, the certificate is stored in the following location on the
host where your chef-server runs:

  /var/opt/opscode/nginx/ca/SERVER_HOSTNAME.crt

Copy that file to your trusted_certs_dir (currently:

  /Users/grantmc/Downloads/chef-repo/.chef/trusted_certs)

using SSH/SCP or some other secure method, then re-run this command to
confirm that the certificate is now trusted.

knife ssl fetch¶

Run the knife ssl fetch to download the self-signed certificate from the Chef server to the /.chef/trusted_certs directory on a workstation.

Verify Checksums

The SSL certificate that is downloaded to the /.chef/trusted_certs directory should be verified to ensure that it is, in fact, the same certificate as the one located on the Chef server. This can be done by comparing the SHA-256 checksums.

View the checksum on the Chef server:

$ ssh ubuntu@chef-server.example.com sudo sha256sum /var/opt/opscode/nginx/ca/chef-server.example.com.crt

The response is similar to:

<ABC123checksum>  /var/opt/opscode/nginx/ca/chef-server.example.com.crt

View the checksum on the workstation:

$ gsha256sum .chef/trusted_certs/chef-server.example.com.crt