Reporting¶

Note

This documentation is meant to support existing Chef customers using Reporting.

Are you a new Chef customer, or looking to gain better insight into your fleet? Take advantage of Chef Automate. You’ll get a graphical interface and query language that gives you insight into operational, compliance, and workflow events. You can try out Chef Automate here.

Use Reporting to keep track of what happens during the execution of chef-client runs across all of the machines that are under management by Chef. Reports can be generated for the entire organization and they can be generated for specific nodes.

Reporting data is collected during the chef-client run and the results are posted to the Chef server at the end of the chef-client run at the same time the node object is uploaded to the Chef server.

Note

This feature is included as part of the Chef Automate license agreement and is available via subscription.

Requirements¶

Reporting has the following minimum requirements:

Chef server 12
chef-client version 11.6.0 (or later)

Reporting can make use of an external database, but to do so Reporting 1.5.5 or later is needed along with Chef server 12.2.0 or later.

The Reporting client is built into the chef-client and can run on all platforms that chef-client is supported on.

Warning

Reporting does not work on chef-client version 11.8.0; upgrade to chef-client version 11.8.2 (or later) if Reporting is being run in your organization.

Warning

The only supported versions of Reporting are 1.5.5 or later.

Reporting is configured as a premium feature of the Chef server and requires acceptance of the Chef MLSA as part of the installation process, starting with version 1.6.0.

The Reporting Run¶

A “chef-client run” is the term used to describe a series of steps that are taken by the chef-client when it is configuring a node. Reporting, when configured, is also tracked during the chef-client run. The following diagram shows how Reporting fits into the chef-client run, and then the list below the diagram describes in greater detail each of those stages.

When reports are generated as part of a chef-client run, in addition to what normally occurs during the chef-client run, the following happens:

Stages	Description
Generate reporting ID	At the beginning of the chef-client run, the chef-client pings Reporting to notify it that a chef-client run is starting and to generate a unique identifier for the chef-client run so that it can be associated with the reporting data.
Send the reporting data to the Chef server	At the end of the chef-client run, the chef-client pings Reporting, and then sends the data that was collected during the chef-client run to the Chef server so that it can be added to the Reporting database.

Chef Manage, Reports¶

The Chef management console enables the management of nodes, data bags, roles, environments, and cookbooks by using a web user interface. In addition, access to nodes, data bags, roles, environments, and cookbooks is configurable using role-based access control (RBAC).

When Reporting and Chef management console are both added to the Chef server, a user interface for Reporting is also available.

Knife Reporting¶

The knife reporting subcommand is used by Reporting to report about chef-client runs across the organization.

Note

Review the list of common options available to this (and all) knife subcommands and plugins.

Install the Plugin¶

To install the knife reporting plugin using RubyGems, run the following command:

$ /opt/chef/embedded/bin/gem install knife-reporting

where /opt/chef/embedded/bin/ is the path to the location where the chef-client expects knife plugins to be located. If the chef-client was installed using RubyGems, omit the path in the previous example.

runs list¶

Use the runs list argument to return a list of chef-client runs by organization or by node.

Syntax¶

This argument has the following syntax:

$ knife runs list NODE_NAME

Options¶

This argument has the following options:

-e MM-DD-YYYY, --endtime MM-DD-YYYY: Find runs with an end time less than or equal to the specified date. This option must be used in conjunction with --starttime and may define up to a 90-day time period.
RUN_ID: Required. The identifier for a chef-client run.
-r N, --rows N: The number of rows to be returned.
-s MM-DD-YYYY, --starttime MM-DD-YYYY: Find runs with a start time greater than or equal to the specified date. This option must be used in conjunction with --endtime and may define up to a 90-day time period.
-u, --unixtimestamps: Show start and end times as UNIX timestamps. This option may be used when the --endtime and --starttime options are used.

Examples¶

View all chef-client runs for an organization

To return a list of chef-client runs by organization (i.e. “every chef-client run that took place across every single node in the organization”), run the following command:

$ knife runs list

View all chef-client runs for a single node

To return a list of chef-client runs that occurred for a node, run the following command:

$ knife runs list node_name

where node_name is the name of a node.

View a specific chef-client run

To return a list of chef-client runs that occurred for a specific node, run the following command:

$ knife runs list node_name 30077269-59d0-4283-81f6-8d23cbed3a7a

where node_name is the name of a node.

runs show¶

Use the runs show argument to return the details for a chef-client run (by run identifier).

Syntax¶

This argument has the following syntax:

$ knife runs show RUN_ID

Options¶

This argument has the following options:

RUN_ID: The identifier for a chef-client run.

Examples¶

Show runs by run identifier

To return details about a specific chef-client run, run the following command:

$ knife runs show c40csdcf2-5007-433c-95570-8e0c7e55f496

For a chef-client run that succeeded, the command will return something similar to:

$ run_detail:
    data:
      end_time:          2013-12-10T17:18:49Z
      node_name:         kitchen-vagrant
      run_id:            c40csdcf2-5007-433c-95570-8e0c7e55f496
      run_list:          ["recipe[createfile]"]
      start_time:        2013-12-10T17:18:49Z
      status:            success
      total_res_count:   1
      updated_res_count: 1
  run_resources:
    cookbook_name:    createfile
    cookbook_version: 0.0.0
    duration:         41
    final_state:
      checksum: f29bc64a9d3123454gfdsffdb3285f5bsdfsdf72414671e0ca3b2e0de
      group:    root
      mode:     0755
      owner:    root
    id:               /tmp/test
    initial_state:
      checksum:
      group:
      mode:
      owner:
    name:             /tmp/test
    result:           touch
    type:             file
    uri:              URI

For a chef-client run that failed (or had some type of error), the command will return something similar to:

$ run_detail:
    data:
      exception:
        backtrace: ["/opt/chef/embedded/lib/ruby/1.9.3/net/http.rb:763:in 'initialize'",
                    "/opt/chef/embedded/lib/ruby/1.9.3/net/http.rb:763:in 'open'",
                    "/opt/chef/embedded/lib/ruby/1.9.3/net/http.rb:763:in 'block in connect'",
                    "/opt/chef/embedded/lib/ruby/1.9.3/timeout.rb:55:in 'timeout'",
                    "/opt/chef/embedded/lib/ruby/1.9.3/timeout.rb:100:in 'timeout'",
                    "/opt/chef/embedded/lib/ruby/1.9.3/net/http.rb:763:in 'connect'",
                    "/opt/chef/embedded/lib/ruby/1.9.3/net/http.rb:756:in 'do_start'",
                    ...
                    "/usr/bin/chef-client:23:in 'load'",
                    "/usr/bin/chef-client:23:in '<main>'"]
        class:       #<SocketError: Error connecting to URL - getaddrinfo: Temporary failure in name resolution>
        description:
          sections:
            Networking Error:: Error connecting to URL - getaddrinfo: Temporary failure in name resolution
                               Your chef_server_url may be misconfigured, or the network could be down.
            Relevant Config Settings:: chef_server_url  "URL"
          title:    Error Syncing Cookbooks:
          message:     Error connecting to URL - getaddrinfo: Temporary failure in name resolution
          end_time:          2013-12-10T14:38:18Z
          node_name:         node_name
          run_id:            c40csdcf2-5007-433c-95570-8e0c7e55f496
          run_list:          ["recipe[createfile]"]
          start_time:        2013-12-10T14:37:37Z
          status:            failure
          total_res_count:   0
          updated_res_count: 0
  run_resources:

state_attrs Method¶

The state_attrs method is used to define the properties that will be tracked by the Reporting server. In general, this should be a list of properties that describe the desired state of the system, such as file permissions, cloud provider data (like snapshots, volumes, identifiers, sizes, and access keys), and so on.

The syntax for the state_attrs method is as follows:

state_attrs :property,
            :property,
            :property

where :property is a comma-delimited list of properties. For example, the ebs_volume resource (available from the aws cookbook) uses the state_attrs method to tell the Reporting server to track the following properties:

state_attrs :availability_zone,
            :aws_access_key,
            :description,
            :device,
            :most_recent_snapshot,
            :piops,
            :size,
            :snapshot_id,
            :snapshots_to_keep,
            :timeout,
            :volume_id,
            :volume_type

Configuration Settings¶

Reporting relies on two settings in the client.rb file:

Setting	Description
`enable_reporting`	Cause the chef-client to send data to the Chef server for use with Reporting. For example: enable_reporting true
`enable_reporting_url_fatals`	Cause the chef-client run to fail when Reporting data cannot be sent to the Chef server (for any reason). For example: enable_reporting_url_fatals false

Reporting API¶

The Reporting API is a REST API that provides access to Reporting data that is collected during a chef-client run. Reporting data is collected only for nodes that have permission to publish Reporting data to the Chef server and only for organizations that have Reporting enabled.

Requirements¶

The Chef server API has the following requirements:

Access to a Chef server running version 0.10.x or above.
The Accept header must be set to application/json.
For PUT and POST requests, the Content-Type header must be set to application/json.
The X-Chef-Version header must be set to the version of the Chef server API that is being used.
A request must be signed using Mixlib::Authentication.
A request must be well-formatted. The easiest way to ensure a well-formatted request is to use the Chef::REST library.

Authentication Headers¶

Authentication to the Chef server occurs when a specific set of HTTP headers are signed using a private key that is associated with the machine from which the request is made. The request is authorized if the Chef server can verify the signature using the public key. Only authorized actions are allowed.

Note

Most authentication requests made to the Chef server are abstracted from the user. Such as when using knife or the Chef server user interface. In some cases, such as when using the knife exec subcommand, the authentication requests need to be made more explicitly, but still in a way that does not require authentication headers. In a few cases, such as when using arbitrary Ruby code or cURL, it may be necessary to include the full authentication header as part of the request to the Chef server.

Header Format¶

By default, all hashing is done using SHA-1 and encoded in Base64. Base64 encoding should have line breaks every 60 characters. Each canonical header should be encoded in the following format:

Method:HTTP_METHOD
Hashed Path:HASHED_PATH
X-Ops-Content-Hash:HASHED_BODY
X-Ops-Timestamp:TIME
X-Ops-UserId:USERID

where:

HTTP_METHOD is the method used in the API request (GET, POST, and so on)
HASHED_PATH is the path of the request: /organizations/NAME/name_of_endpoint. The HASHED_PATH must be hashed using SHA-1 and encoded using Base64, must not have repeated forward slashes (/), must not end in a forward slash (unless the path is /), and must not include a query string.
The private key must be an RSA key in the SSL .pem file format. This signature is then broken into character strings (of not more than 60 characters per line) and placed in the header.

The Chef server decrypts this header and ensures its content matches the content of the non-encrypted headers that were in the request. The timestamp of the message is checked to ensure the request was received within a reasonable amount of time. One approach generating the signed headers is to use mixlib-authentication, which is a class-based header signing authentication object similar to the one used by the chef-client.

Enable SHA-256¶

Chef server versions 12.4.0 and above support signing protocol version 1.3, which adds support for SHA-256 algorithms. It can be enabled on Chef client via the client.rb file:

authentication_protocol_version = '1.3'

And on Chef knife via config.rb:

knife[:authentication_protocol_version] = '1.3'

Required Headers¶

The following authentication headers are required:

Feature	Description
`Accept`	The format in which response data from the Chef server is provided. This header must be set to `application/json`.
`Content-Type`	The format in which data is sent to the Chef server. This header is required for `PUT` and `POST` requests and must be set to `application/json`.
`Host`	The host name (and port number) to which a request is sent. (Port number `80` does not need to be specified.) For example: `api.opscode.com` (which is the same as `api.opscode.com:80`) or `api.opscode.com:443`.
`X-Chef-Version`	The version of the chef-client executable from which a request is made. This header ensures that responses are in the correct format. For example: `12.0.2` or `11.16.x`.
`X-Ops-Authorization-N`	One (or more) 60 character segments that comprise the canonical header. A canonical header is signed with the private key used by the client machine from which the request is sent, and is also encoded using Base64. If more than one segment is required, each should be named sequentially, e.g. `X-Ops-Authorization-1`, `X-Ops-Authorization-2`, `X-Ops-Authorization-N`, where `N` represents the integer used by the last header that is part of the request.
`X-Ops-Content-Hash`	The body of the request. The body should be hashed using SHA-1 and encoded using Base64. All hashing is done using SHA-1 and encoded in Base64. Base64 encoding should have line breaks every 60 characters.
`X-Ops-Reporting-Protocol-Version`	Use to specify the protocol version for the Reporting API. This header must be set to `0.1.0`. A request to the Chef server API that does not include this header and the correct value will return a 404 response code. A request to the Chef server API that includes this header with an incorrect value will return a 406 response code. If the protocol version is incorrect (or unspecified), the chef-client run will proceed normally, but Reporting data will not be collected for that chef-client run unless the `enable_reporting_url_fatals` setting is `true` in the client.rb file for that node.
`X-Ops-Sign`	Set this header to the following value: `version=1.0`.
`X-Ops-Timestamp`	The timestamp, in ISO-8601 format and with UTC indicated by a trailing `Z` and separated by the character `T`. For example: `2013-03-10T14:14:44Z`.
`X-Ops-UserId`	The name of the API client whose private key will be used to create the authorization header.

Example¶

The following example shows an authentication request:

GET /organizations/NAME/nodes HTTP/1.1
  Accept: application/json
  Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
  X-Ops-Sign: algorithm=sha1;version=1.0;
  X-Ops-Userid: user_id
  X-Ops-Timestamp: 2014-12-12T17:13:28Z
  X-Ops-Content-Hash: 2jmj7l5rfasfgSw0ygaVb/vlWAghYkK/YBwk=
  X-Ops-Authorization-1: BE3NnBritishaf3ifuwLSPCCYasdfXaRN5oZb4c6hbW0aefI
  X-Ops-Authorization-2: sL4j1qtEZzi/2WeF67UuytdsdfgbOc5CjgECQwqrym9gCUON
  X-Ops-Authorization-3: yf0p7PrLRCNasdfaHhQ2LWSea+kTcu0dkasdfvaTghfCDC57
  X-Ops-Authorization-4: 155i+ZlthfasfasdffukusbIUGBKUYFjhbvcds3k0i0gqs+V
  X-Ops-Authorization-5: /sLcR7JjQky7sdafIHNfsBQrISktNPower1236hbFIayFBx3
  X-Ops-Authorization-6: nodilAGMb166@haC/fttwlWQ2N1LasdqqGomRedtyhSqXA==
  Host: api.opscode.com:443
  X-Ops-Server-API-Info: 1
  X-Chef-Version: 12.0.2
  User-Agent: Chef Knife/12.0.2 (ruby-2.1.1-p320; ohai-8.0.0; x86_64-darwin12.0.2; +http://chef.io)

Global Endpoints¶

A global endpoint may be used to access all of the organizations on the Chef server.

/reports/status¶

The /reports/status endpoint has the following methods: GET.

GET¶

The GET method is used to return the status of the system components used by Reporting.

This method does not have any parameters.

Request

GET /reports/status

Response

The response is similar to:

{
  "rest_api" : "online",
  "sql_db" : "online",
  "index" : "online"
}

where index is the Chef server search index. If the system component is not online, the response will return offline.

Response Codes

Response Code	Description
`200`	OK. The request was successful.
`404`	Not found. The requested object does not exist.
`406`	Invalid request. The protocol version is incorrect.

Organization Endpoints¶

Each organization-specific authentication request must include /organizations/ORG_NAME as part of the name for the endpoint. For example, the full endpoint for getting the details for a specific reporting run identifier for a node:

GET /organizations/ORG_NAME/reports/nodes/NODE/runs/RUNID

where ORG_NAME is the name of the organization, NODE is the name of the node, and RUNID is the reporting run identifier.

/reports/nodes/NODE/runs¶

The /reports/nodes/NODE/runs endpoint has the following methods: GET and POST.

GET¶

The GET method is used to return Reporting data for a chef-client run.

This method has no parameters.

Request

GET /organizations/ORG/reports/nodes/NODE/runs

Response

The response is similar to:

{
  "node_name" : "pkd01234567",
  "run_id" : "550e4500-e22b-4ad4-a716-446659876500",
  "start_time" : "2014-11-14T23:33:34Z"
  "status" : "started"
}

Response Codes

Response Code	Description
`200`	OK. The request was successful.
`404`	Not found. The requested object does not exist.
`406`	Invalid request. The protocol version is incorrect.

/reports/nodes/NODE/runs/RUNID/RESID¶

The /reports/nodes/NODE/runs/RUNID/RESID endpoint has the following methods: GET.

GET¶

The GET method is used to return a list of what changed during the chef-client run for the specified resource.

This method has no parameters.

Request

GET /organizations/ORG/reports/nodes/NODE/runs/RUNID/RESID

Response

The response is similar to:

{
  resource_detail :
  {
    "content_delta" : string
  }
}

Response Codes

Response Code	Description
`200`	OK. The request was successful.
`404`	Not found. The requested object does not exist.
`406`	Invalid request. The protocol version is incorrect.

/reports/nodes/NODE/runs/RUNID¶

The /reports/nodes/NODE/runs/RUNID endpoint has the following methods: GET and POST.

GET¶

The GET method is used to return a list of resources for a given Reporting run identifier.

This method has the following parameters:

Parameter	Description
`detail`	Optional. When `true`, include the `run_detail` JSON object in the output. Default value: `false`.
`rows`	Optional. The number of resources to return. Default value: `10`.
`start`	Optional. The row at which the results will start. Default value: `0`.

Request

GET /organizations/ORG/reports/nodes/NODE/runs/RUNID

Response

The response is similar to:

{
  run_resources :  [
    {
      "uri" : uri,
      "cookbook_name" : string,
      "cookbook_version" : string,
      "duration" : numeric string - milliseconds,
      "id" : string,
      "type" : string,
      "name" : string,
      "result" : string,
      "initial_state" : json-object,
      "final_state" : json-object,
    }
  ],
  run_detail :
    {
      "node_name" : string,
      "updated_res_count" : integer,
      "total_res_count" : integer,
      "run_list" : string ??? TODO: Verify this is correct
      "start_time" : timestamp
      "end_time" : timestamp
      "data" : { 0..1 exception-record },
      "status"
    }
}

Response Codes

Response Code	Description
`200`	OK. The request was successful.
`404`	Not found. The requested object does not exist.
`406`	Invalid request. The protocol version is incorrect.

/reports/org/runs¶

The /reports/org/runs endpoint has the following methods: GET.

GET¶

The GET method is used to return information about chef-client runs for all nodes in the specified organization.

This method has the following parameters:

Parameter	Description
`from`	Optional. Use to specify the time before which node data will not be returned. Use with `until` to define a range.
`rows`	Optional. The number of resources to return. Default value: `10`.
`start`	Optional. The row at which the results will start. Default value: `0`.
`status`	Optional. Use to specify a status code. When a status code is provided, only nodes with that status will be returned. When a status code is not provided, all nodes will be returned. Possible values: `aborted`, `failure`, or `success`.
`until`	Optional. Use to specify the time after which node data will not be returned. Use with `until` to define a range.

Request

GET /organizations/ORG/reports/org/runs

Response

The response is similar to:

{

}

Response Codes

Response Code	Description
`200`	OK. The request was successful.
`404`	Not found. The requested object does not exist.
`406`	Invalid request. The protocol version is incorrect.

/reports/runs/counts¶

The /reports/runs/counts endpoint has the following methods: GET.

GET¶

The GET method is used to return the frequency of chef-client runs, per-minute, per-hour, per-day, or per-week.

This method has the following parameters:

Parameter	Description
`granularity`	Required. The length of time for which chef-client run counts are returned. Possible values: `hour`, `minute`, `day`, or `week`.

Request

GET /organizations/ORG/reports/runs/counts

Response

The response is similar to:

{

}

Response Codes

Response Code	Description
`200`	OK. The request was successful.
`404`	Not found. The requested object does not exist.
`406`	Invalid request. The protocol version is incorrect.

/reports/runs/durations¶

The /reports/runs/durations endpoint has the following methods: GET.

GET¶

The GET method is used to return the frequency of chef-client runs that occurred within a specified range.

This method has the following parameters:

Parameter	Description
`from`	Optional. Use to specify the time before which node data will not be returned. Use with `until` to define a range.
`until`	Optional. Use to specify the time after which node data will not be returned. Use with `until` to define a range.

Request

GET /organizations/ORG/reports/runs/durations

Response

The response is similar to:

{

}

Response Codes

Response Code	Description
`200`	OK. The request was successful.
`404`	Not found. The requested object does not exist.
`406`	Invalid request. The protocol version is incorrect.