Docs
-->
Elasticsearch Reference [7.0] » Set up Elasticsearch » Configuring monitoring in Elasticsearch » Collecting Elasticsearch monitoring data with Metricbeat
«  Collecting monitoring data     Configuring indices for monitoring  »

Collecting Elasticsearch monitoring data with Metricbeat

In 6.5 and later, you can use Metricbeat to collect data about Elasticsearch and ship it to the monitoring cluster, rather than routing it through exporters as described in Collecting monitoring data.

Example monitoring architecture

To learn about monitoring in general, see Monitoring the Elastic Stack.

  1. Enable the collection of monitoring data. Set xpack.monitoring.collection.enabled to true on each node in the production cluster. By default, it is is disabled (false).

    Note

    You can specify this setting in either the elasticsearch.yml on each node or across the cluster as a dynamic cluster setting. If Elasticsearch security features are enabled, you must have monitor cluster privileges to view the cluster settings and manage cluster privileges to change them.

    For example, you can use the following APIs to review and change this setting:

    GET _cluster/settings

PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}

    For more information, see Monitoring settings and Cluster Update Settings.

  2. Disable the default collection of Elasticsearch monitoring metrics. Set xpack.monitoring.elasticsearch.collection.enabled to false on each node in the production cluster.

    Note

    You can specify this setting in either the elasticsearch.yml on each node or across the cluster as a dynamic cluster setting. If Elasticsearch security features are enabled, you must have monitor cluster privileges to view the cluster settings and manage cluster privileges to change them.

    For example, you can use the following API to change this setting:

    PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.elasticsearch.collection.enabled": false
  }
}

    Leave xpack.monitoring.enabled set to its default value (true).

  3. On each Elasticsearch node in the production cluster:

    1. Install Metricbeat.

    2. Enable the Elasticsearch module in Metricbeat.

      For example, to enable the default configuration in the modules.d directory, run the following command:

      metricbeat modules enable elasticsearch

      For more information, see Specify which modules to run and Elasticsearch module.

    3. Configure the Elasticsearch module in Metricbeat.

      You must specify the following settings in the modules.d/elasticsearch.yml file:

      - module: elasticsearch
  metricsets:
    - ccr
    - cluster_stats
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["http://localhost:9200"] 
  xpack.enabled: true

      This setting identifies the host and port number that are used to access Elasticsearch.

      This setting ensures that Kibana can read this monitoring data successfully. That is to say, it’s stored in the same location and format as monitoring data that is sent by exporters.

    4. If Elastic security features are enabled, you must also provide a user ID and password so that Metricbeat can collect metrics successfully.

      1. Create a user on the production cluster that has the remote_monitoring_collector built-in role. Alternatively, use the remote_monitoring_user built-in user.

      2. Add the username and password settings to the Elasticsearch module configuration file.

        For example, add the following settings in the modules.d/elasticsearch.yml file:

        - module: elasticsearch
  ...
  username: remote_monitoring_user
  password: YOUR_PASSWORD
    5. If you configured Elasticsearch to use encrypted communications, you must access it via HTTPS. For example, use a hosts setting like https://localhost:9200 in the modules.d/elasticsearch.yml file.

    6. Identify where to send the monitoring data.

      Tip

      In production environments, we strongly recommend using a separate cluster (referred to as the monitoring cluster) to store the data. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster.

      For example, specify the Elasticsearch output information in the Metricbeat configuration file (metricbeat.yml):

      output.elasticsearch:
  hosts: ["http://es-mon-1:9200", "http://es-mon2:9200"]

      In this example, the data is stored on a monitoring cluster with nodes es-mon-1 and es-mon-2.

      Important

      The Elasticsearch monitoring features use ingest pipelines, therefore the cluster that stores the monitoring data must have at least one ingest node.

      For more information about these configuration options, see Configure the Elasticsearch output.

    7. If Elasticsearch security features are enabled on the monitoring cluster, you must provide a valid user ID and password so that Metricbeat can send metrics successfully.

      1. Create a user on the monitoring cluster that has the remote_monitoring_agent built-in role. Alternatively, use the remote_monitoring_user built-in user.

      2. Add the username and password settings to the Elasticsearch output information in the Metricbeat configuration file (metricbeat.yml): 

        output.elasticsearch:
  ...
  username: remote_monitoring_user
  password: YOUR_PASSWORD
    8. If you configured the monitoring cluster to use encrypted communications, you must access it via HTTPS. For example, use a hosts setting like https://es-mon-1:9200 in the metricbeat.yml file.
  4. Start Elasticsearch.
  5. Start Metricbeat.
  6. View the monitoring data in Kibana.
«  Collecting monitoring data     Configuring indices for monitoring  »