If you enable the Elastic monitoring features in your cluster, you can optionally collect metrics about Elasticsearch. By default, monitoring is enabled but data collection is disabled.
This method involves sending the metrics to the monitoring cluster by using exporters. For an alternative method, see Collecting monitoring data with Metricbeat.
If you want to collect monitoring data from sources such as Beats and Logstash and route it to a monitoring cluster, you must follow this method. You cannot use Metricbeat to ship the monitoring data for those products yet.
Advanced monitoring settings enable you to control how frequently data is collected, configure timeouts, and set the retention period for locally-stored monitoring indices. You can also adjust how monitoring data is displayed.
To learn about monitoring in general, see Monitoring the Elastic Stack.
Configure your cluster to collect monitoring data:
xpack.monitoring.enabled
setting is true
, which is its
default value, on each node in the cluster. For more information, see
Monitoring settings.
Verify that the xpack.monitoring.elasticsearch.collection.enabled
setting
is true
, which is its default value, on each node in the cluster.
You can specify this setting in either the elasticsearch.yml
on each
node or across the cluster as a dynamic cluster setting. If Elasticsearch
security features are enabled, you must have monitor
cluster privileges to
view the cluster settings and manage
cluster privileges to change them.
For more information, see Monitoring settings and Cluster Update Settings.
Set the xpack.monitoring.collection.enabled
setting to true
on each
node in the cluster. By default, it is is disabled (false
).
You can specify this setting in either the elasticsearch.yml
on each
node or across the cluster as a dynamic cluster setting. If Elasticsearch
security features are enabled, you must have monitor
cluster privileges to
view the cluster settings and manage
cluster privileges to change them.
For example, use the following APIs to review and change this setting:
GET _cluster/settings PUT _cluster/settings { "persistent": { "xpack.monitoring.collection.enabled": true } }
Alternatively, you can enable this setting in Kibana. In the side navigation, click Monitoring. If data collection is disabled, you are prompted to turn it on.
For more information, see Monitoring settings and Cluster Update Settings.
Optional: Specify which indices you want to monitor.
By default, the monitoring agent collects data from all Elasticsearch indices.
To collect data from particular indices, configure the
xpack.monitoring.collection.indices
setting. You can specify multiple indices
as a comma-separated list or use an index pattern to match multiple indices. For
example:
xpack.monitoring.collection.indices: logstash-*, index1, test2
You can prepend -
to explicitly exclude index names or
patterns. For example, to include all indices that start with test
except
test3
, you could specify test*,-test3
. To include system indices such as
.security and .kibana, add .*
to the list of included names.
For example .*,test*,-test3
xpack.monitoring.collection.interval
setting 10 seconds. See
Monitoring settings.
Identify where to store monitoring data.
By default, the data is stored on the same cluster by using a
local
exporter. Alternatively, you can use an http
exporter to send data to
a separate monitoring cluster.
The Elasticsearch monitoring features use ingest pipelines, therefore the cluster that stores the monitoring data must have at least one ingest node.
For more information about typical monitoring architectures, see How Monitoring Works.
If you choose to use an http
exporter:
On the cluster that you want to monitor (often called the production cluster),
configure each node to send metrics to your monitoring cluster. Configure an
HTTP exporter in the xpack.monitoring.exporters
settings in the
elasticsearch.yml
file. For example:
xpack.monitoring.exporters: id1: type: http host: ["http://es-mon-1:9200", "http://es-mon2:9200"]
If the Elastic security features are enabled on the monitoring cluster, you must provide appropriate credentials when data is shipped to the monitoring cluster:
remote_monitoring_agent
built-in role.
Alternatively, use the
remote_monitoring_user
built-in user.
Add the user ID and password settings to the HTTP exporter settings in the
elasticsearch.yml
file on each node.
For example:
xpack.monitoring.exporters: id1: type: http host: ["http://es-mon-1:9200", "http://es-mon2:9200"] auth.username: remote_monitoring_user auth.password: YOUR_PASSWORD
If you configured the monitoring cluster to use
encrypted communications, you must use the HTTPS protocol in
the host
setting. You must also specify the trusted CA certificates that will
be used to verify the identity of the nodes in the monitoring cluster.
To add a CA certificate to an Elasticsearch node’s trusted certificates, you can
specify the location of the PEM encoded certificate with the
certificate_authorities
setting. For example:
xpack.monitoring.exporters: id1: type: http host: ["https://es-mon1:9200", "https://es-mon2:9200"] auth: username: remote_monitoring_user password: YOUR_PASSWORD ssl: certificate_authorities: [ "/path/to/ca.crt" ]
Alternatively, you can configure trusted certificates using a truststore (a Java Keystore file that contains the certificates). For example:
xpack.monitoring.exporters: id1: type: http host: ["https://es-mon1:9200", "https://es-mon2:9200"] auth: username: remote_monitoring_user password: YOUR_PASSWORD ssl: truststore.path: /path/to/file truststore.password: password
If you updated settings in the elasticsearch.yml
files on your production
cluster, restart Elasticsearch. See Stopping Elasticsearch and Starting Elasticsearch.
You may want to temporarily disable shard allocation before you restart your nodes to avoid unnecessary shard reallocation during the install process.