elasticsearch_river

Milestone: 2

This output lets you store logs in elasticsearch. It’s similar to the ‘elasticsearch’ output but improves performance by using a queue server, rabbitmq, to send data to elasticsearch.

Upon startup, this output will automatically contact an elasticsearch cluster and configure it to read from the queue to which we write.

You can learn more about elasticseasrch at http://elasticsearch.org More about the elasticsearch rabbitmq river plugin: https://github.com/elasticsearch/elasticsearch-river-rabbitmq/blob/master/README.md

Synopsis

This is what it might look like in your config file:

output {
  elasticsearch_river {
    codec => ... # codec (optional), default: "plain"
    document_id => ... # string (optional), default: nil
    durable => ... # boolean (optional), default: true
    es_bulk_size => ... # number (optional), default: 1000
    es_bulk_timeout_ms => ... # number (optional), default: 100
    es_host => ... # string (required)
    es_ordered => ... # boolean (optional), default: false
    es_port => ... # number (optional), default: 9200
    exchange => ... # string (optional), default: "elasticsearch"
    exchange_type => ... # string, one of ["fanout", "direct", "topic"] (optional), default: "direct"
    index => ... # string (optional), default: "logstash-%{+YYYY.MM.dd}"
    index_type => ... # string (optional), default: "%{type}"
    key => ... # string (optional), default: "elasticsearch"
    password => ... # string (optional), default: "guest"
    persistent => ... # boolean (optional), default: true
    queue => ... # string (optional), default: "elasticsearch"
    rabbitmq_host => ... # string (required)
    rabbitmq_port => ... # number (optional), default: 5672
    user => ... # string (optional), default: "guest"
    vhost => ... # string (optional), default: "/"
    workers => ... # number (optional), default: 1
  }
}

Details

codec

• Value type is codec
• Default value is "plain"

The codec used for output data. Output codecs are a convenient method for encoding your data before it leaves the output, without needing a separate filter in your Logstash pipeline.

document_id

• Value type is string
• Default value is nil

The document ID for the index. Useful for overwriting existing entries in elasticsearch with the same ID.

durable

• Value type is boolean
• Default value is true

RabbitMQ durability setting. Also used for ElasticSearch setting

es_bulk_size

• Value type is number
• Default value is 1000

ElasticSearch river configuration: bulk fetch size

es_bulk_timeout_ms

• Value type is number
• Default value is 100

ElasticSearch river configuration: bulk timeout in milliseconds

es_host (required setting)

• Value type is string
• There is no default value for this setting.

The name/address of an ElasticSearch host to use for river creation

es_ordered

• Value type is boolean
• Default value is false

ElasticSearch river configuration: is ordered?

es_port

• Value type is number
• Default value is 9200

ElasticSearch API port

exchange

• Value type is string
• Default value is "elasticsearch"

RabbitMQ exchange name

exchange_type

• Value can be any of: "fanout", "direct", "topic"
• Default value is "direct"

The exchange type (fanout, topic, direct)

exclude_tags DEPRECATED

• DEPRECATED WARNING: This config item is deprecated. It may be removed in a further version.
• Value type is array
• Default value is []

Only handle events without any of these tags. Note this check is additional to type and tags.

index

• Value type is string
• Default value is "logstash-%{+YYYY.MM.dd}"

The index to write events to. This can be dynamic using the %{foo} syntax. The default value will partition your indeces by day so you can more easily delete old data or only search specific date ranges.

index_type

• Value type is string
• Default value is "%{type}"

The index type to write events to. Generally you should try to write only similar events to the same ‘type’. String expansion ‘%{foo}’ works here.

key

• Value type is string
• Default value is "elasticsearch"

RabbitMQ routing key

password

• Value type is string
• Default value is "guest"

RabbitMQ password

persistent

• Value type is boolean
• Default value is true

RabbitMQ persistence setting

queue

• Value type is string
• Default value is "elasticsearch"

RabbitMQ queue name

rabbitmq_host (required setting)

• Value type is string
• There is no default value for this setting.

Hostname of RabbitMQ server

rabbitmq_port

• Value type is number
• Default value is 5672

Port of RabbitMQ server

tags DEPRECATED

• DEPRECATED WARNING: This config item is deprecated. It may be removed in a further version.
• Value type is array
• Default value is []

Only handle events with all of these tags. Note that if you specify a type, the event must also match that type. Optional.

type DEPRECATED

• DEPRECATED WARNING: This config item is deprecated. It may be removed in a further version.
• Value type is string
• Default value is ""

The type to act on. If a type is given, then this output will only act on messages with the same type. See any input plugin’s “type” attribute for more. Optional.

user

• Value type is string
• Default value is "guest"

RabbitMQ user

vhost

• Value type is string
• Default value is "/"

RabbitMQ vhost

workers

• Value type is number
• Default value is 1

The number of workers to use for this output. Note that this setting may not be useful for all outputs.

---

This is documentation from lib/logstash/outputs/elasticsearch_river.rb