elasticsearch_http

Milestone: 2

This output lets you store logs in Elasticsearch.

This plugin uses the HTTP/REST interface to Elasticsearch, which usually lets you use any version of Elasticsearch server. It is known to work with elasticsearch 1.4.2

You can learn more about Elasticsearch at http://www.elasticsearch.org

Synopsis

This is what it might look like in your config file:
output {
  elasticsearch_http {
    codec => ... # codec (optional), default: "plain"
    document_id => ... # string (optional), default: nil
    flush_size => ... # number (optional), default: 100
    host => ... # string (required)
    idle_flush_time => ... # number (optional), default: 1
    index => ... # string (optional), default: "logstash-%{+YYYY.MM.dd}"
    index_type => ... # string (optional)
    manage_template => ... # boolean (optional), default: true
    password => ... # password (optional), default: nil
    port => ... # number (optional), default: 9200
    replication => ... # string, one of ["async", "sync"] (optional), default: "sync"
    template => ... # a valid filesystem path (optional)
    template_name => ... # string (optional), default: "logstash"
    template_overwrite => ... # boolean (optional), default: false
    user => ... # string (optional), default: nil
    workers => ... # number (optional), default: 1
  }
}

Details

codec

  • Value type is codec
  • Default value is "plain"

The codec used for output data. Output codecs are a convenient method for encoding your data before it leaves the output, without needing a separate filter in your Logstash pipeline.

document_id

  • Value type is string
  • Default value is nil

The document ID for the index. Useful for overwriting existing entries in Elasticsearch with the same ID.

exclude_tags DEPRECATED

  • DEPRECATED WARNING: This config item is deprecated. It may be removed in a further version.
  • Value type is array
  • Default value is []

Only handle events without any of these tags. Note this check is additional to type and tags.

flush_size

  • Value type is number
  • Default value is 100

This plugin uses the bulk index api for improved indexing performance. To make efficient bulk api calls, we will buffer a certain number of events before flushing that out to Elasticsearch. This setting controls how many events will be buffered before sending a batch of events.

host (required setting)

  • Value type is string
  • There is no default value for this setting.

The hostname or IP address to reach your Elasticsearch server.

idle_flush_time

  • Value type is number
  • Default value is 1

The amount of time since last flush before a flush is forced.

This setting helps ensure slow event rates don’t get stuck in Logstash. For example, if your flush_size is 100, and you have received 10 events, and it has been more than idle_flush_time seconds since the last flush, logstash will flush those 10 events automatically.

This helps keep both fast and slow log streams moving along in near-real-time.

index

  • Value type is string
  • Default value is "logstash-%{+YYYY.MM.dd}"

The index to write events to. This can be dynamic using the %{foo} syntax. The default value will partition your indices by day so you can more easily delete old data or only search specific date ranges.

index_type

  • Value type is string
  • There is no default value for this setting.

The index type to write events to. Generally you should try to write only similar events to the same ‘type’. String expansion ‘%{foo}’ works here.

manage_template

  • Value type is boolean
  • Default value is true

Starting in Logstash 1.3 (unless you set option “manage_template” to false) a default mapping template for Elasticsearch will be applied, if you do not already have one set to match the index pattern defined (default of “logstash-%{+YYYY.MM.dd}”), minus any variables. For example, in this case the template will be applied to all indices starting with logstash-*

If you have dynamic templating (e.g. creating indices based on field names) then you should set “manage_template” to false and use the REST API to upload your templates manually.

password

  • Value type is password
  • Default value is nil

The HTTP Basic Auth password used to access your elasticsearch server.

port

  • Value type is number
  • Default value is 9200

The port for Elasticsearch HTTP interface to use.

replication

  • Value can be any of: "async", "sync"
  • Default value is "sync"

Set the type of Elasticsearch replication to use. If async the index request to Elasticsearch to return after the primary shards have been written. If sync (default), index requests will wait until the primary and the replica shards have been written.

tags DEPRECATED

  • DEPRECATED WARNING: This config item is deprecated. It may be removed in a further version.
  • Value type is array
  • Default value is []

Only handle events with all of these tags. Note that if you specify a type, the event must also match that type. Optional.

template

  • Value type is path
  • There is no default value for this setting.

You can set the path to your own template here, if you so desire. If not the included template will be used.

template_name

  • Value type is string
  • Default value is "logstash"

This configuration option defines how the template is named inside Elasticsearch. Note that if you have used the template management features and subsequently change this you will need to prune the old template manually, e.g. curl -XDELETE http://localhost:9200/_template/OldTemplateName?pretty where OldTemplateName is whatever the former setting was.

template_overwrite

  • Value type is boolean
  • Default value is false

Overwrite the current template with whatever is configured in the template and template_name directives.

type DEPRECATED

  • DEPRECATED WARNING: This config item is deprecated. It may be removed in a further version.
  • Value type is string
  • Default value is ""

The type to act on. If a type is given, then this output will only act on messages with the same type. See any input plugin’s “type” attribute for more. Optional.

user

  • Value type is string
  • Default value is nil

The HTTP Basic Auth username used to access your elasticsearch server.

workers

  • Value type is number
  • Default value is 1

The number of workers to use for this output. Note that this setting may not be useful for all outputs.


This is documentation from lib/logstash/outputs/elasticsearch_http.rb