unique

Milestone: 1

Synopsis

This is what it might look like in your config file:

filter {
  unique {
    add_field => ... # hash (optional), default: {}
    add_tag => ... # array (optional), default: []
    fields => ... # array (required)
    remove_field => ... # array (optional), default: []
    remove_tag => ... # array (optional), default: []
  }
}

Details

add_field

• Value type is hash
• Default value is {}

If this filter is successful, add any arbitrary fields to this event. Field names can be dynamic and include parts of the event using the %{field} Example:

filter {
  unique {
    add_field => { "foo_%{somefield}" => "Hello world, from %{host}" }
  }
}

# You can also add multiple fields at once:

filter {
  unique {
    add_field => { 
      "foo_%{somefield}" => "Hello world, from %{host}"
      "new_field" => "new_static_value"
    }
  }
}

If the event has field “somefield” == “hello” this filter, on success, would add field “foo_hello” if it is present, with the value above and the %{host} piece replaced with that value from the event. The second example would also add a hardcoded field.

add_tag

• Value type is array
• Default value is []

If this filter is successful, add arbitrary tags to the event. Tags can be dynamic and include parts of the event using the %{field} syntax. Example:

filter {
  unique {
    add_tag => [ "foo_%{somefield}" ]
  }
}

# You can also add multiple tags at once:
filter {
  unique {
    add_tag => [ "foo_%{somefield}", "taggedy_tag"]
  }
}

If the event has field “somefield” == “hello” this filter, on success, would add a tag “foo_hello” (and the second example would of course add a “taggedy_tag” tag).

exclude_tags DEPRECATED

• DEPRECATED WARNING: This config item is deprecated. It may be removed in a further version.
• Value type is array
• Default value is []

Only handle events without all/any (controlled by exclude_any config option) of these tags. Optional.

fields (required setting)

• Value type is array
• There is no default value for this setting.

The fields on which to run the unique filter.

remove_field

• Value type is array
• Default value is []

If this filter is successful, remove arbitrary fields from this event. Fields names can be dynamic and include parts of the event using the %{field} Example:

filter {
  unique {
    remove_field => [ "foo_%{somefield}" ]
  }
}

# You can also remove multiple fields at once:

filter {
  unique {
    remove_field => [ "foo_%{somefield}" "my_extraneous_field" ]
  }
}

If the event has field “somefield” == “hello” this filter, on success, would remove the field with name “foo_hello” if it is present. The second example would remove an additional, non-dynamic field.

remove_tag

• Value type is array
• Default value is []

If this filter is successful, remove arbitrary tags from the event. Tags can be dynamic and include parts of the event using the %{field} syntax. Example:

filter {
  unique {
    remove_tag => [ "foo_%{somefield}" ]
  }
}

# You can also remove multiple tags at once:

filter {
  unique {
    remove_tag => [ "foo_%{somefield}", "sad_unwanted_tag"]
  }
}

If the event has field “somefield” == “hello” this filter, on success, would remove the tag “foo_hello” if it is present. The second example would remove a sad, unwanted tag as well.

tags DEPRECATED

• DEPRECATED WARNING: This config item is deprecated. It may be removed in a further version.
• Value type is array
• Default value is []

Only handle events with all/any (controlled by include_any config option) of these tags. Optional.

type DEPRECATED

• DEPRECATED WARNING: This config item is deprecated. It may be removed in a further version.
• Value type is string
• Default value is ""

Note that all of the specified routing options (type,tags.exclude_tags,include_fields,exclude_fields) must be met in order for the event to be handled by the filter. The type to act on. If a type is given, then this filter will only act on messages with the same type. See any input plugin’s “type” attribute for more. Optional.

---

This is documentation from lib/logstash/filters/unique.rb