filter

Dedupe events Transform

The Vector dedupe transform accepts and outputs log events allowing you to prevent duplicate Events from being outputted by using an LRU cache.

Configuration

vector.toml
[transforms.my_transform_id]
  # General
  type = "dedupe" # required
  inputs = ["my-source-id"] # required


  # Fields
  fields.match = ["timestamp", "host", "message"] # optional, default
2 items
  • tableoptional

    cache

    Options controlling how we cache recent Events for future duplicate checking.

    • intcommonoptional

      num_events

      The number of recent Events to cache and compare new incoming Events against.

      • Default: 5000
      • View examples
  • tablecommonrequired

    fields

    Options controlling what fields to match against

    • [string]optional

      ignore

      The field names to ignore when deciding if an Event is a duplicate. Incompatible with the fields.match option.

      • No default
      • View examples
    • [string]commonoptional

      match

      The field names considered when deciding if an Event is a duplicate. This can also be globally set via the global log_schema options.Incompatible with the fields.ignore option.

      • Default: ["timestamp","host","message"]
      • View examples

How It Works

Cache Behavior

This transform is backed by an LRU cache of size cache.num_events. That means that this transform will cache information in memory for the last cache.num_events Events that it has processed. Entries will be removed from the cache in the order they were inserted. If an Event is received that is considered a duplicate of an Event already in the cache that will put that event back to the head of the cache and reset its place in line, making it once again last entry in line to be evicted.

Complex Processing

If you encounter limitations with the dedupe transform then we recommend using a runtime transform. These transforms are designed for complex processing and give you the power of full programming runtime.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.

Memory Usage Details

Each entry in the cache corresponds to an incoming Event and contains a copy of the 'value' data for all fields in the Event being considered for matching. When using fields.match this will be the list of fields specified in that config option. When using fields.ignore that will include all fields present in the incoming event except those specified in fields.ignore. Each entry also uses a single byte per field to store the type information of that field. When using fields.ignore each cache entry additionally stores a copy of each field name being considered for matching. When using fields.match storing the field names is not necessary.

Memory Utilization Estimation

If you want to estimate the memory requirements of this transform for your dataset, you can do so with these formulas:

When using fields.match:

Sum(the average size of the *data* (but not including the field name) for each field in `fields.match`) * `cache.num_events`

When using fields.ignore:

(Sum(the average size of each incoming Event) - (the average size of the field name *and* value for each field in `fields.ignore`)) * `cache.num_events`

Missing Fields

Fields with explicit null values will always be considered different than if that field was omitted entirely. For example, if you run this transform with fields.match = ["a"], the event "{a: null, b:5}" will be considered different than the event "{b:5}".

Support
Contents
Resources