Reduce Transform

The Vector reduce transform accepts and outputs log events, allowing you to combine multiple events into a single event based on a set of identifiers.

Configuration

vector.toml
[transforms.my_transform_id]
# General
type = "reduce" # required
inputs = ["my-source-or-transform-id"] # required
identifier_fields = [] # optional, default
# Ends when
ends_when.type = "check_fields" # optional, default
ends_when."message.eq" = "this is the content to match against" # example
ends_when."message.eq" = ["match this", "or this"] # example
ends_when."message.contains" = "foo" # example
ends_when."message.contains" = ["foo", "bar"] # example
ends_when."environment.ends_with" = "-staging" # example
ends_when."environment.ends_with" = ["-staging", "-running"] # example
ends_when."message.regex" = " (any|of|these|five|words) " # example
ends_when."environment.starts_with" = "staging-" # example
ends_when."environment.starts_with" = ["staging-", "running-"] # example

Options

  • tablecommonoptional

    ends_when

    A condition used to distinguish the final event of a transaction. If this condition resolves to true for an event the transaction it belongs to is immediately flushed.

    • stringenumcommonoptional

      type

      The type of the condition to execute.

      • Default: "check_fields"
      • Enum, must be one of: "check_fields" "is_log" "is_metric"
      • View examples
    • stringcommonoptional

      [field-name].eq

      Check whether a fields contents exactly matches the value specified. This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples
    • booloptional

      [field-name].exists

      Check whether a field exists or does not exist, depending on the provided value being true or false respectively.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples
    • stringoptional

      [field-name].neq

      Check whether a fields contents does not match the value specified. This may be a single string or a list of strings, in which case this evaluates to false if any of the list matches.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples
    • anyoptional

      [field-name].not_[condition]

      Check if the given [condition] does not match.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples
    • stringcommonoptional

      [field_name].contains

      Checks whether a string field contains a string argument. This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples
    • stringcommonoptional

      [field_name].ends_with

      Checks whether a string field ends with a string argument. This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples
    • stringoptional

      [field_name].ip_cidr_contains

      Checks whether an IP field is contained within a given IP CIDR (works with IPv4 and IPv6). This may be a single string or a list of strings, in which case this evaluates to true if the IP field is contained within any of the CIDRs in the list.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples
    • stringcommonoptional

      [field_name].regex

      Checks whether a string field matches a regular expression. Vector uses the documented Rust Regex syntax. Note that this condition is considerably more expensive than a regular string match (such as starts_with or contains) so the use of those conditions are preferred where possible.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples
    • stringcommonoptional

      [field_name].starts_with

      Checks whether a string field starts with a string argument. This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples
  • intoptional

    expire_after_ms

    A maximum period of time to wait after the last event is received before a combined event should be considered complete.

    • Default: 30000
  • intoptional

    flush_period_ms

    Controls the frequency that Vector checks for (and flushes) expired events.

    • Default: 1000
  • [string]commonoptional

    identifier_fields

    An ordered list of fields by which to group events. Each group is combined independently, allowing you to keep independent events separate. When no fields are specified, all events will be combined in a single group. Events missing a specified field will be combined in their own group.

    • Default: []
    • View examples
  • tableoptional

    merge_strategies

    A map of field names to custom merge strategies. For each field specified this strategy will be used for combining events rather than the default behavior.

    The default behavior is as follows:

    1. The first value of a string field is kept, subsequent values are discarded.
    2. For timestamp fields the first is kept and a new field [field-name]_end is added with the last received timestamp value.
    3. Numeric values are summed.
    • stringenumcommonrequired

      [field-name]

      The custom merge strategy to use for a field.

      • No default
      • Enum, must be one of: "array" "concat" "discard" "sum" "max" "min"
      • View examples

Examples

Given the following configuration:

[transforms.transaction_events]
type = "reduce"
inputs = [...]

And these three log events:

{
"message": "A thing is happening",
"custom_string_field_1": "value1",
"this_much": 1
}

Followed by:

{
"message": "That thing is still happening",
"custom_string_field_2": "value2",
"this_much": 3
}

And, finally:

{
"message": "That thing is concluded",
"custom_string_field_3": "value3",
"this_much": 2
}

A single log event will eventually be produced:

{
"message": "A thing is happening",
"custom_string_field_1": "value1",
"custom_string_field_2": "value2",
"custom_string_field_3": "value3",
"this_much": 6
}

Notice that string values have not been overridden and integer values have been summed.

How It Works

Complex Processing

If you encounter limitations with the reduce transform then we recommend using a runtime transform. These transforms are designed for complex processing and give you the power of full programming runtime.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.