Reduce Transform

The Vector reduce transform reduces multiple log events into a single log event based on a set of conditions and merge strategies.

Configuration

[transforms.my_transform_id]
type = "reduce" # required
inputs = ["my-source-or-transform-id"] # required
group_by = [] # optional, default
  • optionaltable

    ends_when

    A condition used to distinguish the final event of a transaction. If this condition resolves to true for an event, the current transaction is immediately flushed with this event.

    • commonoptionalstring

      *.contains

      Checks whether a string field contains a string argument, case sensitive. This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

      • View examples
    • commonoptionalstring

      *.ends_with

      Checks whether a string field ends with a string argument, case sensitive. This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

      • View examples
    • commonoptionalstring

      *.eq

      Check whether a field's contents exactly matches the value specified, case sensitive. This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

      • View examples
    • optionalbool

      *.exists

      Check whether a field exists or does not exist, depending on the provided value being true or false respectively.

      • View examples
    • optionalstring

      *.ip_cidr_contains

      Checks whether an IP field is contained within a given IP CIDR (works with IPv4 and IPv6). This may be a single string or a list of strings, in which case this evaluates to true if the IP field is contained within any of the CIDRs in the list.

      • View examples
    • optionalstring

      .not_

      Allow you to negate any condition listed here.

      • commonoptionalstring

        *.regex

        Checks whether a string field matches a regular expression. Vector uses the documented Rust Regex syntax. Note that this condition is considerably more expensive than a regular string match (such as starts_with or contains) so the use of those conditions are preferred where possible.

        • View examples
      • commonoptionalstring

        *.starts_with

        Checks whether a string field starts with a string argument, case sensitive. This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

        • View examples
      • enumcommonoptionalstring

        type

        The type of the condition to execute.

        • Default: "check_fields"
        • Enum, must be one of: "check_fields" "is_log" "is_metric"
        • View examples
    • optionaluint

      expire_after_ms

      A maximum period of time to wait after the last event is received before a combined event should be considered complete.

      • Default: 30000 (milliseconds)
    • optionaluint

      flush_period_ms

      Controls the frequency that Vector checks for (and flushes) expired events.

      • Default: 1000 (milliseconds)
    • commonoptional[string]

      group_by

      An ordered list of fields by which to group events. Each group is combined independently, allowing you to keep independent events separate. When no fields are specified, all events will be combined in a single group. Events missing a specified field will be combined in their own group.

      • Default: []
      • View examples
    • optionaltable

      merge_strategies

      A map of field names to custom merge strategies. For each field specified this strategy will be used for combining events rather than the default behavior.

      The default behavior is as follows:

      1. The first value of a string field is kept, subsequent values are discarded.
      2. For timestamp fields the first is kept and a new field [field-name]_end is added with the last received timestamp value.
      3. Numeric values are summed.
      • enumcommonrequiredstring

        *

        The custom merge strategy to use for a field.

        • Enum, must be one of: "array" "concat" "concat_newline" "discard" "sum" "max" "min"
        • View examples
    • optionaltable

      starts_when

      A condition used to distinguish the first event of a transaction. If this condition resolves to true for an event, the previous transaction is flushed (without this event) and a new transaction is started.

      • commonoptionalstring

        *.contains

        Checks whether a string field contains a string argument, case sensitive. This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

        • View examples
      • commonoptionalstring

        *.ends_with

        Checks whether a string field ends with a string argument, case sensitive. This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

        • View examples
      • commonoptionalstring

        *.eq

        Check whether a field's contents exactly matches the value specified, case sensitive. This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

        • View examples
      • optionalbool

        *.exists

        Check whether a field exists or does not exist, depending on the provided value being true or false respectively.

        • View examples
      • optionalstring

        *.ip_cidr_contains

        Checks whether an IP field is contained within a given IP CIDR (works with IPv4 and IPv6). This may be a single string or a list of strings, in which case this evaluates to true if the IP field is contained within any of the CIDRs in the list.

        • View examples
      • optionalstring

        .not_

        Allow you to negate any condition listed here.

        • commonoptionalstring

          *.regex

          Checks whether a string field matches a regular expression. Vector uses the documented Rust Regex syntax. Note that this condition is considerably more expensive than a regular string match (such as starts_with or contains) so the use of those conditions are preferred where possible.

          • View examples
        • commonoptionalstring

          *.starts_with

          Checks whether a string field starts with a string argument, case sensitive. This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

          • View examples
        • enumcommonoptionalstring

          type

          The type of the condition to execute.

          • Default: "check_fields"
          • Enum, must be one of: "check_fields" "is_log" "is_metric"
          • View examples

      Telemetry

      This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

      • counter

        stale_events_flushed_total

        The number of stale events that Vector has flushed. This metric includes the following tags:

        • component_kind - The Vector component kind.

        • component_name - The Vector component ID.

        • component_type - The Vector component type.

        • instance - The Vector instance identified by host and port.

        • job - The name of the job producing Vector metrics.

      • counter

        processed_events_total

        The total number of events processed by this component. This metric includes the following tags:

        • component_kind - The Vector component kind.

        • component_name - The Vector component ID.

        • component_type - The Vector component type.

        • file - The file that produced the error

        • instance - The Vector instance identified by host and port.

        • job - The name of the job producing Vector metrics.

      • counter

        processed_bytes_total

        The total number of bytes processed by the component. This metric includes the following tags:

        • component_kind - The Vector component kind.

        • component_name - The Vector component ID.

        • component_type - The Vector component type.

        • instance - The Vector instance identified by host and port.

        • job - The name of the job producing Vector metrics.

      Examples

      Given the following Vector event:

      [
      {
      "log": {
      "timestamp": "2020-10-07T12:33:21.223543Z",
      "message": "Received GET /path",
      "request_id": "abcd1234",
      "request_path": "/path",
      "request_params": {
      "key": "val"
      }
      }
      },
      {
      "log": {
      "timestamp": "2020-10-07T12:33:21.832345Z",
      "message": "Executed query in 5.2ms",
      "request_id": "abcd1234",
      "query": "SELECT * FROM table",
      "query_duration_ms": 5.2
      }
      },
      {
      "log": {
      "timestamp": "2020-10-07T12:33:22.457423Z",
      "message": "Rendered partial _partial.erb in 2.3ms",
      "request_id": "abcd1234",
      "template": "_partial.erb",
      "render_duration_ms": 2.3
      }
      },
      {
      "log": {
      "timestamp": "2020-10-07T12:33:22.543323Z",
      "message": "Executed query in 7.8ms",
      "request_id": "abcd1234",
      "query": "SELECT * FROM table",
      "query_duration_ms": 7.8
      }
      },
      {
      "log": {
      "timestamp": "2020-10-07T12:33:22.742322Z",
      "message": "Sent 200 in 15.2ms",
      "request_id": "abcd1234",
      "response_status": 200,
      "response_duration_ms": 5.2
      }
      }
      ]

      And the following configuration:

      [transforms.reduce]
      type = "reduce"

      The following Vector log event will be output:

      {
      "timestamp": "2020-10-07T12:33:21.223543Z",
      "timestamp_end": "2020-10-07T12:33:22.742322Z",
      "request_id": "abcd1234",
      "request_path": "/path",
      "request_params": {
      "key": "val"
      },
      "query_duration_ms": 13.0,
      "render_duration_ms": 2.3,
      "status": 200,
      "response_duration_ms": 5.2
      }