Filter Transform

The Vector filter transform filters logs

Configuration

[transforms.my_transform_id]
# General
type = "filter" # required
inputs = ["my-source-or-transform-id"] # required
# Condition
condition.type = "check_fields" # example
condition."message.eq" = "foo" # example
condition."message.not_eq" = "foo" # example
condition."message.exists" = true # example
condition."message.not_exists" = true # example
condition."message.contains" = "foo" # example
condition."message.not_contains" = "foo" # example
condition."message.ends_with" = "foo" # example
condition."message.not_ends_with" = "foo" # example
condition."message.ip_cidr_contains" = "10.0.0.0/8" # example
condition."message.not_ip_cidr_contains" = "10.0.0.0/8" # example
condition."message.regex" = " (any|of|these|five|words) " # example
condition."message.not_regex" = " (any|of|these|five|words) " # example
condition."message.starts_with" = "foo" # example
condition."message.not_starts_with" = "foo" # example
  • commonrequiredtable

    condition

    The set of logical conditions to be matched against every input event. Only messages that pass all conditions will be forwarded.

    • commonoptionalstring

      *.contains

      Checks whether a string field contains a string argument, case sensitive. This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

      • View examples
    • commonoptionalstring

      *.ends_with

      Checks whether a string field ends with a string argument, case sensitive. This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

      • View examples
    • commonoptionalstring

      *.eq

      Check whether a field's contents exactly matches the value specified, case sensitive. This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

      • View examples
    • optionalbool

      *.exists

      Check whether a field exists or does not exist, depending on the provided value being true or false respectively.

      • View examples
    • optionalstring

      *.ip_cidr_contains

      Checks whether an IP field is contained within a given IP CIDR (works with IPv4 and IPv6). This may be a single string or a list of strings, in which case this evaluates to true if the IP field is contained within any of the CIDRs in the list.

      • View examples
    • optionalstring

      .not_

      Allow you to negate any condition listed here.

      • commonoptionalstring

        *.regex

        Checks whether a string field matches a regular expression. Vector uses the documented Rust Regex syntax. Note that this condition is considerably more expensive than a regular string match (such as starts_with or contains) so the use of those conditions are preferred where possible.

        • View examples
      • commonoptionalstring

        *.starts_with

        Checks whether a string field starts with a string argument, case sensitive. This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

        • View examples
      • enumcommonoptionalstring

        type

        The type of the condition to execute.

        • Default: "check_fields"
        • Enum, must be one of: "check_fields" "is_log" "is_metric"
        • View examples

    Output

    Telemetry

    This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

    • counter

      processed_events_total

      The total number of events processed by this component. This metric includes the following tags:

      • component_kind - The Vector component kind.

      • component_name - The Vector component ID.

      • component_type - The Vector component type.

      • file - The file that produced the error

      • instance - The Vector instance identified by host and port.

      • job - The name of the job producing Vector metrics.

    • counter

      processed_bytes_total

      The total number of bytes processed by the component. This metric includes the following tags:

      • component_kind - The Vector component kind.

      • component_name - The Vector component ID.

      • component_type - The Vector component type.

      • instance - The Vector instance identified by host and port.

      • job - The name of the job producing Vector metrics.

    Examples

    Given the following Vector event:

    [
    {
    "log": {
    "level": "debug",
    "message": "I'm a noisy debug log"
    }
    },
    {
    "log": {
    "level": "info",
    "message": "I'm a normal info log"
    }
    }
    ]

    And the following configuration:

    [transforms.filter]
    type = "filter"
    condition."level.neq" = "debug"

    The following Vector log event will be output:

    [
    {
    "log": {
    "level": "info",
    "message": "I'm a normal info log"
    }
    }
    ]

    How It Works