Grok Parser Transform

The Vector grok_parser transform parses logs

Configuration

[transforms.my_transform_id]
# General
type = "grok_parser" # required
inputs = ["my-source-or-transform-id"] # required
drop_field = true # optional, default
field = "message" # optional, default
pattern = "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}" # required
# Types
types.status = "int" # example
types.duration = "float" # example
types.success = "bool" # example
types.timestamp_iso8601 = "timestamp|%F" # example
types.timestamp_custom = "timestamp|%a %b %e %T %Y" # example
types.parent.child = "int" # example
  • commonoptionalbool

    drop_field

    If true will drop the specified field after parsing.

    • Default: true
    • View examples
  • commonoptionalstring

    field

    The log field to execute the pattern against. Must be a string value.

    • Default: "message"
    • View examples
  • commonrequiredstring

    pattern

    The Grok pattern

    • View examples
  • commonoptionaltable

    types

    Key/value pairs representing mapped log field names and types. This is used to coerce log fields into their proper types.

Output

Telemetry

This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

  • counter

    processing_errors_total

    The total number of processing errors encountered by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • error_type - The type of the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    processed_events_total

    The total number of events processed by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • file - The file that produced the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    processed_bytes_total

    The total number of bytes processed by the component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

How It Works

Available Patterns

Vector uses the Rust grok library. All patterns listed here are supported. It is recommended to use maintained patterns when possible since they will be improved over time by the community.

Testing

We recommend the Grok debugger for Grok testing.