Key-value Parser Transform

The Vector key_value_parser transform loosely parses a log field's value in key-value format.

Configuration

• Common
• Advanced

• vector.toml
• vector.yaml
• vector.json

[transforms.my_transform_id]
  # General
  type = "key_value_parser" # required
  inputs = ["my-source-or-transform-id"] # required
  drop_field = true # optional, default
  field = "message" # optional, default

  # Types
  types.status = "int" # example
  types.duration = "float" # example
  types.success = "bool" # example
  types.timestamp_iso8601 = "timestamp|%F" # example
  types.timestamp_custom = "timestamp|%a %b %e %T %Y" # example
  types.timestamp_unix = "timestamp|%F %T" # example
  types.parent.child = "int" # example

9 items

common

• commonoptionalbool

  drop_field

  If true will drop the specified field after parsing.

  • Default: true
  • View examples
• commonoptionalstring

  field

  The log field containing key/value pairs to parse. Must be a string value.

  • Default: "message"
  • View examples
• optionalstring

  field_split

  The character(s) to split a key/value pair on which results in a new field with an associated value. Must be a string value. See Description for more info.

  • Default: "="
  • View examples
• optionalbool

  overwrite_target

  If [`target_field`](#target_field) is set and the log contains a field of the same name
  as the target, it will only be overwritten if this is set to `true`.

  • Default: false
  • View examples
• optionalstring

  separator

  The character(s) that separate key/value pairs. Must be a string value. See Description for more info.

  • Default: "[whitespace]"
  • View examples
• optionalstring

  target_field

  If this setting is present, the parsed JSON will be inserted into the
  log as a sub-object with this name.
  If a field with the same name already exists, the parser will fail and
  produce an error.

  • View examples
• optionalstring

  trim_key

  Removes characters from the beginning and end of a key until a character that

  is not listed. ex: <key>=value would result in key: value with this option set to <>.

  • View examples
• optionalstring

  trim_value

  Removes characters from the beginning and end of a value until a character

  that is not listed. ex: key=<<>value>> would result in key: value with this option set to <>.

  • View examples
• commonoptionaltable

  types

  Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.

  Timestamp coercions need to be prefaced with timestamp|, for example "timestamp|%F". Timestamp specifiers can use either of the following:

  1. One of the built-in-formats listed in the Timestamp Formats table below.
  2. The time format specifiers from Rust's chrono library.

  Types

  • array
  • bool
  • bytes
  • float
  • int
  • map
  • null
  • timestamp (see the table below for formats)

  Timestamp Formats

  Format	Description	Example
  %F %T	YYYY-MM-DD HH:MM:SS	2020-12-01 02:37:54
  %v %T	DD-Mmm-YYYY HH:MM:SS	01-Dec-2020 02:37:54
  %FT%T	ISO 8601[RFC
  3339](https://tools.ietf.org/html/rfc3339) format without time zone

  2020-12-01T02:37:54 %a, %d %b %Y %T | RFC 822/2822 without time zone | Tue, 01 Dec 2020 02:37:54 %a %d %b %T %Y | date command output without time zone | Tue 01 Dec 02:37:54 2020 %a %b %e %T %Y | ctime format | Tue Dec 1 02:37:54 2020 %s | UNIX timestamp | 1606790274 %FT%TZ | ISO 8601/RFC 3339 UTC | 2020-12-01T09:37:54Z %+ | ISO 8601/RFC 3339 UTC with time zone | 2020-12-01T02:37:54-07:00 %a %d %b %T %Z %Y | date command output with time zone | Tue 01 Dec 02:37:54 PST 2020 %a %d %b %T %z %Y| date command output with numeric time zone | Tue 01 Dec 02:37:54 -0700 2020 %a %d %b %T %#z %Y | date command output with numeric time zone (minutes can be missing or present) | Tue 01 Dec 02:37:54 -07 2020

  Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.

Telemetry

This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

• counter

  processed_events_total

  The total number of events processed by this component. This metric includes the following tags:

  • component_kind - The Vector component kind.

  • component_name - The Vector component ID.

  • component_type - The Vector component type.

  • file - The file that produced the error

  • instance - The Vector instance identified by host and port.

  • job - The name of the job producing Vector metrics.

• counter

  processed_bytes_total

  The total number of bytes processed by the component. This metric includes the following tags:

  • component_kind - The Vector component kind.

  • component_name - The Vector component ID.

  • component_type - The Vector component type.

  • instance - The Vector instance identified by host and port.

  • job - The name of the job producing Vector metrics.

Examples

Given the following Vector event:

{
  "log": {
    "message": "action:\"Accept\"; flags:\"802832\"; ifdir:\"inbound\"; ifname:\"eth2-05\"; logid:\"6\"; loguid:\"{0x5f0fa4d6,0x1,0x696ac072,0xc28d839a}\";"
  }
}

And the following configuration:

[transforms.key_value_parser]
type = "key_value_parser"
field = "message"
field_split = ":"
separator = ";"
target_field = "data"
trim_key = "\""
trim_value = "\""
type = "key_value_parser"

The following Vector log event will be output:

{
  "message": "action:\"Accept\"; flags:\"802832\"; ifdir:\"inbound\"; ifname:\"eth2-05\"; logid:\"6\"; loguid:\"{0x5f0fa4d6,0x1,0x696ac072,0xc28d839a}\";",
  "data": {
    "action": "Accept",
    "flags": "802832",
    "ifdir": "inbound",
    "ifname": "eth2-05",
    "logid": "6",
    "loguid": "{0x5f0fa4d6,0x1,0x696ac072,0xc28d839a}"
  }
}

How It Works

Description

The Key Value Parser accepts structured data that can be split on a character, or group of characters, and extracts it into a
json object (dictionary) of key/value pairs. The [`separator`](#separator) option allows you to define the character(s) to perform the initial
splitting of the message into pairs. The [`field_split`](#field_split) option allows you to define the character(s) which split the key from the value.