Key-value Parser Transform
The Vector key_value_parser
transform
loosely parses a log field's value in key-value format.
Configuration
- Common
- Advanced
- vector.toml
- vector.yaml
- vector.json
[transforms.my_transform_id]# Generaltype = "key_value_parser" # requiredinputs = ["my-source-or-transform-id"] # requireddrop_field = true # optional, defaultfield = "message" # optional, default# Typestypes.status = "int" # exampletypes.duration = "float" # exampletypes.success = "bool" # exampletypes.timestamp_iso8601 = "timestamp|%F" # exampletypes.timestamp_custom = "timestamp|%a %b %e %T %Y" # exampletypes.timestamp_unix = "timestamp|%F %T" # exampletypes.parent.child = "int" # example
- optionalbool
drop_field
If
true
will drop the specifiedfield
after parsing.- Default:
true
- View examples
- Default:
- optionalstring
field
The log field containing key/value pairs to parse. Must be a
string
value.- Default:
"message"
- View examples
- Default:
- optionalstring
field_split
The character(s) to split a key/value pair on which results in a new field with an associated value. Must be a
string
value. See Description for more info.- Default:
"="
- View examples
- Default:
- optionalbool
overwrite_target
If [`target_field`](#target_field) is set and the log contains a field of the same nameas the target, it will only be overwritten if this is set to `true`.- Default:
false
- View examples
- Default:
- optionalstring
separator
The character(s) that separate key/value pairs. Must be a
string
value. See Description for more info.- Default:
"[whitespace]"
- View examples
- Default:
- optionalstring
target_field
If this setting is present, the parsed JSON will be inserted into thelog as a sub-object with this name.If a field with the same name already exists, the parser will fail andproduce an error.- View examples
- optionalstring
trim_key
Removes characters from the beginning and end of a key until a character thatis not listed. ex:
<key>=value
would result inkey: value
with this option set to<>
.- View examples
- optionalstring
trim_value
Removes characters from the beginning and end of a value until a characterthat is not listed. ex:
key=<<>value>>
would result inkey: value
with this option set to<>
.- View examples
- optionaltable
types
Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.
Timestamp coercions need to be prefaced with
timestamp|
, for example"timestamp|%F"
. Timestamp specifiers can use either of the following:- One of the built-in-formats listed in the Timestamp Formats table below.
- The time format
specifiers
from Rust's
chrono
library.
Types
array
bool
bytes
float
int
map
null
timestamp
(see the table below for formats)
Timestamp Formats
Format Description Example %F %T
YYYY-MM-DD HH:MM:SS
2020-12-01 02:37:54
%v %T
DD-Mmm-YYYY HH:MM:SS
01-Dec-2020 02:37:54
%FT%T
ISO 8601[RFC 3339](https://tools.ietf.org/html/rfc3339) format without time zone 2020-12-01T02:37:54
%a, %d %b %Y %T
| RFC 822/2822 without time zone |Tue, 01 Dec 2020 02:37:54
%a %d %b %T %Y
|date
command output without time zone |Tue 01 Dec 02:37:54 2020
%a %b %e %T %Y
| ctime format |Tue Dec 1 02:37:54 2020
%s
| UNIX timestamp |1606790274
%FT%TZ
| ISO 8601/RFC 3339 UTC |2020-12-01T09:37:54Z
%+
| ISO 8601/RFC 3339 UTC with time zone |2020-12-01T02:37:54-07:00
%a %d %b %T %Z %Y
|date
command output with time zone |Tue 01 Dec 02:37:54 PST 2020
%a %d %b %T %z %Y
|date
command output with numeric time zone |Tue 01 Dec 02:37:54 -0700 2020
%a %d %b %T %#z %Y
|date
command output with numeric time zone (minutes can be missing or present) |Tue 01 Dec 02:37:54 -07 2020
Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.
Telemetry
This component provides the following metrics that can be retrieved through
the internal_metrics
source. See the
metrics section in the
monitoring page for more info.
- counter
processed_events_total
The total number of events processed by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.file
- The file that produced the errorinstance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_bytes_total
The total number of bytes processed by the component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
Examples
Given the following Vector event:
{"log": {"message": "action:\"Accept\"; flags:\"802832\"; ifdir:\"inbound\"; ifname:\"eth2-05\"; logid:\"6\"; loguid:\"{0x5f0fa4d6,0x1,0x696ac072,0xc28d839a}\";"}}
And the following configuration:
[transforms.key_value_parser]type = "key_value_parser"field = "message"field_split = ":"separator = ";"target_field = "data"trim_key = "\""trim_value = "\""type = "key_value_parser"
The following Vector log event will be output:
{"message": "action:\"Accept\"; flags:\"802832\"; ifdir:\"inbound\"; ifname:\"eth2-05\"; logid:\"6\"; loguid:\"{0x5f0fa4d6,0x1,0x696ac072,0xc28d839a}\";","data": {"action": "Accept","flags": "802832","ifdir": "inbound","ifname": "eth2-05","logid": "6","loguid": "{0x5f0fa4d6,0x1,0x696ac072,0xc28d839a}"}}
How It Works
Description
The Key Value Parser accepts structured data that can be split on a character, or group of characters, and extracts it into ajson object (dictionary) of key/value pairs. The [`separator`](#separator) option allows you to define the character(s) to perform the initialsplitting of the message into pairs. The [`field_split`](#field_split) option allows you to define the character(s) which split the key from the value.