Logfmt Parser Transform

The Vector logfmt_parser transform accepts and outputs log events allowing you to parse a log field's value in the logfmt format.

Configuration

vector.toml
[transforms.my_transform_id]
# General
type = "logfmt_parser" # required
inputs = ["my-source-id"] # required
drop_field = true # optional, default
field = "message" # optional, default
# Types
types.status = "int" # example
types.duration = "float" # example
types.success = "bool" # example
types.timestamp = "timestamp|%F" # example
types.timestamp = "timestamp|%a %b %e %T %Y" # example
types.parent.child = "int" # example
  • boolcommonoptional

    drop_field

    If the specified field should be dropped (removed) after parsing.

    • Default: true
    • View examples
  • stringcommonoptional

    field

    The log field to parse.

    See Field Notation Syntax and Format Specification for more info.

    • Default: "message"
    • View examples
  • tablecommonoptional

    types

    Key/value pairs representing mapped log field names and types. This is used to coerce log fields into their proper types.

    • stringenumcommonoptional

      [field-name]

      A definition of log field type conversions. They key is the log field name and the value is the type. strptime specifiers are supported for the timestamp type.

      • No default
      • Enum, must be one of: "bool" "float" "int" "string" "timestamp"
      • View examples

Examples

Given the following Heroku router log line:

log event
{
"message": "at=info method=GET path=/ host=myapp.herokuapp.com request_id=8601b555-6a83-4c12-8269-97c8e32cdb22 fwd="204.204.204.204" dyno=web.1 connect=1ms service=18ms status=200 bytes=13 tls_version=tls1.1 protocol=http"
}

And the following configuration:

vector.toml
[transforms.<transform-id>]
type = "logfmt"
field = "message"
drop_field = true
types.bytes = "int"
types.status = "int"

A log event will be output with the following structure:

log event
{
// ... existing fields
"at": "info",
"method": "GET",
"path": "/",
"host": "myapp.herokuapp.com",
"request_id": "8601b555-6a83-4c12-8269-97c8e32cdb22",
"fwd": "204.204.204.204",
"dyno": "web.1",
"connect": "1ms",
"service": "18ms",
"status": 200,
"bytes": 13,
"tls_version": "tls1.1",
"protocol": "http"
}

A couple of things to note:

  1. The bytes and status fields were coerced into ints via the types options.
  2. The message field was dropped due to setting drop_field to true.

How It Works

Complex Processing

If you encounter limitations with the logfmt_parser transform then we recommend using a runtime transform. These transforms are designed for complex processing and give you the power of full programming runtime.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.

Field Notation Syntax

The field options support Vector's field notation syntax, enabling access to root-level, nested, and array field values. For example:

vector.toml
[transforms.my_logfmt_parser_transform_id]
# ...
field = "message"
field = "parent.child"
field = "array[0]"
# ...

You can learn more about Vector's field notation in the field notation reference.

Format Specification

Logfmt is, unfortunately, a very loosely defined format. There is no official specification for the format and Vector makes a best effort to parse key/value pairs delimited with a =. It works by splitting the field's value on non-quoted white-space and then splitting each token by a non-quoted = character. This makes the parsing process somewhat flexible in that the string does not need to be strictly formatted.

For example, the following log line:

log event
{
"message": "Hello world duration=2s user-agent=\"Firefox/47.3 Mozilla/5.0\""
}

Will be successfully parsed into:

log event
{
"message": "Hello world duration=2s user-agent=\"Firefox/47.3 Mozilla/5.0\"",
"duration": "2s",
"user-agent": "Firefox/47.3 Mozilla/5.0"
}

Key/Value Parsing

This transform can be used for key/value parsing. Logfmt refers to a loosely defined spec that parses a key/value pair delimited by a = character. This section, and it's keywords, is primarily added to assist users in finding this transform for these terms.

Quoting Values

Values can be quoted to capture spaces, and quotes can be escaped with \. For example

key1="value with spaces" key2="value with spaces and \""

Would result in the following log event:

log event
{
"key1": "value with spaces",
"key2": "value with spaces and \""
}

Value Coercion

Values can be coerced upon extraction via the types.* options. This functions exactly like the coercer transform except that its coupled within this transform for convenience.

Timestamps

You can coerce values into timestamps via the timestamp type:

vector.toml
# ...
types.first_timestamp = "timestamp" # best effort parsing
types.second_timestamp = "timestamp|%Y-%m-%dT%H:%M:%S%z" # ISO8601
# ...

As noted above, if you do not specify a specific strftime format, Vector will make a best effort attempt to parse the timestamp against the following common formats:

FormatDescription
Without Timezone
%F %TYYYY-MM-DD HH:MM:SS
%v %TDD-Mmm-YYYY HH:MM:SS
FT%TISO 8601 / RFC 3339 without TZ
m/%d/%Y:%TUS common date format
a, %d %b %Y %TRFC 822/2822 without TZ
a %d %b %T %Ydate command output without TZ
A %d %B %T %Ydate command output without TZ, long names
a %b %e %T %Yctime format
With Timezone
%+ISO 8601 / RFC 3339
%a %d %b %T %Z %Ydate command output
%a %d %b %T %z %Ydate command output, numeric TZ
%a %d %b %T %#z %Ydate command output, numeric TZ
UTC Formats
%sUNIX timestamp
%FT%TZISO 8601 / RFC 3339 UTC