Split Transform

The Vector split transform accepts and outputs log events allowing you to split a field's value on a literal separator and zip the tokens into ordered field names.

Configuration

• Common
• Advanced

vector.toml

[transforms.my_transform_id]
  # General
  type = "split" # required
  inputs = ["my-source-id"] # required
  drop_field = true # optional, default
  field = "message" # optional, default
  field_names = ["timestamp", "level", "message", "parent.child"] # required
  separator = "[whitespace]" # optional, default

  # Types
  types.status = "int" # example
  types.duration = "float" # example
  types.success = "bool" # example
  types.timestamp = "timestamp|%F" # example
  types.timestamp = "timestamp|%a %b %e %T %Y" # example
  types.parent.child = "int" # example

5 items

commonrequired

• boolcommonoptional

  drop_field

  If true the field will be dropped after parsing.

  • Default: true
  • View examples
• stringcommonoptional

  field

  The field to apply the split on.

  See Field Notation Syntax for more info.

  • Default: "message"
  • View examples
• [string]commonrequired

  field_names

  The field names assigned to the resulting tokens, in order.

  See Field Notation Syntax for more info.

  • No default
  • View examples
• [string]commonoptional

  separator

  The separator to split the field on. If no separator is given, it will split on all whitespace. 'Whitespace' is defined according to the terms of the Unicode Derived Core Property White_Space.

  • Default: "[whitespace]"
  • View examples
• tablecommonoptional

  types

  Key/value pairs representing mapped log field names and types. This is used to coerce log fields into their proper types.

  • stringenumcommonoptional

    [field-name]

    A definition of log field type conversions. They key is the log field name and the value is the type. strptime specifiers are supported for the timestamp type.

    • No default
    • Enum, must be one of: "bool" "float" "int" "string" "timestamp"
    • View examples

Examples

Given the following log line:

{
  "message": "5.86.210.12,zieme4647,19/06/2019:17:20:49 -0400,GET /embrace/supply-chains/dynamic/vertical,201,20574"
}

And the following configuration:

[transforms.<transform-id>]
  type = "split"
  field = "message"
  separator = ","
  fields = ["remote_addr", "user_id", "timestamp", "message", "status", "bytes"]

  types.status = "int"
  types.bytes = "int"

A log event will be output with the following structure:

{
  // ... existing fields
  "remote_addr": "5.86.210.12",
  "user_id": "zieme4647",
  "timestamp": "19/06/2019:17:20:49 -0400",
  "message": "GET /embrace/supply-chains/dynamic/vertical",
  "status": 201,
  "bytes": 20574
}

A couple of things to notice:

1. The message field was overwritten since it was provided in the fields option.
2. The status and bytes fields are integers because of type coercion, otherwise they would be strings.

How It Works

Complex Processing

If you encounter limitations with the split transform then we recommend using a runtime transform. These transforms are designed for complex processing and give you the power of full programming runtime.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.

Field Notation Syntax

The field and field_names options support Vector's field notiation syntax, enabling access to root-level, nested, and array field values. For example:

vector.toml

[transforms.my_split_transform_id]
  # ...
  field = "message"
  field = "parent.child"
  # ...

You can learn more about Vector's field notation in the field notation reference.

Value Coercion

Values can be coerced upon extraction via the types.* options. This functions exactly like the coercer transform except that its coupled within this transform for convenience.

Timestamps

You can coerce values into timestamps via the timestamp type:

vector.toml

# ...
types.first_timestamp = "timestamp" # best effort parsing
types.second_timestamp = "timestamp|%Y-%m-%dT%H:%M:%S%z" # ISO8601
# ...

As noted above, if you do not specify a specific strftime format, Vector will make a best effort attempt to parse the timestamp against the following common formats:

Format	Description
Without Timezone
%F %T	YYYY-MM-DD HH:MM:SS
%v %T	DD-Mmm-YYYY HH:MM:SS
FT%T	ISO 8601 / RFC 3339 without TZ
m/%d/%Y:%T	US common date format
a, %d %b %Y %T	RFC 822/2822 without TZ
a %d %b %T %Y	date command output without TZ
A %d %B %T %Y	date command output without TZ, long names
a %b %e %T %Y	ctime format
With Timezone
%+	ISO 8601 / RFC 3339
%a %d %b %T %Z %Y	date command output
%a %d %b %T %z %Y	date command output, numeric TZ
%a %d %b %T %#z %Y	date command output, numeric TZ
UTC Formats
%s	UNIX timestamp
%FT%TZ	ISO 8601 / RFC 3339 UTC