LOG

split transform

The split transform accepts log events and allows you to split a field's value on a given separator and zip the tokens into ordered field names.

Configuration

vector.toml
[transforms.my_transform_id]
  # REQUIRED - General
  type = "split" # example, must be: "split"
  inputs = ["my-source-id"] # example
  field_names = ["timestamp", "level", "message"] # example
  
  # OPTIONAL - General
  drop_field = true # default
  field = "message" # default
  separator = "," # default
  
  # OPTIONAL - Types
  [transforms.my_transform_id.types]
    status = "int"

Options

5 items
commonbooloptional

drop_field

If true thefield will be dropped after parsing.

Default: true
View examples
commonstringoptional

field

The field to apply the split on.

Default: "message"
View examples
common[string]required

field_names

The field names assigned to the resulting tokens, in order.

No default
View examples
common[string]optional

separator

The separator to split the field on. If no separator is given, it will split on whitespace.

Default: "whitespace"
View examples
commontableoptional

types

Key/Value pairs representing mapped log field types.

commonstringenumrequired

[field-name]

A definition of log field type conversions. They key is the log field name and the value is the type. strptime specifiers are supported for the timestamp type.

No default
Enum, must be one of: "bool" "float" "int" "string" "timestamp"
View examples

Output

Given the following log line:

{
  "message": "5.86.210.12,zieme4647,19/06/2019:17:20:49 -0400,GET /embrace/supply-chains/dynamic/vertical,201,20574"
}

And the following configuration:

[transforms.<transform-id>]
type = "split"
field = "message"
fields = ["remote_addr", "user_id", "timestamp", "message", "status", "bytes"]
  [transforms.<transform-id>.types]
    status = "int"
    bytes = "int"

A log event will be output with the following structure:

{
  // ... existing fields
  "remote_addr": "5.86.210.12",
  "user_id": "zieme4647",
  "timestamp": "19/06/2019:17:20:49 -0400",
  "message": "GET /embrace/supply-chains/dynamic/vertical",
  "status": 201,
  "bytes": 20574
}

A few things to note about the output:

  1. The message field was overwritten.
  2. The status and bytes fields are integers because of type coercion.

How It Works

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.

Support
Contents
Resources