Swimlanes Transform

The Vector swimlanes transform accepts and outputs log events allowing you to route events across parallel streams using logical filters.

Configuration

vector.toml
[transforms.my_transform_id]
# General
type = "swimlanes" # required
inputs = ["my-source-id"] # required
# Lanes
[transforms.my_transform_id.lanes.`[swimlane-id]`]
type = "check_fields" # optional, default
"message.eq" = "this is the content to match against" # example
"message.contains" = "foo" # example
"environment.ends_with" = "-staging" # example
"message.regex" = " (any|of|these|five|words) " # example
"environment.starts_with" = "staging-" # example
tablecommonrequired

lanes

A table of swimlane identifiers to logical conditions representing the filter of the swimlane. Each swimlane can then be referenced as an input by other components with the name <transform_name>.<swimlane_id>.

tablecommonrequired

[swimlane-id]

The identifier of a swimlane.

stringenumcommonoptional
type

The type of the condition to execute.

Default: "check_fields"
Enum, must be one of: "check_fields" "is_log" "is_metric"
View examples
stringcommonoptional
[field-name].eq

Check whether a fields contents exactly matches the value specified.

Only relevant when: type = "check_fields"
No default
View examples
booloptional
[field-name].exists

Check whether a field exists or does not exist, depending on the provided value being true or false respectively.

Only relevant when: type = "check_fields"
No default
View examples
stringoptional
[field-name].neq

Check whether a fields contents does not match the value specified.

Only relevant when: type = "check_fields"
No default
View examples
stringcommonoptional
[field_name].contains

Checks whether a string field contains a string argument.

Only relevant when: type = "check_fields"
No default
View examples
stringcommonoptional
[field_name].ends_with

Checks whether a string field ends with a string argument.

Only relevant when: type = "check_fields"
No default
View examples
stringcommonoptional
[field_name].regex

Checks whether a string field matches a regular expression. Vector uses the documented Rust Regex syntax. Note that this condition is considerably more expensive than a regular string match (such as starts_with or contains) so the use of those conditions are preferred where possible.

Only relevant when: type = "check_fields"
No default
View examples
stringcommonoptional
[field_name].starts_with

Checks whether a string field starts with a string argument.

Only relevant when: type = "check_fields"
No default
View examples

Output

The swimlanes transform accepts and outputs log events allowing you to route events across parallel streams using logical filters. For example:

The swimlanes transform, in it's simplest form, can act as a simple if/else stream splitter. For example, we can route events from the host gerry to a sink only_loves_gerry, and all other events to a sink hates_gerry:

vector.toml
[transforms.splitting_gerrys]
inputs = [ "somewhere" ]
type = "swimlanes"
[transforms.splitting_gerrys.lanes.only_gerrys]
type = "check_fields"
"host.eq" = "gerry"
[transforms.splitting_gerrys.lanes.no_gerrys]
type = "check_fields"
"host.neq" = "gerry"
[sinks.only_loves_gerry]
inputs = [ "splitting_gerrys.only_gerrys" ]
type = "something"
[sinks.hates_gerry]
inputs = [ "splitting_gerrys.no_gerrys" ]
type = "something_else"

This syntax makes it easy to create arbitrary numbers of swimlanes, each with their own logic, and easily route them to downstream components.

How It Works

Complex Processing

If you encounter limitations with the swimlanes transform then we recommend using a runtime transform. These transforms are designed for complex processing and give you the power of full programming runtime.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.