0.22 Upgrade Guide

An upgrade guide that addresses breaking changes in 0.22.0

by
Author avatar for spencergilbert Spencer Gilbert , Author avatar for jszwedko Jesse Szwedko

Vector’s 0.22.0 release includes breaking changes:

  1. gcp_stackdriver_metrics configuration change
  2. VRL now supports template strings
  3. encode_key_value and encode_logfmt quote wrapping behavior change

and deprecations:

  1. Imminent removal of transforms replaced by remap

We cover them below to help you upgrade quickly:

Upgrade guide

Breaking changes

gcp_stackdriver_metrics configuration change

The gcp_stackdriver_metrics sink now matches the gcp_stackdriver_logs configuration, and doesn’t require an additional labels section to add labels to submitted metrics.

TOML transform example

Old configuration

[sinks.my_sink_id]
type = "gcp_stackdriver_metrics"
inputs = [ "my-source-or-transform-id" ]
credentials_path = "/path/to/credentials.json"
project_id = "vector-123456"

  [sinks.my_sink_id.resource]
  type = "global"

    [sinks.my_sink_id.resource.labels]
    projectId = "vector-123456"
    instanceId = "Twilight"
    zone = "us-central1-a"

New configuration

[sinks.my_sink_id]
type = "gcp_stackdriver_metrics"
inputs = [ "my-source-or-transform-id" ]
credentials_path = "/path/to/credentials.json"
project_id = "vector-123456"

  [sinks.my_sink_id.resource]
  type = "global"
  projectId = "vector-123456"
  instanceId = "Twilight"
  zone = "us-central1-a"

For more information on the new syntax, you can review the documentation here

VRL now supports template strings

VRL strings can now be templated. It is now possible to insert the values of variables by inserting a placeholder with an embedded variable name into the string using {{..}}. For example in the following code:

beverage = "coffee"
preference = "I love to drink {{ beverage }}!"

assert!(preference == "I love to drink coffee!")

It should be noted that currently the placeholder must contain a simple variable name and that variable must resolve to a string.

This will not work:

stars = 42
sky = "There are {{ stars }} in the sky."

Instead, the variable must be converted to a string first:

stars = to_string(42)
sky = "There are {{ stars }} in the sky."

Also paths are currently not supported, so this will not work:

message = "The message is {{ .message }}."

Assign the field to a variable first:

message = .message
message = "The message is {{ message }}."

If you wish to insert {{ into the string, you can escape using \{{ and \}}. You also still have the option to use raw strings (s'...'):

assert!("\{{ right here \}}" == s'{{ right here }}')

encode_key_value and encode_logfmt quote wrapping behavior change

Values and keys containing whitespace and/or double quotes are now wrapped in double quotes with the original quotes escaped. This change brings encode_logfmt inline with the defined spec used by other libraries.

Previously, only keys and values containing whitespace were wrapped in quotes.

If using encode_logfmt, the previous behavior would result in messages similar to:

lvl=info msg={"some":"val"}

With this change, the message would be encoded as:

lvl=info msg="{\"some\":\"val\"}"

Deprecations

Imminent removal of deprecated transforms replaced by remap and reduce

When the remap transform was introduced, several transforms were deprecated and removed from the documentation, despite still being available in Vector. Additionally the merge transform was replaced by reduce. In 0.23.0 we will be finally removing these transforms as part of some cleanup to our reference documentation.

Transforms to be removed:

  • add_fields
  • add_tags
  • ansi_stripper
  • aws_cloudwatch_logs_subscription_parser
  • coercer
  • concat
  • grok_parser
  • json_parser
  • key_value_parser
  • logfmt_parser
  • merge
  • regex_parser
  • remove_fields
  • remove_tags
  • rename_fields
  • split
  • tokenizer

See below for examples of how to replicate functionality using the remap transform.

add_fields

Before:

[transforms.add_fields]
type = "add_fields"
inputs = ["some_input"]
fields.parent.child2 = "value2"

After:

[transforms.add_fields]
type = "remap"
inputs = ["some_input"]
source = '''
.parent.child2 = "value2"
'''
add_tags

Before:

[transforms.add_tags]
type = "add_tags"
inputs = ["some_input"]
tags.some_tag = "some_value"

After:

[transforms.add_tags]
type = "remap"
inputs = ["some_input"]
source = '''
.tags.some_tag = "some_value"
'''
ansi_stripper

Before:

[transforms.ansi_stripper]
type = "ansi_stripper"
inputs = ["some_input"]

After:

[transforms.ansi_stripper]
type = "remap"
inputs = ["some_input"]
drop_on_error = false
source = '''
.message = strip_ansi_escape_codes(string!(.message))
'''
aws_cloudwatch_logs_subscription_parser

Before:

[transforms.aws_cloudwatch_logs_subscription_parser]
type = "aws_cloudwatch_logs_subscription_parser"
inputs = ["some_input"]

After:

[transforms.aws_cloudwatch_logs_subscription_parser]
type = "remap"
inputs = ["some_input"]
drop_on_error = false
source = '''
. |= parse_aws_cloudwatch_log_subscription_message!(.message)
'''
coercer

Before:

[transforms.coercer]
type = "coercer"
inputs = ["some_input"]
types.some_bool = "bool"
types.some_float = "float"
types.some_int = "int"
types.some_string = "string"
types.some_timestamp = "timestamp"

After:

[transforms.coercer]
type = "remap"
inputs = ["some_input"]
source = '''
.some_bool = to_bool!(.some_bool)
.some_float = to_float!(.some_float)
.some_int = to_int!(.some_int)
.some_string = to_string!(.some_string)
.some_timestamp = to_timestamp!(.some_timestamp)
'''
concat

Before:

[transforms.concat]
type = "concat"
inputs = ["some_input"]
items = ["month", "day", "year"]
target = "date"
joiner = "/"

After:

[transforms.concat]
type = "remap"
inputs = ["some_input"]
drop_on_error = false
source = '''
.date = join!([.month, .day, .year], "/")
'''
grok_parser

Before:

[transforms.grok_parser]
type = "grok_parser"
inputs = ["some_input"]
pattern = "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
types.timestamp = "timestamp|%+"
types.level = "string"
types.message = "string"

After:

[transforms.grok_parser]
type = "remap"
inputs = ["some_input"]
drop_on_error = false
source = '''
. |= parse_grok!(.message, "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}")
.timestamp = parse_timestamp!(.timestamp , format: "%+")
'''
json_parser

Before:

[transforms.json_parser]
type = "json_parser"
inputs = ["some_input"]

After:

[transforms.json_parser]
type = "remap"
inputs = ["some_input"]
drop_on_error = false
source = '''
. |= object!(parse_json(.message))
'''
key_value_parser

Before:

[transforms.key_value_parser]
type = "key_value_parser"
inputs = ["some_input"]

After:

[transforms.key_value_parser]
type = "remap"
inputs = ["some_input"]
drop_on_error = false
source = '''
. |= parse_key_value!(.message)
'''
logfmt_parser

Before:

[transforms.logfmt_parser]
type = "logfmt_parser"
inputs = ["some_input"]

After:

[transforms.logfmt_parser]
type = "remap"
inputs = ["some_input"]
drop_on_error = false
source = '''
. |= parse_logfmt!(.message)
'''
merge

Before:

[transforms.merge]
type = "merge"
inputs = ["some_input"]

After:

[transforms.merge]
type = "reduce"
inputs = ["some_input"]
starts_when = "._partial == true"
merge_strategies.message = "concat"
regex_parser

Before:

[transforms.regex_parser]
type = "regex_parser"
inputs = ["some_input"]
patterns = ['^(?P<host>[\w\.]+) - (?P<user>[\w]+) (?P<bytes_in>[\d]+) \[(?P<timestamp>.*)\] "(?P<method>[\w]+) (?P<path>.*)" (?P<status>[\d]+) (?P<bytes_out>[\d]+)$']
types.bytes_in = "int"
types.timestamp = "timestamp|%d/%m/%Y:%H:%M:%S %z"
types.status = "int"
types.bytes_out = "int"

After:

[transforms.regex_parser]
type = "remap"
inputs = ["some_input"]
drop_on_error = false
source = '''
. |= parse_regex!(.message, [#"^(?P<host>[\w\.]+) - (?P<user>[\w]+) (?P<bytes_in>[\d]+) \[(?P<timestamp>.*)\] "(?P<method>[\w]+) (?P<path>.*)" (?P<status>[\d]+) (?P<bytes_out>[\d]+)$"#]
.bytes_in = to_int!(.bytes_in)
.some_timestamp = parse_timestamp!(.some_timestamp, "%d/%m/%Y:%H:%M:%S %z")
.status = to_int!(.status)
.bytes_out = to_int!(.bytes_out)
'''
remove_fields

Before:

[transforms.remove_fields]
type = "remove_fields"
inputs = ["some_input"]
fields = ["parent.child"]

After:

[transforms.remove_fields]
type = "remap"
inputs = ["some_input"]
source = '''
del(.parent.child)
'''
remove_tags

Before:

[transforms.remove_tags]
type = "remove_tags"
inputs = ["some_input"]
tags = ["some_tag"]

After:

[transforms.remove_tags]
type = "remap"
inputs = ["some_input"]
source = '''
del(.tags.some_tag)
'''
rename_fields

Before:

[transforms.rename_fields]
type = "rename_fields"
inputs = ["some_input"]
fields.new_name = ["old_name"]

After:

[transforms.rename_fields]
type = "remap"
inputs = ["some_input"]
source = '''
.new_name = del(.old_name)
'''
split

Before:

[transforms.split]
type = "split"
inputs = ["some_input"]
field_names = ["remote_addr", "user_id", "timestamp", "message", "status", "bytes"]
types.status = "int"
types.bytes = "int"

After:

[transforms.split]
type = "remap"
inputs = ["some_input"]
drop_on_error = false
source = '''
values = split(.message)
.remote_addr = values[0]
.user_id = values[1]
.timestamp = values[2]
.message = values[3]
.status = to_int!(values[4])
.bytes = to_int!(values[5])
'''
tokenizer

Before:

[transforms.tokenizer]
type = "tokenizer"
inputs = ["some_input"]
field_names = ["remote_addr", "ident", "user_id", "timestamp", "message", "status", "bytes"]
.types.status = "int"
.types.bytes = "int"

After:

[transforms.tokenizer]
type = "remap"
inputs = ["some_input"]
drop_on_error = false
source = '''
values = parse_tokens!(.message)
.remote_addr = values[0]
.user_id = values[1]
.timestamp = values[2]
.message = values[3]
.status = to_int!(values[4])
.bytes = to_int!(values[5])
'''