0.20 Upgrade Guide

Vector’s 0.20.0 release includes breaking changes:

1. Change to set expiration behavior in prometheus_exporter sink
2. New metrics behavior for route transform
3. New internal events for loki sink
4. if statement predicates in VRL need to handle errors
5. Division by a literal nonzero number in VRL is no longer fallible
6. ‘syslog’ decoder will raise an error if the incoming message cannot be decoded successfully
7. Unit tests no longer allow nonexistent targets in extract_from/no_outputs_from configurations

And deprecations:

1. Version 1 of the Vector protocol has been deprecated

We cover them below to help you upgrade quickly:

Upgrade guide

Breaking changes

Change to set expiration behavior in prometheus_exporter sink

As part of work to better control memory growth in the prometheus_exporter sink, we’ve done some work to ensure that metric expiration behavior is consistent for all metrics. Instead of reporting a count of zero when they expire, sets will now stop reporting entirely when they expire and are flushed.

Due to how the prometheus_exporter sink works, Vector must store a snapshot of the state of all metrics: the latest value of a counter or a gauge, and so on. Sets represent a list of unique values, and these values can be anything from integer to strings, which over time, can grow quite large. To combat memory growth, we “expire” metrics from the sink if they have not been updated within the time window configured by flush_period_secs.

For all metrics besides sets, we would delete the metric from the internal state and stop reporting it. For sets, however, we would simply clear their values and thus report a set size of zero. This behavior was not consistent with how we expired other metric types, and additionally, could still lead to a growth of unique series in a scrape, over time, as we would clear the set but not stop reporting it.

New metrics behavior for route transform

Previously, a route transform was internally expanded into a separate transform for each defined route. Each of these expanded transforms then emitted its own metrics. Now, with improved internal support for components with multiple outputs, this expansion mechanism has been removed.

A single route transform will now emit one set of metrics. For the following metrics, specific route information will be recorded as a metric tag output.

• component_sent_events_total
• component_sent_event_bytes_total
• events_discarded_total

For example, for a route transform foo with a route first configured, the events_discarded_total metric will now look like the following:

- {"counter":{"value":10.0},"name":"events_discarded_total"... "tags":{"component_id":"foo.first","component_kind":"transform","component_name":"foo.first","component_type":"route"}}
+ {"counter":{"value":10.0},"name":"events_discarded_total"... "tags":{"component_id":"foo","component_kind":"transform","component_name":"foo","component_type":"route","output":"first"}}

New internal events for loki sink

The loki sink can be configured to either drop or rewrite the timestamp of events that are out-of-order. With this release we’ve updated the logs and metrics related to both options for out_of_order_action.

drop:

• Associated log now emitted at DEBUG, rather than WARN
• processing_errors_total metric deprecated, will be removed next release events_discarded_total metric
• component_discarded_events_total added to align the sink with new component spec

rewrite_timestamp:

• Associated log now emitted at DEBUG, rather than WARN
• processing_errors_total metric deprecated, will be removed next release
• rewritten_timestamp_events_total metric added

if statement predicates in VRL need to handle errors

Previously, predicates in an if statement were not checked at compile time to ensure they could not error. The result of this was that if an error did occur at run time whilst evaluating the predicate Vector would panic.

This is now caught at compile time. Any unhandled errors in a predicate will result in a compilation error and Vector will not start.

For example, the following VRL used to compile:

if contains(.message, "true") {
  log("true")
}

If the message was not a string an error would be raised, resulting in a panic.

To make this run now, the error needs to be handled:

if contains(.message, "true") ?? false {
  log("true")
}

Division by a literal nonzero number in VRL is no longer fallible

Previously, all division in VRL was considered a fallible operation, to handle any division by zero errors. This was even the case when you were dividing by a literal number which could obviously never be zero. This has now been fixed so dividing by a non-zero literal does not require the error to be handled.

Previously you were required to handle the error like:

.zorp = .spog / 43 ?? "this error can never happen"

Now, you will not be required to handle the error. In fact handling this error will now result in a compilation error. To fix remove the error coalesce.

.zorp = .spog / 43

‘syslog’ decoder will raise an error if the incoming message cannot be decoded successfully

Previously if the syslog codec was used on a source and the incoming message could not be parsed as a syslog message it would not raise an error - instead the whole message would be received in the .message field.

This has now been changed to raise an error instead.

Unit tests no longer allow nonexistent targets in extract_from/no_outputs_from configurations

Previously, including nonexistent targets in extract_from or no_outputs_from in a unit test configuration was allowed. However, these nonexistent targets had no outputs and could result in misleading test results. Now, errors will be raised at unit test build time if nonexistent targets are included. Note in v0.20.0, a panic occurs while in v0.20.1 informative error messages are output.

Deprecations

Version 1 of the Vector protocol has been deprecated

With this release, we’ve deprecated version 1 of the Vector protocol used by the vector sink to write data to a remote vector source. The vector source and sink now default the version field to "2" (in 0.19.0 the default of “v1” was removed). Support for version 1 of the protocol will be removed in a future release.

See the announcement post for additional details about why we introduced this new gRPC-based protocol.