Vector v0.18.0 release notes

The Vector team is pleased to announce version 0.18.0!

Be sure to check out the upgrade guide for breaking changes in this release.

In case you missed it, we’ve also released a new unified vector helm chart! This new chart can deployed as either as either an agent or aggregator role and so deprecates our existing vector-agent and vector-aggregator charts. See the chart upgrade guide for how to transition from the old charts.

Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

Known issues

  • The elasticsearch sink incorrectly prints a message for each delivered event. Fixed in v0.18.1.
  • A change to internal telemetry causes aggregated histograms emitted by the prometheus_exporter and prometheus_remote_write sinks to be incorrectly tallied. Fixed in v0.18.1.
  • The new automatic namespacing feature broke running Vector from the published RPM due to it trying to load directories from /etc/vector that are not valid. Fixed in v0.18.1.
  • The new reroute_dropped feature of remap always creates the dropped output even if reroute_dropped = false. Fixed in v0.18.1.
  • The headers_key option for the kafka sink was inadvertantly changed to headers_field. Fixed in v0.19.0.
  • If --config-dir is used, Vector incorrectly tries to load files with unknown extensions. Fixed in v0.19.0.

  • encoding.only_fields failed to deserialize correctly for sinks that used fixed encodings (i.e. those that don’t have encoding.codec). Fixed in v0.19.2. As a workaround, you can split the paths up in your configuration like:

    encoding.only_fields = ["message", "foo.bar"]

    to

    encoding.only_fields = [["message"], ["foo", "bar"]]

    You will need to convert it back to its original representation when upgrading to >= v0.19.2.

Changelog

3 deprecations

  • The deprecated batch.max_size parameter has been removed in this release. See the upgrade guide for more.
  • The deprecated request.in_flight_limit has been removed in this release. See the upgrade guide for more.
  • The deprecated host and namespace field on the datadog_metrics sink has been removed. See the upgrade guide for more.

17 enhancements

  • Instrumentation has been added to sink buffers to help give more visibility into their operation. The following metrics have been added:

    • buffer_byte_size (disk buffer only): The number of bytes in the buffer
    • buffer_events (in-memory buffer only): The number of events in the buffer
    • buffer_received_event_bytes_total: The number of bytes that have been received by this buffer. This count does not include discarded events.
    • buffer_sent_event_bytes_total: The number of bytes that have been sent from the buffer to its associated sink.
    • buffer_received_events_total: The number of events that have been received by this buffer. This count does not include discarded events.
    • buffer_sent_events_total: The number of events that have been sent from the buffer to its associated sink.
    • buffer_discarded_events_total: The number of events that have been discarded from the buffer because it is full (relevant when when_full is drop_newest)
  • The $LOG environment variable for configuring the Vector log level has been renamed to $VECTOR_LOG. $LOG is still also accepted for backwards compatibility. This change makes logging configuration more in-line with Vector’s other environment variable based options, and isolates Vector from being affected by other generic environment variables.
  • VRL diagnostic error messages have been improved to suggest null, true, or false for undefined variables. This helps guide users to realize when they are trying to use a keyword like nil that doesn’t actually exist in VRL.
  • The log_to_metric transform now also allows emitting absolute counters in addition to relative counters via kind = "absolute".
  • The status tag for the http_client_responses_total internal metric was updated to be just the integer (e.g. 200) rather than including the text portion of the HTTP response code (e.g. 200 OK).
  • The kubernetes_logs source now annotates logs with the pod_owner when available.
  • The papertrail sink now allows process field to be set to a event field value the templatable process key.
  • The aws_s3 sink now has less connections terminated prematurely as it optimistically terminates connections before AWS’s timeout.
  • The prometheus_exporter now expires metrics that haven’t been seen since the last flush (controlled by flush_interval_secs) to avoid holding onto stale metrics indefinitely and consuming increasing amounts of memory.
  • Added support for end-to-end acknowledgements to the aws_kinesis_firehose source, journald source, and file sink.
  • The utilization metric for most transforms no longer count time spent blocked on downstream components as busy. This means they should more accurately represent the time spent in that specific transform and require less interpretation to find bottlenecks.
  • The datadog_metrics sink now supports sending distribution data to Datadog like histograms and aggregated samples.
  • The kubernetes_logs source has been updated to be less demanding on the Kubernetes API server (and backing etcd cluster) by allowing for slightly stale data to be used for log enrichment rather than always requesting the most-up-to-date metadata.
  • The generator source has been renamed to demo_logs. We feel this name better reflects the intent of the source. An alias has been added to maintain compatibility.
  • The framing and decoding options are now available on heroku_logs source. See the framing and decoding highlight from v0.17.0 for more about this new source feature.
  • The upper_limit field for aggregated summaries from the metric_to_log transform has been renamed to q which is a common shorthand for quantile.
  • We have continued to add additional instrumentation to components with the goal of having them all match the Component Specification. Once we have finished this we will post a highlight outlining all of the added metrics.

5 new features

  • Initial support for routing failed events from transforms has been added, starting with the remap transform. See the highlight for more.
  • Initial support for enriching events from external data sources has been added via a new Vector concept, enrichment tables. To start, we’ve added support for enriching events with data from a CSV file. See the highlight for more.
  • A new throttle transform has been added for controlling costs. See the highlight for more.
  • Better support for breaking up Vector configuration into multiple files was added via deriving configuration from file and directory names. See the highlight for more.
  • A new aws_sqs source was added for consuming messages from AWS SQS as log events.

9 bug fixes

  • Configuring the number of threads (via --threads) for Vector now actually takes effect again rather than it always using the number of available cores. This was a regression in v0.13.
  • Vector no longer crashes when configuration was reloaded that include changes to both the order of inputs for a component and configuration of one of those inputs.
  • The obsolete event_per_line configuration option was removed from the exec source. This option became non-functional in 0.17.0 but was left available to be configured. Instead, the new framing option can be used to choose between interpreting th e output of the subcommand as an event per line or all at once as one event.
  • Fix regression in v0.17.0 for aws_s3 sink where it would add a / to the prefix provided. The sink no longer adds this / to replace previous behavior.
  • Fix memory leak that occurred when using Vector as a Windows service.
  • Fix lock-ups with the aws_s3, loki, and datadog_logs sinks.
  • Fix naming of the VRL compact function’s object argument to match the docs. This was incorrectly implemented named map in the implementation.
  • The component_sent_bytes_total internal metric is now reported after events are successfully sent to HTTP-based sinks rather than before they are sent.
  • The influxdb_metrics and influxdb_logs sinks now use /ping for healthchecks rather than /health to work with Influx DB 2 Cloud.

What’s next

Component metric standardization
We are in the process of ensuring that all Vector components report a consistent set of metrics to make it easier to monitor the performance of Vector. These metrics are outlined in this new instrumentation specification).
VRL iteration support
A common request from users when the incoming log event shape is unknown, to be able to iterate over the keys and values in those log events. We recently published an RFC for this and expect to implement this support this quarter.

Download Version 0.18.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...