Vector v0.24.0 release notes

The Vector team is pleased to announce version 0.24.0!

Be sure to check out the upgrade guide for breaking changes in this release.

In addition to the new features, enhancements, and fixes listed below, this release adds:

  • A new axiom sink for sending events to Axiom
  • A new gcp_chronicle_unstructured sink for sending unstructured log events to GCP Chronicle
  • A new file_descriptor source to consume input from file descriptors
  • A new opentelemetry source to receive input from OpenTelemetry collectors and SDKs. Only logs are supported in this release, but support for metrics and traces are in-flight. An opentelemetry sink will follow.
  • Support for expiring high cardinality internal metrics through the global expire_metrics (will be replaced by expire_metrics_secs in 0.24.1). This can alleviate issues with Vector using increased memory over time. For now it is opt-in, but we may make this the default in the future.

Note that this release has a backwards incompatible data model change that users of the vector sink and disk buffers should be aware of while upgrading. See the note in the upgrade guide for more details.

Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

Known issues

  • The new host_metrics metrics for physical and logical CPU counts were incorrectly implemented as new modes for the cpu_seconds_total when they were meant to be new gauges. Fixed in 0.24.1.
  • vector top and some sinks like file incorrectly report metrics from the internal_metrics source as they show the incremental metrics rather than absolute. Fixed in 0.24.1.
  • The expire_metrics_secs option added in this release was not correctly applied. Fixed in 0.24.2.
  • Supplying an empty string ("") for options that take a field name started panicking in 0.24.0 rather than disabling the option as it previously did. Fixed in 0.24.2.
  • This release was intended to add support for sending rate metrics to the datadog_metrics sink, but there was a regression in it prior to release. Fixed in 0.24.2.
  • VRL code using closures sometimes returned an incorrect type error (“block returns invalid value type”). Fixed in 0.24.2.

Vector Changelog

6 new features

  • A new axiom sink was added for sending data to Axiom.
    Thanks to bahlo for contributing this change!
  • A new gelf codec was added for decoding/encoding GELF data in Vector’s sources and sinks. It can be used via encoding.codec on sinks and decoding.codec on sources, for those that support codecs.

  • A new enrichment table type was added, geoip. This can be used with VRL’s enrichment table functions to enrich events using a GeoIP database.

    Additionally the geoip enrichment table has support for Connection-Type databases.

    This takes the place of the geoip, which has been deprecated.


    Thanks to ktff, w4 for contributing this change!

  • A new opentelemetry source was added to ingest logs from the OpenTelemetry collector and OpenTelemetry SDKs.

    We will be following with support for ingesting metrics and traces.


    Thanks to caibirdme for contributing this change!
  • A new apex sink was added to send logs to Apex. This sink was removed in v0.28 as the Apex service moved to EOL.
    Thanks to mcasper for contributing this change!
  • A new file_descriptor source was added to read events from file descriptors.
    Thanks to mcasper for contributing this change!

16 enhancements

  • The VRL compiler now rejects assignments to fields on values known not to be objects or arrays. For example, this now fails:

    foo = 1
foo.bar = 2

    Where previously it would overwrite the value of 1 with { "bar": 2 }. This was done to alleviate accidental assignments. You can still assign like:

    foo = 1
foo = {}
foo.bar = 2
  • The prometheus_exporter sink now has a suppress_timestamp option to avoid adding the timestamp to exposed metrics.
    Thanks to shenxn for contributing this change!
  • The prometheus_scrape source now uses unsigned 64-bit integers for histogram buckets, allowing it to avoid errors when scraping endpoints that have buckets with very counts that didn’t fit in an unsigned 32-bit integer.

  • A filter function was added to VRL to allow easy removal of keys from objects or elements from arrays. It can be used like:

    .kubernetes.pod_annotations = filter(.kubernetes.pod_annotations) { |key, _value|
	!starts_with(key, "checksum")
}
  • VRL’s flatten function now takes an optional parameter, separator, to configure the separator to use when flattening keys. This defaults to ., preserving the current behavior.
    Thanks to trennepohl for contributing this change!
  • The sample transform can now sample trace events.
  • A new gcp_chronicle_unstructured sink was added to send log events to GCP Chronicle as unstructured events. We expect to support UDM events in the future.

  • A number of VRL type definition bugs have been resolved to allow VRL to more precisely know the types of values and fields in more places. In general, this means less need for type coercion functions like string().

    See the release highlight for more details.

  • The host_metrics source now emits mode=iowait for host_cpu_seconds_total.
    Thanks to charmitro for contributing this change!

  • End-to-end acknowledgement support has been added to the following sinks:

    • papertrail
    • socket
    • pulsar
    • prometheus_exporter
    • prometheus_remote_write
  • AWS components now allow specifying a region to use when assuming a role via auth.region. By default, this will use the same region that the component is configured to use via region.
    Thanks to akutta for contributing this change!
  • Two new functions were added to the VRL standard library to ease detecting whether an IP address is IPv4 or IPv6:is_ipv4 and is_ipv6.
  • A new internal metric, source_lag_time_seconds, was added for sources that is a histogram of the time difference of when Vector ingests an event and the timestamp of the event itself (if it exists).

  • The internal_metrics source now emits an internal_metrics_cardinality gauge indicating the cardinality of the internal metric store.

    Previously we emitted internal_metrics_cardinality_total but this metric is a counter and so cannot account for metrics being dropped from the internal metric store. internal_metrics_cardinality_total has been deprecated and will be removed in a future release.

  • The websocket sink now allows configuration of the same authentication settings the http sink does via the new auth configuration option.
    Thanks to wez470 for contributing this change!
  • Vector’s internal metrics store, which is exposed via the internal_metrics source, now allows for expiration of metrics via a new configurable global expire_metrics configuration option. When set, the store will drop metrics that haven’t been seen in the configured duration. This can be used to expire metrics from the kubernetes_logs source, and others, which tag their internal metrics with high cardinality, but ephemeral, tags like file.

12 bug fixes

  • Vector no longer panics when reloading a config where a component is deleted and then re-added.
    Thanks to zhongzc for contributing this change!
  • The elasticsearch sink now sends any configured headers when making healthchecks.
  • Vector now has a concept of “rate” metrics which are a counter with an additional interval associated. This is used by the datadog_agent source the datadog_metrics sink to correctly pass “rate” metrics from the Datadog Agent to Datadog.
  • VRL’s parse_apache_log function now handles additional error log formats.
  • The mongodb_metrics source now correctly decodes bytes_written_from values that exceed 32-bit integers, up to 64-bit.
    Thanks to KernelErr for contributing this change!
  • Vector no longer panics after reloading an existing enrichment table that had index updates.
  • The host_metrics source has improved handling of fetching cgroups metrics from hybrid cgroups. It now checks all possible locations for stats.
  • Certificate verification now correctly works for proxied connections (when proxy is configured on a component).
    Thanks to ntim for contributing this change!
  • The docker_logs source now correctly tags internal metrics that it emits with the normal component tags (component_id, component_kind, and component_type).
    Thanks to zhongzc for contributing this change!
  • The host_metrics source now emits two new gauges for the number of CPUs on the host: physical_cpus and logical_cpus. These can be used to better interpret the load* metrics that are emitted.
  • The exec source now gracefully waits for the subprocess to exit when shutting down by sending a SIGTERM. This has only been implemented for *nix hosts. On Windows, the subprocess will be abruptly killed. We hope to improve this in the future.
  • The file, journald, and kafka sources no longer halt when end-to-end acknowledgements are enabled and an attached sink returns an error. This was a change in v0.23.0, but we backed it out to pursue improved error handling in sinks.

1 chore

  • The deprecated codec configuration for sinks was removed so that codecs must be specified as encoding.codec rather than just encoding. See the upgrade note for details.

What’s next

Expanded OpenTelemetry support
This release ships with initial OpenTelemetry support in the form of an opentelemetry source for consuming logs from OpenTelemetry collectors and SDKs. This support will be expanded to cover metrics and traces as well as an opentelemetry sink.

Download Version 0.24.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...