Vector v0.43.0 release notes

The Vector team is pleased to announce version 0.43.0!

This release contains a few notable new features along with the numerous enhancements and fixes as listed below:

  • A new opentelemetry sink with initial support for emitting logs via OTLP over HTTP. We expect this to expand to support gRPC and emission of other data types.
  • A new exclusive_route transform to route events exclusively using an ordered set of conditions
  • A new cef encoder to emit events encoded as in Common Event Format
  • A new chunked_gelf framing decoder to receive chunked GELF messages
  • Vector’s configuration now allows for use of the YAML merge operator allowing simplification of configurations with duplication
  • Two new secrets backends for loading secrets into Vector configuration: file, which loads secrets from a JSON file, and directory, which loads secrets from a tree of files.

There are no breaking changes or deprecations with this release and so no upgrade guide.

Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

Exclusive Route Transform

Author photo for
type: new feature domain: transforms

Changelog

1 chore

  • The global expire_metrics_secs configuration option now defaults to 300s rather than being disabled. To preserve the old behavior, set to a negative value to disable expiration. Thanks to jszwedko for contributing this change!

13 enhancements

  • Add ability to encode messages to Common Event Format (CEF) with the cef encoder (widely used in SIEM systems). Thanks to nabokihms for contributing this change!
  • The Kubernetes Logs source can now enrich logs with pod information on Windows. Thanks to damoxc for contributing this change!
  • The sample transform can now take in a group_by configuration option that allows logs with unique values for the patterns passed in to be sampled independently. This can reduce the complexity of the topology, since users no longer need to create separate samplers with similar configurations to handle different log streams. Thanks to hillmandj for contributing this change!

  • Expose connection_retry_options in the Pulsar sink configuration to allow customizing the connection retry behaviour of the pulsar client. This includes the following options:

    • min_backoff_ms: Minimum delay between connection retries.
    • max_backoff_secs: Maximum delay between reconnection retries.
    • max_retries: Maximum number of connection retries.
    • connection_timeout_secs: Time limit to establish a connection.
    • keep_alive_secs: Keep-alive interval for each broker connection.
    Thanks to FRosner for contributing this change!
  • The http sink now retries requests when the response is a request timeout (HTTP 408). Thanks to noble-varghese pront for contributing this change!
  • Pipeline name is now an optional configuration item for GreptimeDB log sink. Thanks to sunng87 for contributing this change!
  • The dnstap source now supports decoding of EDE code 30 (Invalid Query Type) (added in Compact Denial of Existence in DNSSEC) and has the correct purpose attached to it. Thanks to esensar for contributing this change!
  • Add VECTOR_HOSTNAME env variable to override the hostname used in the Vector events and internal metrics. This is useful when Vector is running on a system where the hostname is not meaningful, such as in a container (Kubernetes). Thanks to nabokihms for contributing this change!
  • Support compression for the Honeycomb sink. The zstd format is enabled by default. Thanks to hgiasac for contributing this change!
  • Vector now supports two additional back-ends for loading secrets: file, for reading a set of secrets from a JSON file, and directory, for loading secrets from a list of files. Thanks to tie for contributing this change!
  • Add Gzip compression support to the gcp_chronicle_unstructured sink. See the documentation. Thanks to chocpanda for contributing this change!
  • The sample transform now has a sample_rate_key configuration option, which default to sample_rate. It allows configuring which key is used to attach the sample rate to sampled events. If set to an empty string, the sample rate will not be attached to sampled events. Thanks to dekelpilli for contributing this change!
  • Support for watching config file changes by polling at certain interval rather than relying on notifications. This can be enabled setting --watch-config-method to poll where the interval can be configured via --watch-config-poll-interval-seconds. Thanks to amribm for contributing this change!

9 new features

  • VRL was updated to v0.20.0. This includes the following changes:

    Breaking Changes & Upgrade Guide

    New Features

    Enhancements

    Fixes

  • The Elasticsearch sink can now write to Amazon OpenSearch Serverless via the opensearch_service_type = "serverless" option. Thanks to handlerbot AvihaiSam for contributing this change!

  • Allows for chunked GELF decoding in message-based sources, such as UDP sockets or unix datagram sockets. Implementation is based on Graylog’s documentation. The implementation also supports payload decompression.

    This framing method can be configured via the framing.method = "chunked_gelf" option in the source configuration.

    Thanks to jorgehermo9 for contributing this change!
  • Adds support for loading and concatenating multiple VRL files in the remap transform via the files option. This allows users to break down Vector remaps into smaller, more manageable units of configuration, improving organization, reusability, and maintainability of VRL code. Thanks to brittonhayes for contributing this change!
  • The opentelemetry source can now be configured to enrich log events with HTTP headers received in the OTLP/HTTP request. Thanks to jblazquez for contributing this change!
  • Introduce a new exclusive_route transform, which functions as a switch statement to route events based on user-defined conditions. See the release highlight for more details on how to use this new transform. Thanks to pront for contributing this change!

  • The elasticsearch sink now supports publishing events as bulk “update"s by configuring bulk.action to update. While using this mode has a couple of constraints:

    1. The message must be added in .doc and have .doc_as_upsert to true.
    2. id_key must be set, and the encoding field should specify doc and doc_as_upsert as values
    Thanks to blackrez for contributing this change!
  • Introducing the first version of the OpenTelemetry sink. This initial implementation supports emitting logs as OTLP over HTTP. Support is expected to expand in the future. Thanks to pront for contributing this change!
  • The host_metrics now supports process metrics collection, configurable via the process option. Thanks to leeteng2001 for contributing this change!

6 bug fixes

  • exec and http_server sources no longer attach a redundant timestamp field when log namespacing is enabled. Thanks to rwakulszowa for contributing this change!
  • The gcp_pubsub sink now supports emitting metrics and traces. Thanks to genadipost for contributing this change!
  • Vector now supports YAML merges in configuration files, a part of the YAML 1.1 specification. This functionality is useful for reducing the size of transform configurations. See the YAML documentation. Thanks to lattwood for contributing this change!
  • The request body of the Honeycomb sink should be encoded as an array according to the API docs. Thanks to hgiasac for contributing this change!
  • Fix bug in implementation of Datadog search syntax which causes queries based on attributes with boolean values to be ignored. Thanks to ArunPiduguDD for contributing this change!
  • The gelf codec now correctly deserializes the subsecond portion of timestamps rather than dropping them. Thanks to jszwedko for contributing this change!

Download Version 0.43.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...