Vector v0.33.0 release notes

The Vector team is pleased to announce version 0.33.0!

Be sure to check out the upgrade guide for breaking changes in this release.

This release marks the switch of the default configuration language for Vector from TOML to YAML. We continue to support both (as well as JSON), but the documentation will prefer YAML in its configuration examples. We think this will lead to more legible configurations as well as more consistency with Vector’s Helm chart which requires YAML configuration. See the associated release highlight for more details on the motivation for this change.

In addition to the usual enhancements and bug fixes, this release includes also includes:

  • A new AWS SNS sink
Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

Known issues

  • The Debian package installer will overwrite existing /etc/default/vector and /etc/vector/vector.yaml files. This was fixed in v0.33.1.
  • The clickhouse sink health check will fail due to the accidentally inclusion of an extra / in the request URI. This was fixed in v0.34.0.

Vector Changelog

2 new features

  • The exec source now allows customization of the environment variables exposed to the subprocess via two new options:

    • clear_environment to remove environment variables propagated from Vector process
    • environment to set custom environment variables for the subprocess

    Thanks to hhromic for contributing this change!
  • A new aws_sns sink has been added to send events to AWS SNS.
    Thanks to wochinge for contributing this change!

15 enhancements

  • Vector now has the ability to disable OpenSSL probing for remote certificates via --openssl-no-probe (or VECTOR_OPENSSL_NO_PROBE).
    Thanks to hhromic for contributing this change!
  • Vector now parses more configuration fields that represent “event paths” (like the kafka source key_field) at config parse time rather than at runtime. This results in earlier surfacing of errors.
  • The http_server source can now have the response code it sends configured via the new response_code configuration parameter.
    Thanks to kunalmohan for contributing this change!
  • The route transform can now have the _unmatched route disabled via the reroute_unmatched parameter. This helps with suppressing the warning that is emitted if this output is not consumed by any other components.
    Thanks to hhromic for contributing this change!
  • The websocket sink now accepts any data type that the configured codec accepts. For example, this means it supports logs, metrics, and traces when the native or native_json codecs are in use.

  • We now publish armv7hl RPM packages. These are the same as the armv7 packages but are more accurately named per the RPM packaging guidelines.

    As part of this change we are deprecating the armv7 packages. This change should be transparent to users using yum but if you are mirroring or directly downloading the RPM files you will want to switch to the new naming scheme: vector-<version>-1.armv7hl.rpm.

    See the upgrade guide for more details.

  • The kubernetes_logs source now has an oldest_first option to configure the source to always consume the oldest file first. This is the same option that exists on the file source. It can enable better behavior by releasing file handles to rotated files before moving on to reading newer files.

  • AWS components now support the AWS FIPS endpoints by specifying use_fips_endpoint in your shared AWS config file or by setting the AWS_USE_FIPS_ENDPOINT environment variable.

    Note this does not yet work when accessing the STS endpoints to fetch authentication credentials. This is being tracked by #18382.

  • The csv codec now allows additional configuration options when encoding:

    • capacity: the capacity of the internal buffer in bytes (default 4098)
    • delimiter: the delimiter to use (defaults to ,)
    • double_quote: when enabled (default) escapes double quotes by doubling them (" is encoded as ""). If disabled, then uses the configured escape character to escape them instead.
    • escape: The escape character to use when escaping quotes (defaults to \). Only applies when double_quote is false.

    Thanks to scMarkus for contributing this change!

  • The deprecated legacy OpenSSL provider support now defaults to disabled. It can be enabled via --openssl-legacy-provider=true.

    The --openssl-legacy-provider flag will be removed in a future release but loading of this provider will still be available via OPENSSL_CONF as described in the upgrade guide.

  • The datadog_metrics sink now transmits the origin of the metric to Datadog for display in metrics explorer. Metrics from the datadog_agent source have their origin passed through but otherwise an appropriate origin header is set for other metrics sources.
  • Vector’s protobuf definition for events was updated to have a new field to store metadata. Most users will not care about this, but if you are consuming Vector’s protobuf encoded events directly please see upgrade guide.
  • The kafka sink now builds requests concurrently which is expected to improve performance.
  • The journald source now has an extra_args configuration option to allow specifying additional arguments to pass through to journalctl when fetching events.

  • VRL was updated to v0.7.0 which brings the following:

    Bug Fixes:

    • parse_nginx_log doesn’t fail if the values of key-value pairs in error logs are missing
    • encode_gzip and encode_zlib now correctly check that the compression level is valid, preventing a panic
    • type definitions for arrays and objects with undefined values was improved
    • parse_aws_vpc_flow_log now handles account-id value as a string, avoiding loss of leading zeros and handling the case where value is unknown

    Features:

    • parse_key_value can now parse values enclosed in single quote characters
    • added pretty parameter for encode_json function to produce pretty-printed JSON string
    • added community_id function for generation of Community IDs
    • parse_aws_vpc_flow_log can now handle VPC logs using version 5 fields
    • the deprecated to_timestamp function was removed (use parse_timestamp and from_unix_timestamp instead)
    • the truncate function now takes a suffix argument to control the suffix to append. This deprecates the existing ellipsis function

18 bug fixes

  • The type definition for the port field included on events emitted from the socket source is now correctly marked as an integer rather than a string. This removes unnecessary casting when the log_namespacing feature is enabled.
  • The elasticsearch sink now correctly ignores the pipeline configuration when it is an empty string ("") to allow disabling this option. Previously it would result in failed Elasticsearch requests due to an empty pipeline name being passed.
  • Fix log event generation from the kubernetes_logs source when log_namespacing is enabled. Previously it would emit empty log events.
  • When log_namespacing is enabled, the dedupe transform can now access metadata fields using the %some_field syntax.
  • When log_namespacing is enabled, the sample transform now adds the sample_rate field to metadata rather than to the event itself.
  • When log_namespacing is enabled, the remap transform now correctly handles converting an array result into log events by avoiding wrapping non-object array elements as {"message": "<the value>"} and instead using the value directly as the log event.
  • When log_namespacing is enabled, the json codec will now allow decoding of non-object values to create the log event rather than erroring if the incoming value is not an object.
  • The gelf codec now defaults to framing using null byte (\0) when encoding. Previously it defaulted to newline (\n) but the GELF server implementation expects the null byte.
    Thanks to MartinEmrich for contributing this change!

  • The published Debian packages no longer contain a conffiles control file as it was unnecessary (all configuration files are under /etc and are automatically flagged as conffiles by dh_installdeb).

    This existing conffiles file was contained an invalid trailing empty line which caused issues on some package managers such as Uyuni with SUSE Manager.

  • The component_errors_total metric is no longer incremented for HTTP client errors that are automatically retried.
  • Vector avoids a panic that occurred when attempting to encode an empty sketch metric.

  • A few fix related fixes to the New Relic sink handling of metrics were made:

    • Metric tags are sent to New Relic as attributes
    • The metric type is sent rather than New Relic always treating the metrics as gauges

    Thanks to asllop for contributing this change!

  • Vector now defaults the worker thread concurrency to the detection provided by the Rust standard library, which attempts to take into account constraints applied by containerization, rather than just the number of detected CPUs. This should result in better resource utilization in container environments.

    See the upgrade guide for more details about this.

  • The published Vector artifacts are now compiled with an allocator page size of 64 kb rather than 4 kb. This allows Vector to run on systems with 64 kb pages (e.g. CentOS 7/8 when on AARCH64) as well as continuing to run on systems with the more typical page size of 4 kb.
  • Warnings about maximum allocation groups are now suppressed when the allocation tracking feature is not in use.
  • GCP components will now automatically retry unauthorized responses. The expectation is that users would rather intervene than drop data in this scenario.

  • The following sinks now avoid ballooning in memory consumption in the pretense of back-pressure by limiting request building concurrency:

    • amqp
    • appsignal
    • azure_monitor_logs
    • clickhouse
    • gcp_stackdriver
    • kafka
    • honeycomb
    • http
    • nats
    • pulsar

  • The request.concurrency had a couple of tweaks to fix the documentation:

    • A value of none now configures “no concurrency” (same as 1). This was the documented behavior of none, but previously it would actually configure adaptive concurrency.
    • As none was the default for most sinks, to maintain the expected default behavior of adaptive concurrency, the default for request.concurrency for these sinks is now adaptive rather than none.

    See the upgrade guide for more details.

2 chore

  • The default configuration language for Vector was updated from TOML to YAML. See the highlight for more details on the motivation for this change.

    As part of this migration, Vector will prefer the /etc/vector/vector.yaml as the default location in the future rather than /etc/vector/vector.toml.

  • The datadog_logs endpoint configuration is now treated as the “base URL” where the expected path, /api/v2/logs is appended. This makes this option more consistent with the endpoint options appearing on the other Datadog sinks.

    See the upgrade guide for details.

Download Version 0.33.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...