Vector v0.21.0 release notes

Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.
Known issues
  • The kubernetes_logs source can panic when while processing Kubernetes watcher events when there is an error. #12245. Fixed in 0.21.1.
  • The elasticsearch sink fails to include the security token when signing requests for AWS authentication to OpenSearch. #12249. Fixed in 0.21.1.
  • The nats source and sink authentication options were not configurable. #12262. Fixed in 0.21.1.
  • The internal_logs source includes excess trace logs whenever vector top is used. #12251. Fixed in 0.21.1.
  • The aws_cloudwatch_logs source does not handle throttle responses from AWS. #12253. Fixed in 0.21.1.
  • Vector panics when loading configuration that includes event paths like encoding.only_fields. #12256. Fixed in 0.21.1.
  • Vector panicked when reloading configuration that added components to a running topology. #12273. Fixed in 0.21.1.
  • Using assume_role on AWS components did not function correctly. #12314. Fixed in 0.21.1.
  • The Vector VRL REPL loses variable assignments if the expression being evaluated errors. #12400. Fixed in 0.21.2.
  • Vector docker images require a volume to be mounted at /var/lib/vector to start correctly when the default data_dir of is used. #12413. Fixed in 0.21.2.
  • For AWS components, the timeout for loading credentials was dropped from 30 seconds to 5 seconds. #12421. 0.21.2 adds a new option, load_timeout_secs that can be configured to a higher value.
  • vector generate works again with the datadog_agent source. #12469. Fixed in 0.21.2.
  • Using assume_role configuration for AWS components doesn’t cache the credentials, resulting in a high number of calls to AssumeRole. This was fixed in 0.22.0 via awslabs/smithy-rs#1296.

The Vector team is pleased to announce version 0.21.0!

Be sure to check out the upgrade guide for breaking changes in this release.

In addition to the new features, enhancements, and fixes listed below, this release adds:

  • A new implementation of the VRL runtime as a Virtual Machine (VM). This new implementation improves performance over VRL’s current tree-walking interpreter implementation. For its initial release, this is an opt in feature (see the highlight for how) but will become the default VRL implementation in the future once it has stabilized. We encourage you to try it out and report any issues you find.
  • A new redis source to complement the redis sink.
  • Initial support for ingesting traces from the Datadog Agent (version < 6/7.33) and forwarding them to Datadog. We are working on adding support for newer Datadog Agents.
  • The kubernetes_logs source has been rewritten to use the community supported kube-rs library. We expect that this will resolve some long outstanding bugs with Vector ceasing to process container logs. It also adds support for Kubernetes authentication token rotation.

We made additional performance improvements this release increasing the average throughput by up to 50% for common topologies (see our soak test framework).

Also, check out our new guide on using vector tap for observing events running through Vector instances

Vector Changelog

3 new features

  • This release includes a beta of a new implementation of VRL as a Virtual Machine. This new implementation improves on VRL performance and otherwise exactly compatible with the existing VRL implementation. This is an opt-in feature for this release to gather feedback (see the highlight for how to enable) but will become the default VRL implementation in the future once it stabilizes.
  • The nats sink and nats source now support TLS and authentication via username/password, JWT, Token, NKey, and client certificate.
  • A new redis source was added to complement the existing redis sink. It supports fetching data via subscribing to a pub/sub channel or popping from a list.

27 enhancements

  • We are in the process of updating all Vector components with consistent instrumentation as described in Vector’s component specification).

    With this release we have instrumented the following sources with these new metrics:

    • mongodb_metrics
    • postgresql_metrics
    • socket
    • statsd

    As well as all transforms.

  • The tls options can now be configured on AWS sinks. This is useful when using AWS compatible endpoints where the certificates may not be trusted by the local store.

  • The end-to-end acknowledgements configuration, acknowledgements, was moved from sources to sinks. When set on a sink, all connected sources that support acknowledgements are configured to wait for the sink to acknowledge before acknowledging the client. Setting acknowledgements on sources is now deprecated.

    See the upgrade guide for more details.

  • vector tap and vector top have had a few enhancements.

    vector tap:

    • It now supports tapping inputs via --inputs-of
    • It now reports when the component id patterns provided do not match. It supports --quiet to suppress these messages.
    • A --meta flag was added to include metadata about which component the output events came from.

    Both vector top and vector tap now automatically reconnect if the remote Vector instance goes away. This behavior can be disabled by passing --no-reconnect.

  • Initial support was added for ingesting traces from the Datadog Agent into Vector (via the datadog_agent source) and forwarding them to the Datadog API (via the new datadog_traces sink). Note that currently APM metrics are dropped and so you will be missing these statistics in Datadog if you forward traces to it through Vector. We will be following up to add support APM metrics to Vector.

    Datadog docs are forthcoming but the Agent configuration option, apm_config.apm_dd_url, can be used to forward traces from the Datadog Agent to Vector.

  • The scrape_interval_secs configuration option of the internal_metrics source can now be fractional seconds.
  • VRL’s to_timestamp now accepts an optional unit argument to control how numeric unix timestamp arguments are interpreted. For example, unit can be set to milliseconds if the incoming timestamps are unix millisecond timestamps. It defaults to seconds to maintain current behavior.
  • For Debian packages, the created vector user is now added to the systemd-journal-remote group, if it exists, to facilitate Vector being used to collect remote journald logs.
  • The loki sink now supports setting out_of_order_action to accept to instruct Vector to not modify event timestamps. Vector would previously modify timestamps to attempt to satisfy Loki’s ordering constraints, but these constraints were relaxed in Loki 2.4. If you are running Loki >= 2.4 it is recommended to set out_of_order_action to accept to enable Vector to send data concurrently.

  • All AWS components were migrated to the new AWS SDK from the end-of-life rusoto SDK. This new SDK supports IMSDv2 for authentication.

    See the upgrade guide for more information.

  • The journald source now supports a since_now option to instruct Vector to only fetch journal entries that occur after Vector starts.
  • Reporting for the blackhole sink can now be disabled via setting print_interval_secs to 0.
  • The route transform now has an _unmatched route that can be consumed to receive events that did not match any of the other defined routes.
  • Add ip_ntop and ip_pton VRL functions which can convert IPv6 addresses to and from their byte and string representations.
  • Add is_empty VRL function which returns whether the given object, array, or string is empty.
  • The splunk_hec source now accepts events on /services/collector. This route is an alias for /services/collector/event.
  • The datadog_metrics sink now allows configuration of TLS via the standard tls options.
  • The aws_ec2_metadata transform now allows fetching the account-id field (this field must be opted into).

  • Additional options have been added to the aws_sqs source:

    • delete_message to control whether messages are deleted after processing. This is useful for testing out the source.
    • visibility_timeout_secs to control how long messages are locked for before being rereleased to be processed again. Tuning this is useful for controlling how long a message will be “sent” if a Vector instance crashes before deleting the message.

    These options mirror those that existed for the aws_s3 source for its SQS configuration.

  • The kubernetes_logs source has been rewritten to use the community supported kube-rs library. We expect that this will resolve some long outstanding bugs with Vector ceasing to process container logs. It also adds support for Kubernetes authentication token rotation.

    See the highlight for more details.

  • Component proxy configuration can now include username/password encoded into the URL of the proxy like http://john:password@my.proxy.com.
  • A strlen function was added to VRL to complement the length function. The length function, when given a string, returns the number of bytes in that string. The strlen function returns the number of characters.

  • Users can now provide dynamic label names to the loki sink via a trailing wildcard. Example:

    labels:
	pod_labels_*: {{ kubernetes.pod_labels }}

    This is similar to the promtail configuration of:

    - action: labelmap
  regex: __meta_kubernetes_pod_label_(.+)
  replacement: pod_labels_$1
  • Users can now configure query parameters on the prometheus_scrape source that are sent to all configured endpoints via the new query option. This is useful when using Vector with a federated Prometheus endpoint.
  • The vector sink can now enable gzip compression by setting compression to true.
  • The splunk_hec source’s healthcheck that is exposed at /service/collector/health no longer requires a HEC token. This matches the behavior of the Splunk forwarder and makes it easier to use with load balancers that cannot set this header.
  • The batch.timeout configuration on sinks can now be include fractional seconds.

20 bug fixes

  • Vector’s /health endpoint (mounted when api.enabled is true) now returns a 503 when Vector is shutting down. This is useful when using a load balancer so that traffic is routed to other running Vector instances.
  • The lua transform now returns an error at configuration load time if an unknown field is present on hooks. This helps make typos more visible.
  • docker_logs source now exits, shutting down Vector, if it hits an unrecoverable deserialization error. Previously it would just stall.
  • AWS components now allow region and endpoint to be configured simultaneously. This is useful when using an AWS compatible API.
  • The gcp_stackdriver_logs sink now recognizes a severity of ER as ERROR.
  • The remainder operation in VRL is now fallible if the right-hand side is a field or variable as this can fail at runtime if the right-hand side is 0. This matches the behavior of division.
  • A case where Vector would panic during reload was fixed that would occur whenever a component has changing inputs, but some inputs are the same.
  • The journald source now flushes internal batches every 10 milliseconds, regardless of whether the batch is full. This avoids an issue where the source would wait a very long time to send data downstream when the volume was low but the batch size was configured high to handle spikes.
  • The geoip transform now avoids re-reading the database from disk randomly. This was unintended behavior. We have an open issue for reloading the database from disk during Vector’s reload process.
  • Ensure that instrumentation labels on internal logs and metrics are not lost when Vector is run in quiet mode (-q or -qq). Previously running with a log level below INFO would cause some instrumentation labels to be lost (like component_id).
  • Escape quotes and backslashes in metric tags for the prometheus_exporter sink. Previously these were not escaped and so resulted in invalid Prometheus export output that could not be scraped.
  • The Debian package names were updated to confirm to the Debian packaging standards by changing from vector-${VECTOR_VERSION}-${PLATFORM}.deb to vector_${VECTOR_VERSION}_${PLATFORM}-${REV}.deb. REV is typically 1.
  • The kafka source now reads the incoming message as raw bytes rather than trying to deserialize it as a UTF-8 string.
  • End-to-end acknowledgements were fixed for the aws_sqs source which would previously only acknowledge the last message in each batch from SQS when acknowledgements were enabled.
  • The aws_sqs source would previously acknowledge events in SQS even if it failed to push them to downstream components. This has been corrected.
  • The VRL parse_xml function now correctly parses the node attributes for solo nodes which have no siblings. Previously these node attributes were dropped.
  • vector top now reports error metrics correctly again.
  • The buffer received event metrics (buffer_received_events_total and buffer_received_bytes_total) now include the counts from discarded events. This was done to match the component metrics and to support future buffer on_full modes which may not discard events right away.

  • The socket source when in udp mode would previously include the port of the remote address in the enriched host field. This differed from the tcp mode where only the host part of the remote address is enriched. Instead, this source now does not include the port in the enriched host field.

    However, the socket source now has a port_key that can be set to opt into enrichment of the remote peer port as part of the address.

  • Vector’s published docker images no longer include VOLUME declarations. Instead, users should provide a volume at runtime if they require one. This avoids the behavior of Vector creating a volume for its data directory even if it is unused.

    See the upgrade guide for more information.

1 chore

  • Deprecated GraphQL API routes were removed. See the upgrade guide for more details

What’s next

An OpenTelemetry source

We are in the process of adding a source for ingesting data from the OpenTelemetry collector and OpenTelemetry compatible tools. We are starting with traces, since this has stabilized, but will move on to metrics and logs.

We’ll also be adding an OpenTelemetry sink for forwarding data from Vector to OpenTelemetry-compatible APIs.

Component metric standardization
We continue to be in the process of ensuring that all Vector components report a consistent set of metrics to make it easier to monitor the performance of Vector. These metrics are outlined in this new instrumentation specification.
Official release of end-to-end acknowledgements feature
We have started to add support for end-to-end acknowledgements from sources to sinks where sources will not ack data until the data has been processed by all associated sinks. It is usable by most components now, but we expect to officially release this feature after some final revisions, testing, and documentation.
VRL iteration support

At long last, support for iteration in VRL is almost ready. We expect it to be included in the next release.

See the RFC for a preview of how this will work.

VRL now has lexical scoping for blocks. This means that variables defined inside of a block in VRL (e.g. an if condition block) are no longer accessible from outside of this block. This breaking change was done to support VRL’s forthcoming iteration feature which requires it.

See the upgrade guide for how to migrate your VRL programs.

Download Version 0.21.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64