Vector v0.18.0 release notes

The Vector team is pleased to announce version 0.18.0!

Be sure to check out the upgrade guide for breaking changes in this release.

In case you missed it, we’ve also released a new unified vector helm chart! This new chart can deployed as either as either an agent or aggregator role and so deprecates our existing vector-agent and vector-aggregator charts. See the chart upgrade guide for how to transition from the old charts.

Known Issues

  • The elasticsearch sink incorrectly prints a message for each delivered event

Features

  • Initial support for routing failed events from transforms has been added, starting with the remap transform. See the highlight for more.
  • Initial support for enriching events from external data sources has been added via a new Vector concept, enrichment tables. To start, we’ve added support for enriching events with data from a CSV file. See the highlight for more.
  • A new throttle transform has been added for controlling costs. See the highlight for more.
  • Better support for breaking up Vector configuration into multiple files was added via deriving configuration from file and directory names. See the highlight for more.
  • A new aws_sqs source was added for consuming messages from AWS SQS as log events.

Enhancements

  • Instrumentation has been added to sink buffers to help give more visibility into their operation. The following metrics have been added:
    • buffer_byte_size (disk buffer only): The number of bytes in the buffer
    • buffer_events (in-memory buffer only): The number of events in the buffer
    • buffer_received_event_bytes_total: The number of bytes that have been received by this buffer. This count does not include discarded events.
    • buffer_sent_event_bytes_total: The number of bytes that have been sent from the buffer to its associated sink.
    • buffer_received_events_total: The number of events that have been received by this buffer. This count does not include discarded events.
    • buffer_sent_events_total: The number of events that have been sent from the buffer to its associated sink.
    • buffer_discarded_events_total: The number of events that have been discarded from the buffer because it is full (relevant when when_full is drop_newest)
  • The $LOG environment variable for configuring the Vector log level has been renamed to $VECTOR_LOG. $LOG is still also accepted for backwards compatibility. This change makes logging configuration more in-line with Vector’s other environment variable based options, and isolates Vector from being affected by other generic environment variables.
  • VRL diagnostic error messages have been improved to suggest null, true, or false for undefined variables. This helps guide users to realize when they are trying to use a keyword like nil that doesn’t actually exist in VRL.
  • The log_to_metric transform now also allows emitting absolute counters in addition to relative counters via kind = "absolute".
  • (breaking!) The status tag for the http_client_responses_total internal metric was updated to be just the integer (e.g. 200) rather than including the text portion of the HTTP response code (e.g. 200 OK).
  • The kubernetes_logs source now annotates logs with the pod_owner when available.
  • The papertrail sink now allows process field to be set to a event field value the templatable process key.
  • The aws_s3 sink now has less connections terminated prematurely as it optimistically terminates connections before AWS’s timeout.
  • The prometheus_exporter now expires metrics that haven’t been seen since the last flush (controlled by flush_interval_secs) to avoid holding onto stale metrics indefinitely and consuming increasing amounts of memory.
  • Added support for end-to-end acknowledgements to the aws_kinesis_firehose source, journald source, and file sink.
  • The utilization metric for most transforms no longer count time spent blocked on downstream components as busy. This means they should more accurately represent the time spent in that specific transform and require less interpretation to find bottlenecks.
  • The datadog_metrics sink now supports sending distribution data to Datadog like histograms and aggregated samples.
  • The kubernetes_logs source has been updated to be less demanding on the Kubernetes API server (and backing etcd cluster) by allowing for slightly stale data to be used for log enrichment rather than always requesting the most-up-to-date metadata.
  • The generator source has been renamed to demo_logs. We feel this name better reflects the intent of the source. An alias has been added to maintain compatibility.
  • The framing and decoding options are now available on heroku_logs source. See the framing and decoding highlight from v0.17.0 for more about this new source feature.
  • (breaking!) The upper_limit field for aggregated summaries from the metric_to_log transform has been renamed to q which is a common shorthand for quantile.
  • We have continued to add additional instrumentation to components with the goal of having them all match the Component Specification. Once we have finished this we will post a highlight outlining all of the added metrics.

Bug Fixes:

  • Configuring the number of threads (via --threads) for Vector now actually takes effect again rather than it always using the number of available cores. This was a regression in v0.13.
  • Vector no longer crashes when configuration was reloaded that include changes to both the order of inputs for a component and configuration of one of those inputs.
  • (breaking!) The obsolete event_per_line configuration option was removed from the exec source. This option became non-functional in 0.17.0 but was left available to be configured. Instead, the new framing option can be used to choose between interpreting th e output of the subcommand as an event per line or all at once as one event.
  • Fix regression in v0.17.0 for aws_s3 sink where it would add a / to the prefix provided. The sink no longer adds this / to replace previous behavior.
  • Fix memory leak that occurred when using Vector as a Windows service.
  • Fix lock-ups with the aws_s3, loki, and datadog_logs sinks.
  • Fix naming of the VRL compact function’s object argument to match the docs. This was incorrectly implemented named map in the implementation.
  • The component_sent_bytes_total internal metric is now reported after events are successfully sent to HTTP-based sinks rather than before they are sent.
  • The influxdb_metrics and influxdb_logs sinks now use /ping for heathchecks rather than /health to work with Influx DB 2 Cloud.

Other changes

  • The deprecated batch.max_size parameter has been removed in this release. See the upgrade guide for more.
  • The deprecated request.in_flight_limit has been removed in this release. See the upgrade guide for more.
  • The deprecated host and namespace field on the datadog_metrics sink has been removed. See the upgrade guide for more.

What’s next

Component metric standardization
We are in the process of ensuring that all Vector components report a consistent set of metrics to make it easier to monitor the performance of Vector. These metrics are outlined in this new instrumentation specification).
VRL iteration support
A common request from users when the incoming log event shape is unknown, to be able to iterate over the keys and values in those log events. We recently published an RFC for this and expect to implement this support this quarter.

Changelog

40 enhancements

observability Adjust component spec and sinks to reflect sent bytes and events only after sending successfully 9503 vrl suggest null, true or false for undefined variables 9517 enriching enable enrichment file reload on SIGHUP 9371 log_to_metric transform add absolute_kind to log_to_metric counters 9463 codecs heroku_logs source Integrate Decoder/DecodingConfig with heroku_logs source 9432 kubernetes_logs source Adding additional metadata from kubernetes 9505 observability sinks Emit EventsSent in batch sinks 9504 observability sources Add event processing metrics to TcpSource sources 9540 buffers Instrument buffer total bytes/events received, sent, dropped 9327 apache_metrics source observability Update to comply with component spec 9596 buffers Instrument buffer max size and disk buffer initial size 9584 observability splunk_hec source Instrument splunk_hec source according to component spec 9586 file sink observability Add event processing metrics 9589 observability aws_s3 sink Add event processing metrics 9614 papertrail Allow process to be set 9685 external docs Show release notes on /releases page 9731 external docs Lighthouse scores for the website 9753 external docs Netlify Lighthouse plugin 9775 cli Rename LOG env var to VECTOR_LOG 9743 config Add SHA256 hash to config for enterprise reporting 9575 pipelines Adds host_metrics reporting to DD Pipelines 9784 external docs Initial website perf improvements 9795 sinks enforce EventsSent instrumentation for new-style sinks 9801 datadog_metrics sink rewrite to the new model + add sketch support 9178 config Test ConfigBuilderHash serialization ordering 9807 external docs Update S3 source example 9838 observability vector sink Add event processing metric 9830 observability vector source Instrument with event processing metrics 9683 config Implicit namespacing based on the config directory structure 9701 external docs Supply defaults for target triples 9885 pipelines Report datadog.configuration as a metrics tag for DD Pipelines 9850 external docs Use defaults more extensively in CUE 9836 prometheus_exporter sink expire metrics 9769 aws_kinesis_firehose source Add support for end-to-end acknowledgements 9891 vrl Add tally_value function 9890 file sink Add support for end-to-end acknowledgements 9892 journald source Add support for end-to-end acknowledgements 9893 observability improve transform utilization metrics 9828 journald source observability Implement component spec 9637 external docs Add docs for no_outputs_from in unit tests 10023

12 new features

topology add transforms with multiple outputs 9169 loki sink rewrite using stream 9506 new transform Initial throttle transform 9378 pipelines transform load and handle pipelines tranforms 9733 config refactor loading for loading recursively 9881 datadog_archives sink Add gcp_cloud_storage support 9403 vrl Support DataDog grok parser - parsing DD grok rules 8850 datadog_archives sink add azure_blog_storage support 9495 kubernetes_logs source Use resource_version of 0 to use cache 9974 new source add AWS SQS source 9968 remap transform add error output stream 9417 kubernetes_logs source Add configurable delay to deletion of k8s metadata 10031

29 bug fixes

observability sinks Fix double emit of EventsSent in HttpSink 9508 cli actually apply threads option 9527 observability Label HTTP response metrics with status code 9510 fluent source logstash source Remove non-working tests 9576 topology handle rebuilding connected components 9536 ci satisfy markdownlint on throttle rfc 9585 observability sinks Fix missing increment in batch counter and add component tests to HttpSink sinks 9525 ci Resolve markdown lint error in buffer specification 9607 ci Fix markdown check guard 9610 file sink Fix compilation of tests 9684 aws_s3 sink close idle connections for aws s3 sinks 9703 sinks Make more concurrency defaults adaptive 9726 ci Use OS X compatible option in soak test script 9788 deps Bump main Cargo.toml to 2021 edition 9800 ci Wire up humio metrics soak correctly 9810 architecture fix bug in Driver that causes hang 9804 aws_s3 sink Remove unintentional prefix 9848 external docs Fix sink compression options 9869 external docs Add missing environment variables to CLI docs 9913 windows platform call Application::run when running vector as a service on windows instead of manually handling the topology life cycle. 9950 vrl Fix naming of compact’s object argument 9972 external docs fix transform name in example config 9988 ci Fix soak test artifact uploading 10008 ci Fix artifact uploading for soak tests 10013 external docs Add client_concurrency to sqs source 10029 blog website Fix topology query in GraphQL API blog post 10030 ci This parameter was dropped 10033 sinks Survey batch configurations 10039 throttle transform Rename window parameter 10095

Download Version 0.18.0

Linux (deb)
deb
Linux (rpm)
rpm
macOS
tar.gz
Windows
zip
Windows (MSI)
msi