Vector v0.17.0 release notes

Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

The Vector team is pleased to announce version 0.17.0!

Be sure to check out the upgrade guide for breaking changes in this release.

This release includes:

It also contains a number of additional enhancements and bug fixes. Check out the highlights and changelog for more details.

Known issues

  • events_out_total and processed_events_total ceased being published by sources and transforms. This will be fixed in a subsequent release, but these metrics have also been replaced by component_sent_events_total which is being implemented for all components.

Changelog

13 new features

pipelines create WIP pipeline parser 8572 pipelines compiler verification 8810 pipelines add –pipeline-dir argument to cli 9002 pipelines add component_scope field to internal metrics 9051 new sink Add ‘splunk_hec_metrics’ sink 8947 vrl add find function 9090 new sink Add ‘datadog_archives’ sink with S3 support 8929 config handle BufferConfig parsing without a type tag 9151 kubernetes_logs source add pod annotations 9180 datadog_logs sink use v2 endpoint for logs 9277 datadog_agent source support v1 and v2 log apis 9176 vrl dynamic path functions 9284 aws service add support to use AWS credentials file for AWS authentication 9060

46 enhancements

vrl add undefined variable compiler error 8910 sinks Use variance calculation to better adapt to RTT times in ARC 8325 host_metrics source Add initial cgroups metrics 8806 sinks Make adaptive concurrency the default 8598 blackhole sink print on interval instead of after X events 8968 reduce transform Add additional merge strategies 8559 loki sink Define events 8958 pipelines rename ComponentId to ComponentKey 8955 loki sink Add logfmt encoding 8983 vrl Add function for retrieving deduplicating an array 9012 journald source Add include_matches/exclude_matches options 8974 humio_logs sink humio_metrics sink Add index and indexed_fields options to Humio sink 8981 enriching add between date search to enrichment tables 9021 aws_kinesis_firehose sink Check batch configuration limits 9088 enriching add find_enrichment_table_records VRL function 9069 vrl Added case sensitive option to enrichment VRL functions 9085 enriching Add select parameter to enrichment VRL functions 9102 releasing Add section for debian package 9164 codecs kafka source Integrate Decoder/DecodingConfig with kafka source 9071 cli Write logs to stderr 9166 enriching error if enrichment functions reference a non existent column 9174 cli Fix API client subscription drop 9250 vrl allow statements in if predicates over multiple lines. 9177 external docs Serve metadata as JSON 9276 observability file source Improve file source observability 9205 observability sinks Emit events for sent events in all HttpSink sinks 9275 kafka sink add header support to kafka sink 9171 vrl Move rate limiting span to just around log function 9294 enriching enrichment functions need to specify date conditions 9296 vrl Add tally function 9015 unit tests Add testing for component specification features 9312 vrl Allow to_float to take timestamps 9373 vrl added reverse_dns remap function 8717 external docs Update unit testing docs to cover VRL conditions 9377 codecs exec source Integrate Decoder/DecodingConfig with exec source 9337 external docs Link to Helm chart repo 9396 dev Added check-build-tools to Makefile (#8146) 8147 graphql api Refactor outputEvents subscription -> outputEventsByComponentIdPatterns 9339 prometheus_scrape source Allowing tagging metrics with instance and endpoint 9330 datadog_logs sink Set DD origin headers 9367 sources observability Add events to HttpSource framework 9379 codecs exec source Integrate Decoder/DecodingConfig with nats source 9400 codecs datadog_agent source Integrate Decoder/DecodingConfig with datadog source 9430 vrl Add default_known_ports parameter to parse_url function 9376 codecs aws_kinesis_firehose source Integrate Decoder/DecodingConfig with aws_kinesis_firehose source 9332 aws_ec2_metadata transform dont error when receiving a 404 when fetching metadata 9492

48 bug fixes

releasing Add enrichment tables to all builds 8939 css website github icon should have hover effect in dark mode. 8964 splunk_hec source Do not blow up on trailing whitespace 8997 vrl handle complex ruby objects 8956 external docs Add template note to config params 9032 config error on duplicate inputs 9011 ci Regenerate intermediate test CA 9053 external docs Fix component docs rendering nits 9074 config ensure components don’t have dots in their names 9083 aws_cloudwatch_logs sink Allow adaptive concurrency 9007 ci Put collects back 9097 tests work around internal_logs lack of isolation 9099 tests Fix smoke test first line function for e2e kubernetes tests 9096 enriching Usable in Test 9095 aws_s3 sink Fix batching parameters 9106 ci Overwrite wix install if exists 9116 aws service more precise aws retry logic 9104 architecture use a fixed vec allocation for batches 9128 prometheus_scrape source allow : in metric names 9123 vrl Re-add VRL posiition to log() calls 8995 ci Use use_v2_api: false for k8s datadog agent tests 9155 http source adjust deflate compression handling 9121 networking reorder service layers to prevent unbounded retries 9026 vrl Add ipv6 support to parse_aws_alb_log.rs 9163 aws_s3 sink ship final batches when shutting down 9184 host_metrics source Detect cgroup memory controller before scanning 9168 cli Fix top quitting when the host has gone away 9139 datadog_archives sink Correct encoding tags 9252 external docs Update GitHub org name throughout the website 9249 external docs Add source/edit buttons to docs and guides 9248 vrl fix byte-swapped integers in ip_aton() and ip_ntoa() functions 9208 cli Fix tap signal handling 9254 releasing Use bash rather than sh for script installer 9266 external docs Improve VRL function templating 9263 ci Re-add markdown and cue checks 9285 external docs Change sidebar ordering of VRL and config docs 9271 transforms add sanity check in route & compound transforms 9301 docs add missing word 9359 vrl reverse if else alternatives before nesting 9363 external docs Provide the correct optionality for the VRL assert message 9384 generator source Give the generator source a default interval 9390 buffers Use tokio runtime in buffer benchmarks 9391 external docs Fix configuration reference 9397 codecs Propagate empty byte frames in message based transports 9399 datadog_logs sink always send unix timestamp in milliseconds 9436 datadog_logs sink expect ACCEPTED(202) instead of OK(200) response from Logs v2 API 9437 vrl Correct parse_url’s value type 9473 sink loki return empty json when buffer is empty 9464

133 chore

deps bump inherent from 0.1.6 to 1.0.0 8832 Add VERSIONING policy 8795 Silence a couple of clippy warnings 8880 Add link to CHANGELOG.md 8885 deps bump redis from 0.21.0 to 0.21.1 8890 deps bump bytes from 1.0.1 to 1.1.0 8891 deps bump libc from 0.2.100 to 0.2.101 8892 deps bump tokio from 1.10.0 to 1.10.1 8897 Introduce a rename_key on LogEvent 8907 deps Remove WASM code 8911 vrl RFC for improved error diagnostics 8894 Add note about RBAC required for kubernetes_logs 8915 deps bump grok from 1.1.0 to 1.2.0 8916 deps bump serde_yaml from 0.8.19 to 0.8.20 8917 generator source Generate config with JSON default 8942 codecs Add RFC for “Framing and Codecs - Sources” 8620 deps bump serde from 1.0.129 to 1.0.130 8952 Add data loss defaults to user experience design 8945 deps bump thiserror from 1.0.26 to 1.0.28 8959 deps bump serde_json from 1.0.66 to 1.0.67 8953 deps bump flate2 from 1.0.20 to 1.0.21 8961 deps bump sha-1 from 0.9.7 to 0.9.8 8962 deps bump arc-swap from 1.3.1 to 1.3.2 8963 Add RFC last call meeting to template 8965 deps bump futures from 0.3.16 to 0.3.17 8966 Add component specification 8858 deps Unpin evmap from fork 8941 deps bump structopt from 0.3.22 to 0.3.23 8971 Introduce a Batcher stream processor 8960 deps bump tokio from 1.10.1 to 1.11.0 8986 enriching move enrichment tables into new crate 8979 deps bump ordered-float from 2.7.0 to 2.8.0 9018 deps bump redis from 0.21.1 to 0.21.2 9019 Attempt to enable disallowed-methods clippy lint 9013 codecs Implement codec traits for sources/decoding 8913 architecture add per-batch timeout support to Batcher 9004 observability Internal tracing RFC 8723 deps bump pulsar from 4.0.0 to 4.1.1 9062 deps bump thiserror from 1.0.28 to 1.0.29 9050 ci bump docker/setup-buildx-action from 1.5.1 to 1.6.0 9058 deps bump assert_cmd from 2.0.0 to 2.0.1 9061 Add how test intermediate certificates were generated 9076 aws_s3 sink refactor the S3 sink 8884 deps bump mongodb from 2.0.0-beta.3 to 2.0.0 9081 deps bump syslog_loose from 0.14.0 to 0.15.0 9082 releasing Run Windows builds on our CI runners 9084 codecs Implement character_delimited framing 8918 compound transform Use array for configuration 8846 codecs Implement length_delimited framing 8936 enriching check for duplicate indexes before adding a new one 9056 host_metrics source Improve error reporting for cgroups metrics 9087 deps Bump grok from 1.1.0 to 1.2.0 8940 codecs Implement newline_delimited framing 9054 codecs Implement octet_counting framing 9063 codecs Implement bytes framing 9059 codecs Implement syslog parser 9065 deps cargo upgrade --aggressive 9089 deps Upgrade Rust to 1.55 9094 observability revert internal tracing proof of concept 9041 deps bump nom from 7.0.0-alpha1 to 7.0.0 8830 ci Run integration tests on PRs with label 9098 deps bump async-nats from 0.9.18 to 0.10.1 9101 observability Add common counters 9111 deps bump serde_yaml from 0.8.20 to 0.8.21 9118 deps bump tracing-core from 0.1.19 to 0.1.20 9119 deps bump anyhow from 1.0.43 to 1.0.44 9120 deps bump flate2 from 1.0.21 to 1.0.22 9133 deps bump tracing from 0.1.26 to 0.1.27 9134 deps bump serde_json from 1.0.67 to 1.0.68 9149 deps bump tracing-subscriber from 0.2.20 to 0.2.22 9136 Add specifications for configuration and instrumentation 9117 deps bump rust_decimal from 1.15.0 to 1.16.0 9148 ci Remove unused advisory 9138 deps bump libc from 0.2.101 to 0.2.102 9156 Rework the Datadog logs sink 8825 codecs stdin source Integrate Decoder/DecodingConfig with stdin source 9070 deps bump socket2 from 0.4.1 to 0.4.2 9167 deps bump mlua from 0.6.2 to 0.6.3 9170 codecs Allow max_length to be missing from framers / empty decoding config 9067 deps bump hyper from 0.14.12 to 0.14.13 9198 Removes sink::Response from the Datadog Logs sink 9172 deps bump metrics-util from 0.10.0 to 0.10.1 9200 codecs Use codecs::Decoder in TcpSource 9154 deps bump tracing-subscriber from 0.2.22 to 0.2.23 9199 external docs Fix typo on pipelines page 9214 deps bump arc-swap from 1.3.2 to 1.4.0 9220 Add buffer instrumentation spec 9165 Add component_ prefix to metric naming 9218 Re-add namespace to instrumentation spec 9232 deps bump tracing from 0.1.27 to 0.1.28 9230 deps bump cidr-utils from 0.5.4 to 0.5.5 9245 deps bump rand_distr from 0.4.1 to 0.4.2 9246 deps bump tracing-subscriber from 0.2.23 to 0.2.24 9247 Re-introduce tokio-console under a feature flag 9240 deps bump bstr from 0.2.16 to 0.2.17 9260 external docs Run cue fmt on the docs 9268 datadog_agent source Accept metrics in the datadog_agent source RFC 9057 Move InternalEvent to vector-core 9264 buffers Fix clippy error in BufferConfig::resources 9274 deps bump tokio from 1.11.0 to 1.12.0 9288 deps bump chrono-tz from 0.5.3 to 0.6.0 9258 deps bump http from 0.2.4 to 0.2.5 9290 graphql api Add received_events_total to the API 9300 sinks make StreamSink::run take self 9331 Move byte_size_of to core_common 9341 ci bump actions/github-script from 4.1 to 5 9345 external docs Actually run cue and markdown checks on CI 9343 architecture refactor PathIter to avoid regex + allocations (plus a bonus change) 9348 deps bump libc from 0.2.102 to 0.2.103 9356 Add buffer max size metric to buffer specification 9318 vrl RFC for iteration support 9001 deps bump pretty_assertions from 0.7.2 to 1.0.0 9364 deps bump mlua from 0.6.3 to 0.6.4 9365 deps bump smallvec from 1.6.1 to 1.7.0 9366 deps bump nix from 0.22.1 to 0.22.2 9369 Add script to surface-scan components against specification 9308 internal docs Fix markdown lint issues in VRL iteration RFC 9380 deps bump tokio-postgres from 0.7.2 to 0.7.3 9385 vrl add development design document 8735 graphql api Add sent_events_total to API 9313 vrl Lint DESIGN.md 9413 Improve highlighting on template syntax page 9419 deps bump atomig from 0.3.1 to 0.3.2 9427 architecture consolidate sink I/O driver logic into reusable component 9215 deps bump lru from 0.6.6 to 0.7.0 9461 deps bump tracing-core from 0.1.20 to 0.1.21 9426 deps bump woothee from 0.11.0 to 0.12.1 9425 buffers Fix compilation issues when disk buffers are not enabled 9358 vrl Add benches for type assertion functions 9412 buffers Add ByteSizeOf for buffer items 9482 deps bump tracing-subscriber from 0.2.24 to 0.2.25 9486 deps bump mlua from 0.6.4 to 0.6.5 9487 deps bump tracing from 0.1.28 to 0.1.29 9488

What’s next

Component metric standardization
We are in the process of ensuring that all Vector components report a consistent set of metrics to make it easier to monitor the performance of Vector. These metrics are outlined in this new instrumentation specification).
VRL iteration support
A common request from users when the incoming log event shape is unknown, to be able to iterate over the keys and values in those log events. We recently published an RFC for this and expect to implement this support this quarter.
New unified Vector helm chart
Taking all of our learnings from the existing agent and aggregator Helm charts, we’ve started work on a new Helm chart that will unify both the agent and aggregator deployments into one chart, which can be configured by deployment role, to simplify deployment and chart maintenance. You can follow along in vectordotdev/helm-charts#38.

Download Version 0.17.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...