Vector v0.9.0 release notes

Code name The Homestretch

Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

New Apache Pulsar Sink

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sinks sink: pulsar
New Encoding Options

Author photo for Ben Johnson
Ben Johnson
type: enhancement domain: sinks
New Honeycomb Sink

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sinks sink: honeycomb
The elasticsearch sink compression option now defaults to none

Author photo for Ben Johnson
Ben Johnson
type: breaking change domain: sinks sink: elasticsearch
Improved Shutdown

Author photo for Ben Johnson
Ben Johnson
type: enhancement domain: topology
4 New Sinks!

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sinks
À la carte Custom Vector Builds

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sources source: vector
New Dedupe Transform

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sources source: vector
New Filter Transform

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: transforms transform: filter
New Tag Cardinality Limit Transform

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: transforms transform: tag_cardinality_limit
The Vector Source Now Accepts Metrics

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sources source: vector
Compression Now Available In The Kafka Sink

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sinks sink: kafka
More Condition Predicates

Author photo for Ben Johnson
Ben Johnson
type: enhancement domain: config domain: transforms transform: filter transform: swimlanes
New Datadog Logs Sink

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sinks sink: datadog_logs
New Papertrail Sink

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sinks sink: papertrail
NixOS Support

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: platforms
The Vector Source and Sink Support TLS

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: sources source: vector
A New Guides Section

Author photo for Ben Johnson
Ben Johnson
type: new feature domain: website
Lua Transform v2

Author photo for Ben Johnson
Ben Johnson
type: enhancement domain: sources source: vector

Changelog

14 new features

cli Add --log-format CLI option 1908 new source Initial http source implementation 1650 kafka sink Add support for compression 1969 new transform Initial dedupe transform implementation 1848 new transform Initial ’tag_cardinality_limit’ transform implementation 1959 new sink Initial papertrail sink implementation 1835 new sink Initial honeycomb sink implementation 1847 new sink Initial pulsar sink implementation 1665 new sink Initial datadog_logs sink implementation 1832 elasticsearch sink Check for and log partial ingestion failures 2185 new transform Initial filter transform implementation 2088 new transform Initial kubernetes_pod_metadata transform implementation 1888 cli Add -qqq command line flag to disable logs 2230 new source Initial internal metrics source implementation 1953

48 enhancements

config Allow setting individual log_schema.* options 1923 socket source syslog source vector source Add TLS support to socket, syslog, and vector sources 1892 config Default to check_fields condition when specifying conditions 1947 config Add is_log and is_metric conditions 1950 operations Create feature flags for all components enabling custom Vector builds 1924 cli Add --fragment flag to generate subcmd 1956 cli Allow names in generate subcmd expression 1955 logplex source splunk_hec source Add TLS support 1968 sinks Add new encoding.only_fields and encoding.except_fields options 1915 networking Move all TLS support over to openssl 1986 config Add contains and prefix predicates 1997 vector sink Add support for TLS 2025 elasticsearch sink Add timeout query parameter 2038 sampler transform Add new key_field option 2039 lua transform Add version configuration option 2056 config Support escaping in the field path notation 2081 add_fields transform Allow native TOML tables 2068 remove_fields transform rename_fields transform Add new drop_empty option to drop empty objects automatically 2077 lua transform Support metric events in version 2 2095 config Make LogSchema more optional 2113 networking Add User-agent header to all outgoing HTTP requests 2130 splunk_hec sink Add new index option to specify custom Splunk indexes 2127 observability The add_fields, remove_fields, and rename_fields now produce a debug log when a field is replaced or nonexistent 2148 aws_s3 sink Increase default rate limit to allow for higher throughput 2161 socket sink Add support for UDP mode 2162 networking Add support for tls.verify_hostname in HTTP based sinks 2164 config Rename prefix condition predicate to starts_with 2181 config Add new ends_with condition predicate 2183 field_filter transform Deprecate in favor of the new filter transform 2195 config Add new regex condition predicate 2198 observability Warn that old field_filter transform is deprecated 2197 regex_parser transform Add new target_field option 2023 networking Detect TCP disconnects earlier 2209 lua transform Implement all hooks and timers in version 2 2126 vector source vector sink Support metric events 2245 ux Use external tagging for metrics serialization 2231 config Add source_key and source_type_key to… 2244 filter transform Accept metrics as input 2267 lua transform Set search_dirs to config dirs by default 2274 gcp_cloud_storage sink Make ACL optional 2283 docker source Add elegant shutdown behavior 2260 sources Add source_type field 2298 cli Show possible values for --log-format, --color, and --format 2300 topology Improve topology shutdown logs 2345 gcp provider Add support for instance accounts 2351 splunk_hec source Wire in new ShutdownSignal for improved shutdown 2373 journald source Wire in ShutdownSignal for improved shutdown 2364 sources Wire in new ShutdownSignal for improved shutdown 2365

20 bug fixes

config Force symlink re-evaluation when detecting a changed Vector config 2034 transforms Fix insertion to arrays in field path notation 2062 merge transform Support nested fields and arrays 1936 journald source Fix handling of message array data 2053 gcp_stackdrive_logging sink Rename gcp_stackdriver_logging sink to gcp_stackdriver_logs 2121 tests Fix disk pressure 2217 elasticsearch sink Default compression to none 2219 loki sink Pass tls settings to healthcheck 2234 influxdb_metrics sink Pass tls settings to healthcheck 2238 gcp_cloud_storage sink Use kebab case for ACL encoding 2259 aws_kinesis_firehose sink Fix healthchecks 2276 aws_ec2_metadata transform Only run required requests based on the supplied fields 2265 website correct typo in testing post 2291 website small fixes for testing post 2292 kafka sink Run healthcheck in rt context 2288 aws_cloudwatch_logs sink Sort events by timestamp 2320 sinks Reduce credentials fetch timeout 2342 lua transform Don’t require isdst, wday, yday in timestamps 2335 aws_cloudwatch_logs sink Allow creation race of groups and streams 2355 lua transform support sub-second timestamp resolution 2333

101 chore

testing Add behavior test for the merge transform 1820 Replace flatten/unflatten by native nesting 1902 Upgrade rdkafka to 0.23.1 1928 Upgrade lib/file-source to futures 0.3 1934 dependencies Rename futures to futures01 1933 gcp_pubsub sink Rewrite to use HttpSink and JsonArrayBuffer 1925 topology Allow transforms to implement Stream 1938 tests Increase test logs size 1949 operations Add TEST_LOG to test-stable CI job 1937 operations Check kube and docker code 1914 dependencies Upgrade to Rust 1.41.1 1958 website Remove Google Analytics 1976 Add Privacy Policy 1977 Add check-component-features target to the main Makefile 1981 Add RFC process 1961 operations Pull fresh containers on test 2019 statsd sink Fix race condition in test 2026 logplex source Refactor to use util/HttpSource 2014 statsd sink Reshuffle timeout in test 2035 networking Refactor internal http client 2029 Add PR checklist 2010 operations Upgrade to rustc 1.42.0 2043 Add markdown linting 2020 splunk_hec sink Rename host_field to host_key 2037 operations Check lockfile over cargo.toml 2050 docs Re-run doc generation 2052 Add target directory to .markdownlintignore 2054 testing Fix check-component-features test 2061 sinks Part 1 of HttpSink refactor 2072 json_parser transform Ensure white-space dooes not break parsing 2060 Update tagline 2079 RFC #1999 - 2020-03-06 - API extensions for lua transform 2000 dependencies Upgrade regex dependency 2083 website Upgrade dependencies 2086 sinks Part 2 of HttpSink refactor 2075 setup Use --release Cargo flag in make build 2087 testing Pin tag for loki Docker image 2091 Switch to tokio-compat to start the transition to our new async runtime 1922 topology Refactor source shutdown and make it two-phase 1994 Add *_flat methods for log events 2082 file sink Upgrade file sink for tokio-compat 1988 Correct doctest at src/stream.rs 2097 console sink Upgrade console sink for tokio 0.2 2096 Correct Cargo.lock 2105 operations Fix x86_64-pc-windows-msvc build 2103 topology Make RunningTopology::stop() signal all sources to shut down using the new ShutdownCoordinator (1091) 2098 Racy buffering tests correction 2106 operations Add test harness GH action 2028 TLS reorganization 2101 networking Revert “Use MaybeTlsListener in TcpSource” 2129 dependencies Rename tokio -> tokio01 and tokio02 -> tokio 2131 operations Relax PR check list to only fire on important changes 2136 dependencies Update bumpalo dependency in Cargo.lock 2141 networking Fix handling of incoming TLS connections 2146 Simplify domains for labels, etc 2152 Upstream github-script action at test harness workflow 2157 operations Switch to upstream repo-permission-check-action 2158 Add LogEvent::new 2154 Remove package managers from install script 2165 Support ARM architectures in the install script 2167 Disable duplicated version output in make version 2169 tests Enable Kubernetes tests 1970 tests Use UTC instead of Europe/London in ClickHouse tests 2178 Add have: * labels 2179 website Add new guides section 2132 website Update to Docusaurus 2.0.0-alpha.49 2196 sinks Refactor internal sinks and simplify 2111 testing Rename map_values_array to array_values_insertion_order test at discriminant 2140 add event-driven observability rfc 2093 website Fix make generate command 2205 operations Improve Windows Makefile support 2150 topology disable flaky test 2215 website Fix sitemap URLs and CSS layouts 2229 operations RPM for aarch64 was using armv7 2220 kubernetes_pod_metadata transform Add field tests 2213 topology Add basic module docs 2233 operations Don’t use buildx plugin by default 2239 website Fix link to the Docker platform on the main page 2249 website Use cat with heredoc instead of echo and quotes 2262 transforms Make RuntimeTransform trait available for all transforms 2281 tests Enable Rust codegen LTO for test harness 2310 networking Upgrade http client to hyper 0.13 2294 operations Fix Windows metrics 2309 website Add /highlights section and rework release notes 2317 sources Wire ShutdownSignal in trait HttpSource 2290 Clarify dependencies of the package-* job in `make hel… 2324 config Remove log_schema.source_key 2297 website Add Twitter author tags 2299 dependencies Bump tower-limit 0.1.2 2346 Use only the filename part of the paths to extract dates 2347 website Update Ana bio 2360 platforms Migrate some PR checks to GitHub Actions from Circle 2352 file source Wire in new ShutdownSignal for improved shutdown 2261 Only run CI workflow when relevant files change 2369 Fix tests 2372 operations Add nightly GH action workflow 2370 Move /test-data to /tests/data 2375 operations Run ci workflow on pull requests only 2378 platforms Reenable component feature check 2371 operations Optimize binaries in test-stable CI to reduce binary sizes. This should make it more stable. 2380 Prepare 0.9.0 2353

What’s next

Kubernetes!
We’re doing this integration right. We’ve been alpha testing with a number of large clusters and we’ve honed in on a very high-quality approach that reflects the UX Vector is going after. You can learn more in our Kubernetes Integration RFC. Feedback is welcome!
WASM Driven Plugin Architecture
This release included an overhaul of our lua transform, which represents a big step towards making Vector more flexible. Lua is an excellent language when you need some quick scripting capabilities. But what if you need something more powerful? Like writing your own custom source, transform, or sink? Vector plans to solve this with WASM foreign module support, enabling you to extend Vector in the language of your choice with little performance penalty. ❤️ WASM. Check out the WASM Foreign Module Support RFC for more info.
Dynamic HTTP Rate-Limiting (AIMD)
Fiddling with rate-limits is a frustrating endeavor. If you set them too high you risk overwhelming the service; too low and you’re unnecessarily limiting throughput. And what happens if you deploy a few more Vector instances? Vector is planning to solve this by automatically detecting the optimal rate limits, taking inspiration from TCP congestion control algorithms. Check out the Dynamic HTTP Rate Limiting RFC.
Vector Observability

Did you know this release includes a new hidden internal source that emits metrics reflecting Vector’s own internal state? 👀 We didn’t highlight this source because we have not finished instrumenting Vector and we’ll be changing the internal schema used.

Fun fact, we took an event-driven approach to Vector’s internal observability, which we strongly believe in. We want Vector to be a good example of our own advice. Checkout the Event-Driven Observability RFC.

Download Version 0.9.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64 ARM64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...