Vector v0.9.0 release notes

We’re doing this integration right. We’ve been alpha testing with a number of large clusters and we’ve honed in on a very high-quality approach that reflects the UX Vector is going after. You can learn more in our Kubernetes Integration RFC. Feedback is welcome!

WASM Driven Plugin Architecture

This release included an overhaul of our lua transform, which represents a big step towards making Vector more flexible. Lua is an excellent language when you need some quick scripting capabilities. But what if you need something more powerful? Like writing your own custom source, transform, or sink? Vector plans to solve this with WASM foreign module support, enabling you to extend Vector in the language of your choice with little performance penalty. ❤️ WASM. Check out the WASM Foreign Module Support RFC for more info.

Dynamic HTTP Rate-Limiting (AIMD)

Fiddling with rate-limits is a frustratin endaevor. If you set them too high you risk overwhelming the service; too low and you’re unecessarily limiting throughput. And what happens if you deploy a few more Vector instances? Vector is planning to solve this by automatically detecting the optimal rate limits, taking inspiration from TCP congestional control algorithms. Check out the Dynamic HTTP Rate Limiting RFC.

Vector Observability

Did you know this release includes a new hidden internal source that emits metrics reflecting Vector’s own internal state? 👀 We didn’t highlight this source because we have not finished instrumenting Vector and we’ll be changing the internal schema used.

Fun fact, we took an event-driven approach to Vector’s internal observability, which we strongly believe in. We want Vector to be a good example of our own advice. Checkout the Event-Driven Observability RFC.

Changelog

48 enhancements

config Allow setting individual log_schema.* options

1923 socket source syslog source vector source Add TLS support to socket, syslog, and vector sources

1892 config Default to check_fields condition when specifying conditions

1947 config Add is_log and is_metric conditions

1950 operations Create feature flags for all components enabling custom Vector builds

1924 cli Add --fragment flag to generate subcmd

1956 cli Allow names in generate subcmd expression

1955 logplex source splunk_hec source Add TLS support

1968 sinks Add new encoding.only_fields and encoding.except_fields options

1915 networking Move all TLS support over to openssl

1986 config Add contains and prefix predicates

1997 vector sink Add support for TLS

2025 elasticsearch sink Add timeout query parameter

2038 sampler transform Add new key_field option

2039 lua transform Add version configuration option

2056 config Support escaping in the field path notation

2081 add_fields transform Allow native TOML tables

2068 remove_fields transform rename_fields transform Add new drop_empty option to drop empty objects automatically

2077 lua transform Support metric events in version 2

2095 config Make LogSchema more optional

2113 networking Add User-agent header to all outgoing HTTP requests

2130 splunk_hec sink Add new index option to specify custom Splunk indexes

2127 observability The add_fields, remove_fields, and rename_fields now produce a debug log when a field is replaced or non-existent

2148 aws_s3 sink Increase default rate limit to allow for higher throughput

2161 socket sink Add support for UDP mode

2162 networking Add support for tls.verify_hostname in HTTP based sinks

2164 config Rename prefix condition predicate to starts_with

2181 config Add new ends_with condition predicate

2183 field_filter transform Deprecate in favor of the new filter transform

2195 config Add new regex condition predicate

2198 observability Warn that old field_filter transform is deprecated

2197 regex_parser transform Add new target_field option

2023 networking Detect TCP disconnects earlier

2209 lua transform Implement all hooks and timers in version 2

2126 vector source vector sink Support metric events

2245 ux Use external tagging for metrics serialization

2231 config Add source_key and source_type_key to…

2244 filter transform Accept metrics as input

2267 lua transform Set search_dirs to config dirs by default

2274 gcp_cloud_storage sink Make ACL optional

2283 docker source Add elegant shutdown behavior

2260 sources Add source_type field

2298 cli Show possible values for --log-format, --color, and --format

2300 topology Improve topology shutdown logs

2345 gcp provider Add support for instance accounts

2351 splunk_hec source Wire in new ShutdownSignal for improved shutdown

2373 journald source Wire in ShutdownSignal for improved shutdown

2364 sources Wire in new ShutdownSignal for improved shutdown

2365

14 new features

cli Add --log-format CLI option

1908 new source Initial http source implementation

1650 kafka sink Add support for compression

1969 new transform Initial dedupe transform implementation

1848 new transform Initial ‘tag_cardinality_limit’ transform implementation

1959 new sink Initial papertrail sink implementation

1835 new sink Initial honeycomb sink implementation

1847 new sink Initial pulsar sink implementation

1665 new sink Initial datadog_logs sink implementation

1832 elasticsearch sink Check for and log partial ingestion failures

2185 new transform Initial filter transform implementation

2088 new transform Inital kubernetes_pod_metadata transform implementation

1888 cli Add -qqq command line flag to disable logs

2230 new source Initial internal metrics source implementation

1953

20 bug fixes

config Force symlink re-evaluation when detecting a changed Vector config

2034 transforms Fix insertion to arrays in field path notation

2062 merge transform Support nested fields and arrays

1936 journald source Fix handling of message array data

2053 gcp_stackdrive_logging sink Rename gcp_stackdriver_logging sink to gcp_stackdriver_logs

2121 tests Fix disk pressure

2217 elasticsearch sink Default compression to none

2219 loki sink Pass tls settings to healthcheck

2234 influxdb_metrics sink Pass tls settings to healthcheck

2238 gcp_cloud_storage sink Use kebab case for ACL encoding

2259 aws_kinesis_firehose sink Fix healthchecks

2276 aws_ec2_metadata transform Only run required requests based on the supplied fields

2265 website correct typo in testing post

2291 website small fixes for testing post

2292 kafka sink Run healthcheck in rt context

2288 aws_cloudwatch_logs sink Sort events by timestamp

2320 sinks Reduce credentials fetch timeout

2342 lua transform Don’t require isdst, wday, yday in timestamps

2335 aws_cloudwatch_logs sink Allow creation race of groups and streams

2355 lua transform support sub-second timestamp resolution

2333