The Vector team is pleased to release 0.9.0. This is a initial dev release. Checkout the Highlights, Changelog, and What's Next section for more detail.
Highlights
New Features
A New Guides Section
High quality guides and education for VectorNew Dedupe Transform
Shed duplicate logsNew Tag Cardinality Limit Transform
Protect downstream metrics storages from runaway metrics tagsNew Filter Transform
Filter and route your logs based on defined conditionsÀ La Carte Custom Vector Builds
Build Vector with select components4 New Sinks!
Papertrail, Honeycomb, Pulsar, and DataDog LogsEnhancements
Lua Transform v2
The next iteration of our Lua transform.New Encoding Options
Vector now lets you whitelist, blacklist, and format fields when events are encodedImproved Shutdown
A faster and more reliable shutdown processBreaking Changes
The `gcp_stackdriver_logging` sink has been renamed to `gcp_stackdriver_logs`
This brings the sink naming inline with consistent naming patternThe `elasticsearch` sink `compression` option now defaults to `none`
Compression is not supported by all Elasticsearch service providersACL is now optional for the Google Cloud Storage sink
ACL is not always required when creating objects in GCP Cloud Storage
Changelog
48 Enhancements
configAllow setting individual `log_schema.*` options
socket sourcesyslog sourcevector sourceAdd TLS support to socket, syslog, and vector sources
configDefault to `check_fields` condition when specifying conditions
configAdd `is_log` and `is_metric` conditions
operationsCreate feature flags for all components enabling custom Vector builds
cliAdd `--fragment` flag to `generate` subcmd
cliAllow names in `generate` subcmd expression
logplex sourcesplunk_hec sourceAdd TLS support
sinksAdd new `encoding.only_fields` and `encoding.except_fields` options
networkingMove all TLS support over to openssl
configAdd `contains` and `prefix` predicates
vector sinkAdd support for TLS
elasticsearch sinkAdd `timeout` query parameter
sampler transformAdd new `key_field` option
lua transformAdd `version` configuration option
configSupport escaping in the field path notation
add_fields transformAllow native TOML tables
remove_fields transformrename_fields transformAdd new `drop_empty` option to drop empty objects automatically
lua transformSupport metric events in version 2
configMake LogSchema more optional
networkingAdd `User-agent` header to all outgoing HTTP requests
splunk_hec sinkAdd new `index` option to specify custom Splunk indexes
observabilityThe `add_fields`, `remove_fields`, and `rename_fields` now produce a debug log when a field is replaced or non-existent
aws_s3 sinkIncrease default rate limit to allow for higher throughput
socket sinkAdd support for UDP mode
networkingAdd support for `tls.verify_hostname` in HTTP based sinks
configRename `prefix` condition predicate to `starts_with`
configAdd new `ends_with` condition predicate
field_filter transformDeprecate in favor of the new `filter` transform
configAdd new `regex` condition predicate
observabilityWarn that old `field_filter` transform is deprecated
regex_parser transformAdd new `target_field` option
networkingDetect TCP disconnects earlier
lua transformImplement all hooks and timers in version 2
vector sourcevector sinkSupport metric events
uxUse external tagging for metrics serialization
configAdd `source_key` and `source_type_key` to…
filter transformAccept metrics as input
lua transformSet `search_dirs` to config dirs by default
gcp_cloud_storage sinkMake ACL optional
docker sourceAdd elegant shutdown behavior
sourcesAdd `source_type` field
cliShow possible values for `--log-format`, `--color`, and `--format`
topologyImprove topology shutdown logs
gcp providerAdd support for instance accounts
splunk_hec sourceWire in new `ShutdownSignal` for improved shutdown
journald sourceWire in `ShutdownSignal` for improved shutdown
sourcesWire in new `ShutdownSignal` for improved shutdown
14 New Features
cliAdd `--log-format` CLI option
new sourceInitial `http` source implementation
kafka sinkAdd support for compression
new transformInitial `dedupe` transform implementation
new transformInitial 'tag_cardinality_limit' transform implementation
new sinkInitial `papertrail` sink implementation
new sinkInitial `honeycomb` sink implementation
new sinkInitial `pulsar` sink implementation
new sinkInitial `datadog_logs` sink implementation
elasticsearch sinkCheck for and log partial ingestion failures
new transformInitial `filter` transform implementation
new transformInital `kubernetes_pod_metadata` transform implementation
cliAdd `-qqq` command line flag to disable logs
new sourceInitial `internal` metrics source implementation
20 Bug Fixes
configForce `symlink` re-evaluation when detecting a changed Vector config
transformsFix insertion to arrays in field path notation
merge transformSupport nested fields and arrays
journald sourceFix handling of message array data
gcp_stackdrive_logging sinkRename `gcp_stackdriver_logging` sink to `gcp_stackdriver_logs`
testsFix disk pressure
elasticsearch sinkDefault `compression` to `none`
loki sinkPass `tls` settings to healthcheck
influxdb_metrics sinkPass `tls` settings to healthcheck
gcp_cloud_storage sinkUse kebab case for ACL encoding
aws_kinesis_firehose sinkFix healthchecks
aws_ec2_metadata transformOnly run required requests based on the supplied `fields`
websitecorrect typo in testing post
websitesmall fixes for testing post
kafka sinkRun healthcheck in rt context
aws_cloudwatch_logs sinkSort events by timestamp
sinksReduce credentials fetch timeout
lua transformDon't require `isdst`, `wday`, `yday` in timestamps
aws_cloudwatch_logs sinkAllow creation race of groups and streams
lua transformsupport sub-second timestamp resolution
5 Perf Improvements
Lift the internal thread limit to enable full concurrency
Start removing Atom from the hot path
Add fast path and caching api to event insert
Add `lto = true` and `codegen-units=1` to `Cargo.toml`
tokenizer transformCache paths for parsed fields
What's Next
Kubernetes!
We're doing this integration right. We've been alpha testing with a number of large clusters and we've honed in on a very high-quality approach that reflects the UX Vector is going after. You can learn more in our Kubernetes Integration RFC. Feedback is welcome!
WASM Driven Plugin Architecture
This release included an overhaul of our
luatransform, which represents a big step towards making Vector more flexible. Lua is an excellent language when you need some quick scripting capabilities. But what if you need something more powerful? Like writing your own custom source, transform, or sink? Vector plans to solve this with WASM foreign module support, enabling you to extend Vector in the language of your choice with little performance penalty. ❤️ WASM. Check out the WASM Foreign Module Support RFC for more info.Dynamic HTTP Rate-Limiting (AIMD)
Fiddling with rate-limits is a frustratin endaevor. If you set them too high you risk overwhelming the service; too low and you're unecessarily limiting throughput. And what happens if you deploy a few more Vector instances? Vector is planning to solve this by automatically detecting the optimal rate limits, taking inspiration from TCP congestional control algorithms. Check out the Dynamic HTTP Rate Limiting RFC.
Vector Observability
Did you know this release includes a new hidden
internalsource that emits metrics reflecting Vector's own internal state? 👀 We didn't highlight this source because we have not finished instrumenting Vector and we'll be changing the internal schema used.Fun fact, we took an event-driven approach to Vector's internal observability, which we strongly beleive in. We want Vector to be a good example of our own advice. Checkout the Event-Driven Observability RFC.
The Vector roadmap is publicly available and represents forward looking progress. Vector is open-source and community driven. Your feedback is welcome!