The Vector team is pleased to release 0.10.0. This is a initial dev release. Checkout the Highlights, Changelog, and What's Next section for more detail.
Highlights
New Features
Journald Unit Filtering Exclusions
The journald source can now exclude units.Kafka components support SASL
Vector has a new way to authenticate with Kafka!New compression options for some sinks
Stuff more data down a smaller hose in less time for cheaper.New Reduce transform
Canonical Log Lines in VectorEnhancements
RegexSet support to `regex` transform
Efficiently run multiple regexes in the same transform!Now supporting the bearer auth strategy
Vector can now bear authentication tokens for relevant components.Support for loading multiple CAs
Complicated PEM? No PEM-blem with Vector!Beautification of the validate command
A little polish on a useful feature.Vector gracefully exits when specific sources finish
One step closer to Vector replacing awk and sed!Breaking Changes
Changelog
26 Enhancements
splunk_hec sinkAdd `sourcetype`
topologyShutdown Vector if all sources finish
filter transformswimlanes transformAdd support for negating conditions in check_fields
elasticsearch sinkAdd explicit AWS region
journald sourceAdd support for include/exclude units
regex_parser transformAdd RegexSet support to regex
json_parser transformlog parse errors as warnings
http sinkadd bearer auth strategy
securityAdd support for loading multiple CAs
networkingAdd support for inline TLS keys
humio sinknew relic sinkAdd compression to humio and new_relic
elasticsearch sinkadd pipeline config
cliConsolidate and beautify `validate`
gcp_stackdriver_logs sinkAdd severity key
sinksUse to_raw_value instead of to_string in JsonArrayBuffer
aws_s3 sinkAdd content options
elasticsearch sinkShow client errors as warning logs
elasticsearch sinkadd assume_role to aws auth
kafka sourcekafka sinkAdd SASL to kafka
sinksBatch buffer rework
journald sourceAllow remapping priority numbers to words
file sourceOption to remove file after some time of reaching `eof`
sinksadd compression option for aws sinks
kafka sourceExtract timestamp from event
operationsAdd musl and glibc support to install.sh
syslog sourceAccept `octet-counting` encoded messages
12 New Features
new sourceNew `generator` source
operationsOptimize CI test workflow with slim builds
kafka sinkAdd support for templates in the topic config
internal_metrics sourceInstrument few more components with metrics
new sinkInitial `influxdb_logs` sink implementation
platformsAdd dev workflow simplification rfc
new transformIntroduce WASM Plugins
operationsIntroduce Dev Workflow improvements
filter transformAdding list of string as filter option
operationsMore usable event RFC
filter transformAdding a condition to filter using IP CIDR
new transformadd reduce transform
44 Bug Fixes
kafka sinkRun entire healthcheck lazily
socket sinkProperly detect shutdown
loki sinkreplace OrigID with OrgID
operationsIterate over target dirs to avoid missing mapall command on macOS
datadog_metrics sinkUse post requests
splunk_hec sinkSend time in <sec>.<ms> format
humio_logs sinkremove superfluous warning
elasticsearch sinkremove id key from body
journald sourceFix `units`/`include_units` switch
socket sinkFix possible infinite loop decoding too-long lines
aws_s3 sinkAllow removed keys to be used for partitioning
datadog_logs sinkEnable tls by default
testsFixup loki tests
testsRemove race between sink and source
file sourceFix the bug with channel closing at file server
kafka sinkkafa sourceFix the Kafka TLS key parameter handling
topologyCheck if `transform`/`sink` is present
sourcesShutdown starting from `tcp` and `unix` sockets
influxdb_logs sinkFix buffer compression
kafka sinkThree KafkaCompression typos in tests
splunk_hec sinkRemove extra slash on Uri join
pulsar sinkAdd new EncodingConfig
sinksincrement request id
configHandle templates with invalid strftime elements
aws_cloudwatch_sinkMake throttling errors retryable
aws_cloudwatch_logs sinkDrop excessive large events
sinksupgrade tower-limit to fix rate limiting
kafka sinkFix typo integration tests
sinksaddress potential deadlock
influxdb_metrics sinkUse `i` for unsigned ints on Influx v1
splunk_hec sourcefix wrap path
elasticsearch sinkrefresh aws creds when required
aws_cloudwatch_logs sinkHandle and enforce batch.max_bytes
operationsFix the cmark-gfm installation at builder image
operationsUse perl rename at builder
gcp providersync token request
loki sinkSet a proper default for the batch buffer initialization
depsupdate crates in lock file after merges
cliCleanup `generate` command
docsadjust log event in coercer transform
reduce transformimprove array handling
loki sinkDrop the batch byte size limit
file sinkappend instead of overwriting files
elasticsearch sinkstop logging every received event
1 Perf Improvement
buffersadd batching to disk buffer reads
What's Next
Kubernetes! ⎈
We're doing this integration right. We've been alpha testing with a number of large clusters and we've honed in on a very high-quality approach that reflects the UX Vector is going after. You can learn more in our Kubernetes Integration RFC. Do you want to beta test? Tell us!
WASM plugins 🔌
This release included an experimental implementation of a WASM transform behind a
wasmbuild flag. We plan to stabilize WASM transforms, as well as add WASM sinks and sources, enabling you to extend Vector in the language of your choice with little performance penalty. ❤️ WASM. Check out the WASM Plugin RFC for more info.Dynamic HTTP Rate-Limiting (AIMD)
Fiddling with rate-limits is a frustrating endaevor. If you set them too high you risk overwhelming the service; too low and you're unecessarily limiting throughput. What happens if you deploy a few more Vector instances? Vector is planning to solve this by automatically detecting the optimal rate limits, taking inspiration from TCP congestional control algorithms. Check out the Dynamic HTTP Rate Limiting RFC.
The Vector roadmap is publicly available and represents forward looking progress. Vector is open-source and community driven. Your feedback is welcome!