Vector v0.10.0 Release Notes

Lamport Clock
Jul 23rd, 2020 /

The Vector team is pleased to release 0.10.0. This is a initial dev release. Checkout the Highlights, Changelog, and What's Next section for more detail.

Highlights

Changelog

  • 26 Enhancements

    • splunk_hec sinkAdd `sourcetype`

    • topologyShutdown Vector if all sources finish

    • filter transformswimlanes transformAdd support for negating conditions in check_fields

    • elasticsearch sinkAdd explicit AWS region

    • journald sourceAdd support for include/exclude units

    • regex_parser transformAdd RegexSet support to regex

    • json_parser transformlog parse errors as warnings

    • http sinkadd bearer auth strategy

    • securityAdd support for loading multiple CAs

    • networkingAdd support for inline TLS keys

    • humio sinknew relic sinkAdd compression to humio and new_relic

    • elasticsearch sinkadd pipeline config

    • cliConsolidate and beautify `validate`

    • gcp_stackdriver_logs sinkAdd severity key

    • sinksUse to_raw_value instead of to_string in JsonArrayBuffer

    • aws_s3 sinkAdd content options

    • elasticsearch sinkShow client errors as warning logs

    • elasticsearch sinkadd assume_role to aws auth

    • kafka sourcekafka sinkAdd SASL to kafka

    • sinksBatch buffer rework

    • journald sourceAllow remapping priority numbers to words

    • file sourceOption to remove file after some time of reaching `eof`

    • sinksadd compression option for aws sinks

    • kafka sourceExtract timestamp from event

    • operationsAdd musl and glibc support to install.sh

    • syslog sourceAccept `octet-counting` encoded messages

  • 12 New Features

    • new sourceNew `generator` source

    • operationsOptimize CI test workflow with slim builds

    • kafka sinkAdd support for templates in the topic config

    • internal_metrics sourceInstrument few more components with metrics

    • new sinkInitial `influxdb_logs` sink implementation

    • platformsAdd dev workflow simplification rfc

    • new transformIntroduce WASM Plugins

    • operationsIntroduce Dev Workflow improvements

    • filter transformAdding list of string as filter option

    • operationsMore usable event RFC

    • filter transformAdding a condition to filter using IP CIDR

    • new transformadd reduce transform

  • 44 Bug Fixes

    • kafka sinkRun entire healthcheck lazily

    • socket sinkProperly detect shutdown

    • loki sinkreplace OrigID with OrgID

    • operationsIterate over target dirs to avoid missing mapall command on macOS

    • datadog_metrics sinkUse post requests

    • splunk_hec sinkSend time in <sec>.<ms> format

    • humio_logs sinkremove superfluous warning

    • elasticsearch sinkremove id key from body

    • journald sourceFix `units`/`include_units` switch

    • socket sinkFix possible infinite loop decoding too-long lines

    • aws_s3 sinkAllow removed keys to be used for partitioning

    • datadog_logs sinkEnable tls by default

    • testsFixup loki tests

    • testsRemove race between sink and source

    • file sourceFix the bug with channel closing at file server

    • kafka sinkkafa sourceFix the Kafka TLS key parameter handling

    • topologyCheck if `transform`/`sink` is present

    • sourcesShutdown starting from `tcp` and `unix` sockets

    • influxdb_logs sinkFix buffer compression

    • kafka sinkThree KafkaCompression typos in tests

    • splunk_hec sinkRemove extra slash on Uri join

    • pulsar sinkAdd new EncodingConfig

    • sinksincrement request id

    • configHandle templates with invalid strftime elements

    • aws_cloudwatch_sinkMake throttling errors retryable

    • aws_cloudwatch_logs sinkDrop excessive large events

    • sinksupgrade tower-limit to fix rate limiting

    • kafka sinkFix typo integration tests

    • sinksaddress potential deadlock

    • influxdb_metrics sinkUse `i` for unsigned ints on Influx v1

    • splunk_hec sourcefix wrap path

    • elasticsearch sinkrefresh aws creds when required

    • aws_cloudwatch_logs sinkHandle and enforce batch.max_bytes

    • operationsFix the cmark-gfm installation at builder image

    • operationsUse perl rename at builder

    • gcp providersync token request

    • loki sinkSet a proper default for the batch buffer initialization

    • depsupdate crates in lock file after merges

    • cliCleanup `generate` command

    • docsadjust log event in coercer transform

    • reduce transformimprove array handling

    • loki sinkDrop the batch byte size limit

    • file sinkappend instead of overwriting files

    • elasticsearch sinkstop logging every received event

  • 1 Perf Improvement

    • buffersadd batching to disk buffer reads

What's Next

  • Kubernetes! ⎈

    We're doing this integration right. We've been alpha testing with a number of large clusters and we've honed in on a very high-quality approach that reflects the UX Vector is going after. You can learn more in our Kubernetes Integration RFC. Do you want to beta test? Tell us!

  • WASM plugins 🔌

    This release included an experimental implementation of a WASM transform behind a wasm build flag. We plan to stabilize WASM transforms, as well as add WASM sinks and sources, enabling you to extend Vector in the language of your choice with little performance penalty. ❤️ WASM. Check out the WASM Plugin RFC for more info.

  • Dynamic HTTP Rate-Limiting (AIMD)

    Fiddling with rate-limits is a frustrating endaevor. If you set them too high you risk overwhelming the service; too low and you're unecessarily limiting throughput. What happens if you deploy a few more Vector instances? Vector is planning to solve this by automatically detecting the optimal rate limits, taking inspiration from TCP congestional control algorithms. Check out the Dynamic HTTP Rate Limiting RFC.

The Vector roadmap is publicly available and represents forward looking progress. Vector is open-source and community driven. Your feedback is welcome!

Like What You See?