Vector v0.9.0 Release Notes

The Homestretch
Apr 17th, 2020 /

The Vector team is pleased to release 0.9.0. This is a initial dev release. Checkout the Highlights, Changelog, and What's Next section for more detail.

Highlights

Changelog

  • 48 Enhancements

    • configAllow setting individual `log_schema.*` options

    • socket sourcesyslog sourcevector sourceAdd TLS support to socket, syslog, and vector sources

    • configDefault to `check_fields` condition when specifying conditions

    • configAdd `is_log` and `is_metric` conditions

    • operationsCreate feature flags for all components enabling custom Vector builds

    • cliAdd `--fragment` flag to `generate` subcmd

    • cliAllow names in `generate` subcmd expression

    • logplex sourcesplunk_hec sourceAdd TLS support

    • sinksAdd new `encoding.only_fields` and `encoding.except_fields` options

    • networkingMove all TLS support over to openssl

    • configAdd `contains` and `prefix` predicates

    • vector sinkAdd support for TLS

    • elasticsearch sinkAdd `timeout` query parameter

    • sampler transformAdd new `key_field` option

    • lua transformAdd `version` configuration option

    • configSupport escaping in the field path notation

    • add_fields transformAllow native TOML tables

    • remove_fields transformrename_fields transformAdd new `drop_empty` option to drop empty objects automatically

    • lua transformSupport metric events in version 2

    • configMake LogSchema more optional

    • networkingAdd `User-agent` header to all outgoing HTTP requests

    • splunk_hec sinkAdd new `index` option to specify custom Splunk indexes

    • observabilityThe `add_fields`, `remove_fields`, and `rename_fields` now produce a debug log when a field is replaced or non-existent

    • aws_s3 sinkIncrease default rate limit to allow for higher throughput

    • socket sinkAdd support for UDP mode

    • networkingAdd support for `tls.verify_hostname` in HTTP based sinks

    • configRename `prefix` condition predicate to `starts_with`

    • configAdd new `ends_with` condition predicate

    • field_filter transformDeprecate in favor of the new `filter` transform

    • configAdd new `regex` condition predicate

    • observabilityWarn that old `field_filter` transform is deprecated

    • regex_parser transformAdd new `target_field` option

    • networkingDetect TCP disconnects earlier

    • lua transformImplement all hooks and timers in version 2

    • vector sourcevector sinkSupport metric events

    • uxUse external tagging for metrics serialization

    • configAdd `source_key` and `source_type_key` to…

    • filter transformAccept metrics as input

    • lua transformSet `search_dirs` to config dirs by default

    • gcp_cloud_storage sinkMake ACL optional

    • docker sourceAdd elegant shutdown behavior

    • sourcesAdd `source_type` field

    • cliShow possible values for `--log-format`, `--color`, and `--format`

    • topologyImprove topology shutdown logs

    • gcp providerAdd support for instance accounts

    • splunk_hec sourceWire in new `ShutdownSignal` for improved shutdown

    • journald sourceWire in `ShutdownSignal` for improved shutdown

    • sourcesWire in new `ShutdownSignal` for improved shutdown

  • 14 New Features

    • cliAdd `--log-format` CLI option

    • new sourceInitial `http` source implementation

    • kafka sinkAdd support for compression

    • new transformInitial `dedupe` transform implementation

    • new transformInitial 'tag_cardinality_limit' transform implementation

    • new sinkInitial `papertrail` sink implementation

    • new sinkInitial `honeycomb` sink implementation

    • new sinkInitial `pulsar` sink implementation

    • new sinkInitial `datadog_logs` sink implementation

    • elasticsearch sinkCheck for and log partial ingestion failures

    • new transformInitial `filter` transform implementation

    • new transformInital `kubernetes_pod_metadata` transform implementation

    • cliAdd `-qqq` command line flag to disable logs

    • new sourceInitial `internal` metrics source implementation

  • 20 Bug Fixes

    • configForce `symlink` re-evaluation when detecting a changed Vector config

    • transformsFix insertion to arrays in field path notation

    • merge transformSupport nested fields and arrays

    • journald sourceFix handling of message array data

    • gcp_stackdrive_logging sinkRename `gcp_stackdriver_logging` sink to `gcp_stackdriver_logs`

    • testsFix disk pressure

    • elasticsearch sinkDefault `compression` to `none`

    • loki sinkPass `tls` settings to healthcheck

    • influxdb_metrics sinkPass `tls` settings to healthcheck

    • gcp_cloud_storage sinkUse kebab case for ACL encoding

    • aws_kinesis_firehose sinkFix healthchecks

    • aws_ec2_metadata transformOnly run required requests based on the supplied `fields`

    • websitecorrect typo in testing post

    • websitesmall fixes for testing post

    • kafka sinkRun healthcheck in rt context

    • aws_cloudwatch_logs sinkSort events by timestamp

    • sinksReduce credentials fetch timeout

    • lua transformDon't require `isdst`, `wday`, `yday` in timestamps

    • aws_cloudwatch_logs sinkAllow creation race of groups and streams

    • lua transformsupport sub-second timestamp resolution

  • 5 Perf Improvements

    • Lift the internal thread limit to enable full concurrency

    • Start removing Atom from the hot path

    • Add fast path and caching api to event insert

    • Add `lto = true` and `codegen-units=1` to `Cargo.toml`

    • tokenizer transformCache paths for parsed fields

What's Next

  • Kubernetes!

    We're doing this integration right. We've been alpha testing with a number of large clusters and we've honed in on a very high-quality approach that reflects the UX Vector is going after. You can learn more in our Kubernetes Integration RFC. Feedback is welcome!

  • WASM Driven Plugin Architecture

    This release included an overhaul of our lua transform, which represents a big step towards making Vector more flexible. Lua is an excellent language when you need some quick scripting capabilities. But what if you need something more powerful? Like writing your own custom source, transform, or sink? Vector plans to solve this with WASM foreign module support, enabling you to extend Vector in the language of your choice with little performance penalty. ❤️ WASM. Check out the WASM Foreign Module Support RFC for more info.

  • Dynamic HTTP Rate-Limiting (AIMD)

    Fiddling with rate-limits is a frustratin endaevor. If you set them too high you risk overwhelming the service; too low and you're unecessarily limiting throughput. And what happens if you deploy a few more Vector instances? Vector is planning to solve this by automatically detecting the optimal rate limits, taking inspiration from TCP congestional control algorithms. Check out the Dynamic HTTP Rate Limiting RFC.

  • Vector Observability

    Did you know this release includes a new hidden internal source that emits metrics reflecting Vector's own internal state? 👀 We didn't highlight this source because we have not finished instrumenting Vector and we'll be changing the internal schema used.

    Fun fact, we took an event-driven approach to Vector's internal observability, which we strongly beleive in. We want Vector to be a good example of our own advice. Checkout the Event-Driven Observability RFC.

The Vector roadmap is publicly available and represents forward looking progress. Vector is open-source and community driven. Your feedback is welcome!

Like What You See?