Vector v0.9.0 Release Notes

The Homestretch
Apr 17th, 2020 /
download

The Vector team is pleased to release 0.9.0. This is a initial dev release. Checkout the Highlights, Changelog, and What's Next section for more detail.

Highlights

Noteworthy changes in this release

Changelog

A complete list of changes
87 items

  • 48 Enhancements

    • 1923

      configAllow setting individual `log_schema.*` options

    • 1892

      socket sourcesyslog sourcevector sourceAdd TLS support to socket, syslog, and vector sources

    • 1947

      configDefault to `check_fields` condition when specifying conditions

    • 1950

      configAdd `is_log` and `is_metric` conditions

    • highlight 1924

      operationsCreate feature flags for all components enabling custom Vector builds

    • 1956

      cliAdd `--fragment` flag to `generate` subcmd

    • 1955

      cliAllow names in `generate` subcmd expression

    • 1968

      logplex sourcesplunk_hec sourceAdd TLS support

    • highlight 1915

      sinksAdd new `encoding.only_fields` and `encoding.except_fields` options

    • 1986

      networkingMove all TLS support over to openssl

    • highlight 1997

      configAdd `contains` and `prefix` predicates

    • highlight 2025

      vector sinkAdd support for TLS

    • 2038

      elasticsearch sinkAdd `timeout` query parameter

    • 2039

      sampler transformAdd new `key_field` option

    • 2056

      lua transformAdd `version` configuration option

    • 2081

      configSupport escaping in the field path notation

    • 2068

      add_fields transformAllow native TOML tables

    • 2077

      remove_fields transformrename_fields transformAdd new `drop_empty` option to drop empty objects automatically

    • 2095

      lua transformSupport metric events in version 2

    • 2113

      configMake LogSchema more optional

    • 2130

      networkingAdd `User-agent` header to all outgoing HTTP requests

    • 2127

      splunk_hec sinkAdd new `index` option to specify custom Splunk indexes

    • 2148

      observabilityThe `add_fields`, `remove_fields`, and `rename_fields` now produce a debug log when a field is replaced or non-existent

    • 2161

      aws_s3 sinkIncrease default rate limit to allow for higher throughput

    • 2162

      socket sinkAdd support for UDP mode

    • 2164

      networkingAdd support for `tls.verify_hostname` in HTTP based sinks

    • 2181

      configRename `prefix` condition predicate to `starts_with`

    • highlight 2183

      configAdd new `ends_with` condition predicate

    • 2195

      field_filter transformDeprecate in favor of the new `filter` transform

    • highlight 2198

      configAdd new `regex` condition predicate

    • 2197

      observabilityWarn that old `field_filter` transform is deprecated

    • 2023

      regex_parser transformAdd new `target_field` option

    • 2209

      networkingDetect TCP disconnects earlier

    • highlight 2126

      lua transformImplement all hooks and timers in version 2

    • highlight 2245

      vector sourcevector sinkSupport metric events

    • breaking 2231

      uxUse external tagging for metrics serialization

    • 2244

      configAdd `source_key` and `source_type_key` to…

    • 2267

      filter transformAccept metrics as input

    • breaking 2274

      lua transformSet `search_dirs` to config dirs by default

    • breaking 2283

      gcp_cloud_storage sinkMake ACL optional

    • 2260

      docker sourceAdd elegant shutdown behavior

    • 2298

      sourcesAdd `source_type` field

    • 2300

      cliShow possible values for `--log-format`, `--color`, and `--format`

    • 2345

      topologyImprove topology shutdown logs

    • 2351

      gcp providerAdd support for instance accounts

    • 2373

      splunk_hec sourceWire in new `ShutdownSignal` for improved shutdown

    • 2364

      journald sourceWire in `ShutdownSignal` for improved shutdown

    • 2365

      sourcesWire in new `ShutdownSignal` for improved shutdown

  • 14 New Features

    • 1908

      cliAdd `--log-format` CLI option

    • 1650

      new sourceInitial `http` source implementation

    • highlight 1969

      kafka sinkAdd support for compression

    • highlight 1848

      new transformInitial `dedupe` transform implementation

    • highlight 1959

      new transformInitial 'tag_cardinality_limit' transform implementation

    • highlight 1835

      new sinkInitial `papertrail` sink implementation

    • highlight 1847

      new sinkInitial `honeycomb` sink implementation

    • highlight 1665

      new sinkInitial `pulsar` sink implementation

    • highlight 1832

      new sinkInitial `datadog_logs` sink implementation

    • 2185

      elasticsearch sinkCheck for and log partial ingestion failures

    • highlight 2088

      new transformInitial `filter` transform implementation

    • 1888

      new transformInital `kubernetes_pod_metadata` transform implementation

    • 2230

      cliAdd `-qqq` command line flag to disable logs

    • 1953

      new sourceInitial `internal` metrics source implementation

  • 20 Bug Fixes

    • 2034

      configForce `symlink` re-evaluation when detecting a changed Vector config

    • 2062

      transformsFix insertion to arrays in field path notation

    • 1936

      merge transformSupport nested fields and arrays

    • 2053

      journald sourceFix handling of message array data

    • breaking 2121

      gcp_stackdrive_logging sinkRename `gcp_stackdriver_logging` sink to `gcp_stackdriver_logs`

    • 2217

      testsFix disk pressure

    • breaking 2219

      elasticsearch sinkDefault `compression` to `none`

    • 2234

      loki sinkPass `tls` settings to healthcheck

    • 2238

      influxdb_metrics sinkPass `tls` settings to healthcheck

    • 2259

      gcp_cloud_storage sinkUse kebab case for ACL encoding

    • 2276

      aws_kinesis_firehose sinkFix healthchecks

    • 2265

      aws_ec2_metadata transformOnly run required requests based on the supplied `fields`

    • 2291

      websitecorrect typo in testing post

    • 2292

      websitesmall fixes for testing post

    • 2288

      kafka sinkRun healthcheck in rt context

    • 2320

      aws_cloudwatch_logs sinkSort events by timestamp

    • 2342

      sinksReduce credentials fetch timeout

    • 2335

      lua transformDon't require `isdst`, `wday`, `yday` in timestamps

    • 2355

      aws_cloudwatch_logs sinkAllow creation race of groups and streams

    • 2333

      lua transformsupport sub-second timestamp resolution

  • 5 Perf Improvements

    • 2145

      Lift the internal thread limit to enable full concurrency

    • 2295

      Start removing Atom from the hot path

    • 2296

      Add fast path and caching api to event insert

    • 2321

      Add `lto = true` and `codegen-units=1` to `Cargo.toml`

    • 2325

      tokenizer transformCache paths for parsed fields

What's Next

  • Kubernetes!

    We're doing this integration right. We've been alpha testing with a number of large clusters and we've honed in on a very high-quality approach that reflects the UX Vector is going after. You can learn more in our Kubernetes Integration RFC. Feedback is welcome!

  • WASM Driven Plugin Architecture

    This release included an overhaul of our lua transform, which represents a big step towards making Vector more flexible. Lua is an excellent language when you need some quick scripting capabilities. But what if you need something more powerful? Like writing your own custom source, transform, or sink? Vector plans to solve this with WASM foreign module support, enabling you to extend Vector in the language of your choice with little performance penalty. ❤️ WASM. Check out the WASM Foreign Module Support RFC for more info.

  • Dynamic HTTP Rate-Limiting (AIMD)

    Fiddling with rate-limits is a frustratin endaevor. If you set them too high you risk overwhelming the service; too low and you're unecessarily limiting throughput. And what happens if you deploy a few more Vector instances? Vector is planning to solve this by automatically detecting the optimal rate limits, taking inspiration from TCP congestional control algorithms. Check out the Dynamic HTTP Rate Limiting RFC.

  • Vector Observability

    Did you know this release includes a new hidden internal source that emits metrics reflecting Vector's own internal state? 👀 We didn't highlight this source because we have not finished instrumenting Vector and we'll be changing the internal schema used.

    Fun fact, we took an event-driven approach to Vector's internal observability, which we strongly beleive in. We want Vector to be a good example of our own advice. Checkout the Event-Driven Observability RFC.

The Vector roadmap is publicly available and represents forward looking progress. Vector is open-source and community driven. Your feedback is welcome!

Like What You See?

Follow @vectordotdev
Get real-time updates!
Star timberio/vector
Star the repo to support us.