Vector v0.8.0 Release Notes

A wholesome platter of features,
The easiest way to stay up-to-date. One email on the 1st of every month. No spam, ever.

The features for this release have been pouring in. Clearly the Vector team need better hobbies.

We're excited to release Vector v0.8.0! Vector follows semantic versioning, and this is an initial dev release. This release brings 14 new features, 24 enhancements, 11 bug fixes. Checkout the highlights for notable features and, as always, let us know what you think!

Highlights

Automatically Merge Split Docker Events!

Anyone that was worked with Docker logs knows how frustrating this problem can be. Docker, by default, splits log messages that exceed 16kb. While 16kb seems like a lot, it can easily be exceeded if you're logging rich structured events. This can be a very difficult and frustrating problem to solve with other tools (we speak from experience). In this release Vector solves this automatically with a new `auto_partial_merge` option in the `docker` source.

We love assimilation and look forward to a future where our individualistic human personalities can also be merged into a societal hive mind.

Improved Control Flow With Swimlanes

The new swimlanes transform makes it much easier to configure conditional branches of transforms and sinks. For example, you can easily create if/else pipelines.

Remember to occasionally let your branches mingle so that they don't completely lose touch.

Config Unit Test Improvements

Unit testing your configuration files has been supported by Vector for a while now. Based on feedback we've received and some healthy dog-fooding we've expanded their functionality and made a lot of improvements to how they work.

If you aren't yet using unit tests in your configs we strongly recommend you try it out. You can find out more in this guide.

Custom Field Schemas

Not everyone enjoys Marmite, we get it, some people have wrong different tastes. Similarly, not everyone is going to want to follow our default field schema (`host`, `message`, `timestamp`). Rather than making you work around this with transforms we've added some handy global options to set these universally:

# OPTIONAL - Log schema
[log_schema]
  host_key = "host" # default
  message_key = "message" # default
  timestamp_key = "timestamp" # default

Support for Multiple Config Files

Vector is now able to execute a topology spread across multiple config files, which allows you to break large pipelines down into bite size, easier managed, chunks. Running them is as simple as:

vector -c ./configs/first.toml -c ./configs/second.toml -c ./more/*.toml

Subscribe to our newsletter and you'll be notified when we learn how to do this with human emotions.

Lots of New Sinks

We all love a good sink. Now you can enjoy our latest offerings: gcp_stackdriver_logging, gcp_cloud_storage, logdna, humio_logs, loki and influxdb_metrics.

Changelog

14 New Features

cli Support multiple configuration files.

config Option to reload config on file change

config Global default log schemas

kubernetes source Add more `include_container_names`, `include_pod_uids`, and `include_namespaces` filters

new sink Initial `gcp_stackdriver_logging` sink implementation

new sink Initial `logdna` sink implementation

new sink Initial `humio_logs` sink implementation

new sink Initial `loki` sink

new sink Initial `influxdb_metrics` sink implementation

new sink Initial `gcp_cloud_storage` sink implementation

new transform New `merge` transform

new transform Initial `rename_fields` transform implementation

new transform Add `swimlanes` transform

socket sink Add unix domain socket support to the `socket` sink

24 Enhancements

add_fields transform Support templating for dynamic values

aws provider Add support for `assume_role` to all AWS sinks

aws_cloudwatch_logs sink Enable more logging by default

aws_s3 sink Add several S3 object-level options (`grant_full_control`, `server_side_encryption`, and more)

cli Colourize help

cli Improve `vector generate` syntax and formatting

cli Test sub command improvements

clickhouse sink Add `timestamp_format` field

coercer transform Add new `drop_unspecified` option to drop unspecified fields

config Clarify topology warnings

config Allow multiple unit test inputs

config Add `no_outputs_from` field to unit tests

docker source Add `auto_partial_merge` to automatically merge partial events

file source Better multi-line support

kafka sink Support advanced `librdkafka` options

kafka source Support advanced `librdkafka` options

kafka source Add support for TLS

breaking 1836

log data model Support `Array` and `Map` kinds in `Value`

breaking 1898

log data model Support null values in log events

observability Warn log for non retryable requests

observability Add proper non retry messages

observability Enable `rdkafka` by default

syslog source Improve best effort support for Syslog formats (5424, 3164, nginx style, etc)

topology Improve cyclic deps error message

11 Bug Fixes

aws_kinesis_firehose sink Use `describe_delivery_stream` AWS API method for healthcheck

aws_s3 sink Fix a bug with `retry_limit` is set to 1

docker source Always check containers for self

journald source Fix the journalctl start date to work across all timezones

json_parser transform only drop source field if JSON parse succeeds

kubernetes source Use checksum fingerprinting

lua transform Respect Lua types when converting back to events (#857)

rpm platform Create `vector` user when installing RPM package

rpm platform Fix group creation in the RPM package

sematext sink Fix EU endpoint and docs

socket sink Allow for non-IP address host names


Download this release