Vector v0.7.0 Release Notes

The Great Omelet,
The easiest way to stay up-to-date. One email on the 1st of every month. No spam, ever.

You can't make an omelet without cracking some eggs. We have our eyes set on version 1 and a backlog of breaking changes we wanted to merge before we reach it. Version 0.7.0 is our attempt to cluster as many breaking changes as we can into a single release.

We're excited to release Vector v0.7.0! Vector follows semantic versioning, and this is an initial dev release. This release brings 13 new features, 10 enhancements, 12 bug fixes, 1 perf improvement. Checkout the highlights for notable features and, as always, let us know what you think!

Breaking Changes

The `tcp` and `udp` sources & sinks have been merged into `socket`

The `tcp` and `udp` sources no longer exist and have been merged into a new `socket` type. Likewise, the `tcp` sink has been merged into the `socket` sink. Migration is straight forward, simply change the `type` to `socket` and add the field `mode` to match the socket type (`tcp` or `udp`):

 [sources.my_tcp_source]
-  type = "tcp"
+  type = "socket"
   address = "0.0.0.0:9000"
+  mode = "tcp"

 [sources.my_tcp_sink]
-  type = "tcp"
+  type = "socket"
   address = "92.12.333.224:5000"
+  mode = "tcp"

Request, batching and auth fields refactored for sink components

Request based sinks have had their request fields nested under the table `request` and no longer use fixed retry intervals, instead using a fibonacci sequence for backing off retry attempts.

Batching fields are now nested under the table `batch`, with the field `size` replaced with either `max_events` or `max_size` in order to clarify its purpose (capping discrete events or bytes respectively).

Finally, authentication fields have been moved from the table `basic_auth` into a general purpose `auth` table complemented with a `strategy` field.

These changes effect the following sinks:

  • `aws_cloudwatch_logs`
  • `aws_kinesis_firehose`
  • `aws_kinesis_streams`
  • `aws_s3`
  • `clickhouse`
  • `datadog_metrics`
  • `elasticsearch`
  • `gcp_pubsub`
  • `http`
  • `new_relic_logs`
  • `splunk_hec`

In order to migrate all fields prefixed with `request_` must be placed within a `request` table with the prefix removed. The config field `retry_backoff_secs` must also be replaced with two new fields `retry_initial_backoff_secs` and `retry_max_duration_secs`. Fields prefixed with `batch_` must be placed within a `batch` table with the prefix removed. Instances of `batch_size` should be renamed `max_size` or `max_events` (refer to the relevant sink docs for the correct variant). Finally, the table `basic_auth` should renamed `auth` with a field `strategy` added:

 [sinks.my_sink_id]
   type = "http"
   inputs = ["my-source-id"]
   uri = "https://10.22.212.22:9000/endpoint"

-  batch_size = 1049000
+  [sinks.my_sink_id.batch]
+    max_size = 1049000

-  [sinks.my_sink_id.basic_auth]
+  [sinks.my_sink_id.auth]
+    strategy = "basic"
     user = "${USERNAME_ENV_VAR}"
     password = "${PASSWORD_ENV_VAR}"

-  request_in_flight_limit = 5
-  request_retry_backoff_secs = 1
+  [sinks.my_sink_id.request]
+    in_flight_limit = 5
+    retry_initial_backoff_secs = 1
+    retry_max_duration_secs = 10

The buffer field `num_items` has been renamed to `max_events`

The buffer field `num_items` has been renamed to `max_events` in order to clarify its purpose. Migration involves simply changing all occurrences of `buffer.num_items` to `buffer.max_events`:

 [sinks.my_sink_id]
   type = "aws_s3"
   inputs = ["my-source-id"]
   bucket = "my-bucket"

   [sinks.my_sink_id.buffer]
     type = "memory"
-    num_items = 500
+    max_events = 500

The `kafka` sink now takes a comma separated list of bootstrap servers

The `kafka` sink field `bootstrap_servers` has been changed from an array to a string, expecting a comma separated list of bootstrap servers similar to the `kafka` source:

 [sinks.my_sink_id]
   type = "kafka"
   inputs = ["my-source-id"]
-  bootstrap_servers = ["10.14.22.123:9092", "10.14.23.332:9092"]
+  bootstrap_servers = "10.14.22.123:9092,10.14.23.332:9092"

The `journald` source now pipes from journalctl

The `journald` source originally used libsystemd in order to consume logs from Journald. Since this was causing issues for some distributions of Vector it has now been changed to pipe logs from journalctl.

Therefore, when using a `journald` source users must now ensure that the `journalctl` binary is available. The field `local_only` has also been removed as it is no longer applicable.

The `splunk_hec` sink does not index any fields by default

There is no longer a distinction within Vector between explicit and implicit event fields. All fields are now implicit and therefore the `splunk_hec` sink will not index any fields by default.

In order to mark desired fields as indexed you can use the optional configuration option `indexed_fields`:

 [sinks.my_sink_id]
   type = "splunk_hec"
   inputs = ["my-source-id"]
+  indexed_fields = ["foo", "bar"]

Changelog

13 New Features

breaking 1526

journald source Switch from libsystemd to piping journalctl

networking Add custom DNS option

new sink Initial `gcp_pubsub` sink implementation

new sink Initial `aws_kinesis_firehose` sink

breaking 1404

new sink Move existing 'tcp' sink into 'socket' sink.

new source Inital `kubernetes` source implementation

breaking 1485

new source Merge existing 'tcp' and 'udp' sources into a single 'socket' source

new source Initial `prometheus` source implementation

new source Initial `logplex` source implementation

new transform Initial `concat` transform implementation

new transform Initial `aws_ec2_metadata` transform implementation

new transform Initial `logfmt_parser` transform impl

socket source Add unix domain socket support to the 'socket` source

10 Enhancements

aws_cloudwatch_logs sink add assume_role

cli Handle interruptions on Windows

datadog_metrics sink Send min for distributions

docker source Add additional container filters

docker source Use shiplift's since for filtering logs

kubernetes source Parse additional metadata

breaking 1006

networking Use fibonacci instead of fixed backoff

breaking 1537

splunk_hec sink Add a new option for specifying indexed fields to `splunk_hec` sink

splunk_hec source Provide timestamp

breaking 1457

splunk_hec source Rename `line` field to `message`

12 Bug Fixes

buffers Don't ack with zero items

buffers Re-enable buffering reclaim test

operations Fix `release-github` job

operations Use Bundler 2.0.x for the checker and releaser images

operations Fix `rustup` for `x86_64-pc-windows-msvc` in CI

operations Ensure that all files are checked by `check-style.sh`

operations Fix newline style

operations Fix tests for NixOS

operations Remove trailing spaces to fix `check-fmt`

regex_parser transform make truncation utf8-aware

splunk_hec sink Stop testing with ext host

website Fix performance comparison link

1 Perf Improvement

file source Backoff reads to inactive files


Download this release