Vector v0.13.0 Release Notes

Apr 21st, 2021 /

The Vector team is pleased to announce version 0.13.0!

This release brings a new datadog_logs source to accept logs from dd-agents, a new gcp_stackdriver_metrics sink to send metrics to GCP Stackdriver, and a new subcommand, vector tap that allows for inspecting events flowing out of a component. It also brings a number of smaller enhancements, particularly around the Vector Remap Language used by our remap, filter, and route transforms.

Check out the highlights and changelog for more details.

Highlights

What's Next

  • End to end acknowledgements

    We've heard from a number of users that they'd like improved delivery guarantees for events flowing through Vector. We are working on a feature to allow, for components that are able to support it, to only acknowledging data flowing into source components after that data has been sent by any associated sinks. For example, this would avoid acknowledging messages in Kafka until the data in those messages has been sent via all associated sinks.

  • Mapping and iteration in VRL

    Shortly you will be able to iterate over data in VRL to:

    • allow processing of fields containing arrays of unknown length
    • map over keys and values in objects to transform them

    Kubernetes aggregator role

    We are hard at work at expanding the ability to run Vector as an aggregator in Kubernetes. This will allow you to build end-to-end observability pipelines in Kubernetes with Vector. Distributing processing on the edge, centralizing it with an aggregator, or both. If you are interested in beta testing, please join our chat and let us know.

    This was mentioned in the release notes for 0.11.0, but didn't quite make it for 0.12 as anticipated. We do expect it to be released in the next few months.

The Vector roadmap is publicly available and represents forward looking progress. Vector is open-source and community driven. Your feedback is welcome!

Changelog

  • 14 Enhancements

    • ciAdd new VRL comparison benchmark

    • observabilityAdd `events_in_total` internal metric to sources

    • releasingCorrect the glibc requirements for packages

    • cliSupport VECTOR_LOG_FORMAT and VECTOR_COLOR environment variables

    • remapSupport newlines in if-statement

    • kafka sourceAdd defaults for metadata keys so that they are added by default

    • route transform`route` transform can now process metrics

    • remapintroduce `is_TYPE` helper functions such as `is_string`

    • graphql apiAdditions to GraphQL API to allow tapping the events flowing out of a component

    • graphql apiAdd `json` to `Log` type to expose the log event as JSON

    • kafka sourceKafka message headers are now added as fields on the event

    • remap`parse_regex` now supresses the numeric capture groups by default but they can be re-added via `numeric_groups: true` in the function call

    • splunk_hec sourceSupport reading the channel from the `channel` query parameter rather than just the `X-Splunk-Request-Channel` header

    • remapSupport identifiers with leading numeric characters as fields in paths

  • 10 New Features

    • new sourceNew `datadog_logs` source

    • remapAdd `abort` statement to halt processing

    • remapAdd `parse_query_string` function

    • remapNew `parse_linux_authorization` function

    • remapSupport dropping events when remap fails or is aborted via new `abort` statement

    • remapAdd `parse_csv` function

    • new sinkAdd gcp_stackdriver_metrics sink.

    • observabilityVector `tap` subcommand to view events flowing out of a component

    • remapAdd `parse_nginx_log` function for parsing common nginx log formats

    • remapAdd `to_regex` function to convert strings to regexes

  • 32 Bug Fixes

    • testsUse next_addr instead of fixed addresses for prometheus_exporter tests

    • internal docsUse regular markdown image syntax in reference section

    • ciFix workflow for packaging for Debian

    • testsExplicitly mount docker socket for AWS ECS metadata mock container

    • external docs`--no-topology` has been removed from `vector validate`

    • syslog sourceNo longer hang indefinitely on invalid TCP data

    • internal docsUse regular markdown image syntax in reference section

    • remapAllow newlines in REPL

    • remapAllow non-matching brackets in REPL; would previously hang

    • exceptionsTypo on the expectation for building regexps

    • ciFix release-homebrew workflow dependency

    • releasingFix package naming for latest artifacts

    • testsExport default container tool setting

    • depsUpgrade reqwest in k8s-e2e-tests

    • testsRetry errors in Kafka integration tests

    • host_metrics sourceFix filtering of filesystem metrics by device

    • releasingEnsure /etc/default/vector is not automatically overwritten

    • testsFix formatting of timestamps in tests

    • testsFix aws_ec2_metadata transform test

    • remapPrevent lexer getting stuck on an unterminated literal which would cause VRL to hang

    • ciUpdate homebrew when bootstrapping OSX

    • datadog serviceDocument existence of region parameter

    • socket sourceRemove socket file on source shutdown which would otherwise prevent Vector from restarting

    • clickhouse sinkClickhouse sink now correctly encodes fields containing an array

    • docker_logs sourceCan now read logs from containers that have an attached TTY

    • ciEnsure all benchmark artifacts are included

    • remapPreserve type defs when assigning fields to avoid unncessary type assertions and confusing error messages

    • kubernetes_logs sourceActually use the checkpoint files to avoid re-ingesting logs when Vector is restarted

    • remapDefaults for `case_sensitive` parameter for string matching functions (like `starts_with`) were updated to default to case sensitive matching to match docs

    • compressionFix compression handling for `aws_s3`, `aws_kinesis_firehose`, `splunk_hec`, and `http` sources to handle multi-part files

    • docker_logs sourceEnsure docker labels are not injected as nested objects if they contain a `.`

    • kubernetes_logs sourceRefactor stream and invocation errors to support recoverable error types

Like What You See?