Vector v0.20.0 release notes

A multiple_outputs option was added to the datadog_agent source. When set to true, rather than emitting both logs and metrics to any components using the component id of the source in inputs, the events will be split into two separate streams, <component_id>.logs and <component_id>.metrics, so that logs and metrics from the Datadog agent can be processed separately.

When set to true, the internal metrics from this source are also updated as mentioned in the highlight article for named output metrics.

multiple_outputs defaults to false for compatibility with existing configurations.

The telemetry for a few components was updated as a result expanding the use of named outputs in Vector for components with multiple output streams:

• route
• datadog_agent (when multiple_outputs is true)
• remap (when reroute_dropped is true)

These components now add an output tag to their metrics. See highlight article for named output metrics.

The abort function can now take an optional string message to include in logs and the metadata for dropped events. For example:

if .foo == 5 {
	abort "foo is " + .foo + "!"
}

The foo is 5! message will appear in Vector’s internal logs as well as being used as metadata.message for rerouted dropped events (if reroute_dropped is true).

We are in the process of updating all Vector components with consistent instrumentation as described in Vector’s component specification).

With this release we have instrumented the following sources with these new metrics:

• aws_ecs_metrics
• aws_kinesis_firehose
• aws_s3
• datadog_agent
• demo_logs
• dnstap
• docker_logs
• eventstoredb_metrics
• exec
• host_metrics
• internal_logs
• internal_metrics
• kafka
• kubernetes_logs
• nats
• nginx_metrics
• prometheus_scrape
• stdin
• vector

And these transforms:

• aws_ec2_metadata

The internal telemetry for the loki sink has been improved by:

• The addition of component_discarded_events_total metric for discarded events
• The addition of the rewritten_timestamp_events_total metric to count events whose timestamp was rewritten
• The log messages for out-of-order events were dropped in severity from warn to debug since out-of-order events are a common occurrence, something the sink is designed to handle, and the handling behavior is explicitly configured by the user

See the upgrade guide for more details.

The vector tap command has had two enhancements:

• It can now tap metric events in addition to log events
• It has a new logfmt option that can be used to format the tapped output as logfmt data

A new new_relic sink was added that can handle both logs and metrics. It can also send logs to New Relic as New Relic events. It replaces the existing new_relic_logs sink that has been deprecated.

Thanks to @Andreu for this contribution!

Fixed runaway memory growth when the prometheus_exporter sink was used with distributions due to the sink holding onto all of the samples it had seen.

This required a breaking change as documented in the upgrade guide.

A few fixes were made to parse_groks:

• The array filter now supports arrays without brackets, if empty string is provided as brackets. For example: `%{data:field:array("", “-")} will parse “a-b” into [“a”, “b”].

• The keyvalue filter now supports nested paths as keys. For example:

    parse_groks("db.name=my_db,db.operation=insert",
      patterns: ["%{data::keyvalue}"]
    )
  

  will yield

    {
        "db" : {
            "name" : "my_db",
            "operation" : "insert",
        }
    }
  

• The date filter now supports the d and y shorthands. For example:

    parse_groks("Nov 16 2020 13:41:29 GMT", patterns: ["%{date("MMM d y HH:mm:ss z"):field}"] )
  

  Will yield:

  1605534089000
  

• Numeric values are now returned as integers when there is no loss in precision; otherwise they continue to be returned as floats.

• Aliases that match multiple fields now correctly extract as an array of values. extracted values.

• Aliases with a filter now correctly extract.

• The keyvalue filter now correctly ignores keys without values.

• To match the beginning and the end of lines, \A and \Z must be used rather than ^ and $`

• The keyvalue filter now correctly handles parsing keys that start with a number