0.19 Upgrade Guide

Vector’s 0.19.0 release includes breaking changes:

1. Removal of deprecated configuration fields for the Splunk HEC Logs sink: host
2. Updated internal metrics for the Splunk HEC sinks
3. Removal of deprecated configuration fields for the Elasticsearch sink
4. Removal of default for version field for Vector source and sink

And deprecations:

1. Splunk HEC sinks rename token to default_token

And behavioral changes:

1. Splunk HEC sinks now include a channel header in requests

We cover them below to help you upgrade quickly:

Upgrade guide

Breaking changes

Removal of deprecated configuration fields for the Splunk HEC Logs sink: host

We’ve removed a long deprecated configuration field from the Splunk HEC Logs sink: host.

You can migrate your configuration by switching to endpoint instead.

 [sinks.splunk]
   type = "splunk_hec_logs"
-  host = "http://splunk-endpoint"
+  endpoint = "http://splunk-endpoint"
   ...

Updated internal metrics for the Splunk HEC sinks

As part of moving towards more consistent Vector component instrumentation, we’ve updated the following internal metrics in the Splunk HEC sinks. For any removed metric, we’ve added an equivalent alternative.

• Removed encode_errors_total
  • Instead, use component_errors_total with the tag error_type = encode_failed.
• Removed processing_errors_total with tag error_type = invalid_metric_kind
  • Instead, use component_errors_total with the tag error_type = invalid_metric.
• Removed processed_bytes_total
  • Instead, use component_received_event_bytes_total and component_sent_event_bytes_total.
• Added components_discarded_events_total
  • Previously, no metric was emitted when an encoding error occurred and an event was dropped.

Removal of deprecated configuration fields for the Elasticsearch sink

Removal of default for version field for Vector source and sink

In the v0.16.0 release, we have introduced a new v2 version of the protocol that the vector source and sink use to communicate. This new protocol offers a number of advantages over our initial v1 protocol implementation including load balancing and better back-pressure handling.

Up until this release, the vector source and sink continued defaulting to the v1 protocol for compatibility. In this release, we remove this default and require you to specify the version field in the vector source and sink configuration. The intent of this change is to prompt users to migrate.

If you do not specify a version number, you will see an error like:

Configuration error. error=data did not match any variant of untagged enum VectorConfig for key `sinks.my_vector_sink` at line 8 column 1

To continue using the v1 protocol, simply add version = "1" to your configuration like:

[sinks.vector]
  type = "vector"
+ version = "1"

[sources.vector]
  type = "vector"
+ version = "1"

However, we recommend that you transition to the v2 protocol by setting version = "2":

[sinks.vector]
  type = "vector"
+ version = "2"

[sources.vector]
  type = "vector"
+ version = "2"

There are some configuration options that are no longer valid with version 2 of the source and sink:

• vector source: keepalive and receive_buffer_bytes
• vector sink: keepalive and send_buffer_bytes

As these were specific to the TCP-based version 1 of the protocol.

See the announcement post for a guide on how to transition without downtime.

In a subsequent release we will begin defaulting to the v2 protocol and eventually remove support for the v1 protocol.

Deprecations

Splunk HEC sinks rename token to default_token

The token configuration option on the splunk_hec_logs and splunk_hec_metrics sink has been deprecated in-lieu of the new default_token option which was added as part of support for Splunk HEC token pass-through.

You can migrate your configuration by renaming token to default_token:

 [sinks.splunk]
   type = "splunk_hec_logs"
   endpoint = "http://splunk-endpoint"
-  token = "MY-TOKEN"
+  default_token = "MY-TOKEN"
   ...

Behavioral Changes

Splunk HEC sinks now include a channel header in requests

As part of our work to support Splunk HEC indexer acknowledgements, splunk_hec sinks will generate a random UUID to use as the X-Splunk-Request-Channel header value. Splunk accepts this header regardless of whether indexer acknowledgements is enabled, so the splunk_hec sinks always attach the header. In other words, disabling indexer acknowledgements for the sink will not disable this header value. You can read more about channels in the official Splunk docs.

If your setup involves a splunk_hec sink sending into a separate splunk_hec source, you will now see a splunk_channel field in events output from the splunk_hec source. This is because splunk_hec sources have always parsed and included channel information from requests in their events. If you want to remove this field, you can simply drop it with a remap transform like follows:

[transforms.foo]
type = "remap"
inputs = ["bar"] # where bar is the splunk_hec source
source = '''
  del(.splunk_channel)
'''