0.16 Upgrade Guide

Vector’s 0.16.0 release includes three breaking changes:

Component name field renamed to ID
Datadog Log sink encoding option removed
Renaming of memory_use_bytes internal metric
datadog_logs source renamed to datadog_agent
kubernetes_logs source’s new RBAC

We cover them below to help you upgrade quickly:

Upgrade guide

Component name field renamed to ID

Historically we’ve referred to the component ID field as name in some places, id in others. We’ve decided to standardize on ID as we feel this is closer to the intention of the field: an unchanging identifier for components.

For example, with the component config:

[transforms.parse_nginx]
type = "remap"
inputs = []
source = ""

The parse_nginx part of the config is now only referred to as ID in the documentation.

We have preserved compatibility with existing usages of component_name for the internal_metrics sources by keeping component_name and adding component_id as a new tag. However, we recommend switching usages over to component_id as we will be removing component_name in the future: if you were grouping by this tag in your metrics queries, or referring to it in a remap or lua transform, you should update it to refer to component_id.

Within the GraphQL API, all references to name for Components has been updated to be componentId. This is used over simply Id as Id has special semantics within the GraphQL ecosystem and we may add support for this field later.

Datadog Log sink encoding option removed

In previous versions of vector it was possible to configure the Datadog logs sink to send in ‘text’ or ‘json’ encoding. While the logs ingest API does accept text format the native format for that API is json. Sending text comes with limitations and is only useful for backward compatibility with older clients.

We no longer allow you to set the encoding of the payloads in the Datadog logs sink. For instance, if your configuration looks like so:

[sinks.dd_logs_egress]
type = "datadog_logs"
inputs = ["datadog_agent"]
encoding.codec = "json"

You should remove encoding.codec entirely, leaving you with:

[sinks.dd_logs_egress]
type = "datadog_logs"
inputs = ["datadog_agent"]

Encoding fields other than codec are still valid.

Renaming of `memory_use_bytes` internal metric

Vector previously documented the internal_metrics memory_use_bytes metric as being “The total memory currently being used by Vector (in bytes)."; however, this metric was actually published by the lua transform and indicated the memory use of just the Lua runtime.

To make this more clear, the metric has been renamed from memory_use_bytes to lua_memory_use_bytes. If you were previously using memory_use_bytes as a measure of the lua runtime memory usage, you should update to refer to lua_memory_use_bytes.

The documentation for this metric has also been updated.

`datadog_logs` source renamed to `datadog_agent`

With the release of Vector 0.16.0, we’ve renamed the datadog_logs source to datadog_agent.

The naming of the datadog_logs source was somewhat ambiguous as it could be construed to indicate it is compatible with the datadog_logs sink and that it mimics the Datadog Logs API. However, the intention of this source is to collect data specifically from running Datadog Agents and this release contains some more baked in assumptions that the data is specifically coming from agent.

For now, this source only collects logs forwarded by the agent, but in the future it will be expanded to collect metrics and traces.

We decided to make this a breaking change, instead of aliasing datadog_logs, as the released changes are not backwards compatible and the name change reflects this.

It is possible that we will re-add a datadog_logs source in the future that mimics the Datadog API for use with other Datadog clients aside from the agent. Let us know if this would be useful to you!

`kubernetes_logs` source’s new RBAC

The kubernetes_logs source will now enrich events with labels from the Namespace they originate from. This enhancement requires access to an additional resource in Kubernetes. Our Kubernetes manifests and Helm chart have been updated to create a ClusterRole granting access to the namespaces resource.

Upgrade Guide

Rename a datadog_logs source components in your configuration to datadog_agent:

[sources.datadog]
-type = "datadog_logs"
+type = "datadog_agent"
address = "0.0.0.0:8080"
store_api_key = true

Updating to 0.16.0 requires that you update to the equivalent chart or Kubernetes manifests. If you don’t use either of our provided installation methods, you should update your ClusterRole as such:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vector-agent
rules:
  - apiGroups:
      - ""
    resources:
+     - namespaces
      - pods
    verbs:
      - watch