Log to Metric Transform

The Vector log_to_metric transform derives one or more metric events from a log event.

Configuration

Common
Advanced

vector.toml
vector.yaml
vector.json

[transforms.my_transform_id]
  # General
  type = "log_to_metric" # required
  inputs = ["my-source-or-transform-id", "prefix-*"] # required

  # Metrics
  [[transforms.my_transform_id.metrics]]
    # General
    field = "duration" # required
    name = "duration_total" # optional, no default
    namespace = "service" # optional, no default
    type = "counter" # required

    # Tags
    tags.host = "${HOSTNAME}" # example
    tags.region = "us-east-1" # example
    tags.status = "{{status}}" # example

commonrequired[table]
metrics
A table of key/value pairs representing the keys to be added to the event.
- commonrequiredstring
  field
  The log field to use as the metric. See Null Fields for more info.
  - Syntax: literal
  - View examples
- optionalbool
  increment_by_value
  If true the metric will be incremented by the field value. If false the metric will be incremented by 1 regardless of the field value.
  - Only relevant when: type = "counter"
  - Default: false
  - View examples
- commonoptionalstring
  name
  The name of the metric. Defaults to <field>_total for counter and <field> for gauge.
  This field supports Vector's template syntax, which enables the use of dynamic per-event values.
  - Syntax: template
  - View examples
- commonoptionalstring
  namespace
  The namespace of the metric.
  This field supports Vector's template syntax, which enables the use of dynamic per-event values.
  - Syntax: template
  - View examples
- commonoptionaltable
  tags
  Key/value pairs representing metric tags.
  - commonrequired*
    *
    Key/value pairs representing metric tags. Environment variables and field interpolation is allowed.
- enumcommonrequiredstring
  type
  The metric type.
  - Syntax: literal
  - Enum, must be one of: "counter" "gauge" "histogram" "set" "summary"
  - View examples

Output

This component outputs the following metric events:

counter
counter
A single value that can only be incremented or reset to zero value, it cannot be decremented. This metric includes the following tags:
- * - Any tags present on the metric.
distribution
distribution
A distribution represents a distribution of sampled values. It is used with services that support global histograms and summaries. This metric includes the following tags:
- * - Any tags present on the metric.
gauge
gauge
A gauge represents a point-in-time value that can increase and decrease. Vector's internal gauge type represents changes to that value. Gauges should be used to track fluctuations in values, like current memory or CPU usage. This metric includes the following tags:
- * - Any tags present on the metric.
gauge
set
A set represents an array of unique values. This metric includes the following tags:
- * - Any tags present on the metric.

Telemetry

This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

counter
processing_errors_total
The total number of processing errors encountered by this component. This metric includes the following tags:
- component_kind - The Vector component kind.
- component_name - The Vector component ID.
- component_type - The Vector component type.
- error_type - The type of the error
- instance - The Vector instance identified by host and port.
- job - The name of the job producing Vector metrics.
counter
events_in_total
The number of events accepted by this component either from tagged origin like file and uri, or cumulatively from other origins. This metric includes the following tags:
- component_kind - The Vector component kind.
- component_name - The Vector component ID.
- component_type - The Vector component type.
- container_name - The name of the container from which the event originates.
- file - The file from which the event originates.
- instance - The Vector instance identified by host and port.
- job - The name of the job producing Vector metrics.
- mode - The connection mode used by the component.
- peer_addr - The IP from which the event originates.
- peer_path - The pathname from which the event originates.
- pod_name - The name of the pod from which the event originates.
- uri - The sanitized uri from which the event originates.
counter
processed_events_total
The total number of events processed by this component. This metric includes the following tags:
- component_kind - The Vector component kind.
- component_name - The Vector component ID.
- component_type - The Vector component type.
- file - The file that produced the error
- instance - The Vector instance identified by host and port.
- job - The name of the job producing Vector metrics.
counter
events_out_total
The total number of events emitted by this component. This metric includes the following tags:
- component_kind - The Vector component kind.
- component_name - The Vector component ID.
- component_type - The Vector component type.
- instance - The Vector instance identified by host and port.
- job - The name of the job producing Vector metrics.
counter
processed_bytes_total
The number of bytes processed by the component. This metric includes the following tags:
- component_kind - The Vector component kind.
- component_name - The Vector component ID.
- component_type - The Vector component type.
- container_name - The name of the container from which the bytes originate.
- file - The file from which the bytes originate.
- instance - The Vector instance identified by host and port.
- job - The name of the job producing Vector metrics.
- mode - The connection mode used by the component.
- peer_addr - The IP from which the bytes originate.
- peer_path - The pathname from which the bytes originate.
- pod_name - The name of the pod from which the bytes originate.
- uri - The sanitized uri from which the bytes originate.

Examples

Counter

Given the following Vector log event:

{
  "host": "10.22.11.222",
  "message": "Sent 200 in 54.2ms",
  "status": 200
}

And the following configuration:

vector.toml

[transforms.log_to_metric]
type = "log_to_metric"
metrics = [
{type = "counter", field = "status", name = "response_total", namespace = "service", tags = {status = "{{status}}", host = "{{host}}"}}
]

The following Vector metric event will be output:

[
  {
    "kind": "incremental",
    "name": "response_total",
    "namespace": "service",
    "tags": {
      "status": "200",
      "host": "10.22.11.222"
    },
    "counter": {
      "value": 1.0
    }
  }
]

How It Works

Multiple Metrics

For clarification, when you convert a single log event into multiple metric events, the metric events are not emitted as a single array. They are emitted individually, and the downstream components treat them as individual events. Downstream components are not aware they were derived from a single log event.

Null Fields

If the target log field contains a null value it will ignored, and a metric will not be emitted.

Reducing

It's important to understand that this transform does not reduce multiple logs to a single metric. Instead, this transform converts logs into granular individual metrics that can then be reduced at the edge. Where the reduction happens depends on your metrics storage. For example, the prometheus_exporter sink will reduce logs in the sink itself for the next scrape, while other metrics sinks will proceed to forward the individual metrics for reduction in the metrics storage itself.

State

This component is stateless, meaning its behavior is consistent across each input.

#Configuration

#metrics

#field

#increment_by_value

#name

#namespace

#tags

*

#type

#Output

#counter

#distribution

#gauge

#set

#Telemetry

#processing_errors_total

#events_in_total

#processed_events_total

#events_out_total

#processed_bytes_total

#Examples

#How It Works

#Multiple Metrics

#Null Fields

#Reducing

#State

Configuration

metrics

field

increment_by_value

name

namespace

tags

type

Output

counter

distribution

gauge

set

Telemetry

processing_errors_total

events_in_total

processed_events_total

events_out_total