Log to Metric Transform

The Vector log_to_metric transform converts logs

Configuration

[transforms.my_transform_id]
# General
type = "log_to_metric" # required
inputs = ["my-source-or-transform-id"] # required
# Metrics
[[transforms.my_transform_id.metrics]]
# General
field = "duration" # required
name = "duration_total" # optional, no default
namespace = "service" # optional, no default
type = "counter" # required
# Tags
tags.host = "${HOSTNAME}" # example
tags.region = "us-east-1" # example
tags.status = "{{status}}" # example
  • commonrequired[table]

    metrics

    A table of key/value pairs representing the keys to be added to the event.

    • commonrequiredstring

      field

      The log field to use as the metric. See Null Fields for more info.

      • View examples
    • optionalbool

      increment_by_value

      If true the metric will be incremented by the field value. If false the metric will be incremented by 1 regardless of the field value.

      • Only relevant when: type = "counter"
      • Default: false
      • View examples
    • templateablecommonoptionalstring

      name

      The name of the metric. Defaults to <field>_total for counter and <field> for gauge.

      • View examples
    • templateablecommonoptionalstring

      namespace

      The namespace of the metric.

      • View examples
    • commonoptionaltable

      tags

      Key/value pairs representing metric tags.

      • commonrequired*
        *

        Key/value pairs representing metric tags. Environment variables and field interpolation is allowed.

      • enumcommonrequiredstring

        type

        The metric type.

        • Enum, must be one of: "counter" "gauge" "histogram" "set" "summary"
        • View examples

      Output

      This component outputs the following metric events:

      • counter

        counter

        A single value that can only be incremented or reset to zero value, it cannot be decremented. This metric includes the following tags:

        • * - Any tags present on the metric.

      • distribution

        distribution

        A distribution represents a distribution of sampled values. It is used with services that support global histograms and summaries. This metric includes the following tags:

        • * - Any tags present on the metric.

      • gauge

        gauge

        A gauge represents a point-in-time value that can increase and decrease. Vector's internal gauge type represents changes to that value. Gauges should be used to track fluctuations in values, like current memory or CPU usage. This metric includes the following tags:

        • * - Any tags present on the metric.

      • gauge

        set

        A set represents an array of unique values. This metric includes the following tags:

        • * - Any tags present on the metric.

      Telemetry

      This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

      • counter

        processed_events_total

        The total number of events processed by this component. This metric includes the following tags:

        • component_kind - The Vector component kind.

        • component_name - The Vector component ID.

        • component_type - The Vector component type.

        • file - The file that produced the error

        • instance - The Vector instance identified by host and port.

        • job - The name of the job producing Vector metrics.

      • counter

        processed_bytes_total

        The total number of bytes processed by the component. This metric includes the following tags:

        • component_kind - The Vector component kind.

        • component_name - The Vector component ID.

        • component_type - The Vector component type.

        • instance - The Vector instance identified by host and port.

        • job - The name of the job producing Vector metrics.

      Examples

      Counter

      Given the following Vector event:

      {
      "log": {
      "host": "10.22.11.222",
      "message": "Sent 200 in 54.2ms",
      "status": 200
      }
      }

      And the following configuration:

      [transforms.log_to_metric]
      type = "log_to_metric"
      metrics = [
      {type = "counter", field = "status", name = "response_total", namespace = "service", tags = {status = "{{status}}", host = "{{host}}"}}
      ]

      The following Vector metric event will be output:

      [
      {
      "metric": {
      "kind": "incremental",
      "name": "response_total",
      "namespace": "service",
      "tags": {
      "status": "200",
      "host": "10.22.11.222"
      },
      "counter": {
      "value": 1.0
      }
      }
      }
      ]

      How It Works

      Multiple Metrics

      For clarification, when you convert a single log event into multiple metric events, the metric events are not emitted as a single array. They are emitted individually, and the downstream components treat them as individual events. Downstream components are not aware they were derived from a single log event.

      Null Fields

      If the target log field contains a null value it will ignored, and a metric will not be emitted.

      Reducing

      It's important to understand that this transform does not reduce multiple logs to a single metric. Instead, this transform converts logs into granular individual metrics that can then be reduced at the edge. Where the reduction happens depends on your metrics storage. For example, the prometheus_exporter sink will reduce logs in the sink itself for the next scrape, while other metrics sinks will proceed to forward the individual metrics for reduction in the metrics storage itself.