Log to Metric Transform
The Vector log_to_metric
transform
derives one or more metric events from a log event.
Configuration
- Common
- Advanced
- vector.toml
- vector.yaml
- vector.json
[transforms.my_transform_id]# Generaltype = "log_to_metric" # requiredinputs = ["my-source-or-transform-id"] # required# Metrics[[transforms.my_transform_id.metrics]]# Generalfield = "duration" # requiredname = "duration_total" # optional, no defaultnamespace = "service" # optional, no defaulttype = "counter" # required# Tagstags.host = "${HOSTNAME}" # exampletags.region = "us-east-1" # exampletags.status = "{{status}}" # example
- required[table]
metrics
A table of key/value pairs representing the keys to be added to the event.
- requiredstring
field
The log field to use as the metric. See Null Fields for more info.
- View examples
- optionalbool
increment_by_value
If
true
the metric will be incremented by thefield
value. Iffalse
the metric will be incremented by 1 regardless of thefield
value.- Only relevant when: type = "counter"
- Default:
false
- View examples
- optionalstring
name
The name of the metric. Defaults to
<field>_total
forcounter
and<field>
forgauge
.- View examples
- optionalstring
namespace
The namespace of the metric.
- View examples
- optionaltable
tags
Key/value pairs representing metric tags.
- required*
*
Key/value pairs representing metric tags. Environment variables and field interpolation is allowed.
- requiredstring
type
The metric type.
- Enum, must be one of:
"counter"
"gauge"
"histogram"
"set"
"summary"
- View examples
- Enum, must be one of:
Output
This component outputs the following metric events:
- counter
counter
A single value that can only be incremented or reset to zero value, it cannot be decremented. This metric includes the following tags:
*
- Any tags present on the metric.
- distribution
distribution
A distribution represents a distribution of sampled values. It is used with services that support global histograms and summaries. This metric includes the following tags:
*
- Any tags present on the metric.
- gauge
gauge
A gauge represents a point-in-time value that can increase and decrease. Vector's internal gauge type represents changes to that value. Gauges should be used to track fluctuations in values, like current memory or CPU usage. This metric includes the following tags:
*
- Any tags present on the metric.
- gauge
set
A set represents an array of unique values. This metric includes the following tags:
*
- Any tags present on the metric.
Telemetry
This component provides the following metrics that can be retrieved through
the internal_metrics
source. See the
metrics section in the
monitoring page for more info.
- counter
processed_events_total
The total number of events processed by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.file
- The file that produced the errorinstance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_bytes_total
The total number of bytes processed by the component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
Examples
Given the following Vector event:
{"log": {"host": "10.22.11.222","message": "Sent 200 in 54.2ms","status": 200}}
And the following configuration:
[transforms.log_to_metric]type = "log_to_metric"metrics = [{type = "counter", field = "status", name = "response_total", namespace = "service", tags = {status = "{{status}}", host = "{{host}}"}}]
The following Vector metric event will be output:
[{"metric": {"kind": "incremental","name": "response_total","namespace": "service","tags": {"status": "200","host": "10.22.11.222"},"counter": {"value": 1.0}}}]
How It Works
Multiple Metrics
For clarification, when you convert a single log
event into multiple metric
events, the metric
events are not emitted as a single array. They are emitted
individually, and the downstream components treat them as individual events.
Downstream components are not aware they were derived from a single log event.
Null Fields
If the target log field
contains a null
value it will ignored, and a metric
will not be emitted.
Reducing
It's important to understand that this transform does not reduce multiple logs
to a single metric. Instead, this transform converts logs into granular
individual metrics that can then be reduced at the edge. Where the reduction
happens depends on your metrics storage. For example, the
prometheus_exporter
sink will reduce logs in the sink itself
for the next scrape, while other metrics sinks will proceed to forward the
individual metrics for reduction in the metrics storage itself.