Log to Metric Transform
The Vector log_to_metric
transform
derives one or more metric events from a log event.
Configuration
- Common
- Advanced
- vector.toml
- vector.yaml
- vector.json
[transforms.my_transform_id]# Generaltype = "log_to_metric" # requiredinputs = ["my-source-or-transform-id", "prefix-*"] # required# Metrics[[transforms.my_transform_id.metrics]]# Generalfield = "duration" # requiredname = "duration_total" # optional, no defaultnamespace = "service" # optional, no defaulttype = "counter" # required# Tagstags.host = "${HOSTNAME}" # exampletags.region = "us-east-1" # exampletags.status = "{{status}}" # example
- commonrequired[table]
metrics
A table of key/value pairs representing the keys to be added to the event.
- commonrequiredstring
field
The log field to use as the metric. See Null Fields for more info.
- Syntax:
literal
- View examples
- Syntax:
- optionalbool
increment_by_value
If
true
the metric will be incremented by thefield
value. Iffalse
the metric will be incremented by 1 regardless of thefield
value.- Only relevant when: type = "counter"
- Default:
false
- View examples
- commonoptionalstring
name
The name of the metric. Defaults to
<field>_total
forcounter
and<field>
forgauge
.This field supports Vector's template syntax, which enables the use of dynamic per-event values.
- Syntax:
template
- View examples
- Syntax:
- commonoptionalstring
namespace
The namespace of the metric.
This field supports Vector's template syntax, which enables the use of dynamic per-event values.
- Syntax:
template
- View examples
- Syntax:
- commonoptionaltable
tags
Key/value pairs representing metric tags.
- commonrequired*
*
Key/value pairs representing metric tags. Environment variables and field interpolation is allowed.
- enumcommonrequiredstring
type
The metric type.
- Syntax:
literal
- Enum, must be one of:
"counter"
"gauge"
"histogram"
"set"
"summary"
- View examples
- Syntax:
Output
This component outputs the following metric events:
- counter
counter
A single value that can only be incremented or reset to zero value, it cannot be decremented. This metric includes the following tags:
*
- Any tags present on the metric.
- distribution
distribution
A distribution represents a distribution of sampled values. It is used with services that support global histograms and summaries. This metric includes the following tags:
*
- Any tags present on the metric.
- gauge
gauge
A gauge represents a point-in-time value that can increase and decrease. Vector's internal gauge type represents changes to that value. Gauges should be used to track fluctuations in values, like current memory or CPU usage. This metric includes the following tags:
*
- Any tags present on the metric.
- gauge
set
A set represents an array of unique values. This metric includes the following tags:
*
- Any tags present on the metric.
Telemetry
This component provides the following metrics that can be retrieved through
the internal_metrics
source. See the
metrics section in the
monitoring page for more info.
- counter
processing_errors_total
The total number of processing errors encountered by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.error_type
- The type of the errorinstance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
events_in_total
The total number of events accepted by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_events_total
The total number of events processed by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.file
- The file that produced the errorinstance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
events_out_total
The total number of events emitted by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_bytes_total
The total number of bytes processed by the component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
Examples
Given the following Vector log event:
{"host": "10.22.11.222","message": "Sent 200 in 54.2ms","status": 200}
And the following configuration:
[transforms.log_to_metric]type = "log_to_metric"metrics = [{type = "counter", field = "status", name = "response_total", namespace = "service", tags = {status = "{{status}}", host = "{{host}}"}}]
The following Vector metric event will be output:
[{"kind": "incremental","name": "response_total","namespace": "service","tags": {"status": "200","host": "10.22.11.222"},"counter": {"value": 1.0}}]
How It Works
Multiple Metrics
For clarification, when you convert a single log
event into multiple metric
events, the metric
events are not emitted as a single array. They are emitted
individually, and the downstream components treat them as individual events.
Downstream components are not aware they were derived from a single log event.
Null Fields
If the target log field
contains a null
value it will ignored, and a metric
will not be emitted.
Reducing
It's important to understand that this transform does not reduce multiple logs
to a single metric. Instead, this transform converts logs into granular
individual metrics that can then be reduced at the edge. Where the reduction
happens depends on your metrics storage. For example, the
prometheus_exporter
sink will reduce logs in the sink itself
for the next scrape, while other metrics sinks will proceed to forward the
individual metrics for reduction in the metrics storage itself.
State
This component is stateless, meaning its behavior is consistent across each input.