Aggregate
Aggregate metrics passing through a topology
Configuration
Example configurations
{
"transforms": {
"my_transform_id": {
"type": "aggregate",
"inputs": [
"my-source-or-transform-id"
]
}
}
}
[transforms.my_transform_id]
type = "aggregate"
inputs = [ "my-source-or-transform-id" ]
transforms:
my_transform_id:
type: aggregate
inputs:
- my-source-or-transform-id
{
"transforms": {
"my_transform_id": {
"type": "aggregate",
"inputs": [
"my-source-or-transform-id"
],
"interval_ms": 10000
}
}
}
[transforms.my_transform_id]
type = "aggregate"
inputs = [ "my-source-or-transform-id" ]
interval_ms = 10_000
transforms:
my_transform_id:
type: aggregate
inputs:
- my-source-or-transform-id
interval_ms: 10000
inputs
required [string]A list of upstream source or transform IDs.
Wildcards (*
) are supported.
See configuration for more info.
interval_ms
optional uintThe interval between flushes, in milliseconds.
During this time frame, metrics (beta) with the same series data (name, namespace, tags, and so on) are aggregated.
10000
Outputs
<component_id>
Telemetry
Metrics
linkaggregate_events_recorded_total
counteraggregate_failed_updates
counterincremental
adds, encountered by the aggregate transform.aggregate_flushes_total
countercomponent_discarded_events_total
counterfilter
transform, or false if due to an error.component_errors_total
countercomponent_received_event_bytes_total
countercomponent_received_events_count
histogramA histogram of the number of events passed in each internal batch in Vector’s internal topology.
Note that this is separate than sink-level batching. It is mostly useful for low level debugging performance issues in Vector due to small internal batches.
component_received_events_total
countercomponent_sent_event_bytes_total
countercomponent_sent_events_total
counterutilization
gaugeExamples
Aggregate over 5 seconds
Given this event...[{"metric":{"counter":{"value":1.1},"kind":"incremental","name":"counter.1","tags":{"host":"my.host.com"},"timestamp":"2021-07-12T07:58:44.223543Z"}},{"metric":{"counter":{"value":2.2},"kind":"incremental","name":"counter.1","tags":{"host":"my.host.com"},"timestamp":"2021-07-12T07:58:45.223543Z"}},{"metric":{"counter":{"value":1.1},"kind":"incremental","name":"counter.1","tags":{"host":"different.host.com"},"timestamp":"2021-07-12T07:58:45.223543Z"}},{"metric":{"counter":{"value":22.33},"kind":"absolute","name":"gauge.1","tags":{"host":"my.host.com"},"timestamp":"2021-07-12T07:58:47.223543Z"}},{"metric":{"counter":{"value":44.55},"kind":"absolute","name":"gauge.1","tags":{"host":"my.host.com"},"timestamp":"2021-07-12T07:58:45.223543Z"}}]
transforms:
my_transform_id:
type: aggregate
inputs:
- my-source-or-transform-id
interval_ms: 5000
[transforms.my_transform_id]
type = "aggregate"
inputs = [ "my-source-or-transform-id" ]
interval_ms = 5_000
{
"transforms": {
"my_transform_id": {
"type": "aggregate",
"inputs": [
"my-source-or-transform-id"
],
"interval_ms": 5000
}
}
}
[{"metric":{"counter":{"value":3.3},"kind":"incremental","name":"counter.1","tags":{"host":"my.host.com"},"timestamp":"2021-07-12T07:58:45.223543Z"}},{"metric":{"counter":{"value":1.1},"kind":"incremental","name":"counter.1","tags":{"host":"different.host.com"},"timestamp":"2021-07-12T07:58:45.223543Z"}},{"metric":{"counter":{"value":44.55},"kind":"absolute","name":"gauge.1","tags":{"host":"my.host.com"},"timestamp":"2021-07-12T07:58:45.223543Z"}}]
How it works
Advantages of Use
Aggregation Behavior
incremental
metrics
are “added” and newer absolute
metrics replace older ones in the same series. This results in a reduction
of volume and less granularity, while maintaining numerical correctness. As an example, two
incremental
counter
metrics with values 10 and 13 processed by the transform during a period would be
aggregated into a single incremental
counter
with a value of 23. Two absolute
gauge
metrics with
values 93 and 95 would result in a single absolute
gauge
with the value of 95. More complex
types like distribution
, histogram
, set
, and summary
behave similarly with incremental
values being combined in a manner that makes sense based on their type.