Throttle
Rate limit logs passing through a topology
Configuration
Example configurations
{
"transforms": {
"my_transform_id": {
"type": "throttle",
"inputs": [
"my-source-or-transform-id"
]
}
}
}
[transforms.my_transform_id]
type = "throttle"
inputs = [ "my-source-or-transform-id" ]
transforms:
my_transform_id:
type: throttle
inputs:
- my-source-or-transform-id
{
"transforms": {
"my_transform_id": {
"type": "throttle",
"inputs": [
"my-source-or-transform-id"
],
"key_field": "{{ message }}"
}
}
}
[transforms.my_transform_id]
type = "throttle"
inputs = [ "my-source-or-transform-id" ]
key_field = "{{ message }}"
transforms:
my_transform_id:
type: throttle
inputs:
- my-source-or-transform-id
key_field: "{{ message }}"
exclude
optional conditionAvailable syntaxes
Syntax | Description | Example |
---|---|---|
vrl | A Vector Remap Language (VRL) Boolean expression. | .status_code != 200 && !includes(["info", "debug"], .severity) |
datadog_search | A Datadog Search query string. | *stack |
is_log | Whether the incoming event is a log. |
|
is_metric | Whether the incoming event is a metric. |
|
is_trace | Whether the incoming event is a trace. |
|
Shorthand for VRL
If you opt for the vrl
syntax for this condition, you can set the condition
as a string via the condition
parameter, without needing to specify both a source
and a type
. The
table below shows some examples:
Config format | Example |
---|---|
YAML | condition: .status == 200 |
TOML | condition = ".status == 200" |
JSON | "condition": ".status == 200" |
Condition config examples
Standard VRL
exclude:
type: "vrl"
source: ".status == 500"
exclude = { type = "vrl", source = ".status == 500" }
"exclude": {
"type": "vrl",
"source": ".status == 500"
}
inputs
required [string]A list of upstream source or transform IDs.
Wildcards (*
) are supported.
See configuration for more info.
internal_metrics
optional objectinternal_metrics.emit_events_discarded_per_key
optional boolWhether or not to emit the events_discarded_total
internal metric with the key
tag.
If true, the counter will be incremented for each discarded event, including the key value
associated with the discarded event. If false, the counter will not be emitted. Instead, the
number of discarded events can be seen through the component_discarded_events_total
internal
metric.
Note that this defaults to false because the key
tag has potentially unbounded cardinality.
Only set this to true if you know that the number of unique keys is bounded.
false
key_field
optional string templateThe value to group events into separate buckets to be rate limited independently.
If left unspecified, or if the event doesn’t have key_field
, then the event is not rate
limited separately.
threshold
required uintThe number of events allowed for a given bucket per configured window_secs
.
Each unique key has its own threshold
.
window_secs
required floatthreshold
is applied, in seconds.Outputs
<component_id>
Telemetry
Metrics
linkcomponent_discarded_events_total
counterfilter
transform, or false if due to an error.component_errors_total
countercomponent_received_event_bytes_total
countercomponent_received_events_count
histogramA histogram of the number of events passed in each internal batch in Vector’s internal topology.
Note that this is separate than sink-level batching. It is mostly useful for low level debugging performance issues in Vector due to small internal batches.
component_received_events_total
countercomponent_sent_event_bytes_total
countercomponent_sent_events_total
counterevents_discarded_total
counterutilization
gaugeExamples
Rate limiting
Given this event...[{"log":{"host":"host-1.hostname.com","message":"First message","timestamp":"2020-10-07T12:33:21.223543Z"}},{"log":{"host":"host-1.hostname.com","message":"Second message","timestamp":"2020-10-07T12:33:21.223543Z"}}]
transforms:
my_transform_id:
type: throttle
inputs:
- my-source-or-transform-id
threshold: 1
window_secs: 60
[transforms.my_transform_id]
type = "throttle"
inputs = [ "my-source-or-transform-id" ]
threshold = 1
window_secs = 60
{
"transforms": {
"my_transform_id": {
"type": "throttle",
"inputs": [
"my-source-or-transform-id"
],
"threshold": 1,
"window_secs": 60
}
}
}
[{"log":{"host":"host-1.hostname.com","message":"First message","timestamp":"2020-10-07T12:33:21.223543Z"}}]
How it works
Rate Limiting
throttle
transform will spread load across the configured window_secs
, ensuring that each bucket’s
throughput averages out to the threshold
per window_secs
. It utilizes a Generic Cell Rate Algorithm to
rate limit the event stream.Buckets
throttle
transform buckets events into rate limiters based on the provided key_field
, or a
single bucket if not provided. Each bucket is rate limited separately.Quotas
Rate limiters use “cells” to determine if there is sufficient capacity for an event to successfully pass through a rate limiter. Each event passing through the transform consumes an available cell, if there is no available cell the event will be rate limited.
A rate limiter is created with a maximum number of cells equal to the threshold
, and cells replenish
at a rate of window_secs
divided by threshold
. For example, a window_secs
of 60 with a threshold
of 10
replenishes a cell every 6 seconds and allows a burst of up to 10 events.
Rate Limited Events
threshold
number of events through and drop any further events
for that particular bucket when the rate limiter is at capacity. Any event passed when the rate
limiter is at capacity will be discarded and tracked by an events_discarded_total
metric tagged
by the bucket’s key
.