Prometheus Exporter
Output metric events to a Prometheus exporter running on the host
status: stable
delivery: best effort
acknowledgements: no
egress: expose
state: stateful
previously known as: prometheus
Alias
This component was previously called the prometheus
sink. Make sure to update your
Vector configuration to accommodate the name change:
[sinks.my_prometheus_exporter_sink]
+type = "prometheus_exporter"
-type = "prometheus"
Warnings
High cardinality metric names and labels are discouraged by Prometheus as they can provide performance
and reliability problems. You should consider alternative strategies to reduce the cardinality. Vector
offers a
tag_cardinality_limit
transform as a way
to protect against this.Configuration
Example configurations
{
"sinks": {
"my_sink_id": {
"type": "prometheus_exporter",
"inputs": [
"my-source-or-transform-id"
],
"address": "0.0.0.0:9598",
"default_namespace": "service"
}
}
}
[sinks.my_sink_id]
type = "prometheus_exporter"
inputs = [ "my-source-or-transform-id" ]
address = "0.0.0.0:9598"
default_namespace = "service"
---
sinks:
my_sink_id:
type: prometheus_exporter
inputs:
- my-source-or-transform-id
address: 0.0.0.0:9598
default_namespace: service
{
"sinks": {
"my_sink_id": {
"type": "prometheus_exporter",
"inputs": [
"my-source-or-transform-id"
],
"address": "0.0.0.0:9598",
"buckets": [
0.005
],
"flush_period_secs": 60,
"default_namespace": "service",
"quantiles": [
0.5
],
"buffer": null,
"tls": null,
"distributions_as_summaries": null
}
}
}
[sinks.my_sink_id]
type = "prometheus_exporter"
inputs = [ "my-source-or-transform-id" ]
address = "0.0.0.0:9598"
buckets = [ 0.005 ]
flush_period_secs = 60
default_namespace = "service"
quantiles = [ 0.5 ]
---
sinks:
my_sink_id:
type: prometheus_exporter
inputs:
- my-source-or-transform-id
address: 0.0.0.0:9598
buckets:
- 0.005
flush_period_secs: 60
default_namespace: service
quantiles:
- 0.5
buffer: null
tls: null
distributions_as_summaries: null
address
required string literalThe address to expose for scraping. The metrics are exposed at the typical Prometheus exporter path,
/metrics
buckets
optional [float]Default buckets to use for aggregating distribution
metrics into histograms.
default:
[0.005 0.01 0.025 0.05 0.1 0.25 0.5 1 2.5 5 10]
buffer
optional objectConfigures the sink specific buffer behavior.
buffer.max_events
common optional uintThe maximum number of events allowed in the buffer.
Relevant when:
type = "memory"
default:
500
(events)buffer.max_size
required uintThe maximum size of the buffer on the disk. Must be at least 128 megabytes (134217728 bytes).
Note that during normal disk buffer operation, the disk buffer can create one additional 128 megabyte block so the minimum disk space required is actually 256 megabytes.
Relevant when:
type = "disk"
buffer.type
common optional string literal enumThe buffer’s type and storage mechanism.
Enum options
Option | Description |
---|---|
disk | Stores the sink’s buffer on disk. This is less performant, but durable. Data will not be lost between restarts. Will also hold data in memory to enhance performance. WARNING: This may stall the sink if disk performance isn’t on par with the throughput. For comparison, AWS gp2 volumes are usually too slow for common cases. |
memory | Stores the sink’s buffer in memory. This is more performant, but less durable. Data will be lost if Vector is restarted forcefully. |
default:
memory
buffer.when_full
optional string literal enumThe behavior when the buffer becomes full.
Enum options
Option | Description |
---|---|
block | Applies back pressure when the buffer is full. This prevents data loss, but will cause data to pile up on the edge. |
drop_newest | Drops new data as it’s received. This data is lost. This should be used when performance is the highest priority. |
default:
block
default_namespace
common optional string literalUsed as a namespace for metrics that don’t have it. Typically
namespaces are set during ingestion (sources), but it is
optional and when missing, we’ll use this value. It should
follow Prometheus naming conventions.
distributions_as_summaries
optional boolWhether or not to render distributions as a
histogram or
summary, which map one-to-one with the Prometheus
metric types of the same name.
default:
false
flush_period_secs
optional uintTime interval on which metrics are flushed. On the flush interval, if a metric has not been seen since the last flush interval, it is considered expired and is removed. Be sure to configure this value higher than your client’s scrape interval.
default:
60
(seconds)inputs
required [string]A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
quantiles
optional [float]Quantiles to use for aggregating distribution metrics
into a summary.
default:
[0.5 0.75 0.9 0.95 0.99]
tls
optional objectConfigures the TLS options for incoming connections.
tls.ca_file
optional string literalAbsolute path to an additional CA certificate file, in DER or PEM format (X.509), or an in-line CA certificate in PEM format.
tls.crt_file
optional string literalAbsolute path to a certificate file used to identify this server, in DER or PEM format (X.509) or PKCS#12, or an in-line certificate in PEM format. If this is set, and is not a PKCS#12 archive,
key_file
must also be set. This is required if enabled
is set to true
.tls.enabled
optional boolRequire TLS for incoming connections. If this is set, an identity certificate is also required.
default:
false
tls.key_file
optional string literalAbsolute path to a private key file used to identify this server, in DER or PEM format (PKCS#8), or an in-line private key in PEM format.
tls.key_pass
optional string literalPass phrase used to unlock the encrypted key file. This has no effect unless
key_file
is set.tls.verify_certificate
optional boolIf
true
, Vector will require a TLS certificate from the connecting host and terminate the connection if the certificate is not valid. If false
(the default), Vector will not request a certificate from the client.default:
false
Telemetry
Metrics
linkbuffer_byte_size
gaugeThe number of bytes current in the buffer.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
buffer_discarded_events_total
counterThe number of events dropped by this non-blocking buffer.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
buffer_events
gaugeThe number of events currently in the buffer.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
buffer_received_event_bytes_total
counterThe number of bytes received by this buffer.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
buffer_received_events_total
counterThe number of events received by this buffer.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
buffer_sent_event_bytes_total
counterThe number of bytes sent by this buffer.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
buffer_sent_events_total
counterThe number of events sent by this buffer.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
component_received_event_bytes_total
counterThe number of event bytes accepted by this component either from
tagged origins like file and uri, or cumulatively from other origins.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
container_name
optional
The name of the container from which the data originated.
file
optional
The file from which the data originated.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the data originated.
peer_path
optional
The pathname from which the data originated.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the data originated.
uri
optional
The sanitized URI from which the data originated.
component_received_events_count
histogramA histogram of Vector the number of events passed in each internal batch in Vector’s internal topology.
Note that this is separate than sink-level batching. It is mostly useful for low level debugging
performance issues in Vector due to small internal batches.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
container_name
optional
The name of the container from which the data originated.
file
optional
The file from which the data originated.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the data originated.
peer_path
optional
The pathname from which the data originated.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the data originated.
uri
optional
The sanitized URI from which the data originated.
component_received_events_total
counterThe number of events accepted by this component either from tagged
origins like file and uri, or cumulatively from other origins.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
container_name
optional
The name of the container from which the data originated.
file
optional
The file from which the data originated.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the data originated.
peer_path
optional
The pathname from which the data originated.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the data originated.
uri
optional
The sanitized URI from which the data originated.
events_in_total
counterThe number of events accepted by this component either from tagged
origins like file and uri, or cumulatively from other origins.
This metric is deprecated and will be removed in a future version.
Use
component_received_events_total
instead.component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
container_name
optional
The name of the container from which the data originated.
file
optional
The file from which the data originated.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the data originated.
peer_path
optional
The pathname from which the data originated.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the data originated.
uri
optional
The sanitized URI from which the data originated.
utilization
gaugeA ratio from 0 to 1 of the load on a component. A value of 0 would indicate a completely idle component that is simply waiting for input. A value of 1 would indicate a that is never idle. This value is updated every 5 seconds.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
Examples
Counter
Given this event...{
"metric": {
"counter": {
"value": 1.5
},
"kind": "incremental",
"name": "logins",
"tags": {
"host": "my-host.local"
}
}
}
[sinks.my_sink_id]
type = "prometheus_exporter"
inputs = [ "my-source-or-transform-id" ]
default_namespace = "service"
---
sinks:
my_sink_id:
type: prometheus_exporter
inputs:
- my-source-or-transform-id
default_namespace: service
{
"sinks": {
"my_sink_id": {
"type": "prometheus_exporter",
"inputs": [
"my-source-or-transform-id"
],
"default_namespace": "service"
}
}
}
# HELP service_logins logins
# TYPE service_logins counter
service_logins{host="my-host.local"} 1.5
Gauge
Given this event...{
"metric": {
"gauge": {
"value": 1.5
},
"kind": "absolute",
"name": "memory_rss",
"namespace": "app",
"tags": {
"host": "my-host.local"
}
}
}
[sinks.my_sink_id]
type = "prometheus_exporter"
inputs = [ "my-source-or-transform-id" ]
---
sinks:
my_sink_id:
type: prometheus_exporter
inputs:
- my-source-or-transform-id
{
"sinks": {
"my_sink_id": {
"type": "prometheus_exporter",
"inputs": [
"my-source-or-transform-id"
]
}
}
}
# HELP app_memory_rss memory_rss
# TYPE app_memory_rss gauge
app_memory_rss{host="my-host.local"} 1.5
Histogram
Given this event...{
"metric": {
"histogram": {
"buckets": [
{
"count": 0,
"upper_limit": 0.005
},
{
"count": 1,
"upper_limit": 0.01
},
{
"count": 0,
"upper_limit": 0.025
},
{
"count": 1,
"upper_limit": 0.05
},
{
"count": 0,
"upper_limit": 0.1
},
{
"count": 0,
"upper_limit": 0.25
},
{
"count": 0,
"upper_limit": 0.5
},
{
"count": 0,
"upper_limit": 1
},
{
"count": 0,
"upper_limit": 2.5
},
{
"count": 0,
"upper_limit": 5
},
{
"count": 0,
"upper_limit": 10
}
],
"count": 2,
"sum": 0.789
},
"kind": "absolute",
"name": "response_time_s",
"tags": {}
}
}
[sinks.my_sink_id]
type = "prometheus_exporter"
inputs = [ "my-source-or-transform-id" ]
---
sinks:
my_sink_id:
type: prometheus_exporter
inputs:
- my-source-or-transform-id
{
"sinks": {
"my_sink_id": {
"type": "prometheus_exporter",
"inputs": [
"my-source-or-transform-id"
]
}
}
}
# HELP response_time_s response_time_s
# TYPE response_time_s histogram
response_time_s_bucket{le="0.005"} 0
response_time_s_bucket{le="0.01"} 1
response_time_s_bucket{le="0.025"} 0
response_time_s_bucket{le="0.05"} 1
response_time_s_bucket{le="0.1"} 0
response_time_s_bucket{le="0.25"} 0
response_time_s_bucket{le="0.5"} 0
response_time_s_bucket{le="1.0"} 0
response_time_s_bucket{le="2.5"} 0
response_time_s_bucket{le="5.0"} 0
response_time_s_bucket{le="10.0"} 0
response_time_s_bucket{le="+Inf"} 0
response_time_s_sum 0.789
response_time_s_count 2
Distribution to histogram
Given this event...{
"metric": {
"distribution": {
"samples": [
{
"rate": 4,
"value": 0
},
{
"rate": 2,
"value": 1
},
{
"rate": 1,
"value": 4
}
],
"statistic": "histogram"
},
"kind": "incremental",
"name": "request_retries",
"tags": {
"host": "my-host.local"
}
}
}
[sinks.my_sink_id]
type = "prometheus_exporter"
inputs = [ "my-source-or-transform-id" ]
buckets = [ 0, 1, 3 ]
---
sinks:
my_sink_id:
type: prometheus_exporter
inputs:
- my-source-or-transform-id
buckets:
- 0
- 1
- 3
{
"sinks": {
"my_sink_id": {
"type": "prometheus_exporter",
"inputs": [
"my-source-or-transform-id"
],
"buckets": [
0,
1,
3
]
}
}
}
# HELP request_retries request_retries
# TYPE request_retries histogram
request_retries_bucket{host="my-host.local",le="0"} 4
request_retries_bucket{host="my-host.local",le="1"} 6
request_retries_bucket{host="my-host.local",le="3"} 6
request_retries_bucket{host="my-host.local",le="+Inf"} 7
request_retries_sum{host="my-host.local"} 6
request_retries_count{host="my-host.local"} 7
Distribution to summary
Given this event...{
"metric": {
"distribution": {
"samples": [
{
"rate": 3,
"value": 0
},
{
"rate": 2,
"value": 1
},
{
"rate": 1,
"value": 4
}
],
"statistic": "summary"
},
"kind": "incremental",
"name": "request_retries",
"tags": {}
}
}
[sinks.my_sink_id]
type = "prometheus_exporter"
inputs = [ "my-source-or-transform-id" ]
quantiles = [ 0.5, 0.75, 0.95 ]
---
sinks:
my_sink_id:
type: prometheus_exporter
inputs:
- my-source-or-transform-id
quantiles:
- 0.5
- 0.75
- 0.95
{
"sinks": {
"my_sink_id": {
"type": "prometheus_exporter",
"inputs": [
"my-source-or-transform-id"
],
"quantiles": [
0.5,
0.75,
0.95
]
}
}
}
# HELP request_retries request_retries
# TYPE request_retries summary
request_retries{quantile="0.5"} 0
request_retries{quantile="0.75"} 1
request_retries{quantile="0.95"} 4
request_retries_sum 6
request_retries_count 6
request_retries_min 0
request_retries_max 4
request_retries_avg 1
Summary
Given this event...{
"metric": {
"kind": "absolute",
"name": "requests",
"summary": {
"count": 6,
"quantiles": [
{
"upper_limit": 0.01,
"value": 1.5
},
{
"upper_limit": 0.5,
"value": 2
},
{
"upper_limit": 0.99,
"value": 3
}
],
"sum": 12
},
"tags": {
"host": "my-host.local"
}
}
}
[sinks.my_sink_id]
type = "prometheus_exporter"
inputs = [ "my-source-or-transform-id" ]
---
sinks:
my_sink_id:
type: prometheus_exporter
inputs:
- my-source-or-transform-id
{
"sinks": {
"my_sink_id": {
"type": "prometheus_exporter",
"inputs": [
"my-source-or-transform-id"
]
}
}
}
# HELP requests requests
# TYPE requests summary
requests{host="my-host.local",quantile="0.01"} 1.5
requests{host="my-host.local",quantile="0.5"} 2
requests{host="my-host.local",quantile="0.99"} 3
requests_sum{host="my-host.local"} 12
requests_count{host="my-host.local"} 6
How it works
Buffers
This component buffers events as shown in
the diagram above. This helps to smooth out data processing if the downstream
service applies backpressure. Buffers are controlled via the
buffer.*
options.Histogram Buckets
Choosing the appropriate buckets for Prometheus histograms is a complicated point of
discussion. The Histograms and Summaries Prometheus guide
provides a good overview of histograms, buckets, summaries, and how you should think
about configuring them. The buckets you choose should align with your known range
and distribution of values as well as how you plan to report on them. The
aforementioned guide provides examples on how you should align them.
Memory Usage
Like other Prometheus instances, the
prometheus
sink aggregates
metrics in memory which keeps the memory footprint to a minimum if Prometheus
fails to scrape the Vector instance over an extended period of time. The
downside is that data will be lost if Vector is restarted. This is by design of
Prometheus' pull model approach, but is worth noting if restart Vector
frequently.