Prometheus Sink
The Vector prometheus sink exposes metric events to Prometheus metrics service.
Configuration
- Common
- Advanced
[sinks.my_sink_id]# REQUIREDtype = "prometheus" # must be: "prometheus"inputs = ["my-source-id"] # exampleaddress = "0.0.0.0:9598" # examplenamespace = "service" # example# OPTIONALbuckets = [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0] # default, secondsflush_period_secs = 60 # default, seconds
Options
address
The address to expose for scraping. See Exposing & Scraping for more info.
buckets
Default buckets to use for aggregating distribution metrics into histograms. See Histogram Buckets for more info.
[0.005,0.01,0.025,0.05,0.1,0.25,0.5,1,2.5,5,10]flush_period_secs
Time interval between set values are reset.
60healthcheck
Enables/disables the sink healthcheck upon start.
truenamespace
A prefix that will be added to all metric names. It should follow Prometheus naming conventions.
Output
The prometheus sink exposes metric events to Prometheus metrics service.
For example:
- Histograms
- Counters
- Gauges
Given the following histogram metric events:
[{"name": "response_time_s","kind": "incremental","value": {"type": "distribution","values": [0.243],"sample_rates": [1.0]}},{"name": "response_time_s","kind": "incremental","value": {"type": "distribution","values": [0.546],"sample_rates": [1.0]}}]
This sink will output the following:
# HELP response_time_s response_time_s# TYPE response_time_s histogramresponse_time_s_bucket{le="0.005"} 0response_time_s_bucket{le="0.01"} 1response_time_s_bucket{le="0.025"} 0response_time_s_bucket{le="0.05"} 1response_time_s_bucket{le="0.1"} 0response_time_s_bucket{le="0.25"} 0response_time_s_bucket{le="0.5"} 0response_time_s_bucket{le="1.0"} 0response_time_s_bucket{le="2.5"} 0response_time_s_bucket{le="5.0"} 0response_time_s_bucket{le="10.0"} 0response_time_s_bucket{le="+Inf"} 0response_time_s_sum 0.789response_time_s_count 2
How It Works
Buffers
Due to the nature of Prometheus' pull model design the prometheus
sink does not utilize a buffer. You can read more about this in the in-memory \
aggregation section.
Environment Variables
Environment variables are supported through all of Vector's configuration.
Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable
will be replaced before being evaluated.
You can learn more in the Environment Variables section.
Exposing & Scraping
The prometheus sink exposes data for scraping.
The address option determines the address and port the data is made available
on. You'll need to configure your networking so that the configured port is
accessible by the downstream service doing the scraping.
High Cardinality Names
High cardinality metric names and labels are discouraged by \ Prometheus and you should consider alternative strategies to reduce the cardinality as this can provide performance and operation problems. In general, high cardinality analysis should be left logs and storages designed for this use case (not Promtheus).
Histogram Buckets
Choosing the appropriate buckets for Prometheus histgorams is a complicated point of discussion. The Histograms and Summaries Prometheus \ guide provides a good overview of histograms, buckets, summaries, and how you should think about configuring them. The buckets you choose should align with your known range and distribution of values as well as how you plan to report on them. The aforementioned guide provides examples on how you should align them.
Default buckets
The buckets option defines the global default buckets for histograms:
[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
These defaults are tailored to broadly measure the response time (in seconds) of a network service. Most likely, however, you will be required to define buckets customized to your use case.
Note: These values are in seconds, therefore,
your metric values should also be in seconds.
If this is not the case you should adjust your metric or buckets to coincide.
In-Memory Aggregation
Like other Prometheus instances, the prometheus sink aggregates
metrics in memory which keeps the memory footprint to a minimum if Prometheus
fails to scrape the Vector instance over an extended period of time. The
downside is that data will be lost if Vector is restarted. This is by design of
Prometheus' pull model approach, but is worth noting if restart Vector
frequently.
Metric Definitions
By default, Vector will use the original metric names and labels set upon
creation. For most cases this makes Vector low friction, allowing you to define
the prometheus sink without tedious metrics definitions. You can
see examples of this in the examples section.
Metric Types
As described in the metric data model page, Vector offers a variety of metric types. Their support, as as well as their mappings, are described below:
| Vector Metric Type | Prometheus Metric Type |
|---|---|
counter | 'counter' |
gauge | 'gauge' |
histogram | 'histogram' |
set | ⚠️ not supported |
| - | 'summary' |
Sets
Prometheus does not have a set type. Sets are
generally specific to Statsd, and if a set is received in the
prometheus sink it will be dropped, and a rate limited warning
level log will be output.
Summaries
Summaries are a Prometheus specific type and Vector does not default to them
by default. issue #710 addresses the ability to define metrics,
including the ability change their types (such as changing them to summary
types).
OOM Errors
If you experience out of memory (OOM) errors it's likely you're using extremely high cardinality metric names or labels. This is discouraged by Prometheus and you should consider alternative strategies to reduce the cardinality. Such as leveraging logs for high cardinality analysis. Issue #387 discusses the ability to provide safeguards around this. We encourage you to add to that discussion with your use case if you find this to be a problem.