Prometheus Sink

The Vector prometheus sink exposes metric events to Prometheus metrics service.

Requirements

Configuration

vector.toml
[sinks.my_sink_id]
type = "prometheus" # required
inputs = ["my-source-id"] # required
address = "0.0.0.0:9598" # required
namespace = "service" # required
  • stringcommonrequired

    address

    The address to expose for scraping.

    See Exposing & Scraping for more info.

    • No default
    • View examples
  • [float] (seconds)optional

    buckets

    Default buckets to use for aggregating distribution metrics into histograms.

    See Histogram Buckets for more info.

    • Default: [0.005,0.01,0.025,0.05,0.1,0.25,0.5,1,2.5,5,10] (seconds)
    • View examples
  • int (seconds)optional

    flush_period_secs

    Time interval between set values are reset.

    • Default: 60 (seconds)
    • View examples
  • stringcommonrequired

    namespace

    A prefix that will be added to all metric names. It should follow Prometheus naming conventions.

    • No default
    • View examples

Examples

Given the following histogram metric events:

Example histogram metrics
[
{
"name": "response_time_s",
"kind": "incremental",
"value": {
"type": "distribution",
"values": [0.243],
"sample_rates": [1.0]
}
},
{
"name": "response_time_s",
"kind": "incremental",
"value": {
"type": "distribution",
"values": [0.546],
"sample_rates": [1.0]
}
}
]

This sink will output the following:

Example sink output
# HELP response_time_s response_time_s
# TYPE response_time_s histogram
response_time_s_bucket{le="0.005"} 0
response_time_s_bucket{le="0.01"} 1
response_time_s_bucket{le="0.025"} 0
response_time_s_bucket{le="0.05"} 1
response_time_s_bucket{le="0.1"} 0
response_time_s_bucket{le="0.25"} 0
response_time_s_bucket{le="0.5"} 0
response_time_s_bucket{le="1.0"} 0
response_time_s_bucket{le="2.5"} 0
response_time_s_bucket{le="5.0"} 0
response_time_s_bucket{le="10.0"} 0
response_time_s_bucket{le="+Inf"} 0
response_time_s_sum 0.789
response_time_s_count 2

How It Works

Buffers

Due to the nature of Prometheus' pull model design the prometheus sink does not utilize a buffer. You can read more about this in the in-memory \ aggregation section.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.

Exposing & Scraping

The prometheus sink exposes data for scraping. The address option determines the address and port the data is made available on. You'll need to configure your networking so that the configured port is accessible by the downstream service doing the scraping.

High Cardinality Names

High cardinality metric names and labels are discouraged by \ Prometheus and you should consider alternative strategies to reduce the cardinality as this can provide performance and operation problems. In general, high cardinality analysis should be left logs and storages designed for this use case (not Promtheus).

Histogram Buckets

Choosing the appropriate buckets for Prometheus histgorams is a complicated point of discussion. The Histograms and Summaries Prometheus \ guide provides a good overview of histograms, buckets, summaries, and how you should think about configuring them. The buckets you choose should align with your known range and distribution of values as well as how you plan to report on them. The aforementioned guide provides examples on how you should align them.

Default buckets

The buckets option defines the global default buckets for histograms:

[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]

These defaults are tailored to broadly measure the response time (in seconds) of a network service. Most likely, however, you will be required to define buckets customized to your use case.

In-Memory Aggregation

Like other Prometheus instances, the prometheus sink aggregates metrics in memory which keeps the memory footprint to a minimum if Prometheus fails to scrape the Vector instance over an extended period of time. The downside is that data will be lost if Vector is restarted. This is by design of Prometheus' pull model approach, but is worth noting if restart Vector frequently.

Metric Definitions

By default, Vector will use the original metric names and labels set upon creation. For most cases this makes Vector low friction, allowing you to define the prometheus sink without tedious metrics definitions. You can see examples of this in the examples section.

Metric Types

As described in the metric data model page, Vector offers a variety of metric types. Their support, as as well as their mappings, are described below:

Vector Metric TypePrometheus Metric Type
counter'counter'
gauge'gauge'
histogram'histogram'
set⚠️ not supported
-'summary'

Sets

Prometheus does not have a set type. Sets are generally specific to Statsd, and if a set is received in the prometheus sink it will be dropped, and a rate limited warning level log will be output.

Summaries

Summaries are a Prometheus specific type and Vector does not default to them by default. issue #710 addresses the ability to define metrics, including the ability change their types (such as changing them to summary types).

OOM Errors

If you experience out of memory (OOM) errors it's likely you're using extremely high cardinality metric names or labels. This is discouraged by Prometheus and you should consider alternative strategies to reduce the cardinality. Such as leveraging logs for high cardinality analysis. Issue #387 discusses the ability to provide safeguards around this. We encourage you to add to that discussion with your use case if you find this to be a problem.