METRIC

prometheus sink

The prometheus sink exposes metric events to Prometheus metrics service.

Configuration

  • Common
  • Advanced
vector.toml
[sinks.my_sink_id]
type = "prometheus" # example, must be: "prometheus"
inputs = ["my-source-id"] # example
address = "0.0.0.0:9598" # example
namespace = "service" # example

Options

4 items
commonstringrequired

address

The address to expose for scraping. See Exposing & Scraping for more info.

No default
View examples
[float]secondsoptional

buckets

Default buckets to use for histogram metrics. See Histogram Buckets for more info.

Default: [0.005,0.01,0.025,0.05,0.1,0.25,0.5,1,2.5,5,10]
View examples
booloptional

healthcheck

Enables/disables the sink healthcheck upon start.

Default: true
View examples
commonstringrequired

namespace

A prefix that will be added to all metric names. It should follow Prometheus naming conventions.

No default
View examples

Output

The prometheus sink exposes metric events to Prometheus metrics service. For example:

  • Histograms
  • Counters
  • Gauges

Given the following input:

[
{
"histogram": {
"name": "response_time_s",
"val": 0.243
}
},
{
"histogram": {
"name": "response_time_s",
"val": 0.546
}
}
]

The following output will be produced:

# HELP response_time_s response_time_s
# TYPE response_time_s histogram
response_time_s_bucket{le="0.005"} 0
response_time_s_bucket{le="0.01"} 1
response_time_s_bucket{le="0.025"} 0
response_time_s_bucket{le="0.05"} 1
response_time_s_bucket{le="0.1"} 0
response_time_s_bucket{le="0.25"} 0
response_time_s_bucket{le="0.5"} 0
response_time_s_bucket{le="1.0"} 0
response_time_s_bucket{le="2.5"} 0
response_time_s_bucket{le="5.0"} 0
response_time_s_bucket{le="10.0"} 0
response_time_s_bucket{le="+Inf"} 0
response_time_s_sum 0.789
response_time_s_count 2

How It Works

Buffers

Due to the nature of Prometheus' pull model design the prometheus sink does not utilize a buffer. You can read more about this in the in-memory \ aggregation section.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.

Exposing & Scraping

The prometheus sink exposes data for scraping. Theaddress option determines the address and port the data is made available on. You'll need to configure your networking so that the configured port is accessible by the downstream service doing the scraping.

High Cardinality Names

High cardinality metric names and labels are discouraged by \ Prometheus and you should consider alternative strategies to reduce the cardinality as this can provide performance and operation problems. In general, high cardinality analysis should be left logs and storages designed for this use case (not Promtheus).

Histogram Buckets

Choosing the appropriate buckets for Prometheus histgorams is a complicated point of discussion. The Histograms and Summaries Prometheus \ guide provides a good overview of histograms, buckets, summaries, and how you should think about configuring them. The buckets you choose should align with your known range and distribution of values as well as how you plan to report on them. The aforementioned guide provides examples on how you should align them.

Default buckets

Thebuckets option defines the global default buckets for histograms:

[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]

These defaults are tailored to broadly measure the response time (in seconds) of a network service. Most likely, however, you will be required to define buckets customized to your use case.

In-Memory Aggregation

Like other Prometheus instances, the prometheus sink aggregates metrics in memory which keeps the memory footprint to a minimum if Prometheus fails to scrape the Vector instance over an extended period of time. The downside is that data will be lost if Vector is restarted. This is by design of Prometheus' pull model approach, but is worth noting if restart Vector frequently.

Metric Definitions

By default, Vector will use the original metric names and labels set upon creation. For most cases this makes Vector low friction, allowing you to define the prometheus sink without tedious metrics definitions. You can see examples of this in the examples section.

Metric Types

As described in the metric data model page, Vector offers a variety of metric types. Their support, as as well as their mappings, are described below:

Vector Metric TypePrometheus Metric Type
counter'counter'
gauge'gauge'
histogram'histogram'
set⚠️ not supported
-'summary'

Sets

Prometheus does not have a set type. Sets are generally specific to Statsd, and if a set is received in the prometheus sink it will be dropped, and a rate limited warning level log will be output.

Summaries

Summaries are a Prometheus specific type and Vector does not default to them by default. issue #710 addresses the ability to define metrics, including the ability change their types (such as changing them to summary types).

OOM Errors

If you experience out of memory (OOM) errors it's likely you're using extremely high cardinality metric names or labels. This is discouraged by Prometheus and you should consider alternative strategies to reduce the cardinality. Such as leveraging logs for high cardinality analysis. Issue #387 discusses the ability to provide safeguards around this. We encourage you to add to that discussion with your use case if you find this to be a problem.