Pipeline components reference

This page documents the top-level configuration for pipeline components: sources, transforms, sinks, and enrichment tables.

These fields define the structure of your observability data pipeline. Each component is defined as a table within these sections, with component-specific configuration options.

For other top-level configuration options, see:

  • Global Options - Global settings like data directories and timezone
  • API - Configure Vector's observability API
  • Schema - Configure Vector's internal schema system
  • Secrets - Configure secrets management

enrichment_tables

optional object
All configured enrichment tables.

enrichment_tables.*

required object
An enrichment table.
enrichment_tables.*.file
required object
File-specific settings.
Relevant when: type = "file"
enrichment_tables.*.file.encoding
required object
File encoding configuration.
enrichment_tables.*.file.encoding.delimiter optional string
The delimiter used to separate fields in each row of the CSV file.
default: ,
enrichment_tables.*.file.encoding.include_headers optional bool

Whether or not the file contains column headers.

When set to true, the first row of the CSV file will be read as the header row, and the values will be used for the names of each column. This is the default behavior.

When set to false, columns are referred to by their numerical index.

default: true
enrichment_tables.*.file.encoding.type required string enum
File encoding type.
Enum options
OptionDescription
csvDecodes the file as a CSV (comma-separated values) file.
enrichment_tables.*.file.path
required string

The path of the enrichment table file.

Currently, only CSV files are supported.

enrichment_tables.*.filter
optional object
Set to make the table act as a probabilistic filter instead of storing original values. This will prevent reading values from the table - found keys will have empty value.
Relevant when: type = "memory"
enrichment_tables.*.filter.bucket_size
optional uint
Number of slots in each bucket
Relevant when: type = "cuckoo"
default: 4
enrichment_tables.*.filter.concurrent_scanning
optional bool

If set to true scanning will not block insertions. This may affect behavior since blocking scans would free up space before insertions.

By default, scanning is blocking.

Relevant when: type = "cuckoo"
default: false
enrichment_tables.*.filter.counter_bits
optional uint
Number of bits to use to track counter. This will limit the max value.
Relevant when: type = "cuckoo"
default: 8
enrichment_tables.*.filter.counter_enabled
optional bool
Can be set to true to track a count alongside hashes.
Relevant when: type = "cuckoo"
default: false
enrichment_tables.*.filter.counter_field
optional string
Field in the incoming value used as the counter increment override.
Relevant when: type = "cuckoo"
enrichment_tables.*.filter.counter_insertion_increment
optional int
The amount to increment the counter by on every insertion. Negative values are allowed.
Relevant when: type = "cuckoo"
default: 1
enrichment_tables.*.filter.counter_lookup_increment
optional int
The amount to increment the counter by on every lookup. Negative values are allowed.
Relevant when: type = "cuckoo"
default: 1
enrichment_tables.*.filter.export_interval
optional uint

The interval used for exporting data.

By default, export is only done on exit.

Relevant when: type = "cuckoo"
enrichment_tables.*.filter.fingerprint_bits
optional uint
Number of bits used for fingerprint.
Relevant when: type = "cuckoo"
default: 8
enrichment_tables.*.filter.lru_aging_strategy
optional object
Strategy to use when aging LRU counters at each scan.
Relevant when: type = "cuckoo"
enrichment_tables.*.filter.lru_aging_strategy.strategy optional string enum
The LRU aging strategy to use.
Enum options
OptionDescription
decrementAging LRU counters by decrementing by a fixed value on each scan.
halvingAging LRU counters by halving their value on each scan.
default: halving
enrichment_tables.*.filter.lru_aging_strategy.value required uint
Value to decrement by
Relevant when: strategy = "decrement"
enrichment_tables.*.filter.lru_bits
optional uint
Number of bits to use to track LRU counter. Low bit count will reduce the maximum LRU counter value, making the items expire sooner if unused.
Relevant when: type = "cuckoo"
default: 8
enrichment_tables.*.filter.lru_deletion_enabled
optional bool
Can be set to true to delete unused items on scan when LRU is used.
Relevant when: type = "cuckoo"
default: false
enrichment_tables.*.filter.lru_enabled
optional bool
Can be set to true to use LRU strategy for kicking.
Relevant when: type = "cuckoo"
default: false
enrichment_tables.*.filter.lru_increment
optional uint
Value to increase LRU counter by on each item access.
Relevant when: type = "cuckoo"
default: 1
enrichment_tables.*.filter.lru_starting_value
optional uint
Starting value for LRU counter on item insertion. Higher value will give newer items a higher probability to stay in the filter.
Relevant when: type = "cuckoo"
default: 1
enrichment_tables.*.filter.max_entries
required uint
Maximum number of entries that can be stored in the filter (actual capacity will usually be larger)
enrichment_tables.*.filter.max_kicks
optional uint
Max number of kicks when experiencing hash collisions.
Relevant when: type = "cuckoo"
default: 500
enrichment_tables.*.filter.persistence_path
optional string

Path to the file to export data to periodically and on exit. Data will be imported from this file on startup and reload.

If table reload_behavior is set to clear-state and this is set, the persisted state will still be read after reload.

Relevant when: type = "cuckoo"
enrichment_tables.*.filter.scanning_threads
optional uint

Number of threads to use for scanning and updating LRU/TTL.

By default, scanning is single threaded.

Relevant when: type = "cuckoo"
enrichment_tables.*.filter.ttl_bits
optional uint
Number of bits to use to track TTL. Low bit count will reduce maximum TTL and also require a worse resolution to keep working.
Relevant when: type = "cuckoo"
default: 8
enrichment_tables.*.filter.ttl_enabled
optional bool
Can be set to true to also track TTL for entries.
Relevant when: type = "cuckoo"
default: true
enrichment_tables.*.filter.type
required string enum
The probabilistic filter to use.
Enum options
OptionDescription
bloom

Bloom filter

Only supports insertion and presence check, no TTL

cuckoo

Cuckoo filter

Supports removal by accepting null values for keys, as well as TTL and LRU.

enrichment_tables.*.flush_interval
optional uint

The interval used for making writes visible in the table. Longer intervals might get better performance, but there is a longer delay before the data is visible in the table. Since every TTL scan makes its changes visible, only use this value if it is shorter than the scan_interval.

NOTE: For cuckoo filter, all writes are visible immediately. Flush interval still defines when metrics for cuckoo filter are made visible.

By default, all writes are made visible immediately.

Relevant when: type = "memory"
enrichment_tables.*.graph
optional object

Extra graph configuration

Configure output for component when generated with graph command

enrichment_tables.*.graph.edge_attributes
optional object

Edge attributes to add to the edges linked to this component’s node in resulting graph

They are added to the edge as provided

enrichment_tables.*.graph.edge_attributes.* required object
A collection of graph edge attributes in graphviz DOT language, related to a single input component.
enrichment_tables.*.graph.edge_attributes.*.* required string
A single graph edge attribute in graphviz DOT language.
Examples 
{
  "color": "red",
  "label": "Example Edge",
  "width": "5.0"
}
Examples 
{
  "example_input": {
    "color": "red",
    "label": "Example Edge",
    "width": "5.0"
  }
}
enrichment_tables.*.graph.node_attributes
optional object

Node attributes to add to this component’s node in resulting graph

They are added to the node as provided

enrichment_tables.*.graph.node_attributes.* required string
A single graph node attribute in graphviz DOT language.
Examples 
{
  "color": "red",
  "name": "Example Node",
  "width": "5.0"
}
enrichment_tables.*.inputs
optional [string]

A list of upstream source or transform IDs.

Wildcards (*) are supported.

See configuration for more info.

Examples 
[
  "my-source-or-transform-id",
  "prefix-*"
]
enrichment_tables.*.internal_metrics
optional object
Configuration of internal metrics
Relevant when: type = "memory"
enrichment_tables.*.internal_metrics.include_key_tag
optional bool

Determines whether to include the key tag on internal metrics.

This is useful for distinguishing between different keys while monitoring. However, the tag’s cardinality is unbounded.

default: false
enrichment_tables.*.locale
optional string

The locale to use when querying the database.

MaxMind includes localized versions of some of the fields within their database, such as country name. This setting can control which of those localized versions are returned by the transform.

More information on which portions of the geolocation data are localized, and what languages are available, can be found here.

Relevant when: type = "geoip"
default: en
enrichment_tables.*.max_byte_size
optional uint

Maximum size of the table in bytes. All insertions that make this table bigger than the maximum size are rejected.

By default, there is no size limit.

Relevant when: type = "memory"
enrichment_tables.*.path
required string

Path to the MaxMind GeoIP2 or GeoLite2 binary city database file (GeoLite2-City.mmdb).

Other databases, such as the country database, are not supported. mmdb enrichment table can be used for other databases.

Relevant when: type = "geoip" or type = "mmdb"
enrichment_tables.*.reload_behavior
optional string enum
Behavior for memory table state on configuration reload.
Relevant when: type = "memory"
Enum options
OptionDescription
clear-stateAlways clear state on configuration reload.
preserve-stateTry to preserve state when possible.
default: clear-state
enrichment_tables.*.scan_interval
optional uint
The scan interval used to look for expired records. This is provided as an optimization to ensure that TTL is updated, but without doing too many cache scans.
Relevant when: type = "memory"
default: 30
enrichment_tables.*.schema
optional object

Key/value pairs representing mapped log field names and types.

This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.

Timestamp coercions need to be prefaced with timestamp|, for example "timestamp|%F". Timestamp specifiers can use either of the following:

  1. One of the built-in-formats listed in the Timestamp Formats table below.
  2. The time format specifiers from Rust’s chrono library.

Types

  • bool
  • string
  • float
  • integer
  • date
  • timestamp (see the table below for formats)

Timestamp Formats

FormatDescriptionExample
%F %TYYYY-MM-DD HH:MM:SS2020-12-01 02:37:54
%v %TDD-Mmm-YYYY HH:MM:SS01-Dec-2020 02:37:54
%FT%TISO 8601/RFC 3339, without time zone2020-12-01T02:37:54
%FT%TZISO 8601/RFC 3339, UTC2020-12-01T09:37:54Z
%+ISO 8601/RFC 3339, UTC, with time zone2020-12-01T02:37:54-07:00
%a, %d %b %Y %TRFC 822/RFC 2822, without time zoneTue, 01 Dec 2020 02:37:54
%a %b %e %T %Yctime formatTue Dec 1 02:37:54 2020
%sUNIX timestamp1606790274
%a %d %b %T %Ydate command, without time zoneTue 01 Dec 02:37:54 2020
%a %d %b %T %Z %Ydate command, with time zoneTue 01 Dec 02:37:54 PST 2020
%a %d %b %T %z %Ydate command, with numeric time zoneTue 01 Dec 02:37:54 -0700 2020
%a %d %b %T %#z %Ydate command, with numeric time zone (minutes can be missing or present)Tue 01 Dec 02:37:54 -07 2020
Relevant when: type = "file"
enrichment_tables.*.schema.*
required string
Represents mapped log field names and types.
enrichment_tables.*.source_config
optional object
Configuration for source functionality.
Relevant when: type = "memory"
enrichment_tables.*.source_config.export_batch_size
optional uint

Batch size for data exporting. Used to prevent exporting entire table at once and blocking the system.

By default, batches are not used and entire table is exported.

enrichment_tables.*.source_config.export_expired_items
optional bool
Set to true to export expired items via the expired output port. Expired items ignore other settings and are exported as they are flushed from the table.
default: false
enrichment_tables.*.source_config.export_interval
optional uint
Interval for exporting all data from the table when used as a source.
enrichment_tables.*.source_config.remove_after_export
optional bool

If set to true, all data will be removed from cache after exporting. Only valid if used as a source and export_interval > 0

By default, export will not remove data from cache

default: false
enrichment_tables.*.source_config.source_key
required string
Key to use for this component when used as a source. This must be different from the component key.
enrichment_tables.*.ttl
optional uint
TTL (time-to-live in seconds) is used to limit the lifetime of data stored in the cache. When TTL expires, data behind a specific key in the cache is removed. TTL is reset when the key is replaced.
Relevant when: type = "memory"
default: 600
enrichment_tables.*.ttl_field
optional string
Field in the incoming value used as the TTL override.
Relevant when: type = "memory"
enrichment_tables.*.type
required string enum
enrichment table type
Enum options
OptionDescription
fileExposes data from a static file as an enrichment table.
geoipExposes data from a MaxMind GeoIP2 database as an enrichment table.
memoryExposes data from a memory cache as an enrichment table. The cache can be written to using a sink.
mmdbExposes data from a MaxMind database as an enrichment table.

sinks

optional object
All configured sinks.

sinks.*

required object
A sink.
sinks.*.buffer
optional object

Configures the buffering behavior for this sink.

More information about the individual buffer types, and buffer behavior, can be found in the Buffering Model section.

sinks.*.buffer.max_events
optional uint
The maximum number of events allowed in the buffer.
Relevant when: type = "memory"
default: 500
sinks.*.buffer.max_size
required uint

The maximum allowed amount of allocated memory the buffer can hold.

If type = "disk" then must be at least ~256 megabytes (268435488 bytes).

sinks.*.buffer.type
optional string enum
The type of buffer to use.
Enum options
OptionDescription
disk

Events are buffered on disk.

This is less performant, but more durable. Data that has been synchronized to disk will not be lost if Vector is restarted forcefully or crashes.

Data is synchronized to disk every 500ms.

memory

Events are buffered in memory.

This is more performant, but less durable. Data will be lost if Vector is restarted forcefully or crashes.

default: memory
sinks.*.buffer.when_full
optional string enum
Event handling behavior when a buffer is full.
Enum options
OptionDescription
block

Wait for free space in the buffer.

This applies backpressure up the topology, signalling that sources should slow down the acceptance/consumption of events. This means that while no data is lost, data will pile up at the edge.

drop_newest

Drops the event instead of waiting for free space in buffer.

The event will be intentionally dropped. This mode is typically used when performance is the highest priority, and it is preferable to temporarily lose events rather than cause a slowdown in the acceptance/consumption of events.

default: block
sinks.*.graph
optional object

Extra graph configuration

Configure output for component when generated with graph command

sinks.*.graph.edge_attributes
optional object

Edge attributes to add to the edges linked to this component’s node in resulting graph

They are added to the edge as provided

sinks.*.graph.edge_attributes.* required object
A collection of graph edge attributes in graphviz DOT language, related to a single input component.
sinks.*.graph.edge_attributes.*.* required string
A single graph edge attribute in graphviz DOT language.
Examples 
{
  "color": "red",
  "label": "Example Edge",
  "width": "5.0"
}
Examples 
{
  "example_input": {
    "color": "red",
    "label": "Example Edge",
    "width": "5.0"
  }
}
sinks.*.graph.node_attributes
optional object

Node attributes to add to this component’s node in resulting graph

They are added to the node as provided

sinks.*.graph.node_attributes.* required string
A single graph node attribute in graphviz DOT language.
Examples 
{
  "color": "red",
  "name": "Example Node",
  "width": "5.0"
}
sinks.*.healthcheck
optional object
Healthcheck configuration.
sinks.*.healthcheck.enabled
optional bool
Whether or not to check the health of the sink when Vector starts up.
default: true
sinks.*.healthcheck.timeout
optional float
Timeout duration for healthcheck in seconds.
default: 10(seconds)
sinks.*.healthcheck.uri
optional string

The full URI to make HTTP healthcheck requests to.

This must be a valid URI, which requires at least the scheme and host. All other components – port, path, etc – are allowed as well.

sinks.*.inputs
required [string]

A list of upstream source or transform IDs.

Wildcards (*) are supported.

See configuration for more info.

Examples 
[
  "my-source-or-transform-id",
  "prefix-*"
]
sinks.*.proxy
optional object

Proxy configuration.

Configure to proxy traffic through an HTTP(S) proxy when making external requests.

Similar to common proxy configuration convention, you can set different proxies to use based on the type of traffic being proxied. You can also set specific hosts that should not be proxied.

sinks.*.proxy.enabled
optional bool
Enables proxying support.
default: true
sinks.*.proxy.http
optional string

Proxy endpoint to use when proxying HTTP traffic.

Must be a valid URI string.

Examples 
"http://foo.bar:3128"
sinks.*.proxy.https
optional string

Proxy endpoint to use when proxying HTTPS traffic.

Must be a valid URI string.

Examples 
"http://foo.bar:3128"
sinks.*.proxy.no_proxy
optional [string]

A list of hosts to avoid proxying.

Multiple patterns are allowed:

PatternExample match
Domain namesexample.com matches requests to example.com
Wildcard domains.example.com matches requests to example.com and its subdomains
IP addresses127.0.0.1 matches requests to 127.0.0.1
CIDR blocks192.168.0.0/16 matches requests to any IP addresses in this range
Splat* matches all hosts
Examples 
[
  "localhost",
  ".foo.bar",
  "*"
]

sources

optional object
All configured sources.

sources.*

required object
A source.
sources.*.graph
optional object

Extra graph configuration

Configure output for component when generated with graph command

sources.*.graph.edge_attributes
optional object

Edge attributes to add to the edges linked to this component’s node in resulting graph

They are added to the edge as provided

sources.*.graph.edge_attributes.* required object
A collection of graph edge attributes in graphviz DOT language, related to a single input component.
sources.*.graph.edge_attributes.*.* required string
A single graph edge attribute in graphviz DOT language.
Examples 
{
  "color": "red",
  "label": "Example Edge",
  "width": "5.0"
}
Examples 
{
  "example_input": {
    "color": "red",
    "label": "Example Edge",
    "width": "5.0"
  }
}
sources.*.graph.node_attributes
optional object

Node attributes to add to this component’s node in resulting graph

They are added to the node as provided

sources.*.graph.node_attributes.* required string
A single graph node attribute in graphviz DOT language.
Examples 
{
  "color": "red",
  "name": "Example Node",
  "width": "5.0"
}
sources.*.proxy
optional object

Proxy configuration.

Configure to proxy traffic through an HTTP(S) proxy when making external requests.

Similar to common proxy configuration convention, you can set different proxies to use based on the type of traffic being proxied. You can also set specific hosts that should not be proxied.

sources.*.proxy.enabled
optional bool
Enables proxying support.
default: true
sources.*.proxy.http
optional string

Proxy endpoint to use when proxying HTTP traffic.

Must be a valid URI string.

Examples 
"http://foo.bar:3128"
sources.*.proxy.https
optional string

Proxy endpoint to use when proxying HTTPS traffic.

Must be a valid URI string.

Examples 
"http://foo.bar:3128"
sources.*.proxy.no_proxy
optional [string]

A list of hosts to avoid proxying.

Multiple patterns are allowed:

PatternExample match
Domain namesexample.com matches requests to example.com
Wildcard domains.example.com matches requests to example.com and its subdomains
IP addresses127.0.0.1 matches requests to 127.0.0.1
CIDR blocks192.168.0.0/16 matches requests to any IP addresses in this range
Splat* matches all hosts
Examples 
[
  "localhost",
  ".foo.bar",
  "*"
]

transforms

optional object
All configured transforms.

transforms.*

required object
A transform.
transforms.*.graph
optional object

Extra graph configuration

Configure output for component when generated with graph command

transforms.*.graph.edge_attributes
optional object

Edge attributes to add to the edges linked to this component’s node in resulting graph

They are added to the edge as provided

transforms.*.graph.edge_attributes.* required object
A collection of graph edge attributes in graphviz DOT language, related to a single input component.
transforms.*.graph.edge_attributes.*.* required string
A single graph edge attribute in graphviz DOT language.
Examples 
{
  "color": "red",
  "label": "Example Edge",
  "width": "5.0"
}
Examples 
{
  "example_input": {
    "color": "red",
    "label": "Example Edge",
    "width": "5.0"
  }
}
transforms.*.graph.node_attributes
optional object

Node attributes to add to this component’s node in resulting graph

They are added to the node as provided

transforms.*.graph.node_attributes.* required string
A single graph node attribute in graphviz DOT language.
Examples 
{
  "color": "red",
  "name": "Example Node",
  "width": "5.0"
}
transforms.*.inputs
required [string]

A list of upstream source or transform IDs.

Wildcards (*) are supported.

See configuration for more info.

Examples 
[
  "my-source-or-transform-id",
  "prefix-*"
]
transforms.*.measure_cpu_usage
optional bool

Enable CPU usage metrics for this transform.

When set to true, each poll of the transform task is timed using the OS thread CPU clock and the accumulated nanoseconds are reported as the component_cpu_usage_ns_total counter, tagged with component_id, component_kind, and component_type.

Defaults to false. Enable only for transforms where CPU attribution is needed, as it adds a clock_gettime call on every future poll.

default: false
Sidebar