Configuration Reference

Vector is configured through a configuration file. This section contains a comprehensive reference of all Vector configuration file options.

Example

The following is an example of a popular Vector configuration that ingests logs from a file and routes them to both Elasticsearch and AWS S3. Your configuration will differ based on your needs. For guided configuration tutorial, please see the setup configuration page.

# Set global options
data_dir = "/var/lib/vector"


# Vector's API (disabled by default)
# Enable and try it out with the `vector top` command
[api]
enabled = false
# address = "127.0.0.1:8686"


# Ingest data by tailing one or more files
[sources.apache_logs]
type         = "file"
include      = ["/var/log/apache2/*.log"]    # supports globbing
ignore_older = 86400                         # 1 day


# Structure and parse via Timber's Remap Language
[transforms.remap]
inputs       = ["apache_logs"]
type         = "remap"
source       = '''
. = parse_apache_log(.message)
'''


# Sample the data to save on cost
[transforms.apache_sampler]
inputs       = ["apache_parser"]
type         = "sampler"
rate         = 50                            # only keep 50%


# Send structured data to a short-term storage
[sinks.es_cluster]
inputs       = ["apache_sampler"]            # only take sampled data
type         = "elasticsearch"
host         = "http://79.12.221.222:9200"   # local or external host
index        = "vector-%Y-%m-%d"             # daily indices


# Send structured data to a cost-effective long-term storage
[sinks.s3_archives]
inputs       = ["apache_parser"]             # don't sample for S3
type         = "aws_s3"
region       = "us-east-1"
bucket       = "my-log-archives"
key_prefix   = "date=%Y-%m-%d"               # daily partitions, hive friendly format
compression  = "gzip"                        # compress final objects
encoding     = "ndjson"                      # new line delimited JSON
batch.max_size   = 10000000                  # 10mb uncompressed

To use this configuration file, simply specify it with the --config flag upon starting Vector:

vector --config /etc/vector/vector.toml

Reference

Components

Sources
Transforms
Sinks

Advanced

Global options
Template syntax
Unit tests

How It Works

Environment variables

Vector will interpolate environment variables within your configuration file with the following syntax:

vector.toml
[transforms.add_host]
  type = "add_fields"


  [transforms.add_host.fields]
    host = "${HOSTNAME}"
    environment = "${ENV:-development}" # default value when not present

Default values

Default values can be supplied via the :- syntax:

option = "${ENV_VAR:-default}"

Escaping

You can escape environment variable by preceding them with a $ character. For example $${HOSTNAME} will be treated literally in the above environment variable example.

Formats

Vector supports TOML, YAML, and JSON to ensure Vector fits into your workflow. A side benefit of supporting JSON is the enablement of data templating languages like Jsonnet and Cue.

Location

The location of your Vector configuration file depends on your installation method. For most Linux based systems, the file can be found at /etc/vector/vector.toml.

Multiple files

You can pass multiple configuration files when starting Vector:

vector --config vector1.toml --config vector2.toml

Or use a globbing syntax:

vector --config /etc/vector/*.toml

Wildcards in identifiers

Vector supports wildcards (*) in component identifiers when building your topology, but only supports them as the last character. For example:

[sources.app1_logs]
type = "file"
includes = ["/var/log/app1.log"]


[sources.app2_logs]
type = "file"
includes = ["/var/log/app.log"]


[sources.system_logs]
type = "file"
includes = ["/var/log/system.log"]


[sinks.app_logs]
type = "datadog_logs"
inputs = ["app*"]


[sinks.archive]
type = "aws_s3"
inputs = ["app*", "system_logs"]
Contents
Resources