Configuration
This section covers configuring Vector and creating pipelines like the
example below. Vector's configuration supports TOML,
YAML, and JSON to ensure Vector fits into your
workflow. The configuration file(s) must be passed via the
--config
flag when
starting Vector:
- TOML
- YAML
- JSON
vector --config /etc/vector/vector.toml
Example
- vector.toml
- vector.yaml
- vector.json
# Set global optionsdata_dir = "/var/lib/vector"# Ingest data by tailing one or more files[sources.apache_logs]type = "file"include = ["/var/log/apache2/*.log"] # supports globbingignore_older = 86400 # 1 day# Structure and parse via Timber's Remap Language[transforms.remap]inputs = ["apache_logs"]type = "remap"source = '''. = parse_apache_log(.message)'''# Sample the data to save on cost[transforms.apache_sampler]inputs = ["apache_parser"]type = "sampler"rate = 50 # only keep 50%# Send structured data to a short-term storage[sinks.es_cluster]inputs = ["apache_sampler"] # only take sampled datatype = "elasticsearch"host = "http://79.12.221.222:9200" # local or external hostindex = "vector-%Y-%m-%d" # daily indices# Send structured data to a cost-effective long-term storage[sinks.s3_archives]inputs = ["apache_parser"] # don't sample for S3type = "aws_s3"region = "us-east-1"bucket = "my-log-archives"key_prefix = "date=%Y-%m-%d" # daily partitions, hive friendly formatcompression = "gzip" # compress final objectsencoding = "ndjson" # new line delimited JSONbatch.max_size = 10000000 # 10mb uncompressed
The key thing to note above is the use of the inputs
option. This connects
Vector's component to create a pipeline. For a simple introduction, please
refer to the:
Reference
Vector provides a full reference that you can use to build your configuration files.
And for more advanced techniques:
How It Works
Configuration File Location
The location of your Vector configuration file depends on your installation
method. For most Linux based systems, the file can be
found at /etc/vector/vector.toml
.
Environment Variables
Vector will interpolate environment variables within your configuration file with the following syntax:
[transforms.add_host]type = "add_fields"[transforms.add_host.fields]host = "${HOSTNAME}"environment = "${ENV:-development}" # default value when not present
Default Values
Default values can be supplied via the :-
syntax:
option = "${ENV_VAR:-default}"
Escaping
You can escape environment variable by preceding them with a $
character. For
example $${HOSTNAME}
will be treated literally in the above environment
variable example.
Multiple Configuration Files
You can pass multiple configuration files when starting Vector:
vector --config vector1.toml --config vector2.toml
Or use a globbing syntax:
vector --config /etc/vector/*.toml
Multiple Formats
Vector supports TOML, YAML, and JSON to ensure Vector fits into your workflow. A side benefit of supporting JSON is the enablement of data templating languages like Jsonnet and [Cue][ursl.cue].
Template Syntax
Select configuration options support Vector's template syntax to produce dynamic values derived from the event's data. Two syntaxes are supported for fields that support field interpolation:
- Strptime specifiers. Ex:
date=%Y/%m/%d
- Event fields. Ex:
{{ field_name }}
For example:
[sinks.es_cluster]type = "elasticsearch"index = "user-{{ user_id }}-%Y-%m-%d"
The above index
value will be calculated for each event. For example, given
the following event:
{"timestamp": "2019-05-02T00:23:22Z","message": "message","user_id": 2}
The index
value will result in:
index = "user-2-2019-05-02"
Learn more in the template reference.