Template syntax
Vector supports a template syntax for some configuration options. This allows for dynamic values derived from event data. Options that support this syntax will be clearly documented as such in the option description.
Example
For example, let’s partition data on AWS S3 by application_id and date. We can accomplish this with the key_prefix option in the aws_s3 sink:
[sinks.backup]
type = "aws_s3"
bucket = "all_application_logs"
key_prefix = "application_id={{ application_id }}/date=%F/"
Notice that Vector allows direct field references as well as strftime specifiers. If we were to run the following log event through Vector:
{
"timestamp": "2020-02-14T01:22:23.223Z",
"application_id": 1,
"message": "Hello world"
}
The value of the key_prefix
option would equal:
application_id=1/date=2020-02-14
Because the aws_s3
sink batches data, each event would be grouped by its produced value. This effectively enables dynamic partitioning, something fundamental to storing log data in filesystems.
Syntax
Event fields
Individual log event fields can be accessed using {{ <field-path-notation> }}
syntax:
option = "{{ field_path_notation }}"
Vector’s field notation uses .
to target nested fields and [<index>]
to target array values.
strftime specifiers
In addition to directly accessing fields, Vector offers a shortcut for injecting strftime specifiers:
options = "year=%Y/month=%m/day=%d/"
The value is derived from the timestamp
field and the name of this field can be changed via the global timestamp_key
option.
Escaping
You can escape this syntax by prefixing the character with a \
. For example, you can escape the event field syntax like this:
option = "\{{ field_name }}"
And strftime specified like so:
options = "year=\%Y/month=\%m/day=\%d/"
Each of the values above would be treated literally.
How it works
Array Values
Array values can be accessed using Vector’s field notation syntax:
option = "{{ parent.child[0] }}"
Fallback values
Vector doesn’t currently support fallback values. Issue 1692 is open to add this functionality. In the interim, you can use the remap
transform to set a default value:
[transforms.set_defaults]
# REQUIRED
type = "remap"
inputs = ["my-source-id"]
source = '''
if !exists(.my_field) {
.my_field = "default"
}
'''
Missing fields
If a field is missing, an error is logged and Vector drops the event. The component_errors_total
internal
metric is incremented with error_type
= template_failed
.
Nested fields
Nested values can be accessed using Vector’s field notation syntax:
option = "{{ parent.child[0] }}"