Templates
Vector supports a template syntax for select configuration options. This allows you to use event fields when configuring Vector:
field = "application_id={{ application_id }}/date=%F/"
Example
For example, let's partition data on AWS S3 by application_id
and date
.
We can accomplish this with the
key_prefix
option in the
aws_s3
sink:
[sinks.backup]type = "aws_s3"bucket = "all_application_logs"key_prefix = "application_id={{ application_id }}/date=%F/"
Notice that Vector allows direct field references as well as strftime
specifiers. If we were to run the following log
event
though Vector:
{"timestamp": "2020-02-14T01:22:23.223Z","application_id": 1,"message": "Hello world"}
The value of the key_prefix
option would equal:
application_id=1/date=2020-02-14
This effectively enables dynamic partitioning, something fundamental to storing log data.
Syntax
Event Fields
Individual log
event fields can be access with the
{{<field-path-notation>}}
syntax:
option = "{{ field_path_notation }}"
Vector's field notation is simple. It uses
.
to target nested fields and [<index>]
to target array values. Learn more
in the field notation docs.
Strftime Specifiers
In addition to directly accessing fields, Vector offers a shortcut for injecting strftime specifiers:
options = "year=%Y/month=%m/day=%d/"
The value is based off of the timestamp
field,
and the name of this field can be changed via the
global timestamp_key
option.
Escaping
You can escape this syntax by prefixing the character with a \
. For example,
you can escape the event field syntax like so:
option = "\{{ field_name }}"
And strptime specified like so:
options = "year=\%Y/month=\%m/day=\%d/"
Each value above will be treated literally.
How It Works
Array Values
Array values can be accessed using Vector's field notation syntax:
option = "{{ parent.child[0] }}"
Fallback Values
Vector currently does not support fallback values. Issue 1692
is open to add this functionality. In the interim, you can use the
lua
transform to set a default value:
[transforms.set_defaults]# REQUIREDtype = "lua"inputs = ["my-source-id"]source = """if event["my_field"] == nil thenevent["my_field"] = "default"end"""
Missing Fields
If a field is missing, a blank string will be inserted in its place. Vector will not error, drop the event, or log anything.
Nested Fields
Nested fields can be accessed using Vector's field notation syntax:
option = "{{ parent.child }}"