Template Syntax
Vector supports a template syntax for select configuration options. This allows for dynamic values derived from event data. Options that support this syntax will be clearly documented as such in the option description.
Example
For example, let's partition data on AWS S3 by application_id
and date
. We can accomplish this with
the key_prefix
option in the aws_s3
sink:
[sinks.backup]type = "aws_s3"bucket = "all_application_logs"key_prefix = "application_id={{ application_id }}/date=%F/"
Notice that Vector allows direct field references as well as strftime
specifiers. If we were to run the
following log
event though Vector:
{"timestamp": "2020-02-14T01:22:23.223Z","application_id": 1,"message": "Hello world"}
The value of the key_prefix
option would equal:
application_id=1/date=2020-02-14
And because the aws_s3
sink batches data, each event would be grouped by it's produced value. This
effectively enables dynamic partitioning, something fundamental to storing log data on file system
storages.
Syntax
Event Fields
Individual log
event fields can be accessed with the
{{ <field-path-notation> }}
syntax:
option = "{{ field_path_notation }}"
Vector's field notation is simple. It uses
.
to target nested fields and [<index>]
to target array values. Learn more
in the field notation docs.
Strftime Specifiers
In addition to directly accessing fields, Vector offers a shortcut for injecting strftime specifiers:
options = "year=%Y/month=%m/day=%d/"
The value is based off of the timestamp
field,
and the name of this field can be changed via the
global timestamp_key
option.
Escaping
You can escape this syntax by prefixing the character with a \
. For example,
you can escape the event field syntax like so:
option = "\{{ field_name }}"
And strptime specified like so:
options = "year=\%Y/month=\%m/day=\%d/"
Each value above will be treated literally.
How It Works
Array Values
Array values can be accessed using Vector's field notation syntax:
option = "{{ parent.child[0] }}"
Fallback Values
Vector currently does not support fallback values. Issue 1692
is open to add this functionality. In the interim, you can use the
lua
transform to set a default value:
[transforms.set_defaults]# REQUIREDtype = "lua"inputs = ["my-source-id"]source = """if event["my_field"] == nil thenevent["my_field"] = "default"end"""
Missing Fields
If a field is missing, a blank string will be inserted in its place. Vector will not error, drop the event, or log anything.
Nested Fields
Nested fields can be accessed using Vector's field notation syntax:
option = "{{ parent.child }}"