Template Syntax

Vector supports a template syntax for select configuration options. This allows for dynamic values derived from event data. Options that support this syntax will be clearly documented as such in the option description.

Example

For example, let's partition data on AWS S3 by application_id and date. We can accomplish this with the key_prefix option in the aws_s3 sink:

[sinks.backup]
  type = "aws_s3"
  bucket = "all_application_logs"
  key_prefix = "application_id={{ application_id }}/date=%F/"

Notice that Vector allows direct field references as well as strftime specifiers. If we were to run the following log event though Vector:

{
  "timestamp": "2020-02-14T01:22:23.223Z",
  "application_id": 1,
  "message": "Hello world"
}

The value of the key_prefix option would equal:

application_id=1/date=2020-02-14

And because the aws_s3 sink batches data, each event would be grouped by it's produced value. This effectively enables dynamic partitioning, something fundamental to storing log data on file system storages.

Syntax

Event Fields

Individual log event fields can be accessed with the {{ <field-path-notation> }} syntax:

option = "{{ field_path_notation }}"

Vector's field notation is simple. It uses . to target nested fields and [<index>] to target array values. Learn more in the field notation docs.

Strftime Specifiers

In addition to directly accessing fields, Vector offers a shortcut for injecting strftime specifiers:

options = "year=%Y/month=%m/day=%d/"

The value is based off of the timestamp field, and the name of this field can be changed via the global timestamp_key option.

Escaping

You can escape this syntax by prefixing the character with a \. For example, you can escape the event field syntax like so:

option = "\{{ field_name }}"

And strptime specified like so:

options = "year=\%Y/month=\%m/day=\%d/"

Each value above will be treated literally.

How It Works

Array Values

Array values can be accessed using Vector's field notation syntax:

option = "{{ parent.child[0] }}"

Fallback Values

Vector currently does not support fallback values. Issue 1692 is open to add this functionality. In the interim, you can use the lua transform to set a default value:

[transforms.set_defaults]
  # REQUIRED
  type = "lua"
  inputs = ["my-source-id"]
  source = """
if event["my_field"] == nil then
  event["my_field"] = "default"
end
"""

Missing Fields

If a field is missing, a blank string will be inserted in its place. Vector will not error, drop the event, or log anything.

Nested Fields

Nested fields can be accessed using Vector's field notation syntax:

option = "{{ parent.child }}"
Contents
Resources