Template syntax

Vector supports a template syntax for some configuration options. This allows for dynamic values derived from event data. Options that support this syntax will be clearly documented as such in the option description.

Example

For example, let’s partition data on AWS S3 by application_id and date. We can accomplish this with the key_prefix option in the aws_s3 sink:

[sinks.backup]
  type = "aws_s3"
  bucket = "all_application_logs"
  key_prefix = "application_id={{ application_id }}/date=%F/"

Notice that Vector allows direct field references as well as strftime specifiers. If we were to run the following log event through Vector:

{
  "timestamp": "2020-02-14T01:22:23.223Z",
  "application_id": 1,
  "message": "Hello world"
}

The value of the key_prefix option would equal:

application_id=1/date=2020-02-14

Because the aws_s3 sink batches data, each event would be grouped by its produced value. This effectively enables dynamic partitioning, something fundamental to storing log data in filesystems.

Syntax

Event fields

Individual log event fields can be accessed using {{ <field-path-notation> }} syntax:

option = "{{ field_path_notation }}"

Vector’s field notation uses . to target nested fields and [<index>] to target array values.

strftime specifiers

In addition to directly accessing fields, Vector offers a shortcut for injecting strftime specifiers:

options = "year=%Y/month=%m/day=%d/"

The value is derived from the timestamp field and the name of this field can be changed via the global timestamp_key option.

Escaping

You can escape this syntax by prefixing the character with a \. For example, you can escape the event field syntax like this:

option = "\{{ field_name }}"

And strftime specified like so:

options = "year=\%Y/month=\%m/day=\%d/"

Each of the values above would be treated literally.

How it works

Array Values

Array values can be accessed using Vector’s field notation syntax:

option = "{{ parent.child[0] }}"

Fallback values

Vector doesn’t currently support fallback values. Issue 1692 is open to add this functionality. In the interim, you can use the remap transform to set a default value:

[transforms.set_defaults]
  # REQUIRED
  type = "remap"
  inputs = ["my-source-id"]
  source = '''
    if !exists(.my_field) {
      .my_field = "default"
    }
  '''

Missing fields

If a field is missing, an error is logged and Vector drops the event. The component_errors_total internal metric is incremented with error_type = template_failed.

Nested fields

Nested values can be accessed using Vector’s field notation syntax:

option = "{{ parent.child[0] }}"