/ Nov 25th, 2019 /

Unit Testing Your Vector Config Files

Treating your Vector configuration files as code

Today we're excited to announce beta support for unit testing Vector configurations, allowing you to define tests directly within your Vector configuration file. These tests are used to assert the output from topologies of transform components given certain input events, ensuring that your configuration behavior does not regress; a very powerful feature for mission-critical production pipelines that are collaborated on.


Let's look at a basic example that uses the regex_parser transform to parse log lines:

type = "file"
include = ["/var/log/my-app.log"]
inputs = ["my_logs"]
type = "regex_parser"
regex = "^(?P<timestamp>[\\w\\-:\\+]+) (?P<level>\\w+) (?P<message>.*)$"
name = "verify_regex"
insert_at = "parser"
type = "raw"
value = "2019-11-28T12:00:00+00:00 info Hello world"
extract_from = "parser"
type = "check_fields"
"timestamp.equals" = "2019-11-28T12:00:00+00:00"
"level.equals" = "info"
"message.equals" = "Hello world"

And you can run the tests via the new test subcommand:

$ vector test ./vector.toml
Running ./vector.toml tests
Test ./vector.toml: verify_regex ... passed


Many Vector configurations will be simple transformations across straight forward pipelines (such as tailing a file and piping the data to AWS CloudWatch Logs) and don't really need protection from regressions. However, Vector configs are capable of expanding indefinitely with transforms in order to solve as much of your processing needs as possible.

As a config grows, and as the number of owners of a config grow, the potential for regressions also grows just like a regular codebase. The lack of testing capabilities of configuration driven services is therefore a common pain for larger organizations. We hope that natively supporting unit tests in Vector configs will preemptively solve this problem.

Getting Started

To help you get started we put together two documentation pages:

  1. A unit testing guide
  2. A reference of the unit testing config spec

These should be everything you need and will be actively maintained as this feature matures.


We're eager to hear your feedback! Unit testing, as a beta feature, is still in an early phase and we need case studies and comments in order to ensure it works well for everyone. Please let us know what you think either in our community chat or by raising an issue.

Get This Change

This change was made available in 0.6.0. You can get this change by downloading the latest stable release.

Like What You See?