Today we're excited to announce beta support for unit testing Vector configurations, allowing you to define tests directly within your Vector configuration file. These tests are used to assert the output from topologies of transform components given certain input events, ensuring that your configuration behavior does not regress; a very powerful feature for mission-critical production pipelines that are collaborated on.
Let's look at a basic example that uses the
transform to parse log lines:
[sources.my_logs]type = "file"include = ["/var/log/my-app.log"][transforms.parser]inputs = ["my_logs"]type = "regex_parser"regex = "^(?P<timestamp>[\\w\\-:\\+]+) (?P<level>\\w+) (?P<message>.*)$"[[tests]]name = "verify_regex"[tests.input]insert_at = "parser"type = "raw"value = "2019-11-28T12:00:00+00:00 info Hello world"[[tests.outputs]]extract_from = "parser"[[tests.outputs.conditions]]type = "check_fields""timestamp.equals" = "2019-11-28T12:00:00+00:00""level.equals" = "info""message.equals" = "Hello world"
And you can run the tests via the new
$ vector test ./vector.tomlRunning ./vector.toml testsTest ./vector.toml: verify_regex ... passed
Many Vector configurations will be simple transformations across straight forward pipelines (such as tailing a file and piping the data to AWS CloudWatch Logs) and don't really need protection from regressions. However, Vector configs are capable of expanding indefinitely with transforms in order to solve as much of your processing needs as possible.
As a configuration grows, and as the number of owners of a configuration grow, the potential for regressions also grows just like a regular code base. The lack of testing capabilities of configuration driven services is therefore a common pain for larger organizations. We hope that natively supporting unit tests in Vector configs will preemptively solve this problem.
To help you get started we put together two documentation pages:
These should be everything you need and will be actively maintained as this feature matures.
We're eager to hear your feedback! Unit testing, as a
beta feature, is still
in an early phase and we need case studies and comments in order to ensure it
works well for everyone. Please let us know what you think either in our
community chat or by
raising an issue.