/ Feb 21st, 2020 /

Improved Multiline Support In The File Source

Merge multiple lines together based on rules

One of the biggest frustrations we've heard from users in this space is the inability to merge lines together. Such a simple task can be incredibly complex and hard. Fear not! We plan to add first-class support for solving this problem.

In addition to the recently added automatic merging of Docker logs, we also added [better multiline support to our file source. These options are very expressive and should solve the vast majority of multiline merging problems.

For example. Given the following lines:

foobar.rb:6:in `/': divided by 0 (ZeroDivisionError)
from foobar.rb:6:in `bar'
from foobar.rb:2:in `foo'
from foobar.rb:9:in `<main>'

You can merge them with the following config:

vector.toml
[sources.my_file_source]
type = "file"
# ...
[sources.my_file_source.multiline]
start_pattern = "^[^\\s]"
mode = "continue_through"
condition_pattern = "^[\\s]+from"
timeout_ms = 1000

And if this doesn't do it, you can always fallback to our lua transform.

Get This Change

This change was made available in 0.8.0. You can get this change by downloading the latest stable release.

Like What You See?