Improved Multiline Support In The File Source

One of the biggest frustrations we’ve heard from users in this space is the inability to merge lines together. Such a simple task can be incredibly complex and hard. Fear not! We plan to add first-class support for solving this problem.

In addition to the recently added automatic merging of Docker logs, we also added better multi-line support to our file source. These options are very expressive and should solve the vast majority of multi-line merging problems.

For example. Given the following lines:

foobar.rb:6:in `/': divided by 0 (ZeroDivisionError)
  from foobar.rb:6:in `bar'
  from foobar.rb:2:in `foo'
  from foobar.rb:9:in `<main>'

You can merge them with the following config:

[sources.my_file_source]
  type = "file"
  # ...

  [sources.my_file_source.multiline]
    start_pattern = "^[^\\s]"
    mode = "continue_through"
    condition_pattern = "^[\\s]+from"
    timeout_ms = 1000

And if this doesn’t do it, you can always fall back to the lua transform.