Improved Multiline Support In The File Source

Merge multiple lines together based on rules

One of the biggest frustrations we’ve heard from users in this space is the inability to merge lines together. Such a simple task can be incredibly complex and hard. Fear not! We plan to add first-class support for solving this problem.

In addition to the recently added automatic merging of Docker logs, we also added better multi-line support to our file source. These options are very expressive and should solve the vast majority of multi-line merging problems.

For example. Given the following lines:

foobar.rb:6:in `/': divided by 0 (ZeroDivisionError)
  from foobar.rb:6:in `bar'
  from foobar.rb:2:in `foo'
  from foobar.rb:9:in `<main>'

You can merge them with the following config:

  type = "file"
  # ...

    start_pattern = "^[^\\s]"
    mode = "continue_through"
    condition_pattern = "^[\\s]+from"
    timeout_ms = 1000

And if this doesn’t do it, you can always fall back to the lua transform.