Mapping one log event into multiple
Introducing new methods for converting a single log event into multiple using the remap transform
This release enables transforming a single log event into multiple log events using the remap transform. The
groundwork for this was laid in 0.14.0, but we’ve been waiting to announce it until the addition of the
unnest function in this release.
The basic premise is that if you assign an array to the root object, ., in a VRL program run by remap, then remap
will create one log event for each event in the array.
For example:
[transforms.remap]
type = "remap"
inputs = []
source = """
. = [{"message": "hello"}, {"message": "world"}]
"""
Would generate two output events:
{"message": "hello"}
{"message": "world"}
Any array elements that are not key/value objects will be converted to an log event by creating a log event with the
message key set to the array element value.
See the remap transform docs for more examples.
Additionally, to make it easier to convert an incoming log event into an array, we’ve added an unnest
function to VRL that transforms an incoming event where one of the fields is an array into an array of events, each with
one of the elements from the array field. This is easiest to see with an example:
[transforms.remap]
type = "remap"
inputs = []
source = """
. = {"host": "localhost", "events": [{"message": "hello"}, {"message": "world"}]} # to represent the incoming event
. = unnest(.events)
"""
Would output the following log events:
{ "events": { "message": "hello" }, "host": "localhost" }
{ "events": { "message": "world" }, "host": "localhost" }
In the future, we plan to add functionality to VRL to allow iterating over arrays, but, for now, the simple case of
mapping each event in an array separately can be done by having one remap transform do the “exploding” of the event,
and another remap transform to receive each new event.
An example of this:
[transforms.explode]
type = "remap"
inputs = []
source = """
. = {"host": "localhost", "events": [{"message": "hello"}, {"message": "world"}]} # to represent the incoming event
. = unnest(.events)
"""
[transforms.map]
type = "remap"
inputs = ["explode"]
source = """
# example of pulling up the nested field to merge it into the top-level
. |= .events
del(.events)
"""
Would output the following log events:
{ "message": "hello", "host": "localhost" }
{ "message": "world", "host": "localhost" }