Merge Transform

The Vector merge transform merges partial log events into a single event.

Configuration

[transforms.my_transform_id]
type = "merge" # required
inputs = ["my-source-or-transform-id"] # required
fields = ["message"] # optional, default
partial_event_marker_field = "_partial" # optional, default
stream_discriminant_fields = [] # optional, default
  • commonoptional[string]

    fields

    Fields to merge. The values of these fields will be merged into the first partial event. Fields not specified here will be ignored. Merging process takes the first partial event and the base, then it merges in the fields from each successive partial event, until a non-partial event arrives. Finally, the non-partial event fields are merged in, producing the resulting merged event.

    • Default: ["message"]
    • View examples
  • commonoptionalstring

    partial_event_marker_field

    The field that indicates that the event is partial. A consequent stream of partial events along with the first non-partial event will be merged together.

    • Default: "_partial"
    • View examples
  • commonoptional[string]

    stream_discriminant_fields

    An ordered list of fields to distinguish streams by. Each stream has a separate partial event merging state. Should be used to prevent events from unrelated sources from mixing together, as this affects partial event processing.

    • Default: []
    • View examples

Telemetry

This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

  • counter

    processed_events_total

    The total number of events processed by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • file - The file that produced the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    processed_bytes_total

    The total number of bytes processed by the component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

Examples

Given the following Vector event:

[
{
"log": {
"message": "First",
"_partial": true,
"custom_string_field": "value1",
"custom_int_field": 1
}
},
{
"log": {
"message": "Second",
"_partial": true,
"custom_string_field": "value2",
"custom_int_field": 2
}
},
{
"log": {
"message": "Third",
"custom_string_field": "value3",
"custom_int_field": 3
}
}
]

And the following configuration:

[transforms.merge]
type = "merge"

The following Vector log event will be output:

{
"message": "FirstSecondThird",
"custom_string_field": "value1",
"custom_int_field": 1
}