JSON Parser Transform

The Vector json_parser transform parses a log field value as JSON.

Warnings

Configuration

[transforms.my_transform_id]
type = "json_parser" # required
inputs = ["my-source-or-transform-id", "prefix-*"] # required
drop_field = true # optional, default
drop_invalid = true # required
field = "message" # optional, default
  • commonoptionalbool

    drop_field

    If the specified field should be dropped (removed) after parsing. If parsing fails, the field will not be removed, irrespective of this setting.

    • Default: true
  • commonrequiredbool

    drop_invalid

    If true events with invalid JSON will be dropped, otherwise the event will be kept and passed through. See Invalid JSON for more info.

  • commonoptionalstring

    field

    The log field to decode as JSON. Must be a string value type. See Invalid JSON for more info.

    • Syntax: literal
    • Default: "message"
  • optionalbool

    overwrite_target

    If target_field is set and the log contains a field of the same name as the target, it will only be overwritten if this is set to true.

    • Default: false
  • optionalstring

    target_field

    If this setting is present, the parsed JSON will be inserted into the log as a sub-object with this name. If a field with the same name already exists, the parser will fail and produce an error.

    • Syntax: literal

Telemetry

This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

  • counter

    processing_errors_total

    The total number of processing errors encountered by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • error_type - The type of the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    events_in_total

    The number of events accepted by this component either from tagged origin like file and uri, or cumulatively from other origins. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • container_name - The name of the container from which the event originates.

    • file - The file from which the event originates.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

    • mode - The connection mode used by the component.

    • peer_addr - The IP from which the event originates.

    • peer_path - The pathname from which the event originates.

    • pod_name - The name of the pod from which the event originates.

    • uri - The sanitized uri from which the event originates.

  • counter

    processed_events_total

    The total number of events processed by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • file - The file that produced the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    events_out_total

    The total number of events emitted by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    processed_bytes_total

    The number of bytes processed by the component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • container_name - The name of the container from which the bytes originate.

    • file - The file from which the bytes originate.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

    • mode - The connection mode used by the component.

    • peer_addr - The IP from which the bytes originate.

    • peer_path - The pathname from which the bytes originate.

    • pod_name - The name of the pod from which the bytes originate.

    • uri - The sanitized uri from which the bytes originate.

How It Works

Invalid JSON

If the value for the specified field is not valid JSON you can control keeping or discarding the event with the drop_invalid option. Setting it to true will discard the event and drop it entirely. Setting it to false will keep the event and pass it through. Note that passing through the event could cause problems and violate assumptions about the structure of your event.

Merge Conflicts

Key Conflicts

Any key present in the decoded JSON will override existing keys in the event.

Object Conflicts

If the decoded JSON includes nested fields it will be deep merged into the event. For example, given the following event:

{
"message": "{"parent": {"child2": "value2"}}",
"parent": {
"child1": "value1"
}
}

Parsing the "message" field would result the following structure:

{
"parent": {
"child1": "value1",
"child2": "value2"
}
}

Notice that the parent.child1 key was preserved.

State

This component is stateless, meaning its behavior is consistent across each input.