program

Remap Transform

The Vector remap transform is the recommended transform for parsing, shaping, and transforming data in Vector. It implements the Vector Remap Language (VRL), an expression-oriented language designed for processing observability data (logs and metrics) in a safe and performant manner. Please refer to the VRL reference when writing VRL scripts.

Configuration

[transforms.my_transform_id]
  type = "remap" # required
  inputs = ["my-source-or-transform-id", "prefix-*"] # required
  source = '''
  . = parse_json!(.message)
  .new_field = "new value"
  .status = to_int!(.status)
  .duration = parse_duration!(.duration, "s")
  .new_name = del(.old_name)
  '''

Telemetry

This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

  • counter

    processing_errors_total

    The total number of processing errors encountered by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • error_type - The type of the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    events_in_total

    The total number of events accepted by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    processed_events_total

    The total number of events processed by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • file - The file that produced the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    events_out_total

    The total number of events emitted by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    processed_bytes_total

    The total number of bytes processed by the component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

Examples

Parse Syslog logs

Given the following Vector log event:

{
  "message": "<102>1 2020-12-22T15:22:31.111Z vector-user.biz su 2666 ID389 - Something went wrong"
}

And the following configuration:

vector.toml
[transforms.remap]
type = "remap"
source = ". |= parse_syslog!(.message)"

The following Vector log event will be output:

{
  "appname": "su",
  "facility": "ntp",
  "hostname": "vector-user.biz",
  "message": "Something went wrong",
  "msgid": "ID389",
  "procid": 2666,
  "severity": "info",
  "timestamp": "2020-12-22T15:22:31.111Z",
  "version": 1
}

How It Works

Lazy Event Mutation

When you make changes to an event through VRL's path assignment syntax, the change is not immediately applied to the actual event. If the program fails to run to completion, any changes made until that point are dropped, and the event is kept in its original state.

If you want to make sure your event is changed as expected, you have to rewrite your program to never fail at runtime (the compiler will help you with this).

Alternatively, if you want to ignore/drop events that caused the program to fail, you can set the drop_on_error configuration value to true.

Learn more about Runtime Errors in the Vector Remap Language reference.

State

This component is stateless, meaning its behavior is consistent across each input.

Vector Remap Language

The Vector Remap Language (VRL) is a restrictive, fast, and safe language we designed specifically for mapping observability data. It avoids the need to chain together many fundamental Vector transforms to accomplish rudimentary reshaping of data.

The intent is to offer the same robustness of full language runtime (ex: Lua) without paying the performance or safety penalty.

Learn more about Vector's Remap Language in the Vector Remap Language reference.

Support
Contents
Resources