program

Remap Transform

The Vector remap transform is the recommended transform for parsing, shaping, and transforming data in Vector. It implements the Vector Remap Language (VRL), an expression-oriented language designed for processing observability data (logs and metrics) in a safe and performant manner. Please refer to the VRL reference when writing VRL scripts.

Configuration

[transforms.my_transform_id]
  type = "remap" # required
  inputs = ["my-source-or-transform-id", "prefix-*"] # required
  source = '''
  . = parse_json!(.message)
  .new_field = "new value"
  .status = to_int!(.status)
  .duration = parse_duration!(.duration, "s")
  .new_name = del(.old_name)
  '''
  • optionalbool

    drop_on_abort

    Drop the event if the VRL program is manually aborted through the abort statement.

    • Default: true
    • View examples
  • optionalbool

    drop_on_error

    Drop the event if the VRL program returns an error at runtime. See Lazy Event Mutation for more info.

    • Default: false
    • View examples
  • commonrequiredstring

    source

    The Vector Remap Language (VRL) program to execute for each event.

    This field accepts a full Vector Remap Language (VRL) program. Please refer to the Vector Remap Language reference for a list of expressions, functions, and examples.

    • Syntax: remap_program
    • View examples

Telemetry

This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

  • counter

    processing_errors_total

    The total number of processing errors encountered by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • error_type - The type of the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    events_in_total

    The number of events accepted by this component either from tagged origin like file and uri, or cumulatively from other origins. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • container_name - The name of the container from which the event originates.

    • file - The file from which the event originates.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

    • mode - The connection mode used by the component.

    • peer_addr - The IP from which the event originates.

    • peer_path - The pathname from which the event originates.

    • pod_name - The name of the pod from which the event originates.

    • uri - The sanitized uri from which the event originates.

  • counter

    processed_events_total

    The total number of events processed by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • file - The file that produced the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    events_out_total

    The total number of events emitted by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    processed_bytes_total

    The number of bytes processed by the component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • container_name - The name of the container from which the bytes originate.

    • file - The file from which the bytes originate.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

    • mode - The connection mode used by the component.

    • peer_addr - The IP from which the bytes originate.

    • peer_path - The pathname from which the bytes originate.

    • pod_name - The name of the pod from which the bytes originate.

    • uri - The sanitized uri from which the bytes originate.

Examples

Parse Syslog logs

Given the following Vector log event:

{
  "message": "<102>1 2020-12-22T15:22:31.111Z vector-user.biz su 2666 ID389 - Something went wrong"
}

And the following configuration:

vector.toml
[transforms.remap]
type = "remap"
source = ". |= parse_syslog!(.message)"

The following Vector log event will be output:

{
  "appname": "su",
  "facility": "ntp",
  "hostname": "vector-user.biz",
  "message": "Something went wrong",
  "msgid": "ID389",
  "procid": 2666,
  "severity": "info",
  "timestamp": "2020-12-22T15:22:31.111Z",
  "version": 1
}

How It Works

Lazy Event Mutation

When you make changes to an event through VRL's path assignment syntax, the change is not immediately applied to the actual event. If the program fails to run to completion, any changes made until that point are dropped, and the event is kept in its original state.

If you want to make sure your event is changed as expected, you have to rewrite your program to never fail at runtime (the compiler will help you with this).

Alternatively, if you want to ignore/drop events that caused the program to fail, you can set the drop_on_error configuration value to true.

Learn more about Runtime Errors in the Vector Remap Language reference.

State

This component is stateless, meaning its behavior is consistent across each input.

Vector Remap Language

The Vector Remap Language (VRL) is a restrictive, fast, and safe language we designed specifically for mapping observability data. It avoids the need to chain together many fundamental Vector transforms to accomplish rudimentary reshaping of data.

The intent is to offer the same robustness of full language runtime (ex: Lua) without paying the performance or safety penalty.

Learn more about Vector's Remap Language in the Vector Remap Language reference.

Support
Contents
Resources