Remap Transform
The Vector remap
transform
is the recommended transform for parsing, shaping, and transforming data in
Vector. It implements the Vector Remap
Language (VRL), an expression-oriented
language designed for processing observability data (logs and metrics) in a
safe and performant manner. Please refer to the VRL
reference when writing VRL scripts.
Configuration
- vector.toml
- vector.yaml
- vector.json
[transforms.my_transform_id]type = "remap" # requiredinputs = ["my-source-or-transform-id", "prefix-*"] # requiredsource = '''. = parse_json!(.message).new_field = "new value".status = to_int!(.status).duration = parse_duration!(.duration, "s").new_name = del(.old_name)'''
- optionalbool
drop_on_error
Drop the event if the VRL program returns an error at runtime. See Lazy Event Mutation for more info.
- Default:
false
- View examples
- Default:
- commonrequiredstring
source
The Vector Remap Language (VRL) program to execute for each event.
This field accepts a full Vector Remap Language (VRL) program. Please refer to the Vector Remap Language reference for a list of expressions, functions, and examples.
- Syntax:
remap_program
- View examples
- Syntax:
Telemetry
This component provides the following metrics that can be retrieved through
the internal_metrics
source. See the
metrics section in the
monitoring page for more info.
- counter
processing_errors_total
The total number of processing errors encountered by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.error_type
- The type of the errorinstance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
events_in_total
The total number of events accepted by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_events_total
The total number of events processed by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.file
- The file that produced the errorinstance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
events_out_total
The total number of events emitted by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_bytes_total
The total number of bytes processed by the component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
Examples
Given the following Vector log event:
{"message": "<102>1 2020-12-22T15:22:31.111Z vector-user.biz su 2666 ID389 - Something went wrong"}
And the following configuration:
[transforms.remap]type = "remap"source = ". |= parse_syslog!(.message)"
The following Vector log event will be output:
{"appname": "su","facility": "ntp","hostname": "vector-user.biz","message": "Something went wrong","msgid": "ID389","procid": 2666,"severity": "info","timestamp": "2020-12-22T15:22:31.111Z","version": 1}
How It Works
Lazy Event Mutation
When you make changes to an event through VRL's path assignment syntax, the change is not immediately applied to the actual event. If the program fails to run to completion, any changes made until that point are dropped, and the event is kept in its original state.
If you want to make sure your event is changed as expected, you have to rewrite your program to never fail at runtime (the compiler will help you with this).
Alternatively, if you want to ignore/drop events that caused the program to fail,
you can set the drop_on_error
configuration value to true
.
Learn more about Runtime Errors in the Vector Remap Language reference.
State
This component is stateless, meaning its behavior is consistent across each input.
Vector Remap Language
The Vector Remap Language (VRL) is a restrictive, fast, and safe language we designed specifically for mapping observability data. It avoids the need to chain together many fundamental Vector transforms to accomplish rudimentary reshaping of data.
The intent is to offer the same robustness of full language runtime (ex: Lua) without paying the performance or safety penalty.
Learn more about Vector's Remap Language in the Vector Remap Language reference.