Lua Transform

The Vector lua transform transform events with a full embedded Lua 5.4 engine.

Warnings

The lua transform is ~60% slower than the remap transform; we recommend that you use the remap transform whenever possible. The lua transform is designed solely for edge cases not covered by the remap transform and not as a go-to option. If the remap transform doesn't cover your use case, please open an issue and let us know.

Configuration

Common
Advanced

vector.toml
vector.yaml
vector.json

[transforms.my_transform_id]
  # General
  type = "lua" # required
  inputs = ["my-source-or-transform-id", "prefix-*"] # required
  version = "2" # required

  # Hooks
        hooks.process = '''
        function (event, emit)
            event.log.field = "value" -- set value of a field
            event.log.another_field = nil -- remove field
            event.log.first, event.log.second = nil, event.log.first -- rename field
            -- Very important! Emit the processed event.
            emit(event)
        end
        '''

5 items

commonrequired

commonrequiredtable
hooks
Configures hooks handlers.
- optionalstring
  init
  A function which is called when the first event comes, before calling hooks.process
  - Syntax: literal
  - View examples
- commonrequiredstring
  process
  A function which is called for each incoming event. It can produce new events using emit function. See Event Data Model for more info.
  - Syntax: literal
  - View examples
- optionalstring
  shutdown
  A function which is called when Vector is stopped. It can produce new events using emit function.
  - Syntax: literal
  - View examples
optional[string]
search_dirs
A list of directories to search when loading a Lua file via the require function. If not specified, the modules are looked up in the directories of Vector's configs. See Search Directories for more info.
- View examples
optionalstring
source
The source which is evaluated when the transform is created.
- Syntax: literal
- View examples
optional[table]
timers
Configures timers which are executed periodically at given interval.
- commonrequiredstring
  handler
  Defines a handler function which is executed periodially at interval_seconds. It can produce new events using emit function.
  - Syntax: literal
  - View examples
- commonrequireduint
  interval_seconds
  Defines the interval at which the timer handler would be executed.
  - View examples
enumcommonrequiredstring
version
Transform API version. Specifying this version ensures that Vector does not break backward compatibility.
- Syntax: literal
- Enum, must be one of: "2"
- View examples

Telemetry

This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

gauge
memory_used_bytes
The total memory currently being used by Vector (in bytes). This metric includes the following tags:
- instance - The Vector instance identified by host and port.
- job - The name of the job producing Vector metrics.
counter
events_in_total
The number of events accepted by this component either from tagged origin like file and uri, or cumulatively from other origins. This metric includes the following tags:
- component_kind - The Vector component kind.
- component_name - The Vector component ID.
- component_type - The Vector component type.
- container_name - The name of the container from which the event originates.
- file - The file from which the event originates.
- instance - The Vector instance identified by host and port.
- job - The name of the job producing Vector metrics.
- mode - The connection mode used by the component.
- peer_addr - The IP from which the event originates.
- peer_path - The pathname from which the event originates.
- pod_name - The name of the pod from which the event originates.
- uri - The sanitized uri from which the event originates.
counter
processed_events_total
The total number of events processed by this component. This metric includes the following tags:
- component_kind - The Vector component kind.
- component_name - The Vector component ID.
- component_type - The Vector component type.
- file - The file that produced the error
- instance - The Vector instance identified by host and port.
- job - The name of the job producing Vector metrics.
counter
processing_errors_total
The total number of processing errors encountered by this component. This metric includes the following tags:
- component_kind - The Vector component kind.
- component_name - The Vector component ID.
- component_type - The Vector component type.
- error_type - The type of the error
- instance - The Vector instance identified by host and port.
- job - The name of the job producing Vector metrics.
counter
events_out_total
The total number of events emitted by this component. This metric includes the following tags:
- component_kind - The Vector component kind.
- component_name - The Vector component ID.
- component_type - The Vector component type.
- instance - The Vector instance identified by host and port.
- job - The name of the job producing Vector metrics.
counter
processed_bytes_total
The number of bytes processed by the component. This metric includes the following tags:
- component_kind - The Vector component kind.
- component_name - The Vector component ID.
- component_type - The Vector component type.
- container_name - The name of the container from which the bytes originate.
- file - The file from which the bytes originate.
- instance - The Vector instance identified by host and port.
- job - The name of the job producing Vector metrics.
- mode - The connection mode used by the component.
- peer_addr - The IP from which the bytes originate.
- peer_path - The pathname from which the bytes originate.
- pod_name - The name of the pod from which the bytes originate.
- uri - The sanitized uri from which the bytes originate.

Examples

Add, rename, & remove log fields

Given the following Vector log event:

{
  "field_to_rename": "old value",
  "field_to_remove": "remove me"
}

And the following configuration:

vector.toml

[transforms.lua]
type = "lua"
hooks.process = '''
function (event, emit)
    -- Add root level field
    event.log.field = "new value"

    -- Add nested field
    event.log.nested.field = "nested value"

    -- Rename field
    event.log.renamed_field = event.log.field_to_rename
    event.log.field_to_rename = nil

    -- Remove fields
    event.log.field_to_remove = nil

    emit(event)
end
'''

The following Vector log event will be output:

{
  "field": "new value",
  "nested": {
    "field": "nested value"
  },
  "renamed_field": "old value"
}

How It Works

Event Data Model

The process hook takes an event as its first argument. Events are represented as tables in Lua and follow Vector's data model exactly. Please refer to Vector's data model reference for the event schema. How Vector's types map to Lua's type are covered below.

Type Mappings

The correspondence between Vector's data types and Lua data type is summarized by the following table:

Vector Type	Lua Type	Comment
`String`	`string`
`Integer`	`integer`
`Float`	`number`
`Boolean`	`boolean`
`Timestamp`	`table`	There is no dedicated timestamp type in Lua. Timestamps are represented as tables using the convention defined by `os.date` and `os.time`. The table representation of a timestamp contains the fields `year`, `month`, `day`, `hour`, `min`, `sec`, `nanosec`, `yday`, `wday`, and `isdst`. If such a table is passed from Lua to Vector, the fields `yday`, `wday`, and `isdst` can be omitted. In addition to the `os.time` representation, Vector supports sub-second resolution with a `nanosec` field in the table.
`Null`	empty string	In Lua setting the value of a table field to `nil` means deletion of this field. In addition, the length operator `#` does not work in the expected way with sequences containing nulls. Because of that `Null` values are encoded as empty strings.
`Map`	`table`
`Array`	`sequence`	Sequences are a special case of tables. Indexes start from 1, following the Lua convention.

Learning Lua

In order to write non-trivial transforms in Lua, one has to have basic understanding of Lua. Because Lua is an easy to learn language, reading a few first chapters of the official book or consulting the manual would suffice.

Search Directories

Vector provides a search_dirs option that allows you to specify absolute paths that will be searched when using the Lua require function. If this option is not set, the directories of the configuration files will be used instead.

State

This component is stateful, meaning its behavior changes based on previous inputs (events). State is not preserved across restarts, therefore state-dependent behavior will reset between restarts and depend on the inputs (events) received since the most recent restart.

#Warnings

#Configuration

#hooks

#init

#process

#shutdown

#search_dirs

#source

#timers

#handler

#interval_seconds

#version

#Telemetry

#memory_used_bytes

#events_in_total

#processed_events_total

#processing_errors_total

#events_out_total

#processed_bytes_total

#Examples

#How It Works

#Event Data Model

#Type Mappings

#Learning Lua

#Search Directories

#State

Warnings

Configuration

hooks

init

process

shutdown

search_dirs

source

timers

handler

interval_seconds

version

Telemetry

memory_used_bytes

events_in_total

processed_events_total

processing_errors_total

events_out_total

processed_bytes_total