Lua
Modify event data using the Lua programming language
status: stable
egress: stream
state: stateful
Transform events with a full embedded Lua 5.4 engine.
Warnings
The
lua
transform is ~60% slower than the remap
transform; we
recommend that you use the remap
transform whenever possible. The lua
transform is
designed solely for edge cases not covered by the remap
transform and not as a go-to option. If the
remap
transform doesn’t cover your use case, please open an issue and let
us know.Configuration
Example configurations
{
"transforms": {
"my_transform_id": {
"type": "lua",
"inputs": [
"my-source-or-transform-id"
],
"hooks": null,
"version": "1"
}
}
}
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "1"
---
transforms:
my_transform_id:
type: lua
inputs:
- my-source-or-transform-id
hooks: null
version: "1"
{
"transforms": {
"my_transform_id": {
"type": "lua",
"inputs": [
"my-source-or-transform-id"
],
"hooks": null,
"search_dirs": [
"/etc/vector/lua"
],
"source": "function init()\n\tcount = 0\nend\n\nfunction process()\n\tcount = count + 1\nend\n\nfunction timer_handler(emit)\n\temit(make_counter(counter))\n\tcounter = 0\nend\n\nfunction shutdown(emit)\n\temit(make_counter(counter))\nend\n\nfunction make_counter(value)\n\treturn metric = {\n\t\tname = \"event_counter\",\n\t\tkind = \"incremental\",\n\t\ttimestamp = os.date(\"!*t\"),\n\t\tcounter = {\n\t\t\tvalue = value\n\t\t}\n \t}\nend",
"timers": {
"handler": "timer_handler",
"interval_seconds": 1
},
"version": "1"
}
}
}
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
search_dirs = [ "/etc/vector/lua" ]
source = """
function init()
\tcount = 0
end
function process()
\tcount = count + 1
end
function timer_handler(emit)
\temit(make_counter(counter))
\tcounter = 0
end
function shutdown(emit)
\temit(make_counter(counter))
end
function make_counter(value)
\treturn metric = {
\t\tname = "event_counter",
\t\tkind = "incremental",
\t\ttimestamp = os.date("!*t"),
\t\tcounter = {
\t\t\tvalue = value
\t\t}
\t}
end"""
version = "1"
[transforms.my_transform_id.timers]
handler = "timer_handler"
interval_seconds = 1
---
transforms:
my_transform_id:
type: lua
inputs:
- my-source-or-transform-id
hooks: null
search_dirs:
- /etc/vector/lua
source: |-
function init()
count = 0
end
function process()
count = count + 1
end
function timer_handler(emit)
emit(make_counter(counter))
counter = 0
end
function shutdown(emit)
emit(make_counter(counter))
end
function make_counter(value)
return metric = {
name = "event_counter",
kind = "incremental",
timestamp = os.date("!*t"),
counter = {
value = value
}
}
end
timers:
handler: timer_handler
interval_seconds: 1
version: "1"
hooks
required objectConfigures hooks handlers.
hooks.init
optional string literalA function which is called when the first event comes, before calling
hooks.process
hooks.process
required string literalA function which is called for each incoming event. It can produce new events using
emit
function.hooks.shutdown
optional string literalA function which is called when Vector is stopped. It can produce new events using
emit
function.inputs
required [string]A list of upstream source or transform
IDs. Wildcards (*
) are supported.
See configuration for more info.
search_dirs
optional [string]A list of directories to search when loading a Lua file via the
require
function. If not specified, the modules are looked up in the directories of Vector’s configs.source
optional string literalThe source which is evaluated when the transform is created.
Examples
"function init()\n\tcount = 0\nend\n\nfunction process()\n\tcount = count + 1\nend\n\nfunction timer_handler(emit)\n\temit(make_counter(counter))\n\tcounter = 0\nend\n\nfunction shutdown(emit)\n\temit(make_counter(counter))\nend\n\nfunction make_counter(value)\n\treturn metric = {\n\t\tname = \"event_counter\",\n\t\tkind = \"incremental\",\n\t\ttimestamp = os.date(\"!*t\"),\n\t\tcounter = {\n\t\t\tvalue = value\n\t\t}\n \t}\nend"
"-- external file with hooks and timers defined\nrequire('custom_module')"
timers
optional [object]Configures timers which are executed periodically at given interval.
Array
object
version
required string literal enumTransform API version. Specifying this version ensures that Vector does not break backward compatibility.
Enum options
string
literal
Option | Description |
---|---|
1 | Lua transform API version 1 |
2 | Lua transform API version 2 |
Outputs
<component_id>
Default output stream of the component. Use this component’s ID as an input to downstream transforms and sinks.
Telemetry
Metrics
linkcomponent_received_event_bytes_total
counterThe number of event bytes accepted by this component either from
tagged origins like file and uri, or cumulatively from other origins.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
container_name
optional
The name of the container from which the data originated.
file
optional
The file from which the data originated.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the data originated.
peer_path
optional
The pathname from which the data originated.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the data originated.
uri
optional
The sanitized URI from which the data originated.
component_received_events_count
histogramA histogram of Vector the number of events passed in each internal batch in Vector’s internal topology.
Note that this is separate than sink-level batching. It is mostly useful for low level debugging
performance issues in Vector due to small internal batches.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
container_name
optional
The name of the container from which the data originated.
file
optional
The file from which the data originated.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the data originated.
peer_path
optional
The pathname from which the data originated.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the data originated.
uri
optional
The sanitized URI from which the data originated.
component_received_events_total
counterThe number of events accepted by this component either from tagged
origins like file and uri, or cumulatively from other origins.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
container_name
optional
The name of the container from which the data originated.
file
optional
The file from which the data originated.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the data originated.
peer_path
optional
The pathname from which the data originated.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the data originated.
uri
optional
The sanitized URI from which the data originated.
component_sent_event_bytes_total
counterThe total number of event bytes emitted by this component.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
host
optional
The hostname of the system Vector is running on.
output
optional
The specific output of the component.
pid
optional
The process ID of the Vector instance.
component_sent_events_total
counterThe total number of events emitted by this component.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
host
optional
The hostname of the system Vector is running on.
output
optional
The specific output of the component.
pid
optional
The process ID of the Vector instance.
events_in_total
counterThe number of events accepted by this component either from tagged
origins like file and uri, or cumulatively from other origins.
This metric is deprecated and will be removed in a future version.
Use
component_received_events_total
instead.component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
container_name
optional
The name of the container from which the data originated.
file
optional
The file from which the data originated.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the data originated.
peer_path
optional
The pathname from which the data originated.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the data originated.
uri
optional
The sanitized URI from which the data originated.
events_out_total
counterThe total number of events emitted by this component.
This metric is deprecated and will be removed in a future version.
Use
component_sent_events_total
instead.component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
host
optional
The hostname of the system Vector is running on.
output
optional
The specific output of the component.
pid
optional
The process ID of the Vector instance.
lua_memory_used_bytes
gaugeThe total memory currently being used by the Lua runtime.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
processed_bytes_total
counterThe number of bytes processed by the component.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
container_name
optional
The name of the container from which the bytes originate.
file
optional
The file from which the bytes originate.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the bytes originate.
peer_path
optional
The pathname from which the bytes originate.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the bytes originate.
uri
optional
The sanitized URI from which the bytes originate.
processed_events_total
counterThe total number of events processed by this component.
This metric is deprecated in place of using
component_received_events_total
and
component_sent_events_total
metrics.component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
processing_errors_total
counterThe total number of processing errors encountered by this component.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
error_type
required
The type of the error
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
utilization
gaugeA ratio from 0 to 1 of the load on a component. A value of 0 would indicate a completely idle component that is simply waiting for input. A value of 1 would indicate a that is never idle. This value is updated every 5 seconds.
component_id
required
The Vector component ID.
component_kind
required
The Vector component kind.
component_name
required
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
required
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
Examples
Add, rename, and remove log fields
Given this event...{
"log": {
"field_to_remove": "remove me",
"field_to_rename": "old value"
}
}
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
[transforms.my_transform_id.hooks]
process = """
function (event, emit)
\t-- Add root level field
\tevent.log.field = "new value"
\t-- Add nested field
\tevent.log.nested.field = "nested value"
\t-- Rename field
\tevent.log.renamed_field = event.log.field_to_rename
\tevent.log.field_to_rename = nil
\t-- Remove fields
\tevent.log.field_to_remove = nil
\temit(event)
end"""
---
transforms:
my_transform_id:
type: lua
inputs:
- my-source-or-transform-id
version: "2"
hooks:
process: |-
function (event, emit)
-- Add root level field
event.log.field = "new value"
-- Add nested field
event.log.nested.field = "nested value"
-- Rename field
event.log.renamed_field = event.log.field_to_rename
event.log.field_to_rename = nil
-- Remove fields
event.log.field_to_remove = nil
emit(event)
end
{
"transforms": {
"my_transform_id": {
"type": "lua",
"inputs": [
"my-source-or-transform-id"
],
"version": "2",
"hooks": {
"process": "function (event, emit)\n\t-- Add root level field\n\tevent.log.field = \"new value\"\n\t-- Add nested field\n\tevent.log.nested.field = \"nested value\"\n\t-- Rename field\n\tevent.log.renamed_field = event.log.field_to_rename\n\tevent.log.field_to_rename = nil\n\t-- Remove fields\n\tevent.log.field_to_remove = nil\n\temit(event)\nend"
}
}
}
}
{
"field": "new value",
"nested": {
"field": "nested value"
},
"renamed_field": "old value"
}
Add, rename, remove metric tags
Given this event...{
"metric": {
"counter": {
"value": 2
},
"kind": "incremental",
"name": "logins",
"tags": {
"tag_to_remove": "remove me",
"tag_to_rename": "old value"
}
}
}
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
[transforms.my_transform_id.hooks]
process = """
function (event, emit)
\t-- Add tag
\tevent.metric.tags.tag = "new value"
\t-- Rename tag
\tevent.metric.tags.renamed_tag = event.log.tag_to_rename
\tevent.metric.tags.tag_to_rename = nil
\t-- Remove tag
\tevent.metric.tags.tag_to_remove = nil
\temit(event)
end"""
---
transforms:
my_transform_id:
type: lua
inputs:
- my-source-or-transform-id
version: "2"
hooks:
process: |-
function (event, emit)
-- Add tag
event.metric.tags.tag = "new value"
-- Rename tag
event.metric.tags.renamed_tag = event.log.tag_to_rename
event.metric.tags.tag_to_rename = nil
-- Remove tag
event.metric.tags.tag_to_remove = nil
emit(event)
end
{
"transforms": {
"my_transform_id": {
"type": "lua",
"inputs": [
"my-source-or-transform-id"
],
"version": "2",
"hooks": {
"process": "function (event, emit)\n\t-- Add tag\n\tevent.metric.tags.tag = \"new value\"\n\t-- Rename tag\n\tevent.metric.tags.renamed_tag = event.log.tag_to_rename\n\tevent.metric.tags.tag_to_rename = nil\n\t-- Remove tag\n\tevent.metric.tags.tag_to_remove = nil\n\temit(event)\nend"
}
}
}
}
{
"counter": {
"value": 2
},
"kind": "incremental",
"name": "logins",
"tags": {
"renamed_tag": "old value",
"tag": "new value"
}
}
Drop an event
Given this event...{
"log": {
"field_to_remove": "remove me",
"field_to_rename": "old value"
}
}
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
[transforms.my_transform_id.hooks]
process = """
function (event, emit)
\t-- Drop event entirely by not calling the `emit` function
end"""
---
transforms:
my_transform_id:
type: lua
inputs:
- my-source-or-transform-id
version: "2"
hooks:
process: |-
function (event, emit)
-- Drop event entirely by not calling the `emit` function
end
{
"transforms": {
"my_transform_id": {
"type": "lua",
"inputs": [
"my-source-or-transform-id"
],
"version": "2",
"hooks": {
"process": "function (event, emit)\n\t-- Drop event entirely by not calling the `emit` function\nend"
}
}
}
}
Iterate over log fields
Given this event...{
"log": {
"value_to_keep": "keep",
"value_to_remove": "-"
}
}
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
[transforms.my_transform_id.hooks]
process = """
function (event, emit)
\t-- Remove all fields where the value is "-"
\tfor f, v in pairs(event) do
\t\tif v == "-" then
\t\t\tevent[f] = nil
\t\tend
\tend
\temit(event)
end"""
---
transforms:
my_transform_id:
type: lua
inputs:
- my-source-or-transform-id
version: "2"
hooks:
process: |-
function (event, emit)
-- Remove all fields where the value is "-"
for f, v in pairs(event) do
if v == "-" then
event[f] = nil
end
end
emit(event)
end
{
"transforms": {
"my_transform_id": {
"type": "lua",
"inputs": [
"my-source-or-transform-id"
],
"version": "2",
"hooks": {
"process": "function (event, emit)\n\t-- Remove all fields where the value is \"-\"\n\tfor f, v in pairs(event) do\n\t\tif v == \"-\" then\n\t\t\tevent[f] = nil\n\t\tend\n\tend\n\temit(event)\nend"
}
}
}
}
{
"value_to_keep": "keep"
}
Parse timestamps
Given this event...{
"log": {
"timestamp_string": "2020-04-07 06:26:02.643"
}
}
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
[transforms.my_transform_id.hooks]
source = """
timestamp_pattern = "(%d%d%d%d)[-](%d%d)[-](%d%d) (%d%d):(%d%d):(%d%d).?(%d*)"
function parse_timestamp(str)
\tlocal year, month, day, hour, min, sec, millis = string.match(str, timestamp_pattern)
\tlocal ms = 0
\tif millis and millis ~= "" then
\t\tms = tonumber(millis)
\tend
\treturn {
\t\tyear = tonumber(year),
\t\tmonth = tonumber(month),
\t\tday = tonumber(day),
\t\thour = tonumber(hour),
\t\tmin = tonumber(min),
\t\tsec = tonumber(sec),
\t\tnanosec = ms * 1000000
\t}
end
function process(event, emit)
\tevent.log.timestamp = parse_timestamp(event.log.timestamp_string)
\temit(event)
end"""
process = "process"
---
transforms:
my_transform_id:
type: lua
inputs:
- my-source-or-transform-id
version: "2"
hooks:
source: >-2
timestamp_pattern = "(%d%d%d%d)[-](%d%d)[-](%d%d) (%d%d):(%d%d):(%d%d).?(%d*)"
function parse_timestamp(str)
local year, month, day, hour, min, sec, millis = string.match(str, timestamp_pattern)
local ms = 0
if millis and millis ~= "" then
ms = tonumber(millis)
end
return {
year = tonumber(year),
month = tonumber(month),
day = tonumber(day),
hour = tonumber(hour),
min = tonumber(min),
sec = tonumber(sec),
nanosec = ms * 1000000
}
end
function process(event, emit)
event.log.timestamp = parse_timestamp(event.log.timestamp_string)
emit(event)
end
process: process
{
"transforms": {
"my_transform_id": {
"type": "lua",
"inputs": [
"my-source-or-transform-id"
],
"version": "2",
"hooks": {
"source": " timestamp_pattern = \"(%d%d%d%d)[-](%d%d)[-](%d%d) (%d%d):(%d%d):(%d%d).?(%d*)\"\n function parse_timestamp(str)\n\tlocal year, month, day, hour, min, sec, millis = string.match(str, timestamp_pattern)\n\tlocal ms = 0\n\tif millis and millis ~= \"\" then\n\t\tms = tonumber(millis)\n\tend\n\treturn {\n\t\tyear = tonumber(year),\n\t\tmonth = tonumber(month),\n\t\tday = tonumber(day),\n\t\thour = tonumber(hour),\n\t\tmin = tonumber(min),\n\t\tsec = tonumber(sec),\n\t\tnanosec = ms * 1000000\n\t}\n end\n function process(event, emit)\n\tevent.log.timestamp = parse_timestamp(event.log.timestamp_string)\n\temit(event)\n end",
"process": "process"
}
}
}
}
{
"timestamp": "2020-04-07 06:26:02.643",
"timestamp_string": "2020-04-07 06:26:02.643"
}
Count the number of logs
Given this event...{
"log": {}
}
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
source = """
function init()
\tcount = 0
end
function process()
\tcount = count + 1
end
function timer_handler(emit)
\temit(make_counter(count))
\tcount = 0
end
function shutdown(emit)
\temit(make_counter(count))
end
function make_counter(value)
\treturn metric = {
\t\tname = "event_counter",
\t\tkind = "incremental",
\t\ttimestamp = os.date("!*t"),
\t\tcounter = {
\t\t\tvalue = value
\t\t}
\t}
end"""
[transforms.my_transform_id.hooks]
init = "init"
process = "process"
shutdown = "shutdown"
[[transforms.my_transform_id.timers]]
interval_seconds = 5
handler = "timer_handler"
---
transforms:
my_transform_id:
type: lua
inputs:
- my-source-or-transform-id
version: "2"
hooks:
init: init
process: process
shutdown: shutdown
timers:
- interval_seconds: 5
handler: timer_handler
source: |-
function init()
count = 0
end
function process()
count = count + 1
end
function timer_handler(emit)
emit(make_counter(count))
count = 0
end
function shutdown(emit)
emit(make_counter(count))
end
function make_counter(value)
return metric = {
name = "event_counter",
kind = "incremental",
timestamp = os.date("!*t"),
counter = {
value = value
}
}
end
{
"transforms": {
"my_transform_id": {
"type": "lua",
"inputs": [
"my-source-or-transform-id"
],
"version": "2",
"hooks": {
"init": "init",
"process": "process",
"shutdown": "shutdown"
},
"timers": [
{
"interval_seconds": 5,
"handler": "timer_handler"
}
],
"source": "function init()\n\tcount = 0\nend\nfunction process()\n\tcount = count + 1\nend\nfunction timer_handler(emit)\n\temit(make_counter(count))\n\tcount = 0\nend\nfunction shutdown(emit)\n\temit(make_counter(count))\nend\nfunction make_counter(value)\n\treturn metric = {\n\t\tname = \"event_counter\",\n\t\tkind = \"incremental\",\n\t\ttimestamp = os.date(\"!*t\"),\n\t\tcounter = {\n\t\t\tvalue = value\n\t\t}\n\t}\nend"
}
}
}
{
"counter": {
"value": 1
},
"kind": "incremental",
"name": "event_counter",
"tags": {
"renamed_tag": "old value",
"tag": "new value"
}
}
How it works
Event Data Model
The
process
hook takes an event
as its first argument.
Events are represented as tables in Lua
and follow Vector’s data model exactly. Please refer to
Vector’s data model reference for the event
schema. How Vector’s types map to Lua’s type are covered below.Type Mappings
The correspondence between Vector’s data types and Lua data type is summarized by the following table:
Vector Type | Lua Type | Comment |
---|---|---|
String | string | |
Integer | integer | |
Float | number | |
Boolean | boolean | |
Timestamp | table | There is no dedicated timestamp type in Lua. Timestamps are represented as tables using the convention defined by os.date and os.time . The table representation of a timestamp contains the fields year , month , day , hour , min , sec , nanosec , yday , wday , and isdst . If such a table is passed from Lua to Vector, the fields yday , wday , and isdst can be omitted. In addition to the os.time representation, Vector supports sub-second resolution with a nanosec field in the table. |
Null | empty string | In Lua setting the value of a table field to nil means deletion of this field. In addition, the length operator # does not work in the expected way with sequences containing nulls. Because of that Null values are encoded as empty strings. |
Map | table | |
Array | sequence | Sequences are a special case of tables. Indexes start from 1, following the Lua convention. |
Learning Lua
In order to write non-trivial transforms in Lua, one has to have
basic understanding of Lua. Because Lua is an easy to learn
language, reading a few first chapters of
the official book or consulting
the manual would suffice.
Search Directories
Vector provides a
search_dirs
option that allows you to specify
absolute paths that will be searched when using the
Lua require
function. If this option is not
set, the directories of the configuration files will be used instead.