Lua
Modify event data using the Lua programming language
Warnings
lua
transform is ~60% slower than the remap
transform; we
recommend that you use the remap
transform whenever possible. The lua
transform is
designed solely for edge cases not covered by the remap
transform and not as a go-to option. If the
remap
transform doesn’t cover your use case, please open an issue and let
us know.Configuration
Example configurations
{
"transforms": {
"my_transform_id": {
"type": "lua",
"inputs": [
"my-source-or-transform-id"
],
"version": "1"
}
}
}
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "1"
transforms:
my_transform_id:
type: lua
inputs:
- my-source-or-transform-id
version: "1"
{
"transforms": {
"my_transform_id": {
"type": "lua",
"inputs": [
"my-source-or-transform-id"
],
"metric_tag_values": "single",
"search_dirs": [
"/etc/vector/lua"
],
"source": "function init()\n\tcount = 0\nend\n\nfunction process()\n\tcount = count + 1\nend\n\nfunction timer_handler(emit)\n\temit(make_counter(counter))\n\tcounter = 0\nend\n\nfunction shutdown(emit)\n\temit(make_counter(counter))\nend\n\nfunction make_counter(value)\n\treturn metric = {\n\t\tname = \"event_counter\",\n\t\tkind = \"incremental\",\n\t\ttimestamp = os.date(\"!*t\"),\n\t\tcounter = {\n\t\t\tvalue = value\n\t\t}\n \t}\nend",
"timers": {
"handler": "timer_handler",
"interval_seconds": null
},
"version": "1"
}
}
}
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
metric_tag_values = "single"
search_dirs = [ "/etc/vector/lua" ]
source = """
function init()
\tcount = 0
end
function process()
\tcount = count + 1
end
function timer_handler(emit)
\temit(make_counter(counter))
\tcounter = 0
end
function shutdown(emit)
\temit(make_counter(counter))
end
function make_counter(value)
\treturn metric = {
\t\tname = "event_counter",
\t\tkind = "incremental",
\t\ttimestamp = os.date("!*t"),
\t\tcounter = {
\t\t\tvalue = value
\t\t}
\t}
end"""
version = "1"
[transforms.my_transform_id.timers]
handler = "timer_handler"
transforms:
my_transform_id:
type: lua
inputs:
- my-source-or-transform-id
metric_tag_values: single
search_dirs:
- /etc/vector/lua
source: |-
function init()
count = 0
end
function process()
count = count + 1
end
function timer_handler(emit)
emit(make_counter(counter))
counter = 0
end
function shutdown(emit)
emit(make_counter(counter))
end
function make_counter(value)
return metric = {
name = "event_counter",
kind = "incremental",
timestamp = os.date("!*t"),
counter = {
value = value
}
}
end
timers:
handler: timer_handler
interval_seconds: null
version: "1"
graph
optional objectExtra graph configuration
Configure output for component when generated with graph command
graph.node_attributes
optional objectNode attributes to add to this component’s node in resulting graph
They are added to the node as provided
graph.node_attributes.*
required string literalhooks
required objectLifecycle hooks.
These hooks can be set to perform additional processing during the lifecycle of the transform.
hooks.init
optional string literalThe function called when the first event comes in, before hooks.process
is called.
It can produce new events using the emit
function.
This can either be inline Lua that defines a closure to use, or the name of the Lua function to call. In both
cases, the closure/function takes a single parameter, emit
, which is a reference to a function for emitting events.
hooks.process
required string literalThe function called for each incoming event.
It can produce new events using the emit
function.
This can either be inline Lua that defines a closure to use, or the name of the Lua function to call. In both
cases, the closure/function takes two parameters. The first parameter, event
, is the event being processed,
while the second parameter, emit
, is a reference to a function for emitting events.
hooks.shutdown
optional string literalThe function called when the transform is stopped.
It can produce new events using the emit
function.
This can either be inline Lua that defines a closure to use, or the name of the Lua function to call. In both
cases, the closure/function takes a single parameter, emit
, which is a reference to a function for emitting events.
inputs
required [string]A list of upstream source or transform IDs.
Wildcards (*
) are supported.
See configuration for more info.
metric_tag_values
optional string literal enumWhen set to single
, metric tag values are exposed as single strings, the
same as they were before this config option. Tags with multiple values show the last assigned value, and null values
are ignored.
When set to full
, all metric tags are exposed as arrays of either string or null
values.
Option | Description |
---|---|
full | All tags are exposed as arrays of either string or null values. |
single | Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored. |
single
search_dirs
optional [string]A list of directories to search when loading a Lua file via the require
function.
If not specified, the modules are looked up in the configuration directories.
source
optional string literalThe Lua program to initialize the transform with.
The program can be used to import external dependencies, as well as define the functions used for the various lifecycle hooks. However, it’s not strictly required, as the lifecycle hooks can be configured directly with inline Lua source for each respective hook.
"function init()\n\tcount = 0\nend\n\nfunction process()\n\tcount = count + 1\nend\n\nfunction timer_handler(emit)\n\temit(make_counter(counter))\n\tcounter = 0\nend\n\nfunction shutdown(emit)\n\temit(make_counter(counter))\nend\n\nfunction make_counter(value)\n\treturn metric = {\n\t\tname = \"event_counter\",\n\t\tkind = \"incremental\",\n\t\ttimestamp = os.date(\"!*t\"),\n\t\tcounter = {\n\t\t\tvalue = value\n\t\t}\n \t}\nend"
"-- external file with hooks and timers defined\nrequire('custom_module')"
timers
optional [object]version
required string literal enumTransform API version.
Specifying this version ensures that backward compatibility is not broken.
Option | Description |
---|---|
1 | Lua transform API version 1. This version is deprecated and will be removed in a future version. |
2 | Lua transform API version 2. |
Outputs
<component_id>
Telemetry
Metrics
linkcomponent_discarded_events_total
counterfilter
transform, or false if due to an error.component_errors_total
countercomponent_received_event_bytes_total
countercomponent_received_events_count
histogramA histogram of the number of events passed in each internal batch in Vector’s internal topology.
Note that this is separate than sink-level batching. It is mostly useful for low level debugging performance issues in Vector due to small internal batches.
component_received_events_total
countercomponent_sent_event_bytes_total
countercomponent_sent_events_total
counterlua_memory_used_bytes
gaugeutilization
gaugeExamples
Add, rename, and remove log fields
Given this event...{
"log": {
"field_to_remove": "remove me",
"field_to_rename": "old value"
}
}
transforms:
my_transform_id:
type: lua
inputs:
- my-source-or-transform-id
version: "2"
hooks:
process: |-
function (event, emit)
-- Add root level field
event.log.field = "new value"
-- Add nested field
event.log.nested = {}
event.log.nested.field = "nested value"
-- Rename field
event.log.renamed_field = event.log.field_to_rename
event.log.field_to_rename = nil
-- Remove fields
event.log.field_to_remove = nil
emit(event)
end
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
[transforms.my_transform_id.hooks]
process = """
function (event, emit)
\t-- Add root level field
\tevent.log.field = "new value"
\t-- Add nested field
\tevent.log.nested = {}
\tevent.log.nested.field = "nested value"
\t-- Rename field
\tevent.log.renamed_field = event.log.field_to_rename
\tevent.log.field_to_rename = nil
\t-- Remove fields
\tevent.log.field_to_remove = nil
\temit(event)
end"""
{
"transforms": {
"my_transform_id": {
"type": "lua",
"inputs": [
"my-source-or-transform-id"
],
"version": "2",
"hooks": {
"process": "function (event, emit)\n\t-- Add root level field\n\tevent.log.field = \"new value\"\n\t-- Add nested field\n\tevent.log.nested = {}\n\tevent.log.nested.field = \"nested value\"\n\t-- Rename field\n\tevent.log.renamed_field = event.log.field_to_rename\n\tevent.log.field_to_rename = nil\n\t-- Remove fields\n\tevent.log.field_to_remove = nil\n\temit(event)\nend"
}
}
}
}
{
"field": "new value",
"nested": {
"field": "nested value"
},
"renamed_field": "old value"
}
Add, rename, remove metric tags
Given this event...{
"metric": {
"counter": {
"value": 2
},
"kind": "incremental",
"name": "logins",
"tags": {
"tag_to_remove": "remove me",
"tag_to_rename": "old value"
}
}
}
transforms:
my_transform_id:
type: lua
inputs:
- my-source-or-transform-id
version: "2"
hooks:
process: |-
function (event, emit)
-- Add tag
event.metric.tags.tag = "new value"
-- Rename tag
event.metric.tags.renamed_tag = event.log.tag_to_rename
event.metric.tags.tag_to_rename = nil
-- Remove tag
event.metric.tags.tag_to_remove = nil
emit(event)
end
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
[transforms.my_transform_id.hooks]
process = """
function (event, emit)
\t-- Add tag
\tevent.metric.tags.tag = "new value"
\t-- Rename tag
\tevent.metric.tags.renamed_tag = event.log.tag_to_rename
\tevent.metric.tags.tag_to_rename = nil
\t-- Remove tag
\tevent.metric.tags.tag_to_remove = nil
\temit(event)
end"""
{
"transforms": {
"my_transform_id": {
"type": "lua",
"inputs": [
"my-source-or-transform-id"
],
"version": "2",
"hooks": {
"process": "function (event, emit)\n\t-- Add tag\n\tevent.metric.tags.tag = \"new value\"\n\t-- Rename tag\n\tevent.metric.tags.renamed_tag = event.log.tag_to_rename\n\tevent.metric.tags.tag_to_rename = nil\n\t-- Remove tag\n\tevent.metric.tags.tag_to_remove = nil\n\temit(event)\nend"
}
}
}
}
{
"counter": {
"value": 2
},
"kind": "incremental",
"name": "logins",
"tags": {
"renamed_tag": "old value",
"tag": "new value"
}
}
Drop an event
Given this event...{
"log": {
"field_to_remove": "remove me",
"field_to_rename": "old value"
}
}
transforms:
my_transform_id:
type: lua
inputs:
- my-source-or-transform-id
version: "2"
hooks:
process: |-
function (event, emit)
-- Drop event entirely by not calling the `emit` function
end
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
[transforms.my_transform_id.hooks]
process = """
function (event, emit)
\t-- Drop event entirely by not calling the `emit` function
end"""
{
"transforms": {
"my_transform_id": {
"type": "lua",
"inputs": [
"my-source-or-transform-id"
],
"version": "2",
"hooks": {
"process": "function (event, emit)\n\t-- Drop event entirely by not calling the `emit` function\nend"
}
}
}
}
Iterate over log fields
Given this event...{
"log": {
"value_to_keep": "keep",
"value_to_remove": "-"
}
}
transforms:
my_transform_id:
type: lua
inputs:
- my-source-or-transform-id
version: "2"
hooks:
process: |-
function (event, emit)
-- Remove all fields where the value is "-"
for f, v in pairs(event) do
if v == "-" then
event[f] = nil
end
end
emit(event)
end
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
[transforms.my_transform_id.hooks]
process = """
function (event, emit)
\t-- Remove all fields where the value is "-"
\tfor f, v in pairs(event) do
\t\tif v == "-" then
\t\t\tevent[f] = nil
\t\tend
\tend
\temit(event)
end"""
{
"transforms": {
"my_transform_id": {
"type": "lua",
"inputs": [
"my-source-or-transform-id"
],
"version": "2",
"hooks": {
"process": "function (event, emit)\n\t-- Remove all fields where the value is \"-\"\n\tfor f, v in pairs(event) do\n\t\tif v == \"-\" then\n\t\t\tevent[f] = nil\n\t\tend\n\tend\n\temit(event)\nend"
}
}
}
}
{
"value_to_keep": "keep"
}
Parse timestamps
Given this event...{
"log": {
"timestamp_string": "2020-04-07 06:26:02.643"
}
}
transforms:
my_transform_id:
type: lua
inputs:
- my-source-or-transform-id
version: "2"
hooks:
process: process
source: >-2
timestamp_pattern = "(%d%d%d%d)[-](%d%d)[-](%d%d) (%d%d):(%d%d):(%d%d).?(%d*)"
function parse_timestamp(str)
local year, month, day, hour, min, sec, millis = string.match(str, timestamp_pattern)
local ms = 0
if millis and millis ~= "" then
ms = tonumber(millis)
end
return {
year = tonumber(year),
month = tonumber(month),
day = tonumber(day),
hour = tonumber(hour),
min = tonumber(min),
sec = tonumber(sec),
nanosec = ms * 1000000
}
end
function process(event, emit)
event.log.timestamp = parse_timestamp(event.log.timestamp_string)
emit(event)
end
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
source = """
timestamp_pattern = "(%d%d%d%d)[-](%d%d)[-](%d%d) (%d%d):(%d%d):(%d%d).?(%d*)"
function parse_timestamp(str)
\tlocal year, month, day, hour, min, sec, millis = string.match(str, timestamp_pattern)
\tlocal ms = 0
\tif millis and millis ~= "" then
\t\tms = tonumber(millis)
\tend
\treturn {
\t\tyear = tonumber(year),
\t\tmonth = tonumber(month),
\t\tday = tonumber(day),
\t\thour = tonumber(hour),
\t\tmin = tonumber(min),
\t\tsec = tonumber(sec),
\t\tnanosec = ms * 1000000
\t}
end
function process(event, emit)
\tevent.log.timestamp = parse_timestamp(event.log.timestamp_string)
\temit(event)
end"""
[transforms.my_transform_id.hooks]
process = "process"
{
"transforms": {
"my_transform_id": {
"type": "lua",
"inputs": [
"my-source-or-transform-id"
],
"version": "2",
"hooks": {
"process": "process"
},
"source": " timestamp_pattern = \"(%d%d%d%d)[-](%d%d)[-](%d%d) (%d%d):(%d%d):(%d%d).?(%d*)\"\n function parse_timestamp(str)\n\tlocal year, month, day, hour, min, sec, millis = string.match(str, timestamp_pattern)\n\tlocal ms = 0\n\tif millis and millis ~= \"\" then\n\t\tms = tonumber(millis)\n\tend\n\treturn {\n\t\tyear = tonumber(year),\n\t\tmonth = tonumber(month),\n\t\tday = tonumber(day),\n\t\thour = tonumber(hour),\n\t\tmin = tonumber(min),\n\t\tsec = tonumber(sec),\n\t\tnanosec = ms * 1000000\n\t}\n end\n function process(event, emit)\n\tevent.log.timestamp = parse_timestamp(event.log.timestamp_string)\n\temit(event)\n end"
}
}
}
{
"timestamp": "2020-04-07 06:26:02.643",
"timestamp_string": "2020-04-07 06:26:02.643"
}
Count the number of logs
Given this event...{
"log": {}
}
transforms:
my_transform_id:
type: lua
inputs:
- my-source-or-transform-id
version: "2"
hooks:
init: init
process: process
shutdown: shutdown
timers:
- interval_seconds: 5
handler: timer_handler
source: |-
function init()
count = 0
end
function process()
count = count + 1
end
function timer_handler(emit)
emit(make_counter(count))
count = 0
end
function shutdown(emit)
emit(make_counter(count))
end
function make_counter(value)
return metric = {
name = "event_counter",
kind = "incremental",
timestamp = os.date("!*t"),
counter = {
value = value
}
}
end
[transforms.my_transform_id]
type = "lua"
inputs = [ "my-source-or-transform-id" ]
version = "2"
source = """
function init()
\tcount = 0
end
function process()
\tcount = count + 1
end
function timer_handler(emit)
\temit(make_counter(count))
\tcount = 0
end
function shutdown(emit)
\temit(make_counter(count))
end
function make_counter(value)
\treturn metric = {
\t\tname = "event_counter",
\t\tkind = "incremental",
\t\ttimestamp = os.date("!*t"),
\t\tcounter = {
\t\t\tvalue = value
\t\t}
\t}
end"""
[transforms.my_transform_id.hooks]
init = "init"
process = "process"
shutdown = "shutdown"
[[transforms.my_transform_id.timers]]
interval_seconds = 5
handler = "timer_handler"
{
"transforms": {
"my_transform_id": {
"type": "lua",
"inputs": [
"my-source-or-transform-id"
],
"version": "2",
"hooks": {
"init": "init",
"process": "process",
"shutdown": "shutdown"
},
"timers": [
{
"interval_seconds": 5,
"handler": "timer_handler"
}
],
"source": "function init()\n\tcount = 0\nend\nfunction process()\n\tcount = count + 1\nend\nfunction timer_handler(emit)\n\temit(make_counter(count))\n\tcount = 0\nend\nfunction shutdown(emit)\n\temit(make_counter(count))\nend\nfunction make_counter(value)\n\treturn metric = {\n\t\tname = \"event_counter\",\n\t\tkind = \"incremental\",\n\t\ttimestamp = os.date(\"!*t\"),\n\t\tcounter = {\n\t\t\tvalue = value\n\t\t}\n\t}\nend"
}
}
}
{
"counter": {
"value": 1
},
"kind": "incremental",
"name": "event_counter",
"tags": {
"renamed_tag": "old value",
"tag": "new value"
}
}
How it works
Event Data Model
process
hook takes an event
as its first argument.
Events are represented as tables in Lua
and follow Vector’s data model exactly. Please refer to
Vector’s data model reference for the event
schema. How Vector’s types map to Lua’s type are covered below.Type Mappings
The correspondence between Vector’s data types and Lua data type is summarized by the following table:
Vector Type | Lua Type | Comment |
---|---|---|
String | string | |
Integer | integer | |
Float | number | |
Boolean | boolean | |
Timestamp | table | There is no dedicated timestamp type in Lua. Timestamps are represented as tables using the convention defined by os.date and os.time . The table representation of a timestamp contains the fields year , month , day , hour , min , sec , nanosec , yday , wday , and isdst . If such a table is passed from Lua to Vector, the fields yday , wday , and isdst can be omitted. In addition to the os.time representation, Vector supports sub-second resolution with a nanosec field in the table. |
Null | empty string | In Lua setting the value of a table field to nil means deletion of this field. In addition, the length operator # does not work in the expected way with sequences containing nulls. Because of that Null values are encoded as empty strings. |
Map | table | |
Array | sequence | Sequences are a special case of tables. Indexes start from 1, following the Lua convention. |
Learning Lua
Search Directories
search_dirs
option that allows you to specify
absolute paths that will be searched when using the
Lua require
function. If this option is not
set, the directories of the configuration files will be used instead.