program

Lua Transform

The Vector lua transform accepts and outputs log and metric events allowing you to transform events with a full embedded Lua engine.

Configuration

vector.toml
[transforms.my_transform_id]
  # General
  type = "lua" # required
  inputs = ["my-source-id"] # required
  version = "2" # required


  # Hooks
  hooks.process = """
  function (event, emit)
    event.log.field = "value" -- set value of a field
    event.log.another_field = nil -- remove field
    event.log.first, event.log.second = nil, event.log.first -- rename field


    -- Very important! Emit the processed event.
    emit(event)
  end
  """
5 items
  • simpleinlinemodulestringenumcommonrequired

    version

    Transform API version. Specifying this version ensures that Vector does not break backward compatibility.

    • No default
    • Enum, must be one of: "2"
    • View examples
  • module[string]optional

    search_dirs

    A list of directories to search when loading a Lua file via the require function. If not specified, the modules are looked up in the directories of Vector's configs.

    See Search Directories for more info.

    • No default
    • View examples
  • simpleinlinemoduletablecommonrequired

    hooks

    Configures hooks handlers.

    • inlinemodulestringoptional

      init

      A function which is called when the first event comes, before calling hooks.process

      • No default
      • View examples
    • simpleinlinemodulestringcommonrequired

      process

      A function which is called for each incoming event. It can produce new events using emit function.

      • No default
      • View examples
    • inlinemodulestringoptional

      shutdown

      A function which is called when Vector is stopped. It can produce new events using emit function.

      • No default
      • View examples
  • inlinemodule[table]optional

    timers

    Configures timers which are executed periodically at given interval.

    • inlinemodulestringrequired

      handler

      Defines a handler function which is executed periodially at interval_seconds. It can produce new events using emit function.

      • No default
      • View examples
    • inlinemoduleinteger (seconds)required

      interval_seconds

      Defines the interval at which the timer handler would be executed.

      • No default
      • View examples
    • No default
    • View examples
  • modulestringoptional

    source

    The source which is evaluated when the transform is created.

    • No default
    • View examples

Examples

Add, rename, & remove log fields

The following examples demonstrates how to add, rename, and remove a log event's fields:

vector.toml
# ...
hooks.process = """
function (event, emit)
  -- Add root level field
  event.log.field = "new value"


  -- Add nested field
  event.log.nested.field = "nested value"


  -- Rename field
  event.log.new_field = event.log.old_field
  event.log.old_field = nil


  -- Remove fields
  event.log.field = nil


  emit(event)
end
"""

How It Works

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.

Add Fields & Tags

To add log fields and metric tags just set the desired key to the appropriate value:

-- Add root level field
event.log.field = "new value"


-- Add nested field
event.log.nested.field = "nested value"


-- Add tag
event.metric.tags.tag = "new value"

Drop Events

To drop events, simply do not call the emitting function with it. For example:

function (event, emit)
  if not event["message"].match(str, "debug") then
    emit(event)
  end
end

Remove Fields & Tags

You can remove log fields and metric tags by setting them to nil:

-- remove a log field
event.log.field = nil


-- remove a metric tag
event.metric.tags.tag = nil

Representation of Events

Events are represented as tables in Lua. Vector uses externally tagged representation to encode both log and metric events in a consistent fashion:

  • Log events are represented as values of a key named log.
  • Metric events are represented as values of a key named metric.

For instance, a typical log event produced by the stdin source could have been created programmatically using the following code:

event = {
  log = {
    host = "localhost",
    message = "the message",
    timestamp = os.date("!*t")
  }
}

Data Types

The correspondence between Vector's data types and Lua data type is summarized by the following table:

Vector TypeLua TypeComment
Stringstring
Integerinteger
Floatnumber
Booleanboolean
TimestamptableThere is no dedicated timestamp type in Lua. Timestamps are represented as tables using the convention defined by os.date and os.time. The table representation of a timestamp contains the fields year, month, day, hour, min, sec, nanosec, yday, wday, and isdst. If such a table is passed from Lua to Vector, the fields yday, wday, and isdst can be omitted. In addition to the os.time representation, Vector supports sub-second resolution with a nanosec field in the table.
Nullempty stringIn Lua setting the value of a table field to nil means deletion of this field. In addition, the length operator # does not work in the expected way with sequences containing nulls. Because of that Null values are encoded as empty strings.
Maptable
ArraysequenceSequences are a special case of tables. Indexes start from 1, following the Lua convention.

Iterate Over Fields & Tags

To iterate over all fields of an event use the pairs method. For example:

function (event, emit)
  -- Remove all fields where the value is "-"
  for f, v in pairs(event) do
    if v == "-" then
      event[f] = nil
    end
  end


  emit(event)
end

Defining timestamps

To parse a timestamp with an optional milliseconds field, like 2020-04-07 06:26:02.643 or 2020-04-07 06:26:02:

timestamp_pattern = "(%d%d%d%d)[-](%d%d)[-](%d%d) (%d%d):(%d%d):(%d%d).?(%d*)"


function parse_timestamp(str)
  local year, month, day, hour, min, sec, millis = string.match(str, timestamp_pattern)
  local ms = 0
  if millis and millis ~= "" then
    ms = tonumber(millis)
  end
  return {
    year    = tonumber(year),
    month   = tonumber(month),
    day     = tonumber(day),
    hour    = tonumber(hour),
    min     = tonumber(min),
    sec     = tonumber(sec),
    nanosec = ms * 1000000
  }
end


parse_timestamp('2020-04-07 06:26:02.643')
parse_timestamp('2020-04-07 06:26:02')

Search Directories

Vector provides a search_dirs option that allows you to specify absolute paths that will be searched when using the Lua require function. If this option is not set, the directories of the configuration files will be used instead.

Lua Version

Vector uses the rlua Rust crate which currently embeds Lua 5.3.

Learning Lua

In order to write non-trivial transforms in Lua, one has to have basic understanding of Lua. Because Lua is an easy to learn language, reading a few first chapters of the official book or consulting the manual would suffice.

Support
Contents
Resources