Lua Transform

The Vector lua transform programs logs and metrics

Configuration

[transforms.my_transform_id]
# General
type = "lua" # required
inputs = ["my-source-or-transform-id"] # required
version = "2" # required
# Hooks
hooks.process = """
function (event, emit)
event.log.field = "value" -- set value of a field
event.log.another_field = nil -- remove field
event.log.first, event.log.second = nil, event.log.first -- rename field
-- Very important! Emit the processed event.
emit(event)
end
"""
  • commonrequiredtable

    hooks

    Configures hooks handlers.

    • optionalstring

      init

      A function which is called when the first event comes, before calling hooks.process

      • View examples
    • commonrequiredstring

      process

      A function which is called for each incoming event. It can produce new events using emit function. See Event Data Model for more info.

      • View examples
    • optionalstring

      shutdown

      A function which is called when Vector is stopped. It can produce new events using emit function.

      • View examples
  • optional[string]

    search_dirs

    A list of directories to search when loading a Lua file via the require function. If not specified, the modules are looked up in the directories of Vector's configs. See Search Directories for more info.

    • View examples
  • optionalstring

    source

    The source which is evaluated when the transform is created.

    • View examples
  • optional[table]

    timers

    Configures timers which are executed periodically at given interval.

    • commonrequiredstring

      handler

      Defines a handler function which is executed periodially at interval_seconds. It can produce new events using emit function.

      • View examples
    • commonrequireduint

      interval_seconds

      Defines the interval at which the timer handler would be executed.

      • View examples
    • enumcommonrequiredstring

      version

      Transform API version. Specifying this version ensures that Vector does not break backward compatibility.

      • Enum, must be one of: "2"
      • View examples

    Output

    Telemetry

    This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

    • gauge

      memory_used_bytes

      The total memory currently being used by Vector (in bytes). This metric includes the following tags:

      • instance - The Vector instance identified by host and port.

      • job - The name of the job producing Vector metrics.

    • counter

      processed_events_total

      The total number of events processed by this component. This metric includes the following tags:

      • component_kind - The Vector component kind.

      • component_name - The Vector component ID.

      • component_type - The Vector component type.

      • file - The file that produced the error

      • instance - The Vector instance identified by host and port.

      • job - The name of the job producing Vector metrics.

    • counter

      processing_errors_total

      The total number of processing errors encountered by this component. This metric includes the following tags:

      • component_kind - The Vector component kind.

      • component_name - The Vector component ID.

      • component_type - The Vector component type.

      • error_type - The type of the error

      • instance - The Vector instance identified by host and port.

      • job - The name of the job producing Vector metrics.

    • counter

      processed_bytes_total

      The total number of bytes processed by the component. This metric includes the following tags:

      • component_kind - The Vector component kind.

      • component_name - The Vector component ID.

      • component_type - The Vector component type.

      • instance - The Vector instance identified by host and port.

      • job - The name of the job producing Vector metrics.

    Examples

    Add, rename, & remove log fields

    Given the following Vector event:

    {
    "log": {
    "field_to_rename": "old value",
    "field_to_remove": "remove me"
    }
    }

    And the following configuration:

    [transforms.lua]
    type = "lua"
    hooks.process = """
    function (event, emit)
    -- Add root level field
    event.log.field = "new value"
    -- Add nested field
    event.log.nested.field = "nested value"
    -- Rename field
    event.log.renamed_field = event.log.field_to_rename
    event.log.field_to_rename = nil
    -- Remove fields
    event.log.field_to_remove = nil
    emit(event)
    end
    """

    The following Vector log event will be output:

    {
    "field": "new value",
    "nested": {
    "field": "nested value"
    },
    "renamed_field": "old value"
    }

    How It Works

    Event Data Model

    The process hook takes an event as its first argument. Events are represented as tables in Lua and follow Vector's data model exactly. Please refer to Vector's data model reference for the event schema. How Vector's types map to Lua's type are covered below.

    Type Mappings

    The correspondence between Vector's data types and Lua data type is summarized by the following table:

    Vector TypeLua TypeComment
    Stringstring
    Integerinteger
    Floatnumber
    Booleanboolean
    TimestamptableThere is no dedicated timestamp type in Lua. Timestamps are represented as tables using the convention defined by os.date and os.time. The table representation of a timestamp contains the fields year, month, day, hour, min, sec, nanosec, yday, wday, and isdst. If such a table is passed from Lua to Vector, the fields yday, wday, and isdst can be omitted. In addition to the os.time representation, Vector supports sub-second resolution with a nanosec field in the table.
    Nullempty stringIn Lua setting the value of a table field to nil means deletion of this field. In addition, the length operator # does not work in the expected way with sequences containing nulls. Because of that Null values are encoded as empty strings.
    Maptable
    ArraysequenceSequences are a special case of tables. Indexes start from 1, following the Lua convention.

    Learning Lua

    In order to write non-trivial transforms in Lua, one has to have basic understanding of Lua. Because Lua is an easy to learn language, reading a few first chapters of the official book or consulting the manual would suffice.

    Search Directories

    Vector provides a search_dirs option that allows you to specify absolute paths that will be searched when using the Lua require function. If this option is not set, the directories of the configuration files will be used instead.