Timber Remap Language

Overview

The Timber Remap Language (TRL) is a lean, single-purpose data transformation language that enables you to easily map and reshape observability event data (logs and metrics) without sacrificing performance or safety. TRL occupies a cozy middle ground between stitching together fundamental transforms and using a full-blown runtime like Lua. Guiding principles behind TRL include:

  1. Performance - Beyond extremely fast execution, TRL is designed to prevent Vector operators from writing slow scripts.
  2. Safety - TRL is Rust native and performs compile-time checks at boot time to ensure safety. In addition, TRL's simplicity and lack of complex "footguns" are ideal for collaboration.
  3. Self-documenting - A TRL script's intentions are clear even at first glance because the language is designed to have a very gentle learning curve.

TRL is designed and maintained by Timber and built specifically for processing data within Vector.

How it's used in Vector

TRL is used in the remap transform as well as various options throughout Vector. When an option allows for dynamic values (values derived from event data) then TRL is allowed in a read-only fashion. For example, the aws_s3 sink's key_prefix option allows for TRL functions to called to effectively partition data on AWS s3.

Example

Given the following log event:

{
"message": "{\"message\":\"HEAD /initiatives HTTP/1.1\",\"bytes\":\"117\",\"status\":\"504\",\"duration",\"50.2ms\",\"timestamp\":\"03/Oct/2020:16:11:29 +0000\"}"
}

And the following Timber Remap script:

. = parse_json(.message)
.bytes = to_int(.bytes)
.duration = parse_duration(.duration, "s")
.status = to_int(.status)
.timestamp = parse_timestamp(.timestamp)

The following output will be produced:

{
"message": "HEAD /initiatives HTTP/1.1",
"bytes": 117,
"duration": {
"seconds": 0.0502
},
"status": 504,
"timestamp": "2020-10-03T16:11:29Z"
}

To use the Timber Remap Language in your pipeline see the remap transform.

Coerce Functions

  • format_number

    format_number(integer | float, scale = integer, decimal_separator = string, grouping_separator = string) :: string

    Returns a string representation of the given number.

    • Arguments

      • required
        value<integer or float>

        The number to format as a string.

      • scale<integer>

        The number of decimal places to display.

      • decimal_separator<string>

        The character to use between the whole and decimal parts of the number.

      • grouping_separator<string>

        The character to use between each thousands part of the number.

    • Example

      Given the following Vector event:

      {
      "number": 1234567.89
      }

      And the following Timber Remap script:

      .formatted = format_number(.number, 3, decimal_separator=".", grouping_separator=",")

      The following output will be produced:

      {
      "number": 1234567.89,
      "formatted": "1,234,567.890"
      }
  • parse_timestamp

    parse_timestamp(string, string, default = string | timestamp) :: timestamp

    Parses a string representing a timestamp using a provided format string. If the string is unable to be parsed, and a default is specified, use this. default can be either a string or a timestamp. If a string, it is parsed and the result returned. If a timestamp, this is returned.

    The format string used is specified by the Chrono library).

    • Arguments

      • required
        value<string>

        The text of the timestamp.

      • required
        format<string>

        The format string the timestamp text is expected to be in.

      • default<string or timestamp>

        If value cannot be converted to a timestamp, if default is a string attempt to parse this. If it is a timestamp, return this timestamp.

    • Example

      Given the following Vector event:

      {
      "timestamp_bad": "I am not a timestamp",
      "timestamp_good": "10-Oct-2020 16:00"
      }

      And the following Timber Remap script:

      .timestamp = parse_timestamp(.timestamp_bad, format="%v %R", default=.timestamp_good)

      The following output will be produced:

      {
      "timestamp": "10-Oct-2020 16:00:00",
      "timestamp_bad": "I am not a timestamp",
      "timestamp_good": "10-Oct-2020 16:00"
      }
  • to_float

    to_float(float | integer | boolean | string) :: float

    Returns a float whose text representation is string.

    • Arguments

      • required
        value<float, integer, boolean, and string>

        The string that is to be converted to a float. Must be the string representation of a float, otherwise an ArgumentError will be raised.

    • Example

      Given the following Vector event:

      {
      "float": "3.14"
      }

      And the following Timber Remap script:

      .float = to_float(.float)

      The following output will be produced:

      {
      "float": 3.14
      }
  • to_int

    to_int(integer | float | boolean | string) :: integer

    Returns an integer whose text representation is string.

    • Arguments

      • required
        value<integer, float, boolean, and string>

        The string that is to be converted to an int. Must be the string representation of an int, otherwise, an ArgumentError will be raised.

    • Example

      Given the following Vector event:

      {
      "integer": "2"
      }

      And the following Timber Remap script:

      .integer = to_int(.integer)

      The following output will be produced:

      {
      "integer": 2
      }
  • to_string

    to_string(boolean | integer | float | string | timestamp | regex | null, default = boolean | integer | float | string | timestamp | regex | null) :: string

    Returns the string representation of the first parameter. If this parameter is an error, then the second parameter is returned.

    • Arguments

      • required
        value<boolean, integer, float, string, timestamp, regex, and null>

        The value to return a string representation of.

      • default<boolean, integer, float, string, timestamp, regex, and null>

        If the value parameter errors, return this parameter instead.

    • Example

      Given the following Vector event:

      {
      "message": 52
      }

      And the following Timber Remap script:

      .message = to_string(.message)

      The following output will be produced:

      {
      "message": "52"
      }
  • to_timestamp

    to_timestamp(string | integer | timestamp, default = string | integer | timestamp) :: timestamp

    Returns a timestamp from the parameters. If parameter is string, the timestamp is parsed in these formats. If parameter is integer, the timestamp is takes as that number of seconds after January 1st 1970.

    • Arguments

      • required
        value<string, integer, and timestamp>

        The value that is to be converted to a timestamp. If a string, must be a valid representation of a timestamp, and no default exists, an ArgumentError will be raised.

      • default<string, integer, and timestamp>

        If value cannot be converted to a timestamp, attempt to convert default to a timestamp. If a string, must be a valid representation of a timestamp, otherwise an ArgumentError will be raised.

    • Example

      Given the following Vector event:

      {
      "date": "2020-10-21T16:00:00Z"
      }

      And the following Timber Remap script:

      .date = to_timestamp(.date)

      The following output will be produced:

      {
      "date": "2020-10-21T16:00:00Z"
      }

Event Functions

  • assert

    assert(boolean, string) :: null
    Checks a given condition. If that condition evaluates to false the event is aborted with
    an error message provided.
    • Arguments

      • required
        condition<boolean>

        The condition to check.

      • required
        message<string>

        Should condition be false, message will be reported as the failure message.

    • Example

      Given the following Vector event:

      {
      "foo": "bar"
      }

      And the following Timber Remap script:

      assert(.foo == "buzz", message = "Foo must be buzz!")

      The following output will be produced:

      {
      "error": "Foo must be buzz!"
      }
  • del

    del(string) :: null

    Removed the fields specified by the given paths from the root event object. Multiple fields can be specified.

    • Arguments

      • required
        paths<string>

        The paths of the fields to delete.

    • Example

      Given the following Vector event:

      {
      "field1": 1,
      "field2": 2,
      "field3": 3
      }

      And the following Timber Remap script:

      del(.field1, .field3)

      The following output will be produced:

      {
      "field2": 2
      }
  • exists

    exists(path) :: boolean

    Checks if the given path exists. Nested paths and arrays can also be checked.

    • Arguments

      • required
        path<path>

        The paths of the fields to check.

    • Example

      Given the following Vector event:

      {
      "field": 1
      }

      And the following Timber Remap script:

      .exists = exists(.field)
      .doesntexist = exists(.field2)

      The following output will be produced:

      {
      "exists": true,
      "doesntexist": false
      }
  • only_fields

    only_fields(string) :: null

    Remove any fields that are not specified by the given paths from the root event object. Multiple fields can be specified.

    • Arguments

      • required
        paths<string>

        The paths of the fields to keep.

    • Example

      Given the following Vector event:

      {
      "field1": 1,
      "field2": 2,
      "field3": 3
      }

      And the following Timber Remap script:

      only_fields(.field1, .field3)

      The following output will be produced:

      {
      "field1": 1,
      "field3": 3
      }

Hash Functions

  • md5

    md5(string) :: string

    Calculates an md5 hash of a given string.

    • Arguments

      • required
        value<string>

        The string to calculate the hash for.

    • Example

      Given the following Vector event:

      {
      "text": "foo"
      }

      And the following Timber Remap script:

      .hash = md5(.text)

      The following output will be produced:

      {
      "hash": "acbd18db4cc2f85cedef654fccc4a4d8"
      }
  • sha1

    sha1(string) :: string

    Calculates a sha1 hash of a given string.

    • Arguments

      • required
        value<string>

        The string to calculate the hash for.

    • Example

      Given the following Vector event:

      {
      "text": "foo"
      }

      And the following Timber Remap script:

      .hash = sha1(.text)

      The following output will be produced:

      {
      "hash": "0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33"
      }
  • sha2

    sha2(string, variant = string) :: string

    Calculates a sha2 hash of a given string.

    • Arguments

      • required
        value<string>

        The string to calculate the hash for.

      • variant<string>

        The variant of the algorithm to use. The allowed variants are:

        • SHA-224
        • SHA-256
        • SHA-384
        • SHA-512
        • SHA-512/224
        • SHA-512/256
    • Example

      Given the following Vector event:

      {
      "text": "foo"
      }

      And the following Timber Remap script:

      .hash = sha2(.text, variant = "SHA-512/224")

      The following output will be produced:

      {
      "hash": "d68f258d37d670cfc1ec1001a0394784233f88f056994f9a7e5e99be"
      }
  • sha3

    sha3(string, variant = string) :: string

    Calculates a sha3 hash of a given string.

    • Arguments

      • required
        value<string>

        The string to calculate the hash for.

      • variant<string>

        The variant of the algorithm to use. The allowed variants are:

        • SHA3-224
        • SHA3-256
        • SHA3-384
        • SHA3-512
    • Example

      Given the following Vector event:

      {
      "text": "foo"
      }

      And the following Timber Remap script:

      .hash = sha3(.text, variant = "SHA3-224")

      The following output will be produced:

      {
      "hash": "f4f6779e153c391bbd29c95e72b0708e39d9166c7cea51d1f10ef58a"
      }

Networking Functions

  • ip_cidr_contains

    ip_cidr_contains(string, string) :: boolean

    Returns true if the given ip address is contained within the block referenced by the given CIDR mask.

    • Arguments

      • required
        cidr<string>

        The CIDR mask - either v4 or v6.

      • required
        value<string>

        The ip address - either a v4 or a v6 address.

    • Example

      Given the following Vector event:

      {
      "address": "192.168.10.32"
      }

      And the following Timber Remap script:

      .cidr = ip_cidr_contains(.address, "192.168.0.0/16")

      The following output will be produced:

      {
      "address": "192.168.10.32",
      "cidr": true
      }
  • ip_subnet

    ip_subnet(string, string) :: string

    Extracts the subnet address from a given IP address using a supplied subnet mask or prefix length. Works with both IPv4 and IPv6 addresses. The IP version for the mask must be the same as the supplied address.

    • Arguments

      • required
        value<string>

        The ip address - either a v4 or a v6 address.

      • required
        subnet<string>

        The subnet to extract from the ip address. This can be either in the form of a prefix length, eg. /8 or as a net mask - 255.255.0.0. The net mask can be either an IPv4 or IPv6 address.

    • Example

      Given the following Vector event:

      {
      "address": "192.168.10.32"
      }

      And the following Timber Remap script:

      .subnet = ip_subnet(.address, "255.255.255.0")

      The following output will be produced:

      {
      "address": "192.168.10.32",
      "subnet": "192.168.10.0"
      }
  • ip_to_ipv6

    ip_to_ipv6(string) :: string

    Converts an address to an IPv6 address. If the parameter is already an IPv6 address it is passed through untouched. IPv4 addresses are converted to IPv4 mapped IPv6 addresses.

    • Arguments

      • required
        value<string>

        The ip address to convert to IPv6.

    • Example

      Given the following Vector event:

      {
      "address": "192.168.10.32"
      }

      And the following Timber Remap script:

      .v6 = ip_to_ipv6(.address)

      The following output will be produced:

      {
      "address": "192.168.10.32",
      "v6": "::ffff:192.168.10.32"
      }
  • ipv6_to_ipv4

    ipv6_to_ipv4(string) :: string

    Converts an IPv4 mapped IPv6 address to an IPv4 address. This function will raise an error if the input address is not a compatible address.

    • Arguments

      • required
        value<string>

        The IPv4 mapped IPv6 address to convert.

    • Example

      Given the following Vector event:

      {
      "address": "::ffff:192.168.0.1"
      }

      And the following Timber Remap script:

      .v4 = ipv6_to_ipv4(.address)

      The following output will be produced:

      {
      "address": "::ffff:192.168.0.1",
      "v4": "192.168.0.1"
      }

Numeric Functions

  • ceil

    ceil(integer | float, precision = integer) :: timestamp

    Rounds the given number up to the given precision of decimal places.

    • Arguments

      • required
        value<integer or float>

        The number to round up.

      • precision<integer>

        The number of decimal places to round to.

    • Example

      Given the following Vector event:

      {
      "number": 4.345
      }

      And the following Timber Remap script:

      .ceil = ceil(.number, precision = 2)

      The following output will be produced:

      {
      "number": 4.345,
      "ceil": 4.35
      }
  • floor

    floor(integer | float, precision = integer) :: timestamp

    Rounds the given number down to the given precision of decimal places.

    • Arguments

      • required
        value<integer or float>

        The number to round down.

      • precision<integer>

        The number of decimal places to round to.

    • Example

      Given the following Vector event:

      {
      "number": 4.345
      }

      And the following Timber Remap script:

      .floor = floor(.number, precision = 2)

      The following output will be produced:

      {
      "number": 4.345,
      "floor": 4.34
      }
  • round

    round(integer | float, precision = integer) :: timestamp

    Rounds the given number to the given number of decimal places. Rounds up or down depending on which is nearest. Numbers that are half way (5) are rounded up.

    • Arguments

      • required
        value<integer or float>

        The number to round.

      • precision<integer>

        The number of decimal places to round to.

    • Example

      Given the following Vector event:

      {
      "number": 4.345
      }

      And the following Timber Remap script:

      .floor = floor(.number, precision = 2)

      The following output will be produced:

      {
      "number": 4.345,
      "floor": 4.35
      }

Object Functions

  • flatten

    flatten(array | map) :: array | map

    Returns a nested array or map that has been flattened to a single level.

    • Arguments

      • required
        value<array or map>

        The array or map to flatten.

    • Example

      Given the following Vector event:

      {
      "array": [
      1,
      [
      2,
      3,
      4
      ],
      [
      5,
      [
      6,
      7
      ],
      8
      ],
      9
      ]
      }

      And the following Timber Remap script:

      .flattened = flatten(.array)

      The following output will be produced:

      {
      "array": [
      1,
      [
      2,
      3,
      4
      ],
      [
      5,
      [
      6,
      7
      ],
      8
      ],
      9
      ],
      "flattened": [
      1,
      2,
      3,
      4,
      5,
      6,
      7,
      8,
      9
      ]
      }

Parse Functions

  • includes

    includes(array, any) :: boolean

    Determines whether an item is contained in an array. The item can be of any type and arrays can be of mixed types.

    • Arguments

      • required
        value<array>

        The array

      • required
        item<any>

        The item to check

    • Example

      Given the following Vector event:

      {
      "fruits": [
      "apple",
      "orange",
      "banana"
      ]
      }

      And the following Timber Remap script:

      .includes_banana = includes(.fruits, "banana")
      .includes_mango = includes(.fruits, "mango")

      The following output will be produced:

      {
      "includes_banana": true,
      "includes_mango": false
      }
  • parse_aws_alb_log

    parse_aws_alb_log(string) :: map

    Parses a Elastic Load Balancer Access log into it's constituent components.

    • Arguments

      • required
        value<string>

        Access log of the Application Load Balancer.

    • Example

      Given the following Vector event:

      {
      "log": "http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\""
      }

      And the following Timber Remap script:

      .parsed = parse_aws_alb_log(.log)

      The following output will be produced:

      {
      "log": "http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\"",
      "parsed": {
      "type": "http",
      "timestamp": "2018-11-30T22:23:00.186641Z",
      "elb": "app/my-loadbalancer/50dc6c495c0c9188",
      "client_host": "192.168.131.39:2817",
      "target_host": null,
      "request_processing_time": 0.0,
      "target_processing_time": 0.001,
      "response_processing_time": 0.0,
      "elb_status_code": "200",
      "target_status_code": "200",
      "received_bytes": 34,
      "sent_bytes": 366,
      "request_method": "GET",
      "request_url": "http://www.example.com:80/",
      "request_protocol": "HTTP/1.1",
      "user_agent": "curl/7.46.0",
      "ssl_cipher": null,
      "ssl_protocol": null,
      "target_group_arn": "arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067",
      "trace_id": "Root=1-58337364-23a8c76965a2ef7629b185e3",
      "domain_name": null,
      "chosen_cert_arn": null,
      "matched_rule_priority": "0",
      "request_creation_time": "2018-11-30T22:22:48.364000Z",
      "actions_executed": "forward",
      "redirect_url": null,
      "error_reason": null,
      "target_port_list": [
      ],
      "target_status_code_list": [
      ],
      "classification": null,
      "classification_reason": null
      }
      }
  • parse_aws_vpc_flow_log

    parse_aws_vpc_flow_log(string, format = string) :: map

    Parses a [VPC Flow Logs](urls.aws_vpc_flow_logs) into it's constituent components.

    • Arguments

      • required
        value<string>

        VPC Flow Log.

      • format<string>

        VPC Flow Log format.

    • Example

      Given the following Vector event:

      {
      "log": "2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA"
      }

      And the following Timber Remap script:

      .parsed = parse_aws_vpc_flow_log(.log)

      The following output will be produced:

      {
      "log": "2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA",
      "parsed": {
      "version": 2,
      "account_id": 123456789010,
      "interface_id": "eni-1235b8ca123456789",
      "srcaddr": null,
      "dstaddr": null,
      "srcport": null,
      "dstport": null,
      "protocol": null,
      "packets": null,
      "bytes": null,
      "start": 1431280876,
      "end": 1431280934,
      "action": null,
      "log_status": "NODATA"
      }
      }
  • parse_duration

    parse_duration(string, string) :: float

    Parses a string representing a duration and returns a number of this duration in another specified unit.

    Available units:

    • ns - nanoseconds (1 billion nanoseconds in a second)
    • us - microseconds (1 million microseconds in a second)
    • µs - microseconds (1 million microseconds in a second)
    • ms - milliseconds (1 thousand microseconds in a second)
    • cs - centisecond (100 centiseconds in a second)
    • ds - decisecond (10 deciseconds in a second)
    • s - second
    • m - minute (60 seconds in a minute)
    • h - hour (60 minutes in an hour)
    • d - day (24 hours in a day)
    • Arguments

      • required
        value<string>

        The string of the duration.

      • required
        output<string>

        The string of the duration unit the number should be output as.

    • Example

      Given the following Vector event:

      {
      "duration": "1005ms"
      }

      And the following Timber Remap script:

      .seconds = parse_duration(.duration, "s")

      The following output will be produced:

      {
      "seconds": 1.005
      }
  • parse_grok

    parse_grok(string, string, remove_empty = boolean) :: map

    Parses a string using the Rust grok library. All patterns listed here are supported. It is recommended to use maintained patterns when possible since they will be improved over time by the community.

    • Arguments

      • required
        value<string>

        The string to parse.

      • required
        pattern<string>

        The Grok pattern.

      • remove_empty<boolean>

        If set to true, any patterns that resolve to an empty value will be removed from the result.

    • Example

      Given the following Vector event:

      {
      "message": "2020-10-02T23:22:12.223222Z info Hello world"
      }

      And the following Timber Remap script:

      .grokked = parse_grok(.message, "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}")

      The following output will be produced:

      {
      "message": "2020-10-02T23:22:12.223222Z info Hello world",
      "grokked.timestamp": "2020-10-02T23:22:12.223222Z",
      "grokked.level": "info",
      "grokked.message": "Hello world"
      }
  • parse_json

    parse_json(string) :: boolean | integer | float | string | map | array | null

    Returns an object whose text representation is a JSON payload in string form.

    string must be the string representation of a JSON payload. Otherwise, an ParseError will be raised.

    • Arguments

      • required
        value<string>

        The string representation of the JSON to parse.

    • Example

      Given the following Vector event:

      {
      "message": "{\"key\": \"val\"}"
      }

      And the following Timber Remap script:

      . = parse_json(.message)

      The following output will be produced:

      {
      "key": "val"
      }
  • parse_syslog

    parse_syslog(string) :: map

    Parses a syslog message. The function makes a best effort to parse the various Syslog formats out in the wild. This includes RFC 6587, RFC 5424, RFC 3164, and other common variations (such as the Nginx Syslog style). If parsing fails, Vector will include the entire Syslog line in the message field.

    • Arguments

      • required
        value<string>

        The text containing the syslog message to parse.

    • Example

      Given the following Vector event:

      {
      "message": "<13>1 2020-03-13T20:45:38.119Z dynamicwireless.name non 2426 ID931 [exampleSDID@32473 iut=\"3\" eventSource= \"Application\" eventID=\"1011\"] Try to override the THX port, maybe it will reboot the neural interface!"
      }

      And the following Timber Remap script:

      .parsed = parse_syslog(.message)

      The following output will be produced:

      {
      "message": "<13>1 2020-03-13T20:45:38.119Z dynamicwireless.name non 2426 ID931 [exampleSDID@32473 iut=\"3\" eventSource= \"Application\" eventID=\"1011\"] Try to override the THX port, maybe it will reboot the neural interface!",
      "parsed": {
      "severity": "notice",
      "facility": "user",
      "timestamp": "2020-03-13T20:45:38.119Z",
      "hostname": "dynamicwireless.name",
      "appname": "non",
      "procid": "2426",
      "msgid": "ID931",
      "iut": "3",
      "eventSource": "Application",
      "eventID": "1011",
      "message": "Try to override the THX port, maybe it will reboot the neural interface!"
      }
      }
  • parse_url

    parse_url(string) :: map

    Parses a url into it's constituent components.

    • Arguments

      • required
        value<string>

        The text of the url.

    • Example

      Given the following Vector event:

      {
      "url": "ftp://foo:bar@vector.dev:4343/foobar?hello=world#123"
      }

      And the following Timber Remap script:

      .parsed = parse_url(.url)

      The following output will be produced:

      {
      "url": "ftp://foo:bar@vector.dev:4343/foobar?hello=world#123",
      "parsed": {
      "scheme": "ftp",
      "username": "foo",
      "password": "bar",
      "host": "vector.dev",
      "port": 4343,
      "path": "/foobar",
      "query": {
      "hello": "world"
      },
      "fragment": "123"
      }
      }
  • to_syslog_level

    to_syslog_level(integer) :: string

    Converts a Syslog severity level into its corresponding keyword, i.e. 0 into "emerg", 1 into `"alert", etc.

    • Arguments

      • required
        value<integer>

        The severity level.

    • Example

      Given the following Vector event:

      {
      "severity": "5"
      }

      And the following Timber Remap script:

      .log_level = to_syslog_level(.severity)

      The following output will be produced:

      {
      "level": "notice"
      }
  • to_syslog_severity

    to_syslog_severity(string) :: integer

    Converts a Syslog log level keyword into an integer severity level (0 to 7). Throws an error if the level isn't recognized. The now-deprecated keywords panic, error, and warn are converted to 0, 3, and 4 respectively.

    • Arguments

      • required
        value<string>

        The Syslog level keyword to convert.

    • Example

      Given the following Vector event:

      {
      "string": "alert"
      }

      And the following Timber Remap script:

      .severity = to_syslog_severity(.log_level)

      The following output will be produced:

      {
      "integer": 1
      }

Text Functions

  • compact

    compact(array | map, recursive = boolean, null = boolean, string = boolean, map = boolean, array = boolean) :: array | map

    Compacts an Array or Map by removing empty values. What is considered an empty value can be specified with the parameters, null, string, map, and array. Specify recursive, if recursive structures should also be compacted, the routine will recurse along and Arrays or Maps and compact those structures.

    • Arguments

      • required
        value<array or map>

        The map or array to compact.

      • recursive<boolean>

        Should the compact be recursive.

      • null<boolean>

        Should null be treated as an empty value.

      • string<boolean>

        Should an empty string be treated as an empty value.

      • map<boolean>

        Should an empty map be treated as an empty value.

      • array<boolean>

        Should an empty array be treated as an empty value.

    • Example

      Given the following Vector event:

      {
      "array": [
      "foo",
      "bar",
      "",
      null,
      [
      ],
      "buzz"
      ]
      }

      And the following Timber Remap script:

      .compacted = compact(.array, string = true, array = true, null = true)

      The following output will be produced:

      {
      "array": [
      "foo",
      "bar",
      "",
      null,
      [
      ],
      "buzz"
      ],
      "compacted": [
      "foo",
      "bar",
      "buzz"
      ]
      }
  • contains

    contains(string, string, case_sensitive = boolean) :: boolean

    Searches a string, value to determine if it contains a given substring. The search can be optionally case insensitive.

    • Arguments

      • required
        value<string>

        The text to search.

      • required
        substring<string>

        The substring to search for in value.

      • case_sensitive<boolean>

        Should the match be case sensitive?

    • Example

      Given the following Vector event:

      {
      "message": "The Needle In The Haystack"
      }

      And the following Timber Remap script:

      .contains = contains(.message, "needle", case_sensitive = false)

      The following output will be produced:

      {
      "contains": true
      }
  • downcase

    downcase(string) :: string

    Returns a copy of string that has been converted into lowercase.

    • Arguments

      • required
        value<string>

        The string to convert to lowercase.

    • Example

      Given the following Vector event:

      {
      "message": "Here Is A Message"
      }

      And the following Timber Remap script:

      .message = downcase(.message)

      The following output will be produced:

      {
      "message": "here is a message"
      }
  • ends_with

    ends_with(string, string, case_sensitive = boolean) :: boolean

    Determines if a given string ends with a given substring. The search can be optionally case insensitive.

    • Arguments

      • required
        value<string>

        The string to search.

      • required
        substring<string>

        The substring value must end with.

      • case_sensitive<boolean>

        Should the match be case sensitive?

    • Example

      Given the following Vector event:

      {
      "message": "The Needle In The Haystack"
      }

      And the following Timber Remap script:

      .contains = ends_with(.message, "the haystack", case_sensitive = false)

      The following output will be produced:

      {
      "message": "The Needle In The Haystack",
      "contains": true
      }
  • format_timestamp

    format_timestamp(timestamp, string) :: string

    Formats a timestamp as a given string. The format string used is specified by the Chrono library.

    • Arguments

      • required
        value<timestamp>

        The timestamp to format as text.

      • required
        format<string>

        The format string

    • Example

      Given the following Vector event:

      {
      "date": "2020-10-21T16:00:00Z"
      }

      And the following Timber Remap script:

      .timestamp = to_timestamp(.date)
      .formatted = format_timestamp(.timestamp, format = "%v %R")

      The following output will be produced:

      {
      "formatted": "10-Oct-2020 16:00"
      }
  • match

    match(string, regex) :: boolean

    Determines whether a string matches the provided pattern.

    • Arguments

      • required
        value<string>

        The value to match.

      • required
        pattern<regex>

        The regular expression pattern to match against.

    • Example

      Given the following Vector event:

      {
      "phrase": "I'm a little teapot"
      }

      And the following Timber Remap script:

      .has_teapot = match(.phrase, /teapot/)

      The following output will be produced:

      {
      "has_teapot": true
      }
  • merge

    merge(string, map, deep = boolean) :: string

    Merges the from map provided into the to path specified, which must specify an existing map. If a key exists in both maps, the field from the from map is chosen. If deep is specified, if a key exists in both maps, and both these fields are also maps merge will recursively merge these fields.

    • Arguments

      • required
        to<string>

        The path of the object to merge into.

      • required
        from<map>

        The object to merge from.

      • deep<boolean>

        If true a deep merge is performed, otherwise only top level fields are merged.

    • Example

      Given the following Vector event:

      {
      "map1": {
      "parent1": {
      "child1": 1,
      "child2": 2
      },
      "parent2": {
      "child3": 3
      }
      },
      "map2": {
      "parent1": {
      "child2": 4,
      "child5": 5
      }
      }
      }

      And the following Timber Remap script:

      merge(.map1, .map2, deep = false)

      The following output will be produced:

      {
      "map1": {
      "parent1": {
      "child2": 4,
      "child5": 5
      },
      "parent2": {
      "child3": 3
      }
      },
      "map2": {
      "parent1": {
      "child2": 4,
      "child5": 5
      }
      }
      }
  • now

    now :: timestamp

    Returns the current timestamp in the Utc timezone.

    • Arguments

      • Example

        Given the following Vector event:

        {
        }

        And the following Timber Remap script:

        .timestamp = now()

        The following output will be produced:

        {
        "timestamp": "21-Oct-2020 20:53"
        }
    • redact

      redact(string, filters = array, redactor = string, patterns = array) :: string

      Obscures sensitive data, such as personal identification numbers or credit card numbers, in Vector event data.

      • Arguments

        • required
          value<string>

          The value that you want to redact.

        • filters<array>

          A list of filters to apply to the input value.

        • redactor<string>

          The redaction method to be applied, with multiple options available.

        • patterns<array>

          A list of patterns to apply. Patterns can be strings or regular expressions; if a string is supplied, Vector searches for exact matches to redact.

      • Example

        Given the following Vector event:

        {
        "credit_card": "9876123454320123"
        }

        And the following Timber Remap script:

        $cc_pattern = /[0-9]{16}/
        .credit_card = redact(.credit_card, filters = ["pattern"], redactor = "full", patterns = [$cc_pattern])

        The following output will be produced:

        {
        "credit_card": "****"
        }
    • replace

      replace(string, regex | string, string, count = integer) :: string

      Replaces any matches of pattern with the provided string. Pattern can be either a fixed string or a regular expression.

      Regular expressions take the form /regex/flags where flags are one of the following:

      • i perform a case insensitive match.
      • m multiline. When enabled ^ and $ match the beginning and end of multiline strings.
      • x ignore whitespace and comments inside the regex.
      • Arguments

        • required
          value<string>

          The original string.

        • required
          pattern<regex or string>

          Replace all matches of this pattern.

        • required
          with<string>

          The string that the matches are replaced with.

        • count<integer>

          The maximum number of replacements to perform. -1 means replace all matches.

      • Example

        Given the following Vector event:

        {
        "text": "Apples and Bananas"
        }

        And the following Timber Remap script:

        .replaced = replace(.text, "and", "not")

        The following output will be produced:

        {
        "text": "Apples and Bananas",
        "replaced": "Apples not Bananas"
        }
    • slice

      slice(array | string, integer, end = integer) :: string

      Returns a slice of the provided string or array between the start and end positions specified.

      If the start and end parameters are negative, they refer to positions counting from the right of the string or array. If end refers to a position that is greater than the length of the string or array a slice up to the end of the string or array is returned.

      • Arguments

        • required
          value<array or string>

          The string or array to slice.

        • required
          start<integer>

          The start position.

        • end<integer>

          The end position.

      • Example

        Given the following Vector event:

        {
        "text": "Supercalifragilisticexpialidocious"
        }

        And the following Timber Remap script:

        .slice = slice(.text, start=5, end=13)

        The following output will be produced:

        {
        "text": "Supercalifragilisticexpialidocious",
        "slice": "califrag"
        }
    • split

      split(string, string | regex, limit = integer) :: string

      Splits the given string whenever a given pattern is matched. If limit is specified, after limit has been reached the remainder of the string is returned unsplit.

      • Arguments

        • required
          value<string>

          The string to split.

        • required
          pattern<string or regex>

          The string is split whenever this pattern is matched.

        • limit<integer>

          The maximum number of substrings to return.

      • Example

        Given the following Vector event:

        {
        "text": "apples and pears and bananas"
        }

        And the following Timber Remap script:

        .split = split(.text, " and ")

        The following output will be produced:

        {
        "text": "apples and pears and bananas",
        "split": [
        "apples",
        "pears",
        "bananas"
        ]
        }
    • starts_with

      starts_with(string, string, case_sensitive = boolean) :: boolean

      Determines if a given string begins with a given substring. The search can be optionally case insensitive.

      • Arguments

        • required
          value<string>

          The string to search.

        • required
          substring<string>

          The substring value must start with.

        • case_sensitive<boolean>

          Should the match be case sensitive?

      • Example

        Given the following Vector event:

        {
        "message": "The Needle In The Haystack"
        }

        And the following Timber Remap script:

        .starts = starts_with(.message, \"the needle\", case_sensitive = false)

        The following output will be produced:

        {
        "message": "The Needle In The Haystack",
        "starts": true
        }
    • strip_ansi_escape_codes

      strip_ansi_escape_codes(string) :: string

      Removes the any ANSI escape codes from the provided string.

      • Arguments

        • required
          value<string>

          The string to strip.

      • Example

        Given the following Vector event:

        {
        "text": "\\e[46mfoo\\e[0m bar"
        }

        And the following Timber Remap script:

        .text = strip_ansi_escape_codes(.text)

        The following output will be produced:

        {
        "text": "foo bar"
        }
    • strip_whitespace

      strip_whitespace(string) :: string

      Trims the whitespace from the start and end of the string. Whitespace is any unicode character with the property "WSpace=yes".

      • Arguments

        • required
          value<string>

          The string to trim.

      • Example

        Given the following Vector event:

        {
        "text": " A sentence. "
        }

        And the following Timber Remap script:

        .trimmed = strip_whitespace(.text)

        The following output will be produced:

        {
        "text": " A sentence. ",
        "slice": "A sentence."
        }
    • tokenize

      tokenize(string) :: array

      Splits the string up into an array of tokens. A token is considered to be:

      • A word surrounded by whitespace.
      • Text delimited by double quotes - "..". Quotes can be included in the token if they are escaped by a backslash - \.
      • Text delimited by square brackets - [..]. Closing square brackets can be included in the token if they are escaped by a backslash - \.
      • Arguments

        • required
          value<string>

          The string to tokenize.

      • Example

        Given the following Vector event:

        {
        "text": "A sentence \"with \\\"a\\\" sentence inside\" and [some brackets]"
        }

        And the following Timber Remap script:

        .tokens = tokenize(.text)

        The following output will be produced:

        {
        "text": "A sentence \"with \\\"a\\\" sentence inside\" and [some brackets]",
        "slice": [
        "A",
        "sentence",
        "with \\\"a\\\" sentence inside",
        "and",
        "some brackets"
        ]
        }
    • truncate

      truncate(string, integer | float, boolean) :: string

      Truncates a string after a given number of characters. If limit is larger than the length of the string, the string is returned unchanged.

      Optionally, an ellipsis (...) is appended if the string does get appended. Note: this does increase the string length by 3, so if you need the result to fit in a certain length, specify the limit as that length minus 3.

      • Arguments

        • required
          value<string>

          The string to truncate.

        • required
          limit<integer or float>

          The number of characters to truncate the string after.

        • required
          ellipsis<boolean>

          If true, an ellipsis (...) is appended should the string be truncated.

      • Example

        Given the following Vector event:

        {
        "text": "A rather long sentence."
        }

        And the following Timber Remap script:

        .truncated = truncate(.text, limit = 11, ellipsis = true)

        The following output will be produced:

        {
        "text": "A rather long sentence.",
        "truncated": "A rather lo..."
        }
    • upcase

      upcase(string) :: string

      Returns a copy of string that has been converted into uppercase.

      • Arguments

        • required
          value<string>

          The string to convert to uppercase.

      • Example

        Given the following Vector event:

        {
        "message": "Here Is A Message"
        }

        And the following Timber Remap script:

        .message = upcase(.message)

        The following output will be produced:

        {
        "message": "HERE IS A MESSAGE"
        }
    • uuid_v4

      uuid_v4 :: string

      Returns a random UUID (Universally Unique Identifier).

      • Arguments

        • Example

          Given the following Vector event:

          {
          }

          And the following Timber Remap script:

          .id = uuid_v4()

          The following output will be produced:

          {
          "id": "1d262f4f-199b-458d-879f-05fd0a5f0683"
          }