VRL example reference

Here you’ll find a comprehensive list of all VRL program examples. These examples demonstrate the breadth of the language and its observability-focused facilities.

Try Using the VRL subcommand

You can run these examples using the vector vrl subcommand with --input (input is newline delimited JSON file representing a list of events). and --program (VRL program) to pass in the example input and program as well as --print-object to show the modified object. The below examples show pretty-printed JSON for the input events, so collapse these to single lines when passing in via --input.

For example: vector vrl --input input.json --program program.vrl --print-object. This closely matches how VRL will receive the input in a running Vector instance.

VRL REPL

Additionally, if vector vrl is run without any arguments, it will spawn a REPL (Read–eval–print loop).

Assuming you have Vector installed, you can run vector vrl to start the REPL. From there, you can type help and press return to get further help.

The REPL behaves nearly identical to the programs you write for your Vector configuration, and can be used to test individual snippets of complex programs before you commit them to your production configuration.

The vector vrl command has many other capabilities, run the command using vector vrl --help to see more information.

Real world examples

Parse Syslog logs

Given this Vector log event...
{
  "message": "\u003c102\u003e1 2020-12-22T15:22:31.111Z vector-user.biz su 2666 ID389 - Something went wrong"
}
...and this VRL program...
. |= parse_syslog!(.message)
...the following log event is output:
{
  "log": {
    "appname": "su",
    "facility": "ntp",
    "hostname": "vector-user.biz",
    "message": "Something went wrong",
    "msgid": "ID389",
    "procid": 2666,
    "severity": "info",
    "timestamp": "2020-12-22T15:22:31.111Z",
    "version": 1
  }
}

Parse key/value (logfmt) logs

Given this Vector log event...
{
  "message": "@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
}
...and this VRL program...
. = parse_key_value!(.message)
...the following log event is output:
{
  "log": {
    "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
    "id": "ConsumerFetcherManager-1382721708341",
    "level": "info",
    "module": "kafka.consumer.ConsumerFetcherManager",
    "msg": "Stopping all fetchers",
    "tag#production": "stopping_fetchers"
  }
}

Parse custom logs

Given this Vector log event...
{
  "message": "2021/01/20 06:39:15 +0000 [error] 17755#17755: *3569904 open() \"/usr/share/nginx/html/test.php\" failed (2: No such file or directory), client: xxx.xxx.xxx.xxx, server: localhost, request: \"GET /test.php HTTP/1.1\", host: \"yyy.yyy.yyy.yyy\""
}
...and this VRL program...
. |= parse_regex!(.message, r'^(?P<timestamp>\d+/\d+/\d+ \d+:\d+:\d+ \+\d+) \[(?P<severity>\w+)\] (?P<pid>\d+)#(?P<tid>\d+):(?: \*(?P<connid>\d+))? (?P<message>.*)$')

# Coerce parsed fields
.timestamp = parse_timestamp(.timestamp, "%Y/%m/%d %H:%M:%S %z") ?? now()
.pid = to_int!(.pid)
.tid = to_int!(.tid)

# Extract structured data
message_parts = split(.message, ", ", limit: 2)
structured = parse_key_value(message_parts[1], key_value_delimiter: ":", field_delimiter: ",") ?? {}
.message = message_parts[0]
. = merge(., structured)
...the following log event is output:
{
  "log": {
    "client": "xxx.xxx.xxx.xxx",
    "connid": "3569904",
    "host": "yyy.yyy.yyy.yyy",
    "message": "open() \"/usr/share/nginx/html/test.php\" failed (2: No such file or directory)",
    "pid": 17755,
    "request": "GET /test.php HTTP/1.1",
    "server": "localhost",
    "severity": "error",
    "tid": 17755,
    "timestamp": "2021-01-20T06:39:15Z"
  }
}

Multiple parsing strategies

Given this Vector log event...
{
  "message": "\u003c102\u003e1 2020-12-22T15:22:31.111Z vector-user.biz su 2666 ID389 - Something went wrong"
}
...and this VRL program...
structured =
  parse_syslog(.message) ??
  parse_common_log(.message) ??
  parse_regex!(.message, r'^(?P<timestamp>\d+/\d+/\d+ \d+:\d+:\d+) \[(?P<severity>\w+)\] (?P<pid>\d+)#(?P<tid>\d+):(?: \*(?P<connid>\d+))? (?P<message>.*)$')
. = merge(., structured)
...the following log event is output:
{
  "log": {
    "appname": "su",
    "facility": "ntp",
    "hostname": "vector-user.biz",
    "message": "Something went wrong",
    "msgid": "ID389",
    "procid": 2666,
    "severity": "info",
    "timestamp": "2020-12-22T15:22:31.111Z",
    "version": 1
  }
}

Modify metric tags

Given this Vector metric event...
{
  "counter": {
    "value": 102
  },
  "kind": "incremental",
  "name": "user_login_total",
  "tags": {
    "email": "vic@vector.dev",
    "host": "my.host.com",
    "instance_id": "abcd1234"
  }
}
...and this VRL program...
.tags.environment = get_env_var!("ENV") # add
.tags.hostname = del(.tags.host) # rename
del(.tags.email)
...the following metric event is output:
{
  "metric": {
    "counter": {
      "value": 102
    },
    "kind": "incremental",
    "name": "user_login_total",
    "tags": {
      "environment": "production",
      "hostname": "my.host.com",
      "instance_id": "abcd1234"
    }
  }
}

Emitting multiple logs from JSON

Given this Vector log event...
{
  "message": "[{\"message\": \"first_log\"}, {\"message\": \"second_log\"}]"
}
...and this VRL program...
. = parse_json!(.message) # sets `.` to an array of objects
...the following log event is output:
[
  {
    "log": {
      "message": "first_log"
    }
  },
  {
    "log": {
      "message": "second_log"
    }
  }
]

Emitting multiple non-object logs from JSON

Given this Vector log event...
{
  "message": "[5, true, \"hello\"]"
}
...and this VRL program...
. = parse_json!(.message) # sets `.` to an array
...the following log event is output:
[
  {
    "log": {
      "message": 5
    }
  },
  {
    "log": {
      "message": true
    }
  },
  {
    "log": {
      "message": "hello"
    }
  }
]

Invalid argument type

Given this Vector log event...
{
  "not_a_string": 1
}
...and this VRL program...
upcase(42)
...you should see this error:
error[E110]: invalid argument type
  ┌─ :1:8
  
1  upcase(42)
          ^^
          
          this expression resolves to the exact type integer
          but the parameter "value" expects the exact type string
  
  = try: ensuring an appropriate type at runtime
  =
  =     42 = string!(42)
  =     upcase(42)
  =
  = try: coercing to an appropriate type and specifying a default value as a fallback in case coercion fails
  =
  =     42 = to_string(42) ?? "default"
  =     upcase(42)
  =
  = see documentation about error handling at https://errors.vrl.dev/#handling
  = learn more about error code 110 at https://errors.vrl.dev/110
  = see language documentation at https://vrl.dev
  = try your code in the VRL REPL, learn more at https://vrl.dev/examples

Unhandled fallible assignment

Given this Vector log event...
{
  "message": "key1=value1 key2=value2"
}
...and this VRL program...
structured = parse_key_value(.message)
...you should see this error:
error[E103]: unhandled fallible assignment
  ┌─ :1:14
  
1  structured = parse_key_value(.message)
   ------------ ^^^^^^^^^^^^^^^^^^^^^^^^^
               
               this expression is fallible because at least one argument's type cannot be verified to be valid
               update the expression to be infallible by adding a `!`: `parse_key_value!(.message)`
               `.message` argument type is `any` and this function expected a parameter `value` of type `string`
   or change this to an infallible assignment:
   structured, err = parse_key_value(.message)
  
  = see documentation about error handling at https://errors.vrl.dev/#handling
  = see functions characteristics documentation at https://vrl.dev/expressions/#function-call-characteristics
  = learn more about error code 103 at https://errors.vrl.dev/103
  = see language documentation at https://vrl.dev
  = try your code in the VRL REPL, learn more at https://vrl.dev/examples

Array examples

append

Appends each item in the items array to the end of the value array.

Append to an array

Source
append([1, 2], [3, 4])
Return
[1,2,3,4]
Learn more about the append function

chunks

Chunks value into slices of length chunk_size bytes.

Split a string into chunks

Source
chunks("abcdefgh", 4)
Return
["abcd","efgh"]
Learn more about the chunks function

Chunks do not respect unicode code point boundaries

Source
chunks("ab你好", 4)
Return
["ab�","�好"]
Learn more about the chunks function

push

Adds the item to the end of the value array.

Push an item onto an array

Source
push([1, 2], 3)
Return
[1,2,3]
Learn more about the push function

Codec examples

decode_base16

Decodes the value (a Base16 string) into its original string.

Decode Base16 data

Source
decode_base16!("796f752068617665207375636365737366756c6c79206465636f646564206d65")
Return
you have successfully decoded me
Learn more about the decode_base16 function

decode_base64

Decodes the value (a Base64 string) into its original string.

Decode Base64 data (default)

Source
decode_base64!("eW91IGhhdmUgc3VjY2Vzc2Z1bGx5IGRlY29kZWQgbWU=")
Return
you have successfully decoded me
Learn more about the decode_base64 function

Decode Base64 data (URL safe)

Source
decode_base64!("eW91IGNhbid0IG1ha2UgeW91ciBoZWFydCBmZWVsIHNvbWV0aGluZyBpdCB3b24ndA==", charset: "url_safe")
Return
you can't make your heart feel something it won't
Learn more about the decode_base64 function

decode_gzip

Decodes the value (a Gzip string) into its original string.

Decode Gzip data

Source
encoded_text = decode_base64!("H4sIAHEAymMAA6vML1XISCxLVSguTU5OLS5OK83JqVRISU3OT0lNUchNBQD7BGDaIAAAAA==")
decode_gzip!(encoded_text)
Return
you have successfully decoded me
Learn more about the decode_gzip function

decode_mime_q

Replaces q-encoded or base64-encoded encoded-word substrings in the value with their original string.

Decode single encoded-word

Source
decode_mime_q!("=?utf-8?b?SGVsbG8sIFdvcmxkIQ==?=")
Return
Hello, World!
Learn more about the decode_mime_q function

Embedded

Source
decode_mime_q!("From: =?utf-8?b?SGVsbG8sIFdvcmxkIQ==?= <=?utf-8?q?hello=5Fworld=40example=2ecom?=>")
Return
From: Hello, World! <hello_world@example.com>
Learn more about the decode_mime_q function

Without charset

Source
decode_mime_q!("?b?SGVsbG8sIFdvcmxkIQ==")
Return
Hello, World!
Learn more about the decode_mime_q function

decode_percent

Decodes a percent-encoded value like a URL.

Percent decode a value

Source
decode_percent("foo%20bar%3F")
Return
foo bar?
Learn more about the decode_percent function

decode_punycode

Decodes a punycode encoded value, such as an internationalized domain name (IDN). This function assumes that the value passed is meant to be used in IDN context and that it is either a domain name or a part of it.

Decode a punycode encoded internationalized domain name

Source
decode_punycode!("www.xn--caf-dma.com")
Return
www.café.com
Learn more about the decode_punycode function

Decode an ASCII only string

Source
decode_punycode!("www.cafe.com")
Return
www.cafe.com
Learn more about the decode_punycode function

Ignore validation

Source
decode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false)
Return
١٠.٦٦.٣٠.٥.
Learn more about the decode_punycode function

decode_snappy

Decodes the value (a Snappy string) into its original string.

Decode Snappy data

Source
encoded_text = decode_base64!("LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg==")
decode_snappy!(encoded_text)
Return
The quick brown fox jumps over 13 lazy dogs.
Learn more about the decode_snappy function

decode_zlib

Decodes the value (a Zlib string) into its original string.

Decode Zlib data

Source
encoded_text = decode_base64!("eJwNy4ENwCAIBMCNXIlQ/KqplUSgCdvXAS41qPMHshCB2R1zJlWIVlR6UURX2+wx2YcuK3kAb9C1wd6dn7Fa+QH9gRxr")
decode_zlib!(encoded_text)
Return
you_have_successfully_decoded_me.congratulations.you_are_breathtaking.
Learn more about the decode_zlib function

decode_zstd

Decodes the value (a Zstandard string) into its original string.

Decode Zstd data

Source
encoded_text = decode_base64!("KLUv/QBY/QEAYsQOFKClbQBedqXsb96EWDax/f/F/z+gNU4ZTInaUeAj82KqPFjUzKqhcfDqAIsLvAsnY1bI/N2mHzDixRQA")
decode_zstd!(encoded_text)
Return
you_have_successfully_decoded_me.congratulations.you_are_breathtaking.
Learn more about the decode_zstd function

encode_base16

Encodes the value to Base16.

Encode to Base16

Source
encode_base16("please encode me")
Return
706c6561736520656e636f6465206d65
Learn more about the encode_base16 function

encode_base64

Encodes the value to Base64.

Encode to Base64 (default)

Source
encode_base64("please encode me")
Return
cGxlYXNlIGVuY29kZSBtZQ==
Learn more about the encode_base64 function

Encode to Base64 (without padding)

Source
encode_base64("please encode me, no padding though", padding: false)
Return
cGxlYXNlIGVuY29kZSBtZSwgbm8gcGFkZGluZyB0aG91Z2g
Learn more about the encode_base64 function

Encode to Base64 (URL safe)

Source
encode_base64("please encode me, but safe for URLs", charset: "url_safe")
Return
cGxlYXNlIGVuY29kZSBtZSwgYnV0IHNhZmUgZm9yIFVSTHM=
Learn more about the encode_base64 function

encode_gzip

Encodes the value to Gzip.

Encode to Gzip

Source
encoded_text = encode_gzip("please encode me")
encode_base64(encoded_text)
Return
H4sIAAAAAAAA/yvISU0sTlVIzUvOT0lVyE0FAI4R4vcQAAAA
Learn more about the encode_gzip function

encode_json

Encodes the value to JSON.

Encode to JSON

Source
.payload = encode_json({"hello": "world"})
Return
{"hello":"world"}
Learn more about the encode_json function

encode_key_value

Encodes the value into key-value format with customizable delimiters. Default delimiters match the logfmt format.

Encode with default delimiters (no ordering)

Source
encode_key_value({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
Return
lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z
Learn more about the encode_key_value function

Encode with default delimiters (fields ordering)

Source
encode_key_value!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
Return
ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345
Learn more about the encode_key_value function

Encode with default delimiters (nested fields)

Source
encode_key_value({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
Return
agent.name=foo event=log log.file.path=my.log
Learn more about the encode_key_value function

Encode with default delimiters (nested fields ordering)

Source
encode_key_value!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
Return
event=log log.file.path=my.log agent.name=foo
Learn more about the encode_key_value function

Encode with custom delimiters (no ordering)

Source
encode_key_value(
	{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"},
	field_delimiter: ",",
	key_value_delimiter: ":"
)
Return
lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z
Learn more about the encode_key_value function

Encode with custom delimiters and flatten boolean

Source
encode_key_value(
	{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "beta": true, "dropped": false},
	field_delimiter: ",",
	key_value_delimiter: ":",
	flatten_boolean: true
)
Return
beta,lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z
Learn more about the encode_key_value function

encode_logfmt

Encodes the value to logfmt.

Encode to logfmt (no ordering)

Source
encode_logfmt({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
Return
lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z
Learn more about the encode_logfmt function

Encode to logfmt (fields ordering)

Source
encode_logfmt!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
Return
ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345
Learn more about the encode_logfmt function

Encode to logfmt (nested fields)

Source
encode_logfmt({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
Return
agent.name=foo event=log log.file.path=my.log
Learn more about the encode_logfmt function

Encode to logfmt (nested fields ordering)

Source
encode_logfmt!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
Return
event=log log.file.path=my.log agent.name=foo
Learn more about the encode_logfmt function

encode_percent

Encodes a value with percent encoding to safely be used in URLs.

Percent encode all non-alphanumeric characters (default)

Source
encode_percent("foo bar?")
Return
foo%20bar%3F
Learn more about the encode_percent function

Percent encode only control characters

Source
encode_percent("foo 	bar", ascii_set: "CONTROLS")
Return
foo %09bar
Learn more about the encode_percent function

encode_proto

Encodes the value into a protocol buffer payload.

Encode to proto

Source
.payload = encode_base64(encode_proto!({"name": "someone", "phones": [{"number": "123456"}]}, "resources/protobuf_descriptor_set.desc", "test_protobuf.Person"))
Return
Cgdzb21lb25lIggKBjEyMzQ1Ng==
Learn more about the encode_proto function

encode_punycode

Encodes a value to punycode. Useful for internationalized domain names (IDN). This function assumes that the value passed is meant to be used in IDN context and that it is either a domain name or a part of it.

Encode an internationalized domain name

Source
encode_punycode!("www.café.com")
Return
www.xn--caf-dma.com
Learn more about the encode_punycode function

Encode an internationalized domain name with mixed case

Source
encode_punycode!("www.CAFé.com")
Return
www.xn--caf-dma.com
Learn more about the encode_punycode function

Encode an ASCII only string

Source
encode_punycode!("www.cafe.com")
Return
www.cafe.com
Learn more about the encode_punycode function

Ignore validation

Source
encode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false)
Return
xn--8hbb.xn--fiba.xn--8hbf.xn--eib.
Learn more about the encode_punycode function

encode_snappy

Encodes the value to Snappy.

Encode to Snappy

Source
encoded_text = encode_snappy!("The quick brown fox jumps over 13 lazy dogs.")
encode_base64(encoded_text)
Return
LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg==
Learn more about the encode_snappy function

encode_zlib

Encodes the value to Zlib.

Encode to Zlib

Source
encoded_text = encode_zlib("please encode me")
encode_base64(encoded_text)
Return
eJwryElNLE5VSM1Lzk9JVchNBQA0RQX7
Learn more about the encode_zlib function

encode_zstd

Encodes the value to Zstandard.

Encode to Zstd

Source
encoded_text = encode_zstd("please encode me")
encode_base64(encoded_text)
Return
KLUv/QBYgQAAcGxlYXNlIGVuY29kZSBtZQ==
Learn more about the encode_zstd function

Coerce examples

to_bool

Coerces the value into a boolean.

Coerce to a Boolean (string)

Source
to_bool!("yes")
Return
true
Learn more about the to_bool function

Coerce to a Boolean (float)

Source
to_bool(0.0)
Learn more about the to_bool function

Coerce to a Boolean (int)

Source
to_bool(0)
Learn more about the to_bool function

Coerce to a Boolean (null)

Source
to_bool(null)
Learn more about the to_bool function

Coerce to a Boolean (Boolean)

Source
to_bool(true)
Return
true
Learn more about the to_bool function

to_float

Coerces the value into a float.

Coerce to a float

Source
to_float!("3.145")
Return
3.145
Learn more about the to_float function

Coerce to a float (timestamp)

Source
to_float(t'2020-12-30T22:20:53.824727Z')
Return
1609366853.824727
Learn more about the to_float function

to_int

Coerces the value into an integer.

Coerce to an int (string)

Source
to_int!("2")
Return
2
Learn more about the to_int function

Coerce to an int (timestamp)

Source
to_int(t'2020-12-30T22:20:53.824727Z')
Return
1609366853
Learn more about the to_int function

to_regex

Coerces the value into a regex.

Coerce to a regex

Source
to_regex("^foo$") ?? r''
Return
^foo$
Learn more about the to_regex function

to_string

Coerces the value into a string.

Coerce to a string (Boolean)

Source
to_string(true)
Return
true
Learn more about the to_string function

Coerce to a string (int)

Source
to_string(52)
Return
52
Learn more about the to_string function

Coerce to a string (float)

Source
to_string(52.2)
Return
52.2
Learn more about the to_string function

Convert examples

from_unix_timestamp

Converts the value integer from a Unix timestamp to a VRL timestamp.

Converts from the number of seconds since the Unix epoch by default. To convert from milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.

Convert from a Unix timestamp (seconds)

Source
from_unix_timestamp!(5)
Return
1970-01-01T00:00:05Z
Learn more about the from_unix_timestamp function

Convert from a Unix timestamp (milliseconds)

Source
from_unix_timestamp!(5000, unit: "milliseconds")
Return
1970-01-01T00:00:05Z
Learn more about the from_unix_timestamp function

Convert from a Unix timestamp (nanoseconds)

Source
from_unix_timestamp!(5000, unit: "nanoseconds")
Return
1970-01-01T00:00:00.000005Z
Learn more about the from_unix_timestamp function

to_syslog_facility

Converts the value, a Syslog facility code, into its corresponding Syslog keyword. For example, 0 into "kern", 1 into "user", etc.

Coerce to a Syslog facility

Source
to_syslog_facility!(4)
Return
auth
Learn more about the to_syslog_facility function

to_syslog_level

Converts the value, a Syslog severity level, into its corresponding keyword, i.e. 0 into "emerg", 1 into "alert", etc.

Coerce to a Syslog level

Source
to_syslog_level!(5)
Return
notice
Learn more about the to_syslog_level function

to_syslog_severity

Converts the value, a Syslog log level keyword, into a Syslog integer severity level (0 to 7).

Coerce to Syslog severity

Source
to_syslog_severity!("alert")
Return
1
Learn more about the to_syslog_severity function

to_unix_timestamp

Converts the value timestamp into a Unix timestamp.

Returns the number of seconds since the Unix epoch by default. To return the number in milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.

Convert to a Unix timestamp (seconds)

Source
to_unix_timestamp(t'2021-01-01T00:00:00+00:00')
Return
1609459200
Learn more about the to_unix_timestamp function

Convert to a Unix timestamp (milliseconds)

Source
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "milliseconds")
Return
1609459200000
Learn more about the to_unix_timestamp function

Convert to a Unix timestamp (nanoseconds)

Source
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "nanoseconds")
Return
1609459200000000000
Learn more about the to_unix_timestamp function

Debug examples

assert

Asserts the condition, which must be a Boolean expression. The program is aborted with message if the condition evaluates to false.

Assertion (true)

Source
assert!("foo" == "foo", message: "\"foo\" must be \"foo\"!")
Return
true
Learn more about the assert function

Assertion (false)

Source
assert!("foo" == "bar", message: "\"foo\" must be \"foo\"!")
Learn more about the assert function

assert_eq

Asserts that two expressions, left and right, have the same value. The program is aborted with message if they do not have the same value.

Successful assertion

Source
assert_eq!(1, 1)
Return
true
Learn more about the assert_eq function

Unsuccessful assertion

Source
assert_eq!(127, [1, 2, 3])
Learn more about the assert_eq function

Unsuccessful assertion with custom log message

Source
 assert_eq!(1, 0, message: "Unequal integers")
Learn more about the assert_eq function

log

Logs the value to stdout at the specified level.

Log a message

Source
log("Hello, World!", level: "info", rate_limit_secs: 60)
Learn more about the log function

Log an error

Source
_, err = to_int(.field)
if err != null {
	log(err, level: "error")
}
Learn more about the log function

Enrichment examples

find_enrichment_table_records

Searches an enrichment table for rows that match the provided condition.

For file enrichment tables, this condition needs to be a VRL object in which the key-value pairs indicate a field to search mapped to a value to search in that field. This function returns the rows that match the provided condition(s). All fields need to match for rows to be returned; if any fields do not match, then no rows are returned.

There are currently two forms of search criteria:

  1. Exact match search. The given field must match the value exactly. Case sensitivity can be specified using the case_sensitive argument. An exact match search can use an index directly into the dataset, which should make this search fairly “cheap” from a performance perspective.

  2. Date range search. The given field must be greater than or equal to the from date and less than or equal to the to date. A date range search involves sequentially scanning through the rows that have been located using any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. Therefore, use date ranges as the only criteria when the enrichment data set is very small.

For geoip and mmdb enrichment tables, this condition needs to be a VRL object with a single key-value pair whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected and without a value, null is used. This table can return the following fields:

  • ISP databases:

    • autonomous_system_number
    • autonomous_system_organization
    • isp
    • organization
  • City databases:

    • city_name
    • continent_code
    • country_code
    • country_name
    • region_code
    • region_name
    • metro_code
    • latitude
    • longitude
    • postal_code
    • timezone
  • Connection-Type databases:

    • connection_type

To use this function, you need to update your configuration to include an enrichment_tables parameter.

Exact match

Source
find_enrichment_table_records!("test",
  {
	"surname": "smith",
  },
  case_sensitive: false)
Return
[{"firstname":"Bob","id":1,"surname":"Smith"},{"firstname":"Fred","id":2,"surname":"Smith"}]
Learn more about the find_enrichment_table_records function
Source
find_enrichment_table_records!("test",
  {
	"surname": "Smith",
	"date_of_birth": {
	  "from": t'1985-01-01T00:00:00Z',
	  "to": t'1985-12-31T00:00:00Z'
	}
  })
Return
[{"firstname":"Bob","id":1,"surname":"Smith"},{"firstname":"Fred","id":2,"surname":"Smith"}]
Learn more about the find_enrichment_table_records function

get_enrichment_table_record

Searches an enrichment table for a row that matches the provided condition. A single row must be matched. If no rows are found or more than one row is found, an error is returned.

For file enrichment tables, this condition needs to be a VRL object in which the key-value pairs indicate a field to search mapped to a value to search in that field. This function returns the rows that match the provided condition(s). All fields need to match for rows to be returned; if any fields do not match, then no rows are returned.

There are currently two forms of search criteria:

  1. Exact match search. The given field must match the value exactly. Case sensitivity can be specified using the case_sensitive argument. An exact match search can use an index directly into the dataset, which should make this search fairly “cheap” from a performance perspective.

  2. Date range search. The given field must be greater than or equal to the from date and less than or equal to the to date. A date range search involves sequentially scanning through the rows that have been located using any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. Therefore, use date ranges as the only criteria when the enrichment data set is very small.

For geoip and mmdb enrichment tables, this condition needs to be a VRL object with a single key-value pair whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected and without a value, null is used. This table can return the following fields:

  • ISP databases:

    • autonomous_system_number
    • autonomous_system_organization
    • isp
    • organization
  • City databases:

    • city_name
    • continent_code
    • country_code
    • country_name
    • region_code
    • region_name
    • metro_code
    • latitude
    • longitude
    • postal_code
    • timezone
  • Connection-Type databases:

    • connection_type

To use this function, you need to update your configuration to include an enrichment_tables parameter.

Exact match

Source
get_enrichment_table_record!("test",
  {
    "surname": "bob",
    "firstname": "John"
  },
  case_sensitive: false)
Return
{
  "firstname": "Bob",
  "id": 1,
  "surname": "Smith"
}
Learn more about the get_enrichment_table_record function
Source
get_enrichment_table_record!("test",
  {
    "surname": "Smith",
    "date_of_birth": {
      "from": t'1985-01-01T00:00:00Z',
      "to": t'1985-12-31T00:00:00Z'
    }
  })
Return
{
  "firstname": "Bob",
  "id": 1,
  "surname": "Smith"
}
Learn more about the get_enrichment_table_record function

Enumerate examples

compact

Compacts the value by removing empty values, where empty values are defined using the available parameters.

Compact an array

Source
compact(["foo", "bar", "", null, [], "buzz"], string: true, array: true, null: true)
Return
["foo","bar","buzz"]
Learn more about the compact function

Compact an object

Source
compact({"field1": 1, "field2": "", "field3": [], "field4": null}, string: true, array: true, null: true)
Return
{
  "field1": 1
}
Learn more about the compact function

filter

Filter elements from a collection.

This function currently does not support recursive iteration.

The function uses the function closure syntax to allow reading the key-value or index-value combination for each item in the collection.

The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

See the examples below to learn about the closure syntax.

Filter elements

Source
filter(array!(.tags)) -> |_index, value| {
    # keep any elements that aren't equal to "foo"
    value != "foo"
}
Return
["bar","baz"]
Learn more about the filter function

flatten

Flattens the value into a single-level representation.

Flatten array

Source
flatten([1, [2, 3, 4], [5, [6, 7], 8], 9])
Return
[1,2,3,4,5,6,7,8,9]
Learn more about the flatten function

Flatten object

Source
flatten({
	"parent1": {
		"child1": 1,
		"child2": 2
	},
	"parent2": {
		"child3": 3
	}
})
Return
{
  "parent1.child1": 1,
  "parent1.child2": 2,
  "parent2.child3": 3
}
Learn more about the flatten function

for_each

Iterate over a collection.

This function currently does not support recursive iteration.

The function uses the “function closure syntax” to allow reading the key/value or index/value combination for each item in the collection.

The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

See the examples below to learn about the closure syntax.

Tally elements

Source
tally = {}
for_each(array!(.tags)) -> |_index, value| {
    # Get the current tally for the `value`, or
    # set to `0`.
    count = int(get!(tally, [value])) ?? 0

    # Increment the tally for the value by `1`.
    tally = set!(tally, [value], count + 1)
}

tally
Return
{
  "bar": 1,
  "baz": 1,
  "foo": 2
}
Learn more about the for_each function

includes

Determines whether the value array includes the specified item.

Array includes

Source
includes(["apple", "orange", "banana"], "banana")
Return
true
Learn more about the includes function

keys

Returns the keys from the object passed into the function.

Get keys from the object

Source
keys({"key1": "val1", "key2": "val2"})
Return
["key1","key2"]
Learn more about the keys function

length

Returns the length of the value.

  • If value is an array, returns the number of elements.
  • If value is an object, returns the number of top-level keys.
  • If value is a string, returns the number of bytes in the string. If you want the number of characters, see strlen.

Length (object)

Source
length({
	"portland": "Trail Blazers",
	"seattle":  "Supersonics"
})
Return
2
Learn more about the length function

Length (nested object)

Source
length({
	"home": {
		"city":  "Portland",
		"state": "Oregon"
	},
	"name": "Trail Blazers",
	"mascot": {
		"name": "Blaze the Trail Cat"
	}
})
Return
3
Learn more about the length function

Length (array)

Source
length(["Trail Blazers", "Supersonics", "Grizzlies"])
Return
3
Learn more about the length function

Length (string)

Source
length("The Planet of the Apes Musical")
Return
30
Learn more about the length function

map_keys

Map the keys within an object.

If recursive is enabled, the function iterates into nested objects, using the following rules:

  1. Iteration starts at the root.
  2. For every nested object type:
    • First return the key of the object type itself.
    • Then recurse into the object, and loop back to item (1) in this list.
    • Any mutation done on a nested object before recursing into it, are preserved.
  3. For every nested array type:
    • First return the key of the array type itself.
    • Then find all objects within the array, and apply item (2) to each individual object.

The above rules mean that map_keys with recursive enabled finds all keys in the target, regardless of whether nested objects are nested inside arrays.

The function uses the function closure syntax to allow reading the key for each item in the object.

The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

See the examples below to learn about the closure syntax.

Upcase keys

Source
map_keys(.) -> |key| { upcase(key) }
Return
{
  "BAR": "bar",
  "FOO": "foo"
}
Learn more about the map_keys function

De-dot keys

Source
map_keys(., recursive: true) -> |key| { replace(key, ".", "_") }
Return
{
  "labels": {
    "app_kubernetes_io/name": "mysql"
  }
}
Learn more about the map_keys function

map_values

Map the values within a collection.

If recursive is enabled, the function iterates into nested collections, using the following rules:

  1. Iteration starts at the root.
  2. For every nested collection type:
    • First return the collection type itself.
    • Then recurse into the collection, and loop back to item (1) in the list
    • Any mutation done on a collection before recursing into it, are preserved.

The function uses the function closure syntax to allow mutating the value for each item in the collection.

The same scoping rules apply to closure blocks as they do for regular blocks, meaning, any variable defined in parent scopes are accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

Check out the examples below to learn about the closure syntax.

Upcase values

Source
map_values(.) -> |value| { upcase!(value) }
Return
{
  "bar": "BAR",
  "foo": "FOO"
}
Learn more about the map_values function

match_array

Determines whether the elements in the value array matches the pattern. By default, it checks that at least one element matches, but can be set to determine if all the elements match.

Match at least one element

Source
match_array(["foobar", "bazqux"], r'foo')
Return
true
Learn more about the match_array function

Match all elements

Source
match_array(["foo", "foobar", "barfoo"], r'foo', all: true)
Return
true
Learn more about the match_array function

No matches

Source
match_array(["bazqux", "xyz"], r'foo')
Learn more about the match_array function

Not all elements match

Source
match_array(["foo", "foobar", "baz"], r'foo', all: true)
Learn more about the match_array function

strlen

Returns the number of UTF-8 characters in value. This differs from length which counts the number of bytes of a string.

Note: This is the count of Unicode scalar values which can sometimes differ from Unicode code points.

strlen

Source
strlen("ñandú")
Return
5
Learn more about the strlen function

unflatten

Unflattens the value into a nested representation.

Unflatten

Source
unflatten({
    "foo.bar.baz": true,
    "foo.bar.qux": false,
	"foo.quux": 42
})
Return
{
  "foo": {
    "bar": {
      "baz": true,
      "qux": false
    },
    "quux": 42
  }
}
Learn more about the unflatten function

Unflatten recursively

Source
unflatten({
    "flattened.parent": {
		"foo.bar": true,
		"foo.baz": false
	}
})
Return
{
  "flattened": {
    "parent": {
      "foo": {
        "bar": true,
        "baz": false
      }
    }
  }
}
Learn more about the unflatten function

Unflatten non-recursively

Source
unflatten({
    "flattened.parent": {
		"foo.bar": true,
		"foo.baz": false
	}
}, recursive: false)
Return
{
  "flattened": {
    "parent": {
      "foo.bar": true,
      "foo.baz": false
    }
  }
}
Learn more about the unflatten function

Ignore inconsistent keys values

Source
unflatten({
	"a": 3,
	"a.b": 2,
	"a.c": 4
})
Return
{
  "a": {
    "b": 2,
    "c": 4
  }
}
Learn more about the unflatten function

unique

Returns the unique values for an array.

The first occurrence of each element is kept.

Unique

Source
unique(["foo", "bar", "foo", "baz"])
Return
["foo","bar","baz"]
Learn more about the unique function

values

Returns the values from the object passed into the function.

Get values from the object

Source
values({"key1": "val1", "key2": "val2"})
Return
["val1","val2"]
Learn more about the values function

Event examples

get_secret

Returns the value of the given secret from an event.

Get the Datadog API key from the event metadata

Source
get_secret("datadog_api_key")
Return
secret value
Learn more about the get_secret function

remove_secret

Removes a secret from an event.

Removes the Datadog API key from the event

Source
remove_secret("datadog_api_key")
Learn more about the remove_secret function

set_secret

Sets the given secret in the event.

Set the Datadog API key to the given value

Source
set_secret("datadog_api_key", "abc122")
Learn more about the set_secret function

set_semantic_meaning

Sets a semantic meaning for an event. Note: This function assigns meaning at startup, and has no runtime behavior. It is suggested to put all calls to this function at the beginning of a VRL function. The function cannot be conditionally called. For example, using an if statement cannot stop the meaning from being assigned.

Sets custom field semantic meaning

Source
set_semantic_meaning(.foo, "bar")
Learn more about the set_semantic_meaning function

Path examples

del

Removes the field specified by the static path from the target.

For dynamic path deletion, see the remove function.

Delete a field

Source
del(.field1)
Learn more about the del function

Rename a field

Source
.new_field = del(.old_field)
Learn more about the del function

exists

Checks whether the path exists for the target.

This function distinguishes between a missing path and a path with a null value. A regular path lookup, such as .foo, cannot distinguish between the two cases since it always returns null if the path doesn’t exist.

Exists (field)

Source
exists(.field)
Return
true
Learn more about the exists function

Exists (array element)

Source
exists(.array[2])
Return
true
Learn more about the exists function

get

Dynamically get the value of a given path.

If you know the path you want to look up, use static paths such as .foo.bar[1] to get the value of that path. However, if you do not know the path names, use the dynamic get function to get the requested value.

single-segment top-level field

Source
get!(value: { "foo": "bar" }, path: ["foo"])
Return
bar
Learn more about the get function

multi-segment nested field

Source
get!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])
Return
baz
Learn more about the get function

array indexing

Source
get!(value: ["foo", "bar", "baz"], path: [-2])
Return
bar
Learn more about the get function

remove

Dynamically remove the value for a given path.

If you know the path you want to remove, use the del function and static paths such as del(.foo.bar[1]) to remove the value at that path. The del function returns the deleted value, and is more performant than remove. However, if you do not know the path names, use the dynamic remove function to remove the value at the provided path.

single-segment top-level field

Source
remove!(value: { "foo": "bar" }, path: ["foo"])
Learn more about the remove function

multi-segment nested field

Source
remove!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])
Return
{
  "foo": {}
}
Learn more about the remove function

array indexing

Source
remove!(value: ["foo", "bar", "baz"], path: [-2])
Return
["foo","baz"]
Learn more about the remove function

compaction

Source
remove!(value: { "foo": { "bar": [42], "baz": true } }, path: ["foo", "bar", 0], compact: true)
Return
{
  "foo": {
    "baz": true
  }
}
Learn more about the remove function

set

Dynamically insert data into the path of a given object or array.

If you know the path you want to assign a value to, use static path assignments such as .foo.bar[1] = true for improved performance and readability. However, if you do not know the path names, use the dynamic set function to insert the data into the object or array.

single-segment top-level field

Source
set!(value: { "foo": "bar" }, path: ["foo"], data: "baz")
Return
{
  "foo": "baz"
}
Learn more about the set function

multi-segment nested field

Source
set!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"], data: "qux")
Return
{
  "foo": {
    "bar": "qux"
  }
}
Learn more about the set function

array

Source
set!(value: ["foo", "bar", "baz"], path: [-2], data: 42)
Return
["foo",42,"baz"]
Learn more about the set function

Cryptography examples

decrypt

Decrypts a string with a symmetric encryption algorithm.

Supported Algorithms:

  • AES-256-CFB (key = 32 bytes, iv = 16 bytes)
  • AES-192-CFB (key = 24 bytes, iv = 16 bytes)
  • AES-128-CFB (key = 16 bytes, iv = 16 bytes)
  • AES-256-OFB (key = 32 bytes, iv = 16 bytes)
  • AES-192-OFB (key = 24 bytes, iv = 16 bytes)
  • AES-128-OFB (key = 16 bytes, iv = 16 bytes)
  • AES-128-SIV (key = 32 bytes, iv = 16 bytes)
  • AES-256-SIV (key = 64 bytes, iv = 16 bytes)
  • Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
  • Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
  • Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
  • AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
  • AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
  • AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
  • AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
  • AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
  • AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
  • CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
  • XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
  • XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)

Decrypt value

Source
ciphertext = decode_base64!("5fLGcu1VHdzsPcGNDio7asLqE1P43QrVfPfmP4i4zOU=")
iv = decode_base64!("fVEIRkIiczCRWNxaarsyxA==")
key = "16_byte_keyxxxxx"
decrypt!(ciphertext, "AES-128-CBC-PKCS7", key, iv: iv)
Return
super_secret_message
Learn more about the decrypt function

encrypt

Encrypts a string with a symmetric encryption algorithm.

Supported Algorithms:

  • AES-256-CFB (key = 32 bytes, iv = 16 bytes)
  • AES-192-CFB (key = 24 bytes, iv = 16 bytes)
  • AES-128-CFB (key = 16 bytes, iv = 16 bytes)
  • AES-256-OFB (key = 32 bytes, iv = 16 bytes)
  • AES-192-OFB (key = 24 bytes, iv = 16 bytes)
  • AES-128-OFB (key = 16 bytes, iv = 16 bytes)
  • AES-128-SIV (key = 32 bytes, iv = 16 bytes)
  • AES-256-SIV (key = 64 bytes, iv = 16 bytes)
  • Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
  • Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
  • Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
  • AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
  • AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
  • AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
  • AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
  • AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
  • AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
  • CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
  • XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
  • XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)

Encrypt value

Source
plaintext = "super secret message"
iv = "1234567890123456" # typically you would call random_bytes(16)
key = "16_byte_keyxxxxx"
encrypted_message = encrypt!(plaintext, "AES-128-CBC-PKCS7", key, iv: iv)
encode_base64(encrypted_message)
Return
GBw8Mu00v0Kc38+/PvsVtGgWuUJ+ZNLgF8Opy8ohIYE=
Learn more about the encrypt function

hmac

Calculates a HMAC of the value using the given key. The hashing algorithm used can be optionally specified.

For most use cases, the resulting bytestream should be encoded into a hex or base64 string using either encode_base16 or encode_base64.

This function is infallible if either the default algorithm value or a recognized-valid compile-time algorithm string literal is used. Otherwise, it is fallible.

Calculate message HMAC (defaults: SHA-256), encoding to a base64 string

Source
encode_base64(hmac("Hello there", "super-secret-key"))
Return
eLGE8YMviv85NPXgISRUZxstBNSU47JQdcXkUWcClmI=
Learn more about the hmac function

Calculate message HMAC using SHA-224, encoding to a hex-encoded string

Source
encode_base16(hmac("Hello there", "super-secret-key", algorithm: "SHA-224"))
Return
42fccbc2b7d22a143b92f265a8046187558a94d11ddbb30622207e90
Learn more about the hmac function

Calculate message HMAC using a variable hash algorithm

Source
.hash_algo = "SHA-256"
hmac_bytes, err = hmac("Hello there", "super-secret-key", algorithm: .hash_algo)
if err == null {
	.hmac = encode_base16(hmac_bytes)
}
Return
78b184f1832f8aff3934f5e0212454671b2d04d494e3b25075c5e45167029662
Learn more about the hmac function

md5

Calculates an md5 hash of the value.

Create md5 hash

Source
md5("foo")
Return
acbd18db4cc2f85cedef654fccc4a4d8
Learn more about the md5 function

seahash

Calculates a Seahash hash of the value. Note: Due to limitations in the underlying VRL data types, this function converts the unsigned 64-bit integer SeaHash result to a signed 64-bit integer. Results higher than the signed 64-bit integer maximum value wrap around to negative values.

Calculate seahash

Source
seahash("foobar")
Return
5348458858952426000
Learn more about the seahash function

Calculate negative seahash

Source
seahash("bar")
Return
-2796170501982571500
Learn more about the seahash function

sha1

Calculates a SHA-1 hash of the value.

Calculate sha1 hash

Source
sha1("foo")
Return
0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33
Learn more about the sha1 function

sha2

Calculates a SHA-2 hash of the value.

Calculate sha2 hash

Source
sha2("foo", variant: "SHA-512/224")
Return
d68f258d37d670cfc1ec1001a0394784233f88f056994f9a7e5e99be
Learn more about the sha2 function

sha3

Calculates a SHA-3 hash of the value.

Calculate sha3 hash

Source
sha3("foo", variant: "SHA3-224")
Return
f4f6779e153c391bbd29c95e72b0708e39d9166c7cea51d1f10ef58a
Learn more about the sha3 function

IP examples

ip_aton

Converts IPv4 address in numbers-and-dots notation into network-order bytes represented as an integer.

This behavior mimics inet_aton.

IPv4 to integer

Source
ip_aton!("1.2.3.4")
Return
16909060
Learn more about the ip_aton function

ip_cidr_contains

Determines whether the ip is contained in the block referenced by the cidr.

IPv4 contains CIDR

Source
ip_cidr_contains!("192.168.0.0/16", "192.168.10.32")
Return
true
Learn more about the ip_cidr_contains function

IPv6 contains CIDR

Source
ip_cidr_contains!("2001:4f8:4:ba::/64", "2001:4f8:4:ba:2e0:81ff:fe22:d1f1")
Return
true
Learn more about the ip_cidr_contains function

ip_ntoa

Converts numeric representation of IPv4 address in network-order bytes to numbers-and-dots notation.

This behavior mimics inet_ntoa.

Integer to IPv4

Source
ip_ntoa!(16909060)
Return
1.2.3.4
Learn more about the ip_ntoa function

ip_ntop

Converts IPv4 and IPv6 addresses from binary to text form.

This behavior mimics inet_ntop.

Convert IPv4 address from bytes after decoding from Base64

Source
ip_ntop!(decode_base64!("wKgAAQ=="))
Return
192.168.0.1
Learn more about the ip_ntop function

Convert IPv6 address from bytes after decoding from Base64

Source
ip_ntop!(decode_base64!("IAENuIWjAAAAAIouA3BzNA=="))
Return
2001:db8:85a3::8a2e:370:7334
Learn more about the ip_ntop function

ip_pton

Converts IPv4 and IPv6 addresses from text to binary form.

  • The binary form of IPv4 addresses is 4 bytes (32 bits) long.
  • The binary form of IPv6 addresses is 16 bytes (128 bits) long.

This behavior mimics inet_pton.

Convert IPv4 address to bytes and encode to Base64

Source
encode_base64(ip_pton!("192.168.0.1"))
Return
wKgAAQ==
Learn more about the ip_pton function

Convert IPv6 address to bytes and encode to Base64

Source
encode_base64(ip_pton!("2001:db8:85a3::8a2e:370:7334"))
Return
IAENuIWjAAAAAIouA3BzNA==
Learn more about the ip_pton function

ip_subnet

Extracts the subnet address from the ip using the supplied subnet.

IPv4 subnet

Source
ip_subnet!("192.168.10.32", "255.255.255.0")
Return
192.168.10.0
Learn more about the ip_subnet function

IPv6 subnet

Source
ip_subnet!("2404:6800:4003:c02::64", "/32")
Return
2404:6800::
Learn more about the ip_subnet function

ip_to_ipv6

Converts the ip to an IPv6 address.

IPv4 to IPv6

Source
ip_to_ipv6!("192.168.10.32")
Return
::ffff:192.168.10.32
Learn more about the ip_to_ipv6 function

ipv6_to_ipv4

Converts the ip to an IPv4 address. ip is returned unchanged if it’s already an IPv4 address. If ip is currently an IPv6 address then it needs to be IPv4 compatible, otherwise an error is thrown.

IPv6 to IPv4

Source
ipv6_to_ipv4!("::ffff:192.168.0.1")
Return
192.168.0.1
Learn more about the ipv6_to_ipv4 function

is_ipv4

Check if the string is a valid IPv4 address or not.

An [IPv4-mapped][https://datatracker.ietf.org/doc/html/rfc6890] or [IPv4-compatible][https://datatracker.ietf.org/doc/html/rfc6890] IPv6 address is not considered valid for the purpose of this function.

Valid IPv4 address

Source
is_ipv4("10.0.102.37")
Return
true
Learn more about the is_ipv4 function

Valid IPv6 address

Source
is_ipv4("2001:0db8:85a3:0000:0000:8a2e:0370:7334")
Learn more about the is_ipv4 function

Arbitrary string

Source
is_ipv4("foobar")
Learn more about the is_ipv4 function

is_ipv6

Check if the string is a valid IPv6 address or not.

Valid IPv6 address

Source
is_ipv6("2001:0db8:85a3:0000:0000:8a2e:0370:7334")
Return
true
Learn more about the is_ipv6 function

Valid IPv4 address

Source
is_ipv6("10.0.102.37")
Learn more about the is_ipv6 function

Arbitrary string

Source
is_ipv6("foobar")
Learn more about the is_ipv6 function

Number examples

abs

Computes the absolute value of value.

Computes the absolute value of the integer

Source
abs(-42)
Return
42
Learn more about the abs function

Computes the absolute value of the float

Source
abs(-42.2)
Return
42.2
Learn more about the abs function

ceil

Rounds the value up to the specified precision.

Round a number up (without precision)

Source
ceil(4.345)
Return
5
Learn more about the ceil function

Round a number up (with precision)

Source
ceil(4.345, precision: 2)
Return
4.35
Learn more about the ceil function

floor

Rounds the value down to the specified precision.

Round a number down (without precision)

Source
floor(4.345)
Return
4
Learn more about the floor function

Round a number down (with precision)

Source
floor(4.345, precision: 2)
Return
4.34
Learn more about the floor function

format_int

Formats the integer value into a string representation using the given base/radix.

Format as a hexadecimal integer

Source
format_int!(42, 16)
Return
2a
Learn more about the format_int function

Format as a negative hexadecimal integer

Source
format_int!(-42, 16)
Return
-2a
Learn more about the format_int function

format_number

Formats the value into a string representation of the number.

Format a number (3 decimals)

Source
format_number(1234567.89, 3, decimal_separator: ".", grouping_separator: ",")
Return
1,234,567.890
Learn more about the format_number function

mod

Calculates the remainder of value divided by modulus.

Calculate the remainder of two integers

Source
mod(5, 2)
Return
1
Learn more about the mod function

round

Rounds the value to the specified precision.

Round a number (without precision)

Source
round(4.345)
Return
4
Learn more about the round function

Round a number (with precision)

Source
round(4.345, precision: 2)
Return
4.35
Learn more about the round function

Object examples

match_datadog_query

Matches an object against a Datadog Search Syntax query.

OR query

Source
match_datadog_query({"message": "contains this and that"}, "this OR that")
Return
true
Learn more about the match_datadog_query function

AND query

Source
match_datadog_query({"message": "contains only this"}, "this AND that")
Learn more about the match_datadog_query function

Attribute wildcard

Source
match_datadog_query({"name": "foobar"}, "@name:foo*")
Return
true
Learn more about the match_datadog_query function

Tag range

Source
match_datadog_query({"tags": ["a:x", "b:y", "c:z"]}, s'b:["x" TO "z"]')
Return
true
Learn more about the match_datadog_query function

merge

Merges the from object into the to object.

Object merge (shallow)

Source
merge(
	{
		"parent1": {
			"child1": 1,
			"child2": 2
		},
		"parent2": {
			"child3": 3
		}
	},
	{
		"parent1": {
			"child2": 4,
			"child5": 5
		}
	}
)
Return
{
  "parent1": {
    "child2": 4,
    "child5": 5
  },
  "parent2": {
    "child3": 3
  }
}
Learn more about the merge function

Object merge (deep)

Source
merge(
	{
		"parent1": {
			"child1": 1,
			"child2": 2
		},
		"parent2": {
			"child3": 3
		}
	},
	{
		"parent1": {
			"child2": 4,
			"child5": 5
		}
	},
	deep: true
)
Return
{
  "parent1": {
    "child1": 1,
    "child2": 4,
    "child5": 5
  },
  "parent2": {
    "child3": 3
  }
}
Learn more about the merge function

unnest

Unnest an array field from an object to create an array of objects using that field; keeping all other fields.

Assigning the array result of this to . results in multiple events being emitted from remap. See the remap transform docs for more details.

This is also referred to as explode in some languages.

Unnest an array field

Source
. = unnest!(.messages)
Learn more about the unnest function

Unnest nested an array field

Source
. = unnest!(.event.messages)
Learn more about the unnest function

Parse examples

parse_apache_log

Parses Apache access and error log lines. Lines can be in common, combined, or the default error format.

Parse using Apache log format (common)

Source
parse_apache_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326", format: "common")
Return
{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}
Learn more about the parse_apache_log function

Parse using Apache log format (combined)

Source
parse_apache_log!(
	s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth" "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"',
	"combined",
)
Return
{
  "agent": "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0",
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "referrer": "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}
Learn more about the parse_apache_log function

Parse using Apache log format (error)

Source
parse_apache_log!(
	s'[01/Mar/2021:12:00:19 +0000] [ab:alert] [pid 4803:tid 3814] [client 147.159.108.175:24259] I will bypass the haptic COM bandwidth, that should matrix the CSS driver!',
	"error"
)
Return
{
  "client": "147.159.108.175",
  "message": "I will bypass the haptic COM bandwidth, that should matrix the CSS driver!",
  "module": "ab",
  "pid": 4803,
  "port": 24259,
  "severity": "alert",
  "thread": "3814",
  "timestamp": "2021-03-01T12:00:19Z"
}
Learn more about the parse_apache_log function

parse_aws_alb_log

Parse AWS ALB log

Source
parse_aws_alb_log!(
	"http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\""
)
Return
{
  "actions_executed": "forward",
  "chosen_cert_arn": null,
  "classification": null,
  "classification_reason": null,
  "client_host": "192.168.131.39:2817",
  "domain_name": null,
  "elb": "app/my-loadbalancer/50dc6c495c0c9188",
  "elb_status_code": "200",
  "error_reason": null,
  "matched_rule_priority": "0",
  "received_bytes": 34,
  "redirect_url": null,
  "request_creation_time": "2018-11-30T22:22:48.364000Z",
  "request_method": "GET",
  "request_processing_time": 0,
  "request_protocol": "HTTP/1.1",
  "request_url": "http://www.example.com:80/",
  "response_processing_time": 0,
  "sent_bytes": 366,
  "ssl_cipher": null,
  "ssl_protocol": null,
  "target_group_arn": "arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067",
  "target_host": null,
  "target_port_list": [],
  "target_processing_time": 0.001,
  "target_status_code": "200",
  "target_status_code_list": [],
  "timestamp": "2018-11-30T22:23:00.186641Z",
  "trace_id": "Root=1-58337364-23a8c76965a2ef7629b185e3",
  "traceability_id": null,
  "type": "http",
  "user_agent": "curl/7.46.0"
}
Learn more about the parse_aws_alb_log function

parse_aws_cloudwatch_log_subscription_message

Parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) from the aws_kinesis_firehose source.

Parse AWS Cloudwatch Log subscription message

Source
parse_aws_cloudwatch_log_subscription_message!(.message)
Return
{
  "log_events": [
    {
      "id": "35683658089614582423604394983260738922885519999578275840",
      "message": "{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}",
      "timestamp": "2020-09-14T19:09:29.039Z"
    }
  ],
  "log_group": "test",
  "log_stream": "test",
  "message_type": "DATA_MESSAGE",
  "owner": "111111111111",
  "subscription_filters": [
    "Destination"
  ]
}
Learn more about the parse_aws_cloudwatch_log_subscription_message function

parse_aws_vpc_flow_log

Parses value in the VPC Flow Logs format.

Parse AWS VPC Flow log (default format)

Source
parse_aws_vpc_flow_log!("2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA")
Return
{
  "account_id": "123456789010",
  "action": null,
  "bytes": null,
  "dstaddr": null,
  "dstport": null,
  "end": 1431280934,
  "interface_id": "eni-1235b8ca123456789",
  "log_status": "NODATA",
  "packets": null,
  "protocol": null,
  "srcaddr": null,
  "srcport": null,
  "start": 1431280876,
  "version": 2
}
Learn more about the parse_aws_vpc_flow_log function

Parse AWS VPC Flow log (custom format)

Source
parse_aws_vpc_flow_log!(
	"- eni-1235b8ca123456789 10.0.1.5 10.0.0.220 10.0.1.5 203.0.113.5",
	"instance_id interface_id srcaddr dstaddr pkt_srcaddr pkt_dstaddr"
)
Return
{
  "dstaddr": "10.0.0.220",
  "instance_id": null,
  "interface_id": "eni-1235b8ca123456789",
  "pkt_dstaddr": "203.0.113.5",
  "pkt_srcaddr": "10.0.1.5",
  "srcaddr": "10.0.1.5"
}
Learn more about the parse_aws_vpc_flow_log function

Parse AWS VPC Flow log including v5 fields

Source
parse_aws_vpc_flow_log!("5 52.95.128.179 10.0.0.71 80 34210 6 1616729292 1616729349 IPv4 14 15044 123456789012 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-0c50d5961bcb2d47b eni-1235b8ca123456789 ap-southeast-2 apse2-az3 - - ACCEPT 19 52.95.128.179 10.0.0.71 S3 - - ingress OK",
format: "version srcaddr dstaddr srcport dstport protocol start end type packets bytes account_id vpc_id subnet_id instance_id interface_id region az_id sublocation_type sublocation_id action tcp_flags pkt_srcaddr pkt_dstaddr pkt_src_aws_service pkt_dst_aws_service traffic_path flow_direction log_status")
Return
{
  "account_id": "123456789012",
  "action": "ACCEPT",
  "az_id": "apse2-az3",
  "bytes": 15044,
  "dstaddr": "10.0.0.71",
  "dstport": 34210,
  "end": 1616729349,
  "flow_direction": "ingress",
  "instance_id": "i-0c50d5961bcb2d47b",
  "interface_id": "eni-1235b8ca123456789",
  "log_status": "OK",
  "packets": 14,
  "pkt_dst_aws_service": null,
  "pkt_dstaddr": "10.0.0.71",
  "pkt_src_aws_service": "S3",
  "pkt_srcaddr": "52.95.128.179",
  "protocol": 6,
  "region": "ap-southeast-2",
  "srcaddr": "52.95.128.179",
  "srcport": 80,
  "start": 1616729292,
  "sublocation_id": null,
  "sublocation_type": null,
  "subnet_id": "subnet-aaaaaaaa012345678",
  "tcp_flags": 19,
  "traffic_path": null,
  "type": "IPv4",
  "version": 5,
  "vpc_id": "vpc-abcdefab012345678"
}
Learn more about the parse_aws_vpc_flow_log function

parse_cef

Parses the value in CEF (Common Event Format) format. Ignores everything up to CEF header. Empty values are returned as empty strings. Surrounding quotes are removed from values.

Parse output generated by PTA

Source
parse_cef!(
	"CEF:0|CyberArk|PTA|12.6|1|Suspected credentials theft|8|suser=mike2@prod1.domain.com shost=prod1.domain.com src=1.1.1.1 duser=andy@dev1.domain.com dhost=dev1.domain.com dst=2.2.2.2 cs1Label=ExtraData cs1=None cs2Label=EventID cs2=52b06812ec3500ed864c461e deviceCustomDate1Label=detectionDate deviceCustomDate1=1388577900000 cs3Label=PTAlink cs3=https://1.1.1.1/incidents/52b06812ec3500ed864c461e cs4Label=ExternalLink cs4=None"
)
Return
{
  "cefVersion": "0",
  "cs1": "None",
  "cs1Label": "ExtraData",
  "cs2": "52b06812ec3500ed864c461e",
  "cs2Label": "EventID",
  "cs3": "https://1.1.1.1/incidents/52b06812ec3500ed864c461e",
  "cs3Label": "PTAlink",
  "cs4": "None",
  "cs4Label": "ExternalLink",
  "deviceCustomDate1": "1388577900000",
  "deviceCustomDate1Label": "detectionDate",
  "deviceEventClassId": "1",
  "deviceProduct": "PTA",
  "deviceVendor": "CyberArk",
  "deviceVersion": "12.6",
  "dhost": "dev1.domain.com",
  "dst": "2.2.2.2",
  "duser": "andy@dev1.domain.com",
  "name": "Suspected credentials theft",
  "severity": "8",
  "shost": "prod1.domain.com",
  "src": "1.1.1.1",
  "suser": "mike2@prod1.domain.com"
}
Learn more about the parse_cef function

Ignore syslog header

Source
parse_cef!(
	"Sep 29 08:26:10 host CEF:1|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232"
)
Return
{
  "cefVersion": "1",
  "deviceEventClassId": "100",
  "deviceProduct": "threatmanager",
  "deviceVendor": "Security",
  "deviceVersion": "1.0",
  "dst": "2.1.2.2",
  "name": "worm successfully stopped",
  "severity": "10",
  "spt": "1232",
  "src": "10.0.0.1"
}
Learn more about the parse_cef function

Translate custom fields

Source
parse_cef!(
	"CEF:0|Dev|firewall|2.2|1|Connection denied|5|c6a1=2345:0425:2CA1:0000:0000:0567:5673:23b5 c6a1Label=Device IPv6 Address",
	translate_custom_fields: true
)
Return
{
  "Device IPv6 Address": "2345:0425:2CA1:0000:0000:0567:5673:23b5",
  "cefVersion": "0",
  "deviceEventClassId": "1",
  "deviceProduct": "firewall",
  "deviceVendor": "Dev",
  "deviceVersion": "2.2",
  "name": "Connection denied",
  "severity": "5"
}
Learn more about the parse_cef function

parse_common_log

Parses the value using the Common Log Format (CLF).

Parse using Common Log Format (with default timestamp format)

Source
parse_common_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326")
Return
{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}
Learn more about the parse_common_log function

Parse using Common Log Format (with custom timestamp format)

Source
parse_common_log!(
	"127.0.0.1 bob frank [2000-10-10T20:55:36Z] \"GET /apache_pb.gif HTTP/1.0\" 200 2326",
	"%+"
)
Return
{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}
Learn more about the parse_common_log function

parse_csv

Parses a single CSV formatted row. Only the first row is parsed in case of multiline input value.

Parse a single CSV formatted row

Source
parse_csv!("foo,bar,\"foo \"\", bar\"")
Return
["foo","bar","foo \", bar"]
Learn more about the parse_csv function

Parse a single CSV formatted row with custom delimiter

Source
parse_csv!("foo bar", delimiter: " ")
Return
["foo","bar"]
Learn more about the parse_csv function

parse_duration

Parses the value into a human-readable duration format specified by unit.

Parse duration (milliseconds)

Source
parse_duration!("1005ms", unit: "s")
Return
1.005
Learn more about the parse_duration function

parse_etld

Parses the eTLD from value representing domain name.

Parse eTLD

Source
parse_etld!("sub.sussex.ac.uk")
Return
{
  "etld": "ac.uk",
  "etld_plus": "ac.uk",
  "known_suffix": true
}
Learn more about the parse_etld function

Parse eTLD+1

Source
parse_etld!("sub.sussex.ac.uk", plus_parts: 1)
Return
{
  "etld": "ac.uk",
  "etld_plus": "sussex.ac.uk",
  "known_suffix": true
}
Learn more about the parse_etld function

Parse eTLD with unknown suffix

Source
parse_etld!("vector.acmecorp")
Return
{
  "etld": "acmecorp",
  "etld_plus": "acmecorp",
  "known_suffix": false
}
Learn more about the parse_etld function

Parse eTLD with custom PSL

Source
parse_etld!("vector.acmecorp", psl: "resources/public_suffix_list.dat")
Return
{
  "etld": "acmecorp",
  "etld_plus": "acmecorp",
  "known_suffix": false
}
Learn more about the parse_etld function

parse_glog

Parses the value using the glog (Google Logging Library) format.

Parse using glog

Source
parse_glog!("I20210131 14:48:54.411655 15520 main.c++:9] Hello world!")
Return
{
  "file": "main.c++",
  "id": 15520,
  "level": "info",
  "line": 9,
  "message": "Hello world!",
  "timestamp": "2021-01-31T14:48:54.411655Z"
}
Learn more about the parse_glog function

parse_grok

Parses the value using the grok format. All patterns listed here are supported.

Parse using Grok

Source
parse_grok!(
	"2020-10-02T23:22:12.223222Z info Hello world",
	"%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
)
Return
{
  "level": "info",
  "message": "Hello world",
  "timestamp": "2020-10-02T23:22:12.223222Z"
}
Learn more about the parse_grok function

parse_groks

Parses the value using multiple grok patterns. All patterns listed here are supported.

Parse using multiple Grok patterns

Source
parse_groks!(
	"2020-10-02T23:22:12.223222Z info Hello world",
	patterns: [
		"%{common_prefix} %{_status} %{_message}",
		"%{common_prefix} %{_message}",
	],
	aliases: {
		"common_prefix": "%{_timestamp} %{_loglevel}",
		"_timestamp": "%{TIMESTAMP_ISO8601:timestamp}",
		"_loglevel": "%{LOGLEVEL:level}",
		"_status": "%{POSINT:status}",
		"_message": "%{GREEDYDATA:message}"
	}
)
Return
{
  "level": "info",
  "message": "Hello world",
  "timestamp": "2020-10-02T23:22:12.223222Z"
}
Learn more about the parse_groks function

parse_influxdb

Parses the value as an InfluxDB line protocol string, producing a list of Vector-compatible metrics.

Parse InfluxDB line protocol

Source
parse_influxdb!("cpu,host=A,region=us-west usage_system=64i,usage_user=10u,temperature=50.5,on=true,sleep=false 1590488773254420000")
Return
[{"gauge":{"value":64},"kind":"absolute","name":"cpu_usage_system","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":10},"kind":"absolute","name":"cpu_usage_user","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":50.5},"kind":"absolute","name":"cpu_temperature","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":1},"kind":"absolute","name":"cpu_on","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":0},"kind":"absolute","name":"cpu_sleep","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"}]
Learn more about the parse_influxdb function

parse_int

Parses the string value representing a number in an optional base/radix to an integer.

Parse decimal

Source
parse_int!("-42")
Return
-42
Learn more about the parse_int function

Parse binary

Source
parse_int!("0b1001")
Return
9
Learn more about the parse_int function

Parse octal

Source
parse_int!("0o42")
Return
34
Learn more about the parse_int function

Parse hexadecimal

Source
parse_int!("0x2a")
Return
42
Learn more about the parse_int function

Parse explicit base

Source
parse_int!("2a", 17)
Return
44
Learn more about the parse_int function

parse_json

Parses the value as JSON.

Parse JSON

Source
parse_json!("{\"key\": \"val\"}")
Return
{
  "key": "val"
}
Learn more about the parse_json function

Parse JSON with max_depth

Source
parse_json!("{\"top_level\":{\"key\": \"val\"}}", max_depth: 1)
Return
{
  "top_level": "{\"key\": \"val\"}"
}
Learn more about the parse_json function

parse_key_value

Parses the value in key-value format. Also known as logfmt.

  • Keys and values can be wrapped with ".
  • " characters can be escaped using \.

Parse logfmt log

Source
parse_key_value!(
	"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
Return
{
  "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
  "id": "ConsumerFetcherManager-1382721708341",
  "level": "info",
  "module": "kafka.consumer.ConsumerFetcherManager",
  "msg": "Stopping all fetchers",
  "tag#production": "stopping_fetchers"
}
Learn more about the parse_key_value function

Parse comma delimited log

Source
parse_key_value!(
	"path:\"/cart_link\", host:store.app.com, fwd: \"102.30.171.16\", dyno: web.1, connect:0ms, service:87ms, status:304, bytes:632, protocol:https",
	field_delimiter: ",",
	key_value_delimiter: ":"
)
Return
{
  "bytes": "632",
  "connect": "0ms",
  "dyno": "web.1",
  "fwd": "102.30.171.16",
  "host": "store.app.com",
  "path": "/cart_link",
  "protocol": "https",
  "service": "87ms",
  "status": "304"
}
Learn more about the parse_key_value function

Parse comma delimited log with standalone keys

Source
parse_key_value!(
	"env:prod,service:backend,region:eu-east1,beta",
	field_delimiter: ",",
	key_value_delimiter: ":",
)
Return
{
  "beta": true,
  "env": "prod",
  "region": "eu-east1",
  "service": "backend"
}
Learn more about the parse_key_value function

Parse duplicate keys

Source
parse_key_value!(
	"at=info,method=GET,path=\"/index\",status=200,tags=dev,tags=dummy",
	field_delimiter: ",",
	key_value_delimiter: "=",
)
Return
{
  "at": "info",
  "method": "GET",
  "path": "/index",
  "status": "200",
  "tags": [
    "dev",
    "dummy"
  ]
}
Learn more about the parse_key_value function

parse_klog

Parses the value using the klog format used by Kubernetes components.

Parse using klog

Source
parse_klog!("I0505 17:59:40.692994   28133 klog.go:70] hello from klog")
Return
{
  "file": "klog.go",
  "id": 28133,
  "level": "info",
  "line": 70,
  "message": "hello from klog",
  "timestamp": "2024-05-05T17:59:40.692994Z"
}
Learn more about the parse_klog function

parse_linux_authorization

Parses Linux authorization logs usually found under either /var/log/auth.log (for Debian-based systems) or /var/log/secure (for RedHat-based systems) according to Syslog format.

Parse Linux authorization event

Source
parse_linux_authorization!(
	s'Mar 23 2023 01:49:58 localhost sshd[1111]: Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar'
)
Return
{
  "appname": "sshd",
  "hostname": "localhost",
  "message": "Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar",
  "procid": 1111,
  "timestamp": "2023-03-23T01:49:58Z"
}
Learn more about the parse_linux_authorization function

parse_logfmt

Parses the value in logfmt.

  • Keys and values can be wrapped using the " character.
  • " characters can be escaped by the \ character.
  • As per this logfmt specification, the parse_logfmt function accepts standalone keys and assigns them a Boolean value of true.

Parse logfmt log

Source
parse_logfmt!(
	"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
Return
{
  "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
  "id": "ConsumerFetcherManager-1382721708341",
  "level": "info",
  "module": "kafka.consumer.ConsumerFetcherManager",
  "msg": "Stopping all fetchers",
  "tag#production": "stopping_fetchers"
}
Learn more about the parse_logfmt function

parse_nginx_log

Parses Nginx access and error log lines. Lines can be in combined, ingress_upstreaminfo, or error format.

Parse via Nginx log format (combined)

Source
parse_nginx_log!(
    s'172.17.0.1 - alice [01/Apr/2021:12:02:31 +0000] "POST /not-found HTTP/1.1" 404 153 "http://localhost/somewhere" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"',
    "combined",
)
Return
{
  "agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36",
  "client": "172.17.0.1",
  "compression": "2.75",
  "referer": "http://localhost/somewhere",
  "request": "POST /not-found HTTP/1.1",
  "size": 153,
  "status": 404,
  "timestamp": "2021-04-01T12:02:31Z",
  "user": "alice"
}
Learn more about the parse_nginx_log function

Parse via Nginx log format (error)

Source
parse_nginx_log!(
    s'2021/04/01 13:02:31 [error] 31#31: *1 open() "/usr/share/nginx/html/not-found" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "POST /not-found HTTP/1.1", host: "localhost:8081"',
    "error"
)
Return
{
  "cid": 1,
  "client": "172.17.0.1",
  "host": "localhost:8081",
  "message": "open() \"/usr/share/nginx/html/not-found\" failed (2: No such file or directory)",
  "pid": 31,
  "request": "POST /not-found HTTP/1.1",
  "server": "localhost",
  "severity": "error",
  "tid": 31,
  "timestamp": "2021-04-01T13:02:31Z"
}
Learn more about the parse_nginx_log function

Parse via Nginx log format (ingress_upstreaminfo)

Source
parse_nginx_log!(
    s'0.0.0.0 - bob [18/Mar/2023:15:00:00 +0000] "GET /some/path HTTP/2.0" 200 12312 "https://10.0.0.1/some/referer" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 462 0.050 [some-upstream-service-9000] [some-other-upstream-5000] 10.0.50.80:9000 19437 0.049 200 752178adb17130b291aefd8c386279e7',
    "ingress_upstreaminfo"
)
Return
{
  "body_bytes_size": 12312,
  "http_referer": "https://10.0.0.1/some/referer",
  "http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36",
  "proxy_alternative_upstream_name": "some-other-upstream-5000",
  "proxy_upstream_name": "some-upstream-service-9000",
  "remote_addr": "0.0.0.0",
  "remote_user": "bob",
  "req_id": "752178adb17130b291aefd8c386279e7",
  "request": "GET /some/path HTTP/2.0",
  "request_length": 462,
  "request_time": 0.05,
  "status": 200,
  "timestamp": "2023-03-18T15:00:00Z",
  "upstream_addr": "10.0.50.80:9000",
  "upstream_response_length": 19437,
  "upstream_response_time": 0.049,
  "upstream_status": 200
}
Learn more about the parse_nginx_log function

parse_proto

Parses the value as a protocol buffer payload.

Parse proto

Source
parse_proto!(decode_base64!("Cgdzb21lb25lIggKBjEyMzQ1Ng=="), "resources/protobuf_descriptor_set.desc", "test_protobuf.Person")
Return
{
  "name": "someone",
  "phones": [
    {
      "number": "123456"
    }
  ]
}
Learn more about the parse_proto function

parse_query_string

Parses the value as a query string.

Parse query string

Source
parse_query_string("foo=%2B1&bar=2&bar=3&xyz")
Return
{
  "bar": [
    "2",
    "3"
  ],
  "foo": "+1",
  "xyz": ""
}
Learn more about the parse_query_string function

Parse Ruby on Rails' query string

Source
parse_query_string("?foo%5b%5d=1&foo%5b%5d=2")
Return
{
  "foo[]": [
    "1",
    "2"
  ]
}
Learn more about the parse_query_string function

parse_regex

Parses the value using the provided Regex pattern.

This function differs from the parse_regex_all function in that it returns only the first match.

Parse using Regex (with capture groups)

Source
parse_regex!("first group and second group.", r'(?P<number>.*?) group')
Return
{
  "number": "first"
}
Learn more about the parse_regex function

Parse using Regex (without capture groups)

Source
parse_regex!("first group and second group.", r'(\w+) group', numeric_groups: true)
Return
{
  "0": "first group",
  "1": "first"
}
Learn more about the parse_regex function

parse_regex_all

Parses the value using the provided Regex pattern.

This function differs from the parse_regex function in that it returns all matches, not just the first.

Parse using Regex (all matches)

Source
parse_regex_all!("first group and second group.", r'(?P<number>\w+) group', numeric_groups: true)
Return
[{"0":"first group","1":"first","number":"first"},{"0":"second group","1":"second","number":"second"}]
Learn more about the parse_regex_all function

parse_ruby_hash

Parses the value as ruby hash.

Parse ruby hash

Source
parse_ruby_hash!(s'{ "test" => "value", "testNum" => 0.2, "testObj" => { "testBool" => true, "testNull" => nil } }')
Return
{
  "test": "value",
  "testNum": 0.2,
  "testObj": {
    "testBool": true,
    "testNull": null
  }
}
Learn more about the parse_ruby_hash function

parse_syslog

Parses the value in Syslog format.

Parse Syslog log (5424)

Source
parse_syslog!(
	s'<13>1 2020-03-13T20:45:38.119Z dynamicwireless.name non 2426 ID931 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"] Try to override the THX port, maybe it will reboot the neural interface!'
)
Return
{
  "appname": "non",
  "exampleSDID@32473": {
    "eventID": "1011",
    "eventSource": "Application",
    "iut": "3"
  },
  "facility": "user",
  "hostname": "dynamicwireless.name",
  "message": "Try to override the THX port, maybe it will reboot the neural interface!",
  "msgid": "ID931",
  "procid": 2426,
  "severity": "notice",
  "timestamp": "2020-03-13T20:45:38.119Z",
  "version": 1
}
Learn more about the parse_syslog function

parse_timestamp

Parses the value in strptime format.

Parse timestamp

Source
parse_timestamp!("10-Oct-2020 16:00+00:00", format: "%v %R %:z")
Return
2020-10-10T16:00:00Z
Learn more about the parse_timestamp function

parse_tokens

Parses the value in token format. A token is considered to be one of the following:

  • A word surrounded by whitespace.
  • Text delimited by double quotes: "..". Quotes can be included in the token if they are escaped by a backslash (\).
  • Text delimited by square brackets: [..]. Closing square brackets can be included in the token if they are escaped by a backslash (\).

Parse tokens

Source
parse_tokens(
	"A sentence \"with \\\"a\\\" sentence inside\" and [some brackets]"
)
Return
["A","sentence","with \\\"a\\\" sentence inside","and","some brackets"]
Learn more about the parse_tokens function

parse_url

Parses the value in URL format.

Parse URL

Source
parse_url!("ftp://foo:bar@example.com:4343/foobar?hello=world#123")
Return
{
  "fragment": "123",
  "host": "example.com",
  "password": "bar",
  "path": "/foobar",
  "port": 4343,
  "query": {
    "hello": "world"
  },
  "scheme": "ftp",
  "username": "foo"
}
Learn more about the parse_url function

Parse URL with default port

Source
parse_url!("https://example.com", default_known_ports: true)
Return
{
  "fragment": null,
  "host": "example.com",
  "password": "",
  "path": "/",
  "port": 443,
  "query": {},
  "scheme": "https",
  "username": ""
}
Learn more about the parse_url function

Parse URL with internationalized domain name

Source
parse_url!("https://www.café.com")
Return
{
  "fragment": null,
  "host": "www.xn--caf-dma.com",
  "password": "",
  "path": "/",
  "port": null,
  "query": {},
  "scheme": "https",
  "username": ""
}
Learn more about the parse_url function

Parse URL with mixed case internationalized domain name

Source
parse_url!("https://www.CAFé.com")
Return
{
  "fragment": null,
  "host": "www.xn--caf-dma.com",
  "password": "",
  "path": "/",
  "port": null,
  "query": {},
  "scheme": "https",
  "username": ""
}
Learn more about the parse_url function

parse_user_agent

Parses the value as a user agent string, which has a loosely defined format so this parser only provides best effort guarantee.

Fast mode

Source
parse_user_agent(
	"Mozilla Firefox 1.0.1 Mozilla/5.0 (X11; U; Linux i686; de-DE; rv:1.7.6) Gecko/20050223 Firefox/1.0.1"
)
Return
{
  "browser": {
    "family": "Firefox",
    "version": "1.0.1"
  },
  "device": {
    "category": "pc"
  },
  "os": {
    "family": "Linux",
    "version": null
  }
}
Learn more about the parse_user_agent function

Reliable mode

Source
parse_user_agent(
	"Mozilla/4.0 (compatible; MSIE 7.66; Windows NT 5.1; SV1; .NET CLR 1.1.4322)",
	mode: "reliable"
)
Return
{
  "browser": {
    "family": "Internet Explorer",
    "version": "7.66"
  },
  "device": {
    "category": "pc"
  },
  "os": {
    "family": "Windows XP",
    "version": "NT 5.1"
  }
}
Learn more about the parse_user_agent function

Enriched mode

Source
parse_user_agent(
	"Opera/9.80 (J2ME/MIDP; Opera Mini/4.3.24214; iPhone; CPU iPhone OS 4_2_1 like Mac OS X; AppleWebKit/24.783; U; en) Presto/2.5.25 Version/10.54",
	mode: "enriched"
)
Return
{
  "browser": {
    "family": "Opera Mini",
    "major": "4",
    "minor": "3",
    "patch": "24214",
    "version": "10.54"
  },
  "device": {
    "brand": "Apple",
    "category": "smartphone",
    "family": "iPhone",
    "model": "iPhone"
  },
  "os": {
    "family": "iOS",
    "major": "4",
    "minor": "2",
    "patch": "1",
    "patch_minor": null,
    "version": "4.2.1"
  }
}
Learn more about the parse_user_agent function

parse_xml

Parses the value as XML.

Parse XML

Source
value = s'<book category="CHILDREN"><title lang="en">Harry Potter</title><author>J K. Rowling</author><year>2005</year></book>';

parse_xml!(value, text_key: "value", parse_number: false)
Return
{
  "book": {
    "@category": "CHILDREN",
    "author": "J K. Rowling",
    "title": {
      "@lang": "en",
      "value": "Harry Potter"
    },
    "year": "2005"
  }
}
Learn more about the parse_xml function

Random examples

random_bool

Returns a random boolean.

Random boolean

Source
is_boolean(random_bool())
Return
true
Learn more about the random_bool function

random_bytes

A cryptographically secure random number generator. Returns a string value containing the number of random bytes requested.

Generate random base 64 encoded bytes

Source
encode_base64(random_bytes(16))
Return
LNu0BBgUbh7XAlXbjSOomQ==
Learn more about the random_bytes function

random_float

Returns a random float between [min, max).

Random float from 0.0 to 10.0, not including 10.0

Source
f = random_float(0.0, 10.0)
f >= 0 && f < 10
Return
true
Learn more about the random_float function

random_int

Returns a random integer between [min, max).

Random integer from 0 to 10, not including 10

Source
i = random_int(0, 10)
i >= 0 && i < 10
Return
true
Learn more about the random_int function

uuid_from_friendly_id

Convert a Friendly ID (base62 encoding a 128-bit word) to a UUID.

Convert a Friendly ID to a UUID

Source
uuid_from_friendly_id!("3s87yEvnmkiPBMHsj8bwwc")
Return
7f41deed-d5e2-8b5e-7a13-ab4ff93cfad2
Learn more about the uuid_from_friendly_id function

uuid_v4

Generates a random UUIDv4 string.

Create a UUIDv4

Source
uuid_v4()
Return
1d262f4f-199b-458d-879f-05fd0a5f0683
Learn more about the uuid_v4 function

uuid_v7

Generates a random UUIDv7 string.

Create a UUIDv7 with implicit now()

Source
uuid_v7()
Return
06338364-8305-7b74-8000-de4963503139
Learn more about the uuid_v7 function

Create a UUIDv7 with explicit now()

Source
uuid_v7(now())
Return
018e29b3-0bea-7f78-8af3-d32ccb1b93c1
Learn more about the uuid_v7 function

Create a UUIDv7 with custom timestamp

Source
uuid_v7(t'2020-12-30T22:20:53.824727Z')
Return
0176b5bd-5d19-7394-bb60-c21028c6152b
Learn more about the uuid_v7 function

String examples

camelcase

Takes the value string, and turns it into camelCase. Optionally, you can pass in the existing case of the function, or else an attempt is made to determine the case automatically.

camelCase a string

Source
camelcase("input-string")
Return
inputString
Learn more about the camelcase function

camelCase a string

Source
camelcase("input-string", "kebab-case")
Return
inputString
Learn more about the camelcase function

community_id

Generates an ID based on the Community ID Spec.

TCP

Source
community_id!(source_ip: "1.2.3.4", destination_ip: "5.6.7.8", source_port: 1122, destination_port: 3344, protocol: 6)
Return
1:wCb3OG7yAFWelaUydu0D+125CLM=
Learn more about the community_id function

contains

Determines whether the value string contains the specified substring.

String contains (case sensitive)

Source
contains("The Needle In The Haystack", "Needle")
Return
true
Learn more about the contains function

String contains (case insensitive)

Source
contains("The Needle In The Haystack", "needle", case_sensitive: false)
Return
true
Learn more about the contains function

contains_all

Determines whether the value string contains all the specified substrings.

String contains all

Source
contains_all("The Needle In The Haystack", ["Needle", "Haystack"])
Return
true
Learn more about the contains_all function

String contains all (case sensitive)

Source
contains_all("the NEEDLE in the haystack", ["needle", "haystack"])
Learn more about the contains_all function

downcase

Downcases the value string, where downcase is defined according to the Unicode Derived Core Property Lowercase.

Downcase a string

Source
downcase("Hello, World!")
Return
hello, world!
Learn more about the downcase function

ends_with

Determines whether the value string ends with the specified substring.

String ends with (case sensitive)

Source
ends_with("The Needle In The Haystack", "The Haystack")
Return
true
Learn more about the ends_with function

String ends with (case insensitive)

Source
ends_with("The Needle In The Haystack", "the haystack", case_sensitive: false)
Return
true
Learn more about the ends_with function

find

Determines from left to right the start position of the first found element in value that matches pattern. Returns -1 if not found.

Match text

Source
find("foobar", "foo")
Learn more about the find function

Match regex

Source
find("foobar", r'b.r')
Return
3
Learn more about the find function

No matches

Source
find("foobar", "baz")
Return
-1
Learn more about the find function

With an offset

Source
find("foobarfoobarfoo", "bar", 4)
Return
9
Learn more about the find function

join

Joins each string in the value array into a single string, with items optionally separated from one another by a separator.

Join array (no separator)

Source
join!(["bring", "us", "together"])
Return
bringustogether
Learn more about the join function

Join array (comma separator)

Source
join!(["sources", "transforms", "sinks"], separator: ", ")
Return
sources, transforms, sinks
Learn more about the join function

kebabcase

Takes the value string, and turns it into kebab-case. Optionally, you can pass in the existing case of the function, or else we will try to figure out the case automatically.

kebab-case a string

Source
kebabcase("InputString")
Return
input-string
Learn more about the kebabcase function

kebab-case a string

Source
kebabcase("InputString", "PascalCase")
Return
input-string
Learn more about the kebabcase function

match

Determines whether the value matches the pattern.

Regex match on a string

Source
match("I'm a little teapot", r'teapot')
Return
true
Learn more about the match function

String does not match the regular expression

Source
match("I'm a little teapot", r'.*balloon')
Learn more about the match function

match_any

Determines whether value matches any of the given patterns. All patterns are checked in a single pass over the target string, giving this function a potential performance advantage over the multiple calls in the match function.

Regex match on a string

Source
match_any("I'm a little teapot", [r'frying pan', r'teapot'])
Return
true
Learn more about the match_any function

parse_float

Parses the string value representing a floating point number in base 10 to a float.

Parse negative integer

Source
parse_float!("-42")
Return
-42
Learn more about the parse_float function

Parse negative integer

Source
parse_float!("42.38")
Return
42.38
Learn more about the parse_float function

Scientific notation

Source
parse_float!("2.5e3")
Return
2500
Learn more about the parse_float function

pascalcase

Takes the value string, and turns it into PascalCase. Optionally, you can pass in the existing case of the function, or else we will try to figure out the case automatically.

PascalCase a string

Source
pascalcase("input-string")
Return
InputString
Learn more about the pascalcase function

PascalCase a string

Source
pascalcase("input-string", "kebab-case")
Return
InputString
Learn more about the pascalcase function

redact

Redact sensitive data in value such as:

This can help achieve compliance by ensuring sensitive data does not leave your network.

Replace text using a regex

Source
redact("my id is 123456", filters: [r'\d+'])
Return
my id is [REDACTED]
Learn more about the redact function

Replace us social security numbers in any field

Source
redact({ "name": "John Doe", "ssn": "123-12-1234"}, filters: ["us_social_security_number"])
Return
{
  "name": "John Doe",
  "ssn": "[REDACTED]"
}
Learn more about the redact function

Replace with custom text

Source
redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "text", "replacement": "***"})
Return
my id is ***
Learn more about the redact function

Replace with SHA-2 hash

Source
redact("my id is 123456", filters: [r'\d+'], redactor: "sha2")
Return
my id is GEtTedW1p6tC094dDKH+3B8P+xSnZz69AmpjaXRd63I=
Learn more about the redact function

Replace with SHA-3 hash

Source
redact("my id is 123456", filters: [r'\d+'], redactor: "sha3")
Return
my id is ZNCdmTDI7PeeUTFnpYjLdUObdizo+bIupZdl8yqnTKGdLx6X3JIqPUlUWUoFBikX+yTR+OcvLtAqWO11NPlNJw==
Learn more about the redact function

Replace with SHA-256 hash using hex encoding

Source
redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "sha2", "variant": "SHA-256", "encoding": "base16"})
Return
my id is 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92
Learn more about the redact function

replace

Replaces all matching instances of pattern in value.

The pattern argument accepts regular expression capture groups.

Note when using capture groups:

  • You will need to escape the $ by using $$ to avoid Vector interpreting it as an environment variable when loading configuration
  • If you want a literal $ in the replacement pattern, you will also need to escape this with $$. When combined with environment variable interpolation in config files this means you will need to use $$$$ to have a literal $ in the replacement pattern.

Replace literal text

Source
replace("Apples and Bananas", "and", "not")
Return
Apples not Bananas
Learn more about the replace function

Replace using regular expression

Source
replace("Apples and Bananas", r'(?i)bananas', "Pineapples")
Return
Apples and Pineapples
Learn more about the replace function

Replace first instance

Source
replace("Bananas and Bananas", "Bananas", "Pineapples", count: 1)
Return
Pineapples and Bananas
Learn more about the replace function

Replace with capture groups (Note: Use $$num in config files)

Source
replace("foo123bar", r'foo(?P<num>\d+)bar', "$num")
Return
123
Learn more about the replace function

replace_with

Replaces all matching instances of pattern using a closure.

The pattern argument accepts a regular expression that can use capture groups.

The function uses the function closure syntax to compute the replacement values.

The closure takes a single parameter, which is an array, where the first item is always present and contains the entire string that matched pattern. The items from index one on contain the capture groups of the corresponding index. If a capture group is optional, the value may be null if it didn’t match.

The value returned by the closure must be a string and will replace the section of the input that was matched.

This returns a new string with the replacements, the original string is not mutated.

Capitalize words

Source
	replace_with("apples and bananas", r'\b(\w)(\w*)') -> |match| {
		upcase!(match.captures[0]) + string!(match.captures[1])
	}
Return
Apples And Bananas
Learn more about the replace_with function

Replace with hash

Source
	replace_with("email from test@example.com", r'\w+@example.com') -> |match| {
		sha2(match.string, variant: "SHA-512/224")
	}
Return
email from adf6e1bc4415d24912bd93072ad34ef825a7b6eb3bf53f68def1fc17
Learn more about the replace_with function

Replace first instance

Source
	replace_with("Apples and Apples", r'(?i)apples|cones', count: 1) -> |match| {
		"Pine" + downcase(match.string)
	}
Return
Pineapples and Apples
Learn more about the replace_with function

Named capture group

Source
	replace_with("level=error A message", r'level=(?P<level>\w+)') -> |match| {
		lvl = upcase!(match.level)
		"[{{lvl}}]"
	}
Return
[ERROR] A message
Learn more about the replace_with function

screamingsnakecase

Takes the value string, and turns it into SCREAMING_SNAKE case. Optionally, you can pass in the existing case of the function, or else we will try to figure out the case automatically.

SCREAMING_SNAKE a string

Source
screamingsnakecase("input-string")
Return
INPUT_STRING
Learn more about the screamingsnakecase function

SCREAMING_SNAKE a string

Source
screamingsnakecase("input-string", "kebab-case")
Return
INPUT_STRING
Learn more about the screamingsnakecase function

sieve

Keeps only matches of pattern in value.

This can be used to define patterns that are allowed in the string and remove everything else.

Sieve with regex

Source
sieve("test123%456.فوائد.net.", r'[a-z0-9.]')
Return
test123456..net.
Learn more about the sieve function

Custom replacements

Source
sieve("test123%456.فوائد.net.", r'[a-z.0-9]', replace_single: "X", replace_repeated: "<REMOVED>")
Return
test123X456.<REMOVED>.net.
Learn more about the sieve function

slice

Returns a slice of value between the start and end positions.

If the start and end parameters are negative, they refer to positions counting from the right of the string or array. If end refers to a position that is greater than the length of the string or array, a slice up to the end of the string or array is returned.

Slice a string (positive index)

Source
slice!("Supercalifragilisticexpialidocious", start: 5, end: 13)
Return
califrag
Learn more about the slice function

Slice a string (negative index)

Source
slice!("Supercalifragilisticexpialidocious", start: 5, end: -14)
Return
califragilistic
Learn more about the slice function

snakecase

Takes the value string, and turns it into snake-case. Optionally, you can pass in the existing case of the function, or else we will try to figure out the case automatically.

snake-case a string

Source
snakecase("input-string")
Return
input_string
Learn more about the snakecase function

snake-case a string

Source
snakecase("input-string", "kebab-case")
Return
input_string
Learn more about the snakecase function

split

Splits the value string using pattern.

Split a string (no limit)

Source
split("apples and pears and bananas", " and ")
Return
["apples","pears","bananas"]
Learn more about the split function

Split a string (with a limit)

Source
split("apples and pears and bananas", " and ", limit: 2)
Return
["apples","pears and bananas"]
Learn more about the split function

starts_with

Determines whether value begins with substring.

String starts with (case sensitive)

Source
starts_with("The Needle In The Haystack", "The Needle")
Return
true
Learn more about the starts_with function

String starts with (case insensitive)

Source
starts_with("The Needle In The Haystack", "the needle", case_sensitive: false)
Return
true
Learn more about the starts_with function

strip_ansi_escape_codes

Strips ANSI escape codes from value.

Strip ANSI escape codes

Source
strip_ansi_escape_codes("\e[46mfoo\e[0m bar")
Return
foo bar
Learn more about the strip_ansi_escape_codes function

strip_whitespace

Strips whitespace from the start and end of value, where whitespace is defined by the Unicode White_Space property.

Strip whitespace

Source
strip_whitespace("  A sentence.  ")
Return
A sentence.
Learn more about the strip_whitespace function

truncate

Truncates the value string up to the limit number of characters.

Truncate a string

Source
truncate("A rather long sentence.", limit: 11, suffix: "...")
Return
A rather lo...
Learn more about the truncate function

Truncate a string

Source
truncate("A rather long sentence.", limit: 11, suffix: "[TRUNCATED]")
Return
A rather lo[TRUNCATED]
Learn more about the truncate function

upcase

Upcases value, where upcase is defined according to the Unicode Derived Core Property Uppercase.

Upcase a string

Source
upcase("Hello, World!")
Return
HELLO, WORLD!
Learn more about the upcase function

System examples

get_env_var

Returns the value of the environment variable specified by name.

Get an environment variable

Source
get_env_var!("HOME")
Return
/root
Learn more about the get_env_var function

get_hostname

Returns the local system’s hostname.

Get hostname

Source
.hostname = get_hostname!()
Learn more about the get_hostname function

get_timezone_name

Returns the name of the timezone in the Vector configuration (see global configuration options). If the configuration is set to local, then it attempts to determine the name of the timezone from the host OS. If this is not possible, then it returns the fixed offset of the local timezone for the current time in the format "[+-]HH:MM", for example, "+02:00".

Get the IANA name of Vector’s timezone

Source
.vector_timezone = get_timezone_name!()
Learn more about the get_timezone_name function

Timestamp examples

format_timestamp

Formats value into a string representation of the timestamp.

Format a timestamp (ISO8601/RFC 3339)

Source
format_timestamp!(t'2020-10-21T16:00:00Z', format: "%+")
Return
2020-10-21T16:00:00+00:00
Learn more about the format_timestamp function

Format a timestamp (custom)

Source
format_timestamp!(t'2020-10-21T16:00:00Z', format: "%v %R")
Return
21-Oct-2020 16:00
Learn more about the format_timestamp function

now

Returns the current timestamp in the UTC timezone with nanosecond precision.

Generate a current timestamp

Source
now()
Return
2021-03-04T10:51:15.928937Z
Learn more about the now function

Type examples

array

Returns value if it is an array, otherwise returns an error. This enables the type checker to guarantee that the returned value is an array and can be used in any function that expects an array.

Declare an array type

Source
array!(.value)
Return
[1,2,3]
Learn more about the array function

bool

Returns value if it is a Boolean, otherwise returns an error. This enables the type checker to guarantee that the returned value is a Boolean and can be used in any function that expects a Boolean.

Declare a Boolean type

Source
bool!(.value)
Learn more about the bool function

float

Returns value if it is a float, otherwise returns an error. This enables the type checker to guarantee that the returned value is a float and can be used in any function that expects a float.

Declare a float type

Source
float!(.value)
Return
42
Learn more about the float function

int

Returns value if it is an integer, otherwise returns an error. This enables the type checker to guarantee that the returned value is an integer and can be used in any function that expects an integer.

Declare an integer type

Source
int!(.value)
Return
42
Learn more about the int function

is_array

Check if the value’s type is an array.

Valid array

Source
is_array([1, 2, 3])
Return
true
Learn more about the is_array function

Non-matching type

Source
is_array("a string")
Learn more about the is_array function

is_boolean

Check if the value’s type is a boolean.

Valid boolean

Source
is_boolean(false)
Return
true
Learn more about the is_boolean function

Non-matching type

Source
is_boolean("a string")
Learn more about the is_boolean function

is_empty

Check if the object, array, or string has a length of 0.

Empty array

Source
is_empty([])
Return
true
Learn more about the is_empty function

Non-empty string

Source
is_empty("a string")
Learn more about the is_empty function

Non-empty object

Source
is_empty({"foo": "bar"})
Learn more about the is_empty function

is_float

Check if the value’s type is a float.

Valid float

Source
is_float(0.577)
Return
true
Learn more about the is_float function

Non-matching type

Source
is_float("a string")
Learn more about the is_float function

is_integer

Check if the value`’s type is an integer.

Valid integer

Source
is_integer(1)
Return
true
Learn more about the is_integer function

Non-matching type

Source
is_integer("a string")
Learn more about the is_integer function

is_json

Check if the string is a valid JSON document.

Valid JSON object

Source
is_json("{}")
Return
true
Learn more about the is_json function

Non-valid value

Source
is_json("{")
Learn more about the is_json function

Exact variant

Source
is_json("{}", variant: "object")
Return
true
Learn more about the is_json function

Non-valid exact variant

Source
is_json("{}", variant: "array")
Learn more about the is_json function

is_null

Check if value’s type is null. For a more relaxed function, see is_nullish.

Null value

Source
is_null(null)
Return
true
Learn more about the is_null function

Non-matching type

Source
is_null("a string")
Learn more about the is_null function

is_nullish

Determines whether value is nullish, where nullish denotes the absence of a meaningful value.

Null detection (blank string)

Source
is_nullish("")
Return
true
Learn more about the is_nullish function

Null detection (dash string)

Source
is_nullish("-")
Return
true
Learn more about the is_nullish function

Null detection (whitespace)

Source
is_nullish("
  
")
Return
true
Learn more about the is_nullish function

is_object

Check if value’s type is an object.

Valid object

Source
is_object({"foo": "bar"})
Return
true
Learn more about the is_object function

Non-matching type

Source
is_object("a string")
Learn more about the is_object function

is_regex

Check if value’s type is a regex.

Valid regex

Source
is_regex(r'pattern')
Return
true
Learn more about the is_regex function

Non-matching type

Source
is_regex("a string")
Learn more about the is_regex function

is_string

Check if value’s type is a string.

Valid string

Source
is_string("a string")
Return
true
Learn more about the is_string function

Non-matching type

Source
is_string([1, 2, 3])
Learn more about the is_string function

is_timestamp

Check if value’s type is a timestamp.

Valid timestamp

Source
is_timestamp(t'2021-03-26T16:00:00Z')
Return
true
Learn more about the is_timestamp function

Non-matching type

Source
is_timestamp("a string")