APT and RPM repositories at repositories.timber.io will be decommissioned on February 28th Migration instructions

VRL function reference

Here you’ll find a comprehensive list of all built-in VRL functions. Functions are categorized by their purpose and sorted alphabetically for easy discovery. To use these functions in Vector, see the documentation on function call expressions and Vector’s remap transform.

Array functions

append

infallible pure
Appends each item in the items array to the end of the value array.

Function spec

append(value: <array>, items: <array>)
:: <array>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
The initial array.yes
itemsarray
The items to append.yes

Examples

Append to an array
Source
append([1, 2], [3, 4])
Return
[1,2,3,4]

chunks

fallible pure
Chunks value into slices of length chunk_size bytes.

Function spec

chunks(value: <array | string>, chunk_size: <integer>)
:: <array> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
string
The array of bytes to split.yes
chunk_sizeinteger
The desired length of each chunk in bytes. This may be constrained by the host platform architecture.yes

Errors

The chunks function is fallible, which means that error handling is required for these errors:
chunk_size must be at least 1 byte.
chunk_size is too large.

Examples

Split a string into chunks
Source
chunks("abcdefgh", 4)
Return
["abcd","efgh"]
Chunks do not respect unicode code point boundaries
Source
chunks("ab你好", 4)
Return
["ab�","�好"]

push

infallible pure
Adds the item to the end of the value array.

Function spec

push(value: <array>, item: <any>)
:: <array>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
The target array.yes
itemany
The item to push.yes

Examples

Push an item onto an array
Source
push([1, 2], 3)
Return
[1,2,3]

Codec functions

decode_base16

fallible pure
Decodes the value (a Base16 string) into its original string.

Function spec

decode_base16(value: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The Base16 data to decode.yes

Errors

The decode_base16 function is fallible, which means that error handling is required for these errors:
value isn’t a valid encoded Base16 string.

Examples

Decode Base16 data
Source
decode_base16!("796f752068617665207375636365737366756c6c79206465636f646564206d65")
Return
you have successfully decoded me

decode_base64

fallible pure
Decodes the value (a Base64 string) into its original string.

Function spec

decode_base64(value: <string>, [charset: <string>])
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The Base64 data to decode.yes
charsetstring
The character set to use when decoding the data.standardno

Errors

The decode_base64 function is fallible, which means that error handling is required for these errors:
value isn’t a valid encoded Base64 string.

Examples

Decode Base64 data (default)
Source
decode_base64!("eW91IGhhdmUgc3VjY2Vzc2Z1bGx5IGRlY29kZWQgbWU=")
Return
you have successfully decoded me
Decode Base64 data (URL safe)
Source
decode_base64!("eW91IGNhbid0IG1ha2UgeW91ciBoZWFydCBmZWVsIHNvbWV0aGluZyBpdCB3b24ndA==", charset: "url_safe")
Return
you can't make your heart feel something it won't

decode_gzip

fallible pure
Decodes the value (a Gzip string) into its original string.

Function spec

decode_gzip(value: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The Gzip data to decode.yes

Errors

The decode_gzip function is fallible, which means that error handling is required for these errors:
value isn’t a valid encoded Gzip string.

Examples

Decode Gzip data
Source
encoded_text = decode_base64!("H4sIAHEAymMAA6vML1XISCxLVSguTU5OLS5OK83JqVRISU3OT0lNUchNBQD7BGDaIAAAAA==")
decode_gzip!(encoded_text)
Return
you have successfully decoded me

decode_mime_q

fallible pure
Replaces q-encoded or base64-encoded encoded-word substrings in the value with their original string.

Function spec

decode_mime_q(value: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string with encoded-words to decode.yes

Errors

The decode_mime_q function is fallible, which means that error handling is required for these errors:
value has invalid encoded encoded-word string.

Examples

Decode single encoded-word
Source
decode_mime_q!("=?utf-8?b?SGVsbG8sIFdvcmxkIQ==?=")
Return
Hello, World!
Embedded
Source
decode_mime_q!("From: =?utf-8?b?SGVsbG8sIFdvcmxkIQ==?= <=?utf-8?q?hello=5Fworld=40example=2ecom?=>")
Return
From: Hello, World! <hello_world@example.com>
Without charset
Source
decode_mime_q!("?b?SGVsbG8sIFdvcmxkIQ==")
Return
Hello, World!

decode_percent

infallible pure
Decodes a percent-encoded value like a URL.

Function spec

decode_percent(value: <string>)
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to decode.yes

Examples

Percent decode a value
Source
decode_percent("foo%20bar%3F")
Return
foo bar?

decode_punycode

fallible pure
Decodes a punycode encoded value, like an internationalized domain name (IDN).

Function spec

decode_punycode(value: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to decode.yes

Errors

The decode_punycode function is fallible, which means that error handling is required for these errors:
value is not valid punycode

Examples

Decode a punycode encoded internationalized domain name
Source
decode_punycode!("www.xn--caf-dma.com")
Return
www.café.com
Decode an ASCII only string
Source
decode_punycode!("www.cafe.com")
Return
www.cafe.com

decode_snappy

fallible pure
Decodes the value (a Snappy string) into its original string.

Function spec

decode_snappy(value: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The Snappy data to decode.yes

Errors

The decode_snappy function is fallible, which means that error handling is required for these errors:
value isn’t a valid encoded Snappy string.

Examples

Decode Snappy data
Source
encoded_text = decode_base64!("LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg==")
decode_snappy!(encoded_text)
Return
The quick brown fox jumps over 13 lazy dogs.

decode_zlib

fallible pure
Decodes the value (a Zlib string) into its original string.

Function spec

decode_zlib(value: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The Zlib data to decode.yes

Errors

The decode_zlib function is fallible, which means that error handling is required for these errors:
value isn’t a valid encoded Zlib string.

Examples

Decode Zlib data
Source
encoded_text = decode_base64!("eJwNy4ENwCAIBMCNXIlQ/KqplUSgCdvXAS41qPMHshCB2R1zJlWIVlR6UURX2+wx2YcuK3kAb9C1wd6dn7Fa+QH9gRxr")
decode_zlib!(encoded_text)
Return
you_have_successfully_decoded_me.congratulations.you_are_breathtaking.

decode_zstd

fallible pure
Decodes the value (a Zstandard string) into its original string.

Function spec

decode_zstd(value: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The Zstandard data to decode.yes

Errors

The decode_zstd function is fallible, which means that error handling is required for these errors:
value isn’t a valid encoded Zstd string.

Examples

Decode Zstd data
Source
encoded_text = decode_base64!("KLUv/QBY/QEAYsQOFKClbQBedqXsb96EWDax/f/F/z+gNU4ZTInaUeAj82KqPFjUzKqhcfDqAIsLvAsnY1bI/N2mHzDixRQA")
decode_zstd!(encoded_text)
Return
you_have_successfully_decoded_me.congratulations.you_are_breathtaking.

encode_base16

infallible pure
Encodes the value to Base16.

Function spec

encode_base16(value: <string>)
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to encode.yes

Examples

Encode to Base16
Source
encode_base16("please encode me")
Return
706c6561736520656e636f6465206d65

encode_base64

infallible pure
Encodes the value to Base64.

Function spec

encode_base64(value: <string>, [padding: <boolean>, charset: <string>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to encode.yes
paddingboolean
Whether the Base64 output is padded.trueno
charsetstring
The character set to use when encoding the data.standardno

Examples

Encode to Base64 (default)
Source
encode_base64("please encode me")
Return
cGxlYXNlIGVuY29kZSBtZQ==
Encode to Base64 (without padding)
Source
encode_base64("please encode me, no padding though", padding: false)
Return
cGxlYXNlIGVuY29kZSBtZSwgbm8gcGFkZGluZyB0aG91Z2g
Encode to Base64 (URL safe)
Source
encode_base64("please encode me, but safe for URLs", charset: "url_safe")
Return
cGxlYXNlIGVuY29kZSBtZSwgYnV0IHNhZmUgZm9yIFVSTHM=

encode_gzip

infallible pure
Encodes the value to Gzip.

Function spec

encode_gzip(value: <string>, [compression_level: <integer>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to encode.yes
compression_levelinteger
The default compression level.6no

Examples

Encode to Gzip
Source
encoded_text = encode_gzip("please encode me")
encode_base64(encoded_text)
Return
H4sIAAAAAAAA/yvISU0sTlVIzUvOT0lVyE0FAI4R4vcQAAAA

encode_json

infallible pure
Encodes the value to JSON.

Function spec

encode_json(value: <any>, [pretty: <boolean>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value to convert to a JSON string.yes
prettyboolean
Whether to pretty print the JSON string or not.no

Examples

Encode to JSON
Source
.payload = encode_json({"hello": "world"})
Return
{"hello":"world"}

encode_key_value

fallible pure
Encodes the value into key-value format with customizable delimiters. Default delimiters match the logfmt format.

Function spec

encode_key_value(value: <object>, [fields_ordering: <array>, key_value_delimiter: <string>, field_delimiter: <string>, flatten_boolean: <boolean>])
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueobject
The value to convert to a string.yes
fields_orderingarray
The ordering of fields to preserve. Any fields not in this list are listed unordered, after all ordered fields.no
key_value_delimiterstring
The string that separates the key from the value.=no
field_delimiterstring
The string that separates each key-value pair.no
flatten_booleanboolean
Whether to encode key-value with a boolean value as a standalone key if true and nothing if false.no

Notices

This function has special behavior that you should be aware of.
If fields_ordering is specified then the function is fallible else it is infallible.

Errors

The encode_key_value function is fallible, which means that error handling is required for these errors:
fields_ordering contains a non-string element.

Examples

Encode with default delimiters (no ordering)
Source
encode_key_value({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
Return
lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z
Encode with default delimiters (fields ordering)
Source
encode_key_value!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
Return
ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345
Encode with default delimiters (nested fields)
Source
encode_key_value({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
Return
agent.name=foo event=log log.file.path=my.log
Encode with default delimiters (nested fields ordering)
Source
encode_key_value!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
Return
event=log log.file.path=my.log agent.name=foo
Encode with custom delimiters (no ordering)
Source
encode_key_value(
	{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"},
	field_delimiter: ",",
	key_value_delimiter: ":"
)
Return
lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z
Encode with custom delimiters and flatten boolean
Source
encode_key_value(
	{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "beta": true, "dropped": false},
	field_delimiter: ",",
	key_value_delimiter: ":",
	flatten_boolean: true
)
Return
beta,lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z

encode_logfmt

fallible pure
Encodes the value to logfmt.

Function spec

encode_logfmt(value: <object>, [fields_ordering: <array>])
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueobject
The value to convert to a logfmt string.yes
fields_orderingarray
The ordering of fields to preserve. Any fields not in this list are listed unordered, after all ordered fields.no

Notices

This function has special behavior that you should be aware of.
If fields_ordering is specified then the function is fallible else it is infallible.

Errors

The encode_logfmt function is fallible, which means that error handling is required for these errors:
fields_ordering contains a non-string element.

Examples

Encode to logfmt (no ordering)
Source
encode_logfmt({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
Return
lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z
Encode to logfmt (fields ordering)
Source
encode_logfmt!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
Return
ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345
Encode to logfmt (nested fields)
Source
encode_logfmt({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
Return
agent.name=foo event=log log.file.path=my.log
Encode to logfmt (nested fields ordering)
Source
encode_logfmt!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
Return
event=log log.file.path=my.log agent.name=foo

encode_percent

infallible pure
Encodes a value with percent encoding to safely be used in URLs.

Function spec

encode_percent(value: <string>, [ascii_set: <string>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to encode.yes
ascii_setstring
The ASCII set to use when encoding the data.NON_ALPHANUMERICno

Examples

Percent encode all non-alphanumeric characters (default)
Source
encode_percent("foo bar?")
Return
foo%20bar%3F
Percent encode only control characters
Source
encode_percent("foo 	bar", ascii_set: "CONTROLS")
Return
foo %09bar

encode_punycode

fallible pure
Encodes a value to punycode. Useful for internationalized domain names (IDN).

Function spec

encode_punycode(value: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to encode.yes

Errors

The encode_punycode function is fallible, which means that error handling is required for these errors:
value can not be encoded to punycode

Examples

Encode an internationalized domain name
Source
encode_punycode!("www.café.com")
Return
www.xn--caf-dma.com
Encode an internationalized domain name with mixed case
Source
encode_punycode!("www.CAFé.com")
Return
www.xn--caf-dma.com
Encode an ASCII only string
Source
encode_punycode!("www.cafe.com")
Return
www.cafe.com

encode_snappy

fallible pure
Encodes the value to Snappy.

Function spec

encode_snappy(value: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to encode.yes

Errors

The encode_snappy function is fallible, which means that error handling is required for these errors:
value cannot be encoded into a Snappy string.

Examples

Encode to Snappy
Source
encoded_text = encode_snappy!("The quick brown fox jumps over 13 lazy dogs.")
encode_base64(encoded_text)
Return
LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg==

encode_zlib

infallible pure
Encodes the value to Zlib.

Function spec

encode_zlib(value: <string>, [compression_level: <integer>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to encode.yes
compression_levelinteger
The default compression level.6no

Examples

Encode to Zlib
Source
encoded_text = encode_zlib("please encode me")
encode_base64(encoded_text)
Return
eJwryElNLE5VSM1Lzk9JVchNBQA0RQX7

encode_zstd

infallible pure
Encodes the value to Zstandard.

Function spec

encode_zstd(value: <string>, [compression_level: <integer>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to encode.yes
compression_levelinteger
The default compression level.3no

Examples

Encode to Zstd
Source
encoded_text = encode_zstd("please encode me")
encode_base64(encoded_text)
Return
KLUv/QBYgQAAcGxlYXNlIGVuY29kZSBtZQ==

Coerce functions

to_bool

fallible pure
Coerces the value into a boolean.

Function spec

to_bool(value: <boolean | integer | float | null | string>)
:: <boolean> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueboolean
integer
float
null
string
The value to convert to a Boolean.yes

Errors

The to_bool function is fallible, which means that error handling is required for these errors:
value is not a supported boolean representation.

Examples

Coerce to a Boolean (string)
Source
to_bool!("yes")
Return
true
Coerce to a Boolean (float)
Source
to_bool(0.0)
Coerce to a Boolean (int)
Source
to_bool(0)
Coerce to a Boolean (null)
Source
to_bool(null)
Coerce to a Boolean (Boolean)
Source
to_bool(true)
Return
true

to_float

fallible pure
Coerces the value into a float.

Function spec

to_float(value: <integer | float | boolean | string | timestamp>)
:: <float> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
float
boolean
string
timestamp
The value to convert to a float. Must be convertible to a float, otherwise an error is raised.yes

Errors

The to_float function is fallible, which means that error handling is required for these errors:
value is not a supported float representation.

Examples

Coerce to a float
Source
to_float!("3.145")
Return
3.145
Coerce to a float (timestamp)
Source
to_float(t'2020-12-30T22:20:53.824727Z')
Return
1609366853.824727

to_int

fallible pure
Coerces the value into an integer.

Function spec

to_int(value: <integer | float | boolean | string | timestamp>)
:: <integer> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
float
boolean
string
timestamp
The value to convert to an integer.yes

Errors

The to_int function is fallible, which means that error handling is required for these errors:
value is a string but the text is not an integer.
value is not a string, int, or timestamp.

Examples

Coerce to an int (string)
Source
to_int!("2")
Return
2
Coerce to an int (timestamp)
Source
to_int(t'2020-12-30T22:20:53.824727Z')
Return
1609366853

to_regex

fallible pure
Coerces the value into a regex.

Function spec

to_regex(value: <string>)
:: <regex> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The value to convert to a regex.yes

Notices

This function has special behavior that you should be aware of.
Compiling a regular expression is an expensive operation and can limit Vector throughput. Don’t use this function unless you are absolutely sure there is no other way!

Errors

The to_regex function is fallible, which means that error handling is required for these errors:
value is not a string.

Examples

Coerce to a regex
Source
to_regex("^foo$") ?? r''
Return
^foo$

to_string

fallible pure
Coerces the value into a string.

Function spec

to_string(value: <integer | float | boolean | string | timestamp | null>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
float
boolean
string
timestamp
null
The value to convert to a string.yes

Errors

The to_string function is fallible, which means that error handling is required for these errors:
value is not an integer, float, boolean, string, timestamp, or null.

Examples

Coerce to a string (Boolean)
Source
to_string(true)
Return
true
Coerce to a string (int)
Source
to_string(52)
Return
52
Coerce to a string (float)
Source
to_string(52.2)
Return
52.2

Convert functions

from_unix_timestamp

infallible pure

Converts the value integer from a Unix timestamp to a VRL timestamp.

Converts from the number of seconds since the Unix epoch by default. To convert from milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.

Function spec

from_unix_timestamp(value: <integer>, [unit: <string>])
:: <timestamp>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
The Unix timestamp to convert.yes
unitstring
The time unit.secondsno

Examples

Convert from a Unix timestamp (seconds)
Source
from_unix_timestamp!(5)
Return
1970-01-01T00:00:05Z
Convert from a Unix timestamp (milliseconds)
Source
from_unix_timestamp!(5000, unit: "milliseconds")
Return
1970-01-01T00:00:05Z
Convert from a Unix timestamp (nanoseconds)
Source
from_unix_timestamp!(5000, unit: "nanoseconds")
Return
1970-01-01T00:00:00.000005Z

to_syslog_facility

fallible pure
Converts the value, a Syslog facility code, into its corresponding Syslog keyword. For example, 0 into "kern", 1 into "user", etc.

Function spec

to_syslog_facility(value: <integer>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
The facility code.yes

Errors

The to_syslog_facility function is fallible, which means that error handling is required for these errors:
value is not a valid Syslog facility code.

Examples

Coerce to a Syslog facility
Source
to_syslog_facility!(4)
Return
auth

to_syslog_level

fallible pure
Converts the value, a Syslog severity level, into its corresponding keyword, i.e. 0 into "emerg", 1 into "alert", etc.

Function spec

to_syslog_level(value: <integer>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
The severity level.yes

Errors

The to_syslog_level function is fallible, which means that error handling is required for these errors:
value isn’t a valid Syslog severity level.

Examples

Coerce to a Syslog level
Source
to_syslog_level!(5)
Return
notice

to_syslog_severity

fallible pure
Converts the value, a Syslog log level keyword, into a Syslog integer severity level (0 to 7).

Function spec

to_syslog_severity(value: <string>)
:: <integer> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The Syslog level keyword to convert.yes

Errors

The to_syslog_severity function is fallible, which means that error handling is required for these errors:
value is not a valid Syslog level keyword.

Examples

Coerce to Syslog severity
Source
to_syslog_severity!("alert")
Return
1

to_unix_timestamp

infallible pure

Converts the value timestamp into a Unix timestamp.

Returns the number of seconds since the Unix epoch by default. To return the number in milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.

Function spec

to_unix_timestamp(value: <timestamp>, [unit: <string>])
:: <integer>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuetimestamp
The timestamp to convert into a Unix timestamp.yes
unitstring
The time unit.secondsno

Examples

Convert to a Unix timestamp (seconds)
Source
to_unix_timestamp(t'2021-01-01T00:00:00+00:00')
Return
1609459200
Convert to a Unix timestamp (milliseconds)
Source
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "milliseconds")
Return
1609459200000
Convert to a Unix timestamp (nanoseconds)
Source
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "nanoseconds")
Return
1609459200000000000

Debug functions

assert

fallible impure
Asserts the condition, which must be a Boolean expression. The program is aborted with message if the condition evaluates to false.

Function spec

assert(condition: <boolean>, [message: <string>])
:: <null> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
conditionboolean
The condition to check.yes
messagestring
An optional custom error message. If the equality assertion fails, message is appended to the default message prefix. See the examples below for a fully formed log message sample.no

Notices

This function has special behavior that you should be aware of.
The assert function should be used in a standalone fashion and only when you want to abort the program. You should avoid it in logical expressions and other situations in which you want the program to continue if the condition evaluates to false.

Errors

The assert function is fallible, which means that error handling is required for these errors:
condition evaluates to false.

Examples

Assertion (true)
Source
assert!("foo" == "foo", message: "\"foo\" must be \"foo\"!")
Return
true
Assertion (false)
Source
assert!("foo" == "bar", message: "\"foo\" must be \"foo\"!")

assert_eq

infallible impure
Asserts that two expressions, left and right, have the same value. The program is aborted with message if they do not have the same value.

Function spec

assert_eq(left: <any>, right: <any>, [message: <string>])
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
leftany
The value to check for equality against right.yes
rightany
The value to check for equality against left.yes
messagestring
An optional custom error message. If the equality assertion fails, message is appended to the default message prefix. See the examples below for a fully formed log message sample.no

Notices

This function has special behavior that you should be aware of.
The assert_eq function should be used in a standalone fashion and only when you want to abort the program. You should avoid it in logical expressions and other situations in which you want the program to continue if the condition evaluates to false.

Examples

Successful assertion
Source
assert_eq!(1, 1)
Return
true
Unsuccessful assertion
Source
assert_eq!(127, [1, 2, 3])
Unsuccessful assertion with custom log message
Source
 assert_eq!(1, 0, message: "Unequal integers")

log

infallible impure
Logs the value to stdout at the specified level.

Function spec

log(value: <any>, [level: <string>, rate_limit_secs: <integer>])
:: <null>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueany
The value to log.yes
levelstring
The log level.infono
rate_limit_secsinteger
Specifies that the log message is output no more than once per the given number of seconds. Use a value of 0 to turn rate limiting off.1no

Examples

Log a message
Source
log("Hello, World!", level: "info", rate_limit_secs: 60)
Log an error
Source
_, err = to_int(.field)
if err != null {
	log(err, level: "error")
}

Enrichment functions

find_enrichment_table_records

infallible pure

Searches an enrichment table for rows that match the provided condition.

For file enrichment tables, this condition needs to be a VRL object in which the key-value pairs indicate a field to search mapped to a value to search in that field. This function returns the rows that match the provided condition(s). All fields need to match for rows to be returned; if any fields do not match, then no rows are returned.

There are currently two forms of search criteria:

  1. Exact match search. The given field must match the value exactly. Case sensitivity can be specified using the case_sensitive argument. An exact match search can use an index directly into the dataset, which should make this search fairly “cheap” from a performance perspective.

  2. Date range search. The given field must be greater than or equal to the from date and less than or equal to the to date. A date range search involves sequentially scanning through the rows that have been located using any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. Therefore, use date ranges as the only criteria when the enrichment data set is very small.

For geoip enrichment tables, this condition needs to be a VRL object with a single key-value pair whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected and without a value, null is used. This table can return the following fields:

  • ISP databases:

    • autonomous_system_number
    • autonomous_system_organization
    • isp
    • organization
  • City databases:

    • city_name
    • continent_code
    • country_code
    • country_name
    • region_code
    • region_name
    • metro_code
    • latitude
    • longitude
    • postal_code
    • timezone
  • Connection-Type databases:

    • connection_type

To use this function, you need to update your configuration to include an enrichment_tables parameter.

Function spec

find_enrichment_table_records(table: <string>, condition: <object>, [select: <array>, case_sensitive: <boolean>])
:: <array>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
tablestring
The enrichment table to search.yes
conditionobject
The condition to search on. Since the condition is used at boot time to create indices into the data, these conditions must be statically defined.yes
selectarray
A subset of fields from the enrichment table to return. If not specified, all fields are returned.no
case_sensitiveboolean
Whether text fields need to match cases exactly.trueno

Examples

Exact match
Source
find_enrichment_table_records!("test",
  {
	"surname": "smith",
  },
  case_sensitive: false)
Return
[{"firstname":"Bob","id":1,"surname":"Smith"},{"firstname":"Fred","id":2,"surname":"Smith"}]
Source
find_enrichment_table_records!("test",
  {
	"surname": "Smith",
	"date_of_birth": {
	  "from": t'1985-01-01T00:00:00Z',
	  "to": t'1985-12-31T00:00:00Z'
	}
  })
Return
[{"firstname":"Bob","id":1,"surname":"Smith"},{"firstname":"Fred","id":2,"surname":"Smith"}]

get_enrichment_table_record

fallible pure

Searches an enrichment table for a row that matches the provided condition. A single row must be matched. If no rows are found or more than one row is found, an error is returned.

For file enrichment tables, this condition needs to be a VRL object in which the key-value pairs indicate a field to search mapped to a value to search in that field. This function returns the rows that match the provided condition(s). All fields need to match for rows to be returned; if any fields do not match, then no rows are returned.

There are currently two forms of search criteria:

  1. Exact match search. The given field must match the value exactly. Case sensitivity can be specified using the case_sensitive argument. An exact match search can use an index directly into the dataset, which should make this search fairly “cheap” from a performance perspective.

  2. Date range search. The given field must be greater than or equal to the from date and less than or equal to the to date. A date range search involves sequentially scanning through the rows that have been located using any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. Therefore, use date ranges as the only criteria when the enrichment data set is very small.

For geoip enrichment tables, this condition needs to be a VRL object with a single key-value pair whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected and without a value, null is used. This table can return the following fields:

  • ISP databases:

    • autonomous_system_number
    • autonomous_system_organization
    • isp
    • organization
  • City databases:

    • city_name
    • continent_code
    • country_code
    • country_name
    • region_code
    • region_name
    • metro_code
    • latitude
    • longitude
    • postal_code
    • timezone
  • Connection-Type databases:

    • connection_type

To use this function, you need to update your configuration to include an enrichment_tables parameter.

Function spec

get_enrichment_table_record(table: <string>, condition: <object>, [select: <array>, case_sensitive: <boolean>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
tablestring
The enrichment table to search.yes
conditionobject
The condition to search on. Since the condition is used at boot time to create indices into the data, these conditions must be statically defined.yes
selectarray
A subset of fields from the enrichment table to return. If not specified, all fields are returned.no
case_sensitiveboolean
Whether the text fields match the case exactly.trueno

Errors

The get_enrichment_table_record function is fallible, which means that error handling is required for these errors:
The row is not found.
Multiple rows are found that match the condition.

Examples

Exact match
Source
get_enrichment_table_record!("test",
  {
    "surname": "bob",
    "firstname": "John"
  },
  case_sensitive: false)
Return
{
  "firstname": "Bob",
  "id": 1,
  "surname": "Smith"
}
Source
get_enrichment_table_record!("test",
  {
    "surname": "Smith",
    "date_of_birth": {
      "from": t'1985-01-01T00:00:00Z',
      "to": t'1985-12-31T00:00:00Z'
    }
  })
Return
{
  "firstname": "Bob",
  "id": 1,
  "surname": "Smith"
}

Enumerate functions

compact

infallible pure
Compacts the value by removing empty values, where empty values are defined using the available parameters.

Function spec

compact(value: <array | object>, [recursive: <boolean>, null: <boolean>, string: <boolean>, object: <boolean>, array: <boolean>, nullish: <boolean>])
:: <array | object>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
object
The object or array to compact.yes
recursiveboolean
Whether the compaction be recursive.trueno
nullboolean
Whether null should be treated as an empty value.trueno
stringboolean
Whether an empty string should be treated as an empty value.trueno
objectboolean
Whether an empty object should be treated as an empty value.trueno
arrayboolean
Whether an empty array should be treated as an empty value.trueno
nullishboolean
Tests whether the value is “nullish” as defined by the is_nullish function.no

Examples

Compact an array
Source
compact(["foo", "bar", "", null, [], "buzz"], string: true, array: true, null: true)
Return
["foo","bar","buzz"]
Compact an object
Source
compact({"field1": 1, "field2": "", "field3": [], "field4": null}, string: true, array: true, null: true)
Return
{
  "field1": 1
}

filter

infallible pure

Filter elements from a collection.

This function currently does not support recursive iteration.

The function uses the function closure syntax to allow reading the key-value or index-value combination for each item in the collection.

The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

See the examples below to learn about the closure syntax.

Function spec

filter(value: <array | object>)
:: <array | object>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
object
The array or object to filter.yes

Examples

Filter elements
Source
filter(array!(.tags)) -> |_index, value| {
    # keep any elements that aren't equal to "foo"
    value != "foo"
}
Return
["bar","baz"]

flatten

infallible pure
Flattens the value into a single-level representation.

Function spec

flatten(value: <array | object>, [separator: <string>])
:: <array | object>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
object
The array or object to flatten.yes
separatorstring
The separator to be used to join nested keys.no

Examples

Flatten array
Source
flatten([1, [2, 3, 4], [5, [6, 7], 8], 9])
Return
[1,2,3,4,5,6,7,8,9]
Flatten object
Source
flatten({
	"parent1": {
		"child1": 1,
		"child2": 2
	},
	"parent2": {
		"child3": 3
	}
})
Return
{
  "parent1.child1": 1,
  "parent1.child2": 2,
  "parent2.child3": 3
}

for_each

infallible pure

Iterate over a collection.

This function currently does not support recursive iteration.

The function uses the “function closure syntax” to allow reading the key/value or index/value combination for each item in the collection.

The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

See the examples below to learn about the closure syntax.

Function spec

for_each(value: <array | object>)
:: <null>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
object
The array or object to iterate.yes

Examples

Tally elements
Source
tally = {}
for_each(array!(.tags)) -> |_index, value| {
    # Get the current tally for the `value`, or
    # set to `0`.
    count = int(get!(tally, [value])) ?? 0

    # Increment the tally for the value by `1`.
    tally = set!(tally, [value], count + 1)
}

tally
Return
{
  "bar": 1,
  "baz": 1,
  "foo": 2
}

includes

infallible pure
Determines whether the value array includes the specified item.

Function spec

includes(value: <array>, item: <any>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
The array.yes
itemany
The item to check.yes

Examples

Array includes
Source
includes(["apple", "orange", "banana"], "banana")
Return
true

keys

infallible pure
Returns the keys from the object passed into the function.

Function spec

keys(value: <object>)
:: <array>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueobject
The object to extract keys from.yes

Examples

Get keys from the object
Source
keys({"key1": "val1", "key2": "val2"})
Return
["key1","key2"]

length

infallible pure

Returns the length of the value.

  • If value is an array, returns the number of elements.
  • If value is an object, returns the number of top-level keys.
  • If value is a string, returns the number of bytes in the string. If you want the number of characters, see strlen.

Function spec

length(value: <array | object | string>)
:: <integer>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
object
string
The array or object.yes

Examples

Length (object)
Source
length({
	"portland": "Trail Blazers",
	"seattle":  "Supersonics"
})
Return
2
Length (nested object)
Source
length({
	"home": {
		"city":  "Portland",
		"state": "Oregon"
	},
	"name": "Trail Blazers",
	"mascot": {
		"name": "Blaze the Trail Cat"
	}
})
Return
3
Length (array)
Source
length(["Trail Blazers", "Supersonics", "Grizzlies"])
Return
3
Length (string)
Source
length("The Planet of the Apes Musical")
Return
30

map_keys

infallible pure

Map the keys within an object.

If recursive is enabled, the function iterates into nested objects, using the following rules:

  1. Iteration starts at the root.
  2. For every nested object type:
    • First return the key of the object type itself.
    • Then recurse into the object, and loop back to item (1) in this list.
    • Any mutation done on a nested object before recursing into it, are preserved.
  3. For every nested array type:
    • First return the key of the array type itself.
    • Then find all objects within the array, and apply item (2) to each individual object.

The above rules mean that map_keys with recursive enabled finds all keys in the target, regardless of whether nested objects are nested inside arrays.

The function uses the function closure syntax to allow reading the key for each item in the object.

The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

See the examples below to learn about the closure syntax.

Function spec

map_keys(value: <object>, [recursive: <boolean>])
:: <object>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueobject
The object to iterate.yes
recursiveboolean
Whether to recursively iterate the collection.no

Examples

Upcase keys
Source
map_keys(.) -> |key| { upcase(key) }
Return
{
  "BAR": "bar",
  "FOO": "foo"
}
De-dot keys
Source
map_keys(., recursive: true) -> |key| { replace(key, ".", "_") }
Return
{
  "labels": {
    "app_kubernetes_io/name": "mysql"
  }
}

map_values

infallible pure

Map the values within a collection.

If recursive is enabled, the function iterates into nested collections, using the following rules:

  1. Iteration starts at the root.
  2. For every nested collection type:
    • First return the collection type itself.
    • Then recurse into the collection, and loop back to item (1) in the list
    • Any mutation done on a collection before recursing into it, are preserved.

The function uses the function closure syntax to allow mutating the value for each item in the collection.

The same scoping rules apply to closure blocks as they do for regular blocks, meaning, any variable defined in parent scopes are accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

Check out the examples below to learn about the closure syntax.

Function spec

map_values(value: <array | object>, [recursive: <boolean>])
:: <array | object>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
object
The object or array to iterate.yes
recursiveboolean
Whether to recursively iterate the collection.no

Examples

Upcase values
Source
map_values(.) -> |value| { upcase!(value) }
Return
{
  "bar": "BAR",
  "foo": "FOO"
}

match_array

infallible pure
Determines whether the elements in the value array matches the pattern. By default, it checks that at least one element matches, but can be set to determine if all the elements match.

Function spec

match_array(value: <array>, pattern: <regex>, [all: <boolean>])
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
The array.yes
patternregex
The regular expression pattern to match against.yes
allboolean
Whether to match on all elements of value.no

Examples

Match at least one element
Source
match_array(["foobar", "bazqux"], r'foo')
Return
true
Match all elements
Source
match_array(["foo", "foobar", "barfoo"], r'foo', all: true)
Return
true
No matches
Source
match_array(["bazqux", "xyz"], r'foo')
Not all elements match
Source
match_array(["foo", "foobar", "baz"], r'foo', all: true)

strlen

infallible pure

Returns the number of UTF-8 characters in value. This differs from length which counts the number of bytes of a string.

Note: This is the count of Unicode scalar values which can sometimes differ from Unicode code points.

Function spec

strlen(value: <string>)
:: <integer>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string.yes

Examples

strlen
Source
strlen("ñandú")
Return
5

unique

infallible pure

Returns the unique values for an array.

The first occurrence of each element is kept.

Function spec

unique(value: <array>)
:: <array>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuearray
The array to return unique elements from.yes

Examples

Unique
Source
unique(["foo", "bar", "foo", "baz"])
Return
["foo","bar","baz"]

values

infallible pure
Returns the values from the object passed into the function.

Function spec

values(value: <object>)
:: <array>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueobject
The object to extract values from.yes

Examples

Get values from the object
Source
values({"key1": "val1", "key2": "val2"})
Return
["val1","val2"]

Event functions

get_secret

infallible pure
Returns the value of the given secret from an event.

Function spec

get_secret(key: <string>)
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
keystring
The name of the secret.yes

Examples

Get the Datadog API key from the event metadata
Source
get_secret("datadog_api_key")
Return
secret value

remove_secret

infallible pure
Removes a secret from an event.

Function spec

remove_secret(key: <string>)
:: <null>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
keystring
The name of the secret to remove.yes

Examples

Removes the Datadog API key from the event
Source
remove_secret("datadog_api_key")

set_secret

infallible pure
Sets the given secret in the event.

Function spec

set_secret(key: <string>, secret: <string>)
:: <null>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
keystring
The name of the secret.yes
secretstring
The secret value.yes

Examples

Set the Datadog API key to the given value
Source
set_secret("datadog_api_key", "abc122")

set_semantic_meaning

infallible pure
Sets a semantic meaning for an event. Note: This function assigns meaning at startup, and has no runtime behavior. It is suggested to put all calls to this function at the beginning of a VRL function. The function cannot be conditionally called. For example, using an if statement cannot stop the meaning from being assigned.

Function spec

set_semantic_meaning(target: <path>, meaning: <string>)
:: <null>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
targetpath
The path of the value that is assigned a meaning.yes
meaningstring
The name of the meaning to assign.yes

Examples

Sets custom field semantic meaning
Source
set_semantic_meaning(.foo, "bar")

Path functions

del

infallible impure

Removes the field specified by the static path from the target.

For dynamic path deletion, see the remove function.

Function spec

del(path: <path>, [compact: <boolean>])
:: <any>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
pathpath
The path of the field to delete.yes
compactboolean
After deletion, if compact is true and there is an empty object or array left, the empty object or array is also removed, cascading up to the root. This only applies to the path being deleted, and any parent paths.no

Notices

This function has special behavior that you should be aware of.
The del function modifies the current event in place and returns the value of the deleted field.

Examples

Delete a field
Source
del(.field1)
Rename a field
Source
.new_field = del(.old_field)

exists

infallible pure

Checks whether the path exists for the target.

This function distinguishes between a missing path and a path with a null value. A regular path lookup, such as .foo, cannot distinguish between the two cases since it always returns null if the path doesn’t exist.

Function spec

exists(path: <path>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
pathpath
The path of the field to check.yes

Examples

Exists (field)
Source
exists(.field)
Return
true
Exists (array element)
Source
exists(.array[2])
Return
true

get

fallible pure

Dynamically get the value of a given path.

If you know the path you want to look up, use static paths such as .foo.bar[1] to get the value of that path. However, if you do not know the path names, use the dynamic get function to get the requested value.

Function spec

get(value: <object | array>, path: <array>)
:: <any> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueobject
array
The object or array to query.yes
patharray
An array of path segments to look for the value.yes

Errors

The get function is fallible, which means that error handling is required for these errors:
The path segment must be a string or an integer.

Examples

single-segment top-level field
Source
get!(value: { "foo": "bar" }, path: ["foo"])
Return
bar
multi-segment nested field
Source
get!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])
Return
baz
array indexing
Source
get!(value: ["foo", "bar", "baz"], path: [-2])
Return
bar

remove

fallible pure

Dynamically remove the value for a given path.

If you know the path you want to remove, use the del function and static paths such as del(.foo.bar[1]) to remove the value at that path. The del function returns the deleted value, and is more performant than remove. However, if you do not know the path names, use the dynamic remove function to remove the value at the provided path.

Function spec

remove(value: <object | array>, path: <array>, [compact: <boolean>])
:: <object | array> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueobject
array
The object or array to remove data from.yes
patharray
An array of path segments to remove the value from.yes
compactboolean
After deletion, if compact is true, any empty objects or arrays left are also removed.no

Errors

The remove function is fallible, which means that error handling is required for these errors:
The path segment must be a string or an integer.

Examples

single-segment top-level field
Source
remove!(value: { "foo": "bar" }, path: ["foo"])
multi-segment nested field
Source
remove!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])
Return
{
  "foo": {}
}
array indexing
Source
remove!(value: ["foo", "bar", "baz"], path: [-2])
Return
["foo","baz"]
compaction
Source
remove!(value: { "foo": { "bar": [42], "baz": true } }, path: ["foo", "bar", 0], compact: true)
Return
{
  "foo": {
    "baz": true
  }
}

set

fallible pure

Dynamically insert data into the path of a given object or array.

If you know the path you want to assign a value to, use static path assignments such as .foo.bar[1] = true for improved performance and readability. However, if you do not know the path names, use the dynamic set function to insert the data into the object or array.

Function spec

set(value: <object | array>, path: <array>, data: <any>)
:: <object | array> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueobject
array
The object or array to insert data into.yes
patharray
An array of path segments to insert the value into.yes
dataany
The data to be inserted.yes

Errors

The set function is fallible, which means that error handling is required for these errors:
The path segment must be a string or an integer.

Examples

single-segment top-level field
Source
set!(value: { "foo": "bar" }, path: ["foo"], data: "baz")
Return
{
  "foo": "baz"
}
multi-segment nested field
Source
set!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"], data: "qux")
Return
{
  "foo": {
    "bar": "qux"
  }
}
array
Source
set!(value: ["foo", "bar", "baz"], path: [-2], data: 42)
Return
["foo",42,"baz"]

Cryptography functions

decrypt

fallible pure

Decrypts a string with a symmetric encryption algorithm.

Supported Algorithms:

  • AES-256-CFB (key = 32 bytes, iv = 16 bytes)
  • AES-192-CFB (key = 24 bytes, iv = 16 bytes)
  • AES-128-CFB (key = 16 bytes, iv = 16 bytes)
  • AES-256-OFB (key = 32 bytes, iv = 16 bytes)
  • AES-192-OFB (key = 24 bytes, iv = 16 bytes)
  • AES-128-OFB (key = 16 bytes, iv = 16 bytes)
  • AES-256-CTR (key = 32 bytes, iv = 16 bytes)
  • AES-192-CTR (key = 24 bytes, iv = 16 bytes)
  • AES-128-CTR (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)

Function spec

decrypt(ciphertext: <string>, algorithm: <string>, key: <string>, iv: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
ciphertextstring
The string in raw bytes (not encoded) to decrypt.yes
algorithmstring
The algorithm to use.yes
keystring
The key in raw bytes (not encoded) for decryption. The length must match the algorithm requested.yes
ivstring
The IV in raw bytes (not encoded) for decryption. The length must match the algorithm requested. A new IV should be generated for every message. You can use random_bytes to generate a cryptographically secure random value. The value should match the one used during encryption.yes

Errors

The decrypt function is fallible, which means that error handling is required for these errors:
algorithm is not a supported algorithm.
key length does not match the key size required for the algorithm specified.
iv length does not match the iv size required for the algorithm specified.

Examples

Decrypt value
Source
ciphertext = decode_base64!("5fLGcu1VHdzsPcGNDio7asLqE1P43QrVfPfmP4i4zOU=")
iv = decode_base64!("fVEIRkIiczCRWNxaarsyxA==")
key = "16_byte_keyxxxxx"
decrypt!(ciphertext, "AES-128-CBC-PKCS7", key, iv: iv)
Return
super_secret_message

encrypt

fallible pure

Encrypts a string with a symmetric encryption algorithm.

Supported Algorithms:

  • AES-256-CFB (key = 32 bytes, iv = 16 bytes)
  • AES-192-CFB (key = 24 bytes, iv = 16 bytes)
  • AES-128-CFB (key = 16 bytes, iv = 16 bytes)
  • AES-256-OFB (key = 32 bytes, iv = 16 bytes)
  • AES-192-OFB (key = 24 bytes, iv = 16 bytes)
  • AES-128-OFB (key = 16 bytes, iv = 16 bytes)
  • Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
  • Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
  • Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
  • AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
  • AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
  • AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
  • AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
  • AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
  • AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
  • CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
  • XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
  • XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)

Function spec

encrypt(plaintext: <string>, algorithm: <string>, key: <string>, iv: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
plaintextstring
The string to encrypt.yes
algorithmstring
The algorithm to use.yes
keystring
The key in raw bytes (not encoded) for encryption. The length must match the algorithm requested.yes
ivstring
The IV in raw bytes (not encoded) for encryption. The length must match the algorithm requested. A new IV should be generated for every message. You can use random_bytes to generate a cryptographically secure random value.yes

Errors

The encrypt function is fallible, which means that error handling is required for these errors:
algorithm is not a supported algorithm.
key length does not match the key size required for the algorithm specified.
iv length does not match the iv size required for the algorithm specified.

Examples

Encrypt value
Source
plaintext = "super secret message"
iv = "1234567890123456" # typically you would call random_bytes(16)
key = "16_byte_keyxxxxx"
encrypted_message = encrypt!(plaintext, "AES-128-CBC-PKCS7", key, iv: iv)
encode_base64(encrypted_message)
Return
GBw8Mu00v0Kc38+/PvsVtGgWuUJ+ZNLgF8Opy8ohIYE=

hmac

infallible pure

Calculates a HMAC of the value using the given key. The hashing algorithm used can be optionally specified.

For most use cases, the resulting bytestream should be encoded into a hex or base64 string using either encode_base16 or encode_base64.

This function is infallible if either the default algorithm value or a recognized-valid compile-time algorithm string literal is used. Otherwise, it is fallible.

Function spec

hmac(value: <string>, key: <string>, [algorithm: <string>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to calculate the HMAC for.yes
keystring
The string to use as the cryptographic key.yes
algorithmstring
The hashing algorithm to use.SHA-256no

Examples

Calculate message HMAC (defaults: SHA-256), encoding to a base64 string
Source
encode_base64(hmac("Hello there", "super-secret-key"))
Return
eLGE8YMviv85NPXgISRUZxstBNSU47JQdcXkUWcClmI=
Calculate message HMAC using SHA-224, encoding to a hex-encoded string
Source
encode_base16(hmac("Hello there", "super-secret-key", algorithm: "SHA-224"))
Return
42fccbc2b7d22a143b92f265a8046187558a94d11ddbb30622207e90
Calculate message HMAC using a variable hash algorithm
Source
.hash_algo = "SHA-256"
hmac_bytes, err = hmac("Hello there", "super-secret-key", algorithm: .hash_algo)
if err == null {
	.hmac = encode_base16(hmac_bytes)
}
Return
78b184f1832f8aff3934f5e0212454671b2d04d494e3b25075c5e45167029662

md5

infallible pure
Calculates an md5 hash of the value.

Function spec

md5(value: <string>)
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to calculate the hash for.yes

Examples

Create md5 hash
Source
md5("foo")
Return
acbd18db4cc2f85cedef654fccc4a4d8

seahash

infallible pure
Calculates a Seahash hash of the value. Note: Due to limitations in the underlying VRL data types, this function converts the unsigned 64-bit integer SeaHash result to a signed 64-bit integer. Results higher than the signed 64-bit integer maximum value wrap around to negative values.

Function spec

seahash(value: <string>)
:: <integer>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to calculate the hash for.yes

Examples

Calculate seahash
Source
seahash("foobar")
Return
5348458858952426000
Calculate negative seahash
Source
seahash("bar")
Return
-2796170501982571500

sha1

infallible pure
Calculates a SHA-1 hash of the value.

Function spec

sha1(value: <string>)
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to calculate the hash for.yes

Examples

Calculate sha1 hash
Source
sha1("foo")
Return
0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33

sha2

infallible pure
Calculates a SHA-2 hash of the value.

Function spec

sha2(value: <string>, [variant: <string>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to calculate the hash for.yes
variantstring
The variant of the algorithm to use.SHA-512/256no

Examples

Calculate sha2 hash
Source
sha2("foo", variant: "SHA-512/224")
Return
d68f258d37d670cfc1ec1001a0394784233f88f056994f9a7e5e99be

sha3

infallible pure
Calculates a SHA-3 hash of the value.

Function spec

sha3(value: <string>, [variant: <string>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to calculate the hash for.yes
variantstring
The variant of the algorithm to use.SHA3-512no

Examples

Calculate sha3 hash
Source
sha3("foo", variant: "SHA3-224")
Return
f4f6779e153c391bbd29c95e72b0708e39d9166c7cea51d1f10ef58a

IP functions

ip_aton

fallible pure

Converts IPv4 address in numbers-and-dots notation into network-order bytes represented as an integer.

This behavior mimics inet_aton.

Function spec

ip_aton(value: <string>)
:: <integer> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The IP address to convert to binary.yes

Errors

The ip_aton function is fallible, which means that error handling is required for these errors:
value is not a valid IPv4 address.

Examples

IPv4 to integer
Source
ip_aton!("1.2.3.4")
Return
16909060

ip_cidr_contains

fallible pure
Determines whether the ip is contained in the block referenced by the cidr.

Function spec

ip_cidr_contains(cidr: <string>, ip: <string>)
:: <boolean> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
cidrstring
The CIDR mask (v4 or v6).yes
ipstring
The IP address (v4 or v6).yes

Errors

The ip_cidr_contains function is fallible, which means that error handling is required for these errors:
cidr is not a valid CIDR.
ip is not a valid IP address.

Examples

IPv4 contains CIDR
Source
ip_cidr_contains!("192.168.0.0/16", "192.168.10.32")
Return
true
IPv6 contains CIDR
Source
ip_cidr_contains!("2001:4f8:4:ba::/64", "2001:4f8:4:ba:2e0:81ff:fe22:d1f1")
Return
true

ip_ntoa

fallible pure

Converts numeric representation of IPv4 address in network-order bytes to numbers-and-dots notation.

This behavior mimics inet_ntoa.

Function spec

ip_ntoa(value: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The integer representation of an IPv4 address.yes

Errors

The ip_ntoa function is fallible, which means that error handling is required for these errors:
value cannot fit in an unsigned 32-bit integer.

Examples

Integer to IPv4
Source
ip_ntoa!(16909060)
Return
1.2.3.4

ip_ntop

fallible pure

Converts IPv4 and IPv6 addresses from binary to text form.

This behavior mimics inet_ntop.

Function spec

ip_ntop(value: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The binary data to convert from. For IPv4 addresses, it must be 4 bytes (32 bits) long. For IPv6 addresses, it must be 16 bytes (128 bits) long.yes

Notices

This function has special behavior that you should be aware of.
The binary data for this function is not easily printable. However, the results from functions such as decode_base64 or decode_percent can still be used correctly.

Errors

The ip_ntop function is fallible, which means that error handling is required for these errors:
value must be of length 4 or 16 bytes.

Examples

Convert IPv4 address from bytes after decoding from Base64
Source
ip_ntop!(decode_base64!("wKgAAQ=="))
Return
192.168.0.1
Convert IPv6 address from bytes after decoding from Base64
Source
ip_ntop!(decode_base64!("IAENuIWjAAAAAIouA3BzNA=="))
Return
2001:db8:85a3::8a2e:370:7334

ip_pton

fallible pure

Converts IPv4 and IPv6 addresses from text to binary form.

  • The binary form of IPv4 addresses is 4 bytes (32 bits) long.
  • The binary form of IPv6 addresses is 16 bytes (128 bits) long.

This behavior mimics inet_pton.

Function spec

ip_pton(value: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The IP address (v4 or v6) to convert to binary form.yes

Notices

This function has special behavior that you should be aware of.
The binary data from this function is not easily printable. However, functions such as encode_base64 or encode_percent can still process it correctly.

Errors

The ip_pton function is fallible, which means that error handling is required for these errors:
value is not a valid IP (v4 or v6) address in text form.

Examples

Convert IPv4 address to bytes and encode to Base64
Source
encode_base64(ip_pton!("192.168.0.1"))
Return
wKgAAQ==
Convert IPv6 address to bytes and encode to Base64
Source
encode_base64(ip_pton!("2001:db8:85a3::8a2e:370:7334"))
Return
IAENuIWjAAAAAIouA3BzNA==

ip_subnet

fallible pure
Extracts the subnet address from the ip using the supplied subnet.

Function spec

ip_subnet(ip: <string>, subnet: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
ipstring
The IP address (v4 or v6).yes
subnetstring
The subnet to extract from the IP address. This can be either a prefix length like /8 or a net mask like 255.255.0.0. The net mask can be either an IPv4 or IPv6 address.yes

Notices

This function has special behavior that you should be aware of.
Works with both IPv4 and IPv6 addresses. The IP version for the mask must be the same as the supplied address.

Errors

The ip_subnet function is fallible, which means that error handling is required for these errors:
ip is not a valid IP address.
subnet is not a valid subnet.

Examples

IPv4 subnet
Source
ip_subnet!("192.168.10.32", "255.255.255.0")
Return
192.168.10.0
IPv6 subnet
Source
ip_subnet!("2404:6800:4003:c02::64", "/32")
Return
2404:6800::

ip_to_ipv6

fallible pure
Converts the ip to an IPv6 address.

Function spec

ip_to_ipv6(ip: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
ipstring
The IP address to convert to IPv6.yes

Errors

The ip_to_ipv6 function is fallible, which means that error handling is required for these errors:
ip is not a valid IP address.

Examples

IPv4 to IPv6
Source
ip_to_ipv6!("192.168.10.32")
Return
::ffff:192.168.10.32

ipv6_to_ipv4

fallible pure
Converts the ip to an IPv4 address. ip is returned unchanged if it’s already an IPv4 address. If ip is currently an IPv6 address then it needs to be IPv4 compatible, otherwise an error is thrown.

Function spec

ipv6_to_ipv4(ip: <string>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
ipstring
The IPv4-mapped IPv6 address to convert.yes

Errors

The ipv6_to_ipv4 function is fallible, which means that error handling is required for these errors:
ip is not a valid IP address.
ip is an IPv6 address that is not compatible with IPv4.

Examples

IPv6 to IPv4
Source
ipv6_to_ipv4!("::ffff:192.168.0.1")
Return
192.168.0.1

is_ipv4

infallible pure

Check if the string is a valid IPv4 address or not.

An [IPv4-mapped][https://datatracker.ietf.org/doc/html/rfc6890] or [IPv4-compatible][https://datatracker.ietf.org/doc/html/rfc6890] IPv6 address is not considered valid for the purpose of this function.

Function spec

is_ipv4(value: <string>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The IP address to checkyes

Examples

Valid IPv4 address
Source
is_ipv4("10.0.102.37")
Return
true
Valid IPv6 address
Source
is_ipv4("2001:0db8:85a3:0000:0000:8a2e:0370:7334")
Arbitrary string
Source
is_ipv4("foobar")

is_ipv6

infallible pure
Check if the string is a valid IPv6 address or not.

Function spec

is_ipv6(value: <string>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The IP address to checkyes

Examples

Valid IPv6 address
Source
is_ipv6("2001:0db8:85a3:0000:0000:8a2e:0370:7334")
Return
true
Valid IPv4 address
Source
is_ipv6("10.0.102.37")
Arbitrary string
Source
is_ipv6("foobar")

Number functions

abs

infallible pure
Computes the absolute value of value.

Function spec

abs(value: <integer | float>)
:: <integer | float>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
float
The number to calculate the absolute value.yes

Examples

Computes the absolute value of the integer
Source
abs(-42)
Return
42
Computes the absolute value of the float
Source
abs(-42.2)
Return
42.2

ceil

infallible pure
Rounds the value up to the specified precision.

Function spec

ceil(value: <integer | float>, [precision: <integer>])
:: <integer | float>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
float
The number to round up.yes
precisioninteger
The number of decimal places to round to.no

Examples

Round a number up (without precision)
Source
ceil(4.345)
Return
5
Round a number up (with precision)
Source
ceil(4.345, precision: 2)
Return
4.35

floor

infallible pure
Rounds the value down to the specified precision.

Function spec

floor(value: <integer | float>, [precision: <integer>])
:: <integer | float>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
float
The number to round down.yes
precisioninteger
The number of decimal places to round to.no

Examples

Round a number down (without precision)
Source
floor(4.345)
Return
4
Round a number down (with precision)
Source
floor(4.345, precision: 2)
Return
4.34

format_int

fallible pure
Formats the integer value into a string representation using the given base/radix.

Function spec

format_int(value: <integer>, [base: <integer>])
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
The number to format.yes
baseinteger
The base to format the number in. Must be between 2 and 36 (inclusive).10no

Errors

The format_int function is fallible, which means that error handling is required for these errors:
The base is not between 2 and 36.

Examples

Format as a hexadecimal integer
Source
format_int!(42, 16)
Return
2a
Format as a negative hexadecimal integer
Source
format_int!(-42, 16)
Return
-2a

format_number

infallible pure
Formats the value into a string representation of the number.

Function spec

format_number(value: <integer | float>, [scale: <integer>, decimal_separator: <string>, grouping_separator: <string>])
:: <string>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
float
The number to format as a string.yes
scaleinteger
The number of decimal places to display.no
decimal_separatorstring
The character to use between the whole and decimal parts of the number..no
grouping_separatorstring
The character to use between each thousands part of the number.,no

Examples

Format a number (3 decimals)
Source
format_number(1234567.89, 3, decimal_separator: ".", grouping_separator: ",")
Return
1,234,567.890

mod

fallible pure
Calculates the remainder of value divided by modulus.

Function spec

mod(value: <integer | float>, modulus: <integer | float>)
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
float
The value the modulus is applied to.yes
modulusinteger
float
The modulus value.yes

Errors

The mod function is fallible, which means that error handling is required for these errors:
value is not an integer or float.
modulus is not an integer or float.
modulus is equal to 0.

Examples

Calculate the remainder of two integers
Source
mod(5, 2)
Return
1

round

infallible pure
Rounds the value to the specified precision.

Function spec

round(value: <integer | float>, [precision: <integer>])
:: <integer | float>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueinteger
float
The number to round.yes
precisioninteger
The number of decimal places to round to.no

Examples

Round a number (without precision)
Source
round(4.345)
Return
4
Round a number (with precision)
Source
round(4.345, precision: 2)
Return
4.35

Object functions

match_datadog_query

infallible pure
Matches an object against a Datadog Search Syntax query.

Function spec

match_datadog_query(value: <object>, query: <string>)
:: <boolean>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valueobject
The object.yes
querystring
The Datadog Search Syntax query.yes

Examples

OR query
Source
match_datadog_query({"message": "contains this and that"}, "this OR that")
Return
true
AND query
Source
match_datadog_query({"message": "contains only this"}, "this AND that")
Facet wildcard
Source
match_datadog_query({"custom": {"name": "foo"}}, "@name:foo*")
Return
true
Tag range
Source
match_datadog_query({"tags": ["a:x", "b:y", "c:z"]}, s'b:["x" TO "z"]')
Return
true

merge

infallible pure
Merges the from object into the to object.

Function spec

merge(to: <object>, from: <object>, [deep: <boolean>])
:: <object>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
toobject
The object to merge into.yes
fromobject
The object to merge from.yes
deepboolean
A deep merge is performed if true, otherwise only top-level fields are merged.no

Examples

Object merge (shallow)
Source
merge(
	{
		"parent1": {
			"child1": 1,
			"child2": 2
		},
		"parent2": {
			"child3": 3
		}
	},
	{
		"parent1": {
			"child2": 4,
			"child5": 5
		}
	}
)
Return
{
  "parent1": {
    "child2": 4,
    "child5": 5
  },
  "parent2": {
    "child3": 3
  }
}
Object merge (deep)
Source
merge(
	{
		"parent1": {
			"child1": 1,
			"child2": 2
		},
		"parent2": {
			"child3": 3
		}
	},
	{
		"parent1": {
			"child2": 4,
			"child5": 5
		}
	},
	deep: true
)
Return
{
  "parent1": {
    "child1": 1,
    "child2": 4,
    "child5": 5
  },
  "parent2": {
    "child3": 3
  }
}

unnest

fallible pure

Unnest an array field from an object to create an array of objects using that field; keeping all other fields.

Assigning the array result of this to . results in multiple events being emitted from remap. See the remap transform docs for more details.

This is also referred to as explode in some languages.

Function spec

unnest(path: <path>)
:: <array> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
pathpath
The path of the field to unnest.yes

Errors

The unnest function is fallible, which means that error handling is required for these errors:
The field path referred to is not an array.

Examples

Unnest an array field
Source
. = unnest!(.messages)
Unnest nested an array field
Source
. = unnest!(.event.messages)

Parse functions

parse_apache_log

fallible pure
Parses Apache access and error log lines. Lines can be in common, combined, or the default error format.

Function spec

parse_apache_log(value: <string>, format: <string>, [timestamp_format: <string>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
formatstring
The format to use for parsing the log.yes
timestamp_formatstring
The date/time format to use for encoding the timestamp. The time is parsed in local time if the timestamp does not specify a timezone.%d/%b/%Y:%T %zno

Notices

This function has special behavior that you should be aware of.
Missing information in the log message may be indicated by -. These fields are omitted in the result.

Errors

The parse_apache_log function is fallible, which means that error handling is required for these errors:
value does not match the specified format.
timestamp_format is not a valid format string.
The timestamp in value fails to parse using the provided timestamp_format.

Examples

Parse using Apache log format (common)
Source
parse_apache_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326", format: "common")
Return
{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}
Parse using Apache log format (combined)
Source
parse_apache_log!(
	s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth" "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"',
	"combined",
)
Return
{
  "agent": "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0",
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "referrer": "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}
Parse using Apache log format (error)
Source
parse_apache_log!(
	s'[01/Mar/2021:12:00:19 +0000] [ab:alert] [pid 4803:tid 3814] [client 147.159.108.175:24259] I will bypass the haptic COM bandwidth, that should matrix the CSS driver!',
	"error"
)
Return
{
  "client": "147.159.108.175",
  "message": "I will bypass the haptic COM bandwidth, that should matrix the CSS driver!",
  "module": "ab",
  "pid": 4803,
  "port": 24259,
  "severity": "alert",
  "thread": "3814",
  "timestamp": "2021-03-01T12:00:19Z"
}

parse_aws_alb_log

fallible pure

Function spec

parse_aws_alb_log(value: <string>)
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
Access log of the Application Load Balancer.yes

Errors

The parse_aws_alb_log function is fallible, which means that error handling is required for these errors:
value is not a properly formatted AWS ALB log.

Examples

Parse AWS ALB log
Source
parse_aws_alb_log!(
	"http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\""
)
Return
{
  "actions_executed": "forward",
  "chosen_cert_arn": null,
  "classification": null,
  "classification_reason": null,
  "client_host": "192.168.131.39:2817",
  "domain_name": null,
  "elb": "app/my-loadbalancer/50dc6c495c0c9188",
  "elb_status_code": "200",
  "error_reason": null,
  "matched_rule_priority": "0",
  "received_bytes": 34,
  "redirect_url": null,
  "request_creation_time": "2018-11-30T22:22:48.364000Z",
  "request_method": "GET",
  "request_processing_time": 0,
  "request_protocol": "HTTP/1.1",
  "request_url": "http://www.example.com:80/",
  "response_processing_time": 0,
  "sent_bytes": 366,
  "ssl_cipher": null,
  "ssl_protocol": null,
  "target_group_arn": "arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067",
  "target_host": null,
  "target_port_list": [],
  "target_processing_time": 0.001,
  "target_status_code": "200",
  "target_status_code_list": [],
  "timestamp": "2018-11-30T22:23:00.186641Z",
  "trace_id": "Root=1-58337364-23a8c76965a2ef7629b185e3",
  "type": "http",
  "user_agent": "curl/7.46.0"
}

parse_aws_cloudwatch_log_subscription_message

fallible pure
Parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) from the aws_kinesis_firehose source.

Function spec

parse_aws_cloudwatch_log_subscription_message(value: <string>)
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string representation of the message to parse.yes

Errors

The parse_aws_cloudwatch_log_subscription_message function is fallible, which means that error handling is required for these errors:
value is not a properly formatted AWS CloudWatch Log subscription message.

Examples

Parse AWS Cloudwatch Log subscription message
Source
parse_aws_cloudwatch_log_subscription_message!(.message)
Return
{
  "log_events": [
    {
      "id": "35683658089614582423604394983260738922885519999578275840",
      "message": "{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}",
      "timestamp": "2020-09-14T19:09:29.039Z"
    }
  ],
  "log_group": "test",
  "log_stream": "test",
  "message_type": "DATA_MESSAGE",
  "owner": "111111111111",
  "subscription_filters": [
    "Destination"
  ]
}

parse_aws_vpc_flow_log

fallible pure
Parses value in the VPC Flow Logs format.

Function spec

parse_aws_vpc_flow_log(value: <string>, [format: <string>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
VPC Flow Log.yes
formatstring
VPC Flow Log format.no

Errors

The parse_aws_vpc_flow_log function is fallible, which means that error handling is required for these errors:
value is not a properly formatted AWS VPC Flow log.

Examples

Parse AWS VPC Flow log (default format)
Source
parse_aws_vpc_flow_log!("2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA")
Return
{
  "account_id": "123456789010",
  "action": null,
  "bytes": null,
  "dstaddr": null,
  "dstport": null,
  "end": 1431280934,
  "interface_id": "eni-1235b8ca123456789",
  "log_status": "NODATA",
  "packets": null,
  "protocol": null,
  "srcaddr": null,
  "srcport": null,
  "start": 1431280876,
  "version": 2
}
Parse AWS VPC Flow log (custom format)
Source
parse_aws_vpc_flow_log!(
	"- eni-1235b8ca123456789 10.0.1.5 10.0.0.220 10.0.1.5 203.0.113.5",
	"instance_id interface_id srcaddr dstaddr pkt_srcaddr pkt_dstaddr"
)
Return
{
  "dstaddr": "10.0.0.220",
  "instance_id": null,
  "interface_id": "eni-1235b8ca123456789",
  "pkt_dstaddr": "203.0.113.5",
  "pkt_srcaddr": "10.0.1.5",
  "srcaddr": "10.0.1.5"
}
Parse AWS VPC Flow log including v5 fields
Source
parse_aws_vpc_flow_log!("5 52.95.128.179 10.0.0.71 80 34210 6 1616729292 1616729349 IPv4 14 15044 123456789012 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-0c50d5961bcb2d47b eni-1235b8ca123456789 ap-southeast-2 apse2-az3 - - ACCEPT 19 52.95.128.179 10.0.0.71 S3 - - ingress OK",
format: "version srcaddr dstaddr srcport dstport protocol start end type packets bytes account_id vpc_id subnet_id instance_id interface_id region az_id sublocation_type sublocation_id action tcp_flags pkt_srcaddr pkt_dstaddr pkt_src_aws_service pkt_dst_aws_service traffic_path flow_direction log_status")
Return
{
  "account_id": "123456789012",
  "action": "ACCEPT",
  "az_id": "apse2-az3",
  "bytes": 15044,
  "dstaddr": "10.0.0.71",
  "dstport": 34210,
  "end": 1616729349,
  "flow_direction": "ingress",
  "instance_id": "i-0c50d5961bcb2d47b",
  "interface_id": "eni-1235b8ca123456789",
  "log_status": "OK",
  "packets": 14,
  "pkt_dst_aws_service": null,
  "pkt_dstaddr": "10.0.0.71",
  "pkt_src_aws_service": "S3",
  "pkt_srcaddr": "52.95.128.179",
  "protocol": 6,
  "region": "ap-southeast-2",
  "srcaddr": "52.95.128.179",
  "srcport": 80,
  "start": 1616729292,
  "sublocation_id": null,
  "sublocation_type": null,
  "subnet_id": "subnet-aaaaaaaa012345678",
  "tcp_flags": 19,
  "traffic_path": null,
  "type": "IPv4",
  "version": 5,
  "vpc_id": "vpc-abcdefab012345678"
}

parse_cef

fallible pure
Parses the value in CEF (Common Event Format) format. Ignores everything up to CEF header. Empty values are returned as empty strings. Surrounding quotes are removed from values.

Function spec

parse_cef(value: <string>, [translate_custom_fields: <boolean>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
translate_custom_fieldsboolean
Toggles translation of custom field pairs to key:value.no

Notices

This function has special behavior that you should be aware of.
All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_cef function is fallible, which means that error handling is required for these errors:
value is not a properly formatted CEF string.

Examples

Parse output generated by PTA
Source
parse_cef!(
	"CEF:0|CyberArk|PTA|12.6|1|Suspected credentials theft|8|suser=mike2@prod1.domain.com shost=prod1.domain.com src=1.1.1.1 duser=andy@dev1.domain.com dhost=dev1.domain.com dst=2.2.2.2 cs1Label=ExtraData cs1=None cs2Label=EventID cs2=52b06812ec3500ed864c461e deviceCustomDate1Label=detectionDate deviceCustomDate1=1388577900000 cs3Label=PTAlink cs3=https://1.1.1.1/incidents/52b06812ec3500ed864c461e cs4Label=ExternalLink cs4=None"
)
Return
{
  "cefVersion": "0",
  "cs1": "None",
  "cs1Label": "ExtraData",
  "cs2": "52b06812ec3500ed864c461e",
  "cs2Label": "EventID",
  "cs3": "https://1.1.1.1/incidents/52b06812ec3500ed864c461e",
  "cs3Label": "PTAlink",
  "cs4": "None",
  "cs4Label": "ExternalLink",
  "deviceCustomDate1": "1388577900000",
  "deviceCustomDate1Label": "detectionDate",
  "deviceEventClassId": "1",
  "deviceProduct": "PTA",
  "deviceVendor": "CyberArk",
  "deviceVersion": "12.6",
  "dhost": "dev1.domain.com",
  "dst": "2.2.2.2",
  "duser": "andy@dev1.domain.com",
  "name": "Suspected credentials theft",
  "severity": "8",
  "shost": "prod1.domain.com",
  "src": "1.1.1.1",
  "suser": "mike2@prod1.domain.com"
}
Ignore syslog header
Source
parse_cef!(
	"Sep 29 08:26:10 host CEF:1|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232"
)
Return
{
  "cefVersion": "1",
  "deviceEventClassId": "100",
  "deviceProduct": "threatmanager",
  "deviceVendor": "Security",
  "deviceVersion": "1.0",
  "dst": "2.1.2.2",
  "name": "worm successfully stopped",
  "severity": "10",
  "spt": "1232",
  "src": "10.0.0.1"
}
Translate custom fields
Source
parse_cef!(
	"CEF:0|Dev|firewall|2.2|1|Connection denied|5|c6a1=2345:0425:2CA1:0000:0000:0567:5673:23b5 c6a1Label=Device IPv6 Address",
	translate_custom_fields: true
)
Return
{
  "Device IPv6 Address": "2345:0425:2CA1:0000:0000:0567:5673:23b5",
  "cefVersion": "0",
  "deviceEventClassId": "1",
  "deviceProduct": "firewall",
  "deviceVendor": "Dev",
  "deviceVersion": "2.2",
  "name": "Connection denied",
  "severity": "5"
}

parse_common_log

fallible pure
Parses the value using the Common Log Format (CLF).

Function spec

parse_common_log(value: <string>, [timestamp_format: <string>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
timestamp_formatstring
The date/time format to use for encoding the timestamp.%d/%b/%Y:%T %zno

Notices

This function has special behavior that you should be aware of.
Missing information in the log message may be indicated by -. These fields are omitted in the result.

Errors

The parse_common_log function is fallible, which means that error handling is required for these errors:
value does not match the Common Log Format.
timestamp_format is not a valid format string.
The timestamp in value fails to parse using the provided timestamp_format.

Examples

Parse using Common Log Format (with default timestamp format)
Source
parse_common_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326")
Return
{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}
Parse using Common Log Format (with custom timestamp format)
Source
parse_common_log!(
	"127.0.0.1 bob frank [2000-10-10T20:55:36Z] \"GET /apache_pb.gif HTTP/1.0\" 200 2326",
	"%+"
)
Return
{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}

parse_csv

fallible pure
Parses a single CSV formatted row. Only the first row is parsed in case of multiline input value.

Function spec

parse_csv(value: <string>, [delimiter: <string>])
:: <array> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
delimiterstring
The field delimiter to use when parsing. Must be a single-byte utf8 character.,no

Notices

This function has special behavior that you should be aware of.
All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_csv function is fallible, which means that error handling is required for these errors:
The delimiter must be a single-byte UTF-8 character.
value is not a valid CSV string.

Examples

Parse a single CSV formatted row
Source
parse_csv!("foo,bar,\"foo \"\", bar\"")
Return
["foo","bar","foo \", bar"]
Parse a single CSV formatted row with custom delimiter
Source
parse_csv!("foo bar", delimiter: " ")
Return
["foo","bar"]

parse_duration

fallible pure
Parses the value into a human-readable duration format specified by unit.

Function spec

parse_duration(value: <string>, unit: <string>)
:: <float> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string of the duration.yes
unitstring
The output units for the duration.yes

Errors

The parse_duration function is fallible, which means that error handling is required for these errors:
value is not a properly formatted duration.

Examples

Parse duration (milliseconds)
Source
parse_duration!("1005ms", unit: "s")
Return
1.005

parse_etld

fallible pure
Parses the eTLD from value representing domain name.

Function spec

parse_etld(value: <string>, [plus_parts: <integer>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The domain string.yes
plus_partsinteger
Can be provided to get additional parts of the domain name. When 1 is passed, eTLD+1 will be returned, which represents a domain registrable by a single organization. Higher numbers will return subdomains.no

Errors

The parse_etld function is fallible, which means that error handling is required for these errors:
unable to determine eTLD for value

Examples

Parse eTLD
Source
parse_etld!("sub.sussex.ac.uk")
Return
{
  "etld": "ac.uk",
  "etld_plus": "ac.uk",
  "known_suffix": true
}
Parse eTLD+1
Source
parse_etld!("sub.sussex.ac.uk", plus_parts: 1)
Return
{
  "etld": "ac.uk",
  "etld_plus": "sussex.ac.uk",
  "known_suffix": true
}
Parse eTLD with unknown suffix
Source
parse_etld!("vector.acmecorp")
Return
{
  "etld": "acmecorp",
  "etld_plus": "acmecorp",
  "known_suffix": false
}

parse_glog

fallible pure
Parses the value using the glog (Google Logging Library) format.

Function spec

parse_glog(value: <string>)
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes

Errors

The parse_glog function is fallible, which means that error handling is required for these errors:
value does not match the glog format.

Examples

Parse using glog
Source
parse_glog!("I20210131 14:48:54.411655 15520 main.c++:9] Hello world!")
Return
{
  "file": "main.c++",
  "id": 15520,
  "level": "info",
  "line": 9,
  "message": "Hello world!",
  "timestamp": "2021-01-31T14:48:54.411655Z"
}

parse_grok

fallible pure
Parses the value using the grok format. All patterns listed here are supported.

Function spec

parse_grok(value: <string>, pattern: <string>)
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
patternstring
The Grok pattern.yes

Notices

This function has special behavior that you should be aware of.
We recommend using community-maintained Grok patterns when possible, as they’re more likely to be properly vetted and improved over time than bespoke patterns.

Errors

The parse_grok function is fallible, which means that error handling is required for these errors:
value fails to parse using the provided pattern.

Examples

Parse using Grok
Source
parse_grok!(
	"2020-10-02T23:22:12.223222Z info Hello world",
	"%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
)
Return
{
  "level": "info",
  "message": "Hello world",
  "timestamp": "2020-10-02T23:22:12.223222Z"
}

parse_groks

fallible pure
Parses the value using multiple grok patterns. All patterns listed here are supported.

Function spec

parse_groks(value: <string>, patterns: <array>, [aliases: <object>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
patternsarray
The Grok patterns, which are tried in order until the first match.yes
aliasesobject
The shared set of grok aliases that can be referenced in the patterns to simplify them.trueno

Notices

This function has special behavior that you should be aware of.
We recommend using community-maintained Grok patterns when possible, as they’re more likely to be properly vetted and improved over time than bespoke patterns.

Errors

The parse_groks function is fallible, which means that error handling is required for these errors:
value fails to parse using the provided pattern.

Examples

Parse using multiple Grok patterns
Source
parse_groks!(
	"2020-10-02T23:22:12.223222Z info Hello world",
	patterns: [
		"%{common_prefix} %{_status} %{_message}",
		"%{common_prefix} %{_message}",
	],
	aliases: {
		"common_prefix": "%{_timestamp} %{_loglevel}",
		"_timestamp": "%{TIMESTAMP_ISO8601:timestamp}",
		"_loglevel": "%{LOGLEVEL:level}",
		"_status": "%{POSINT:status}",
		"_message": "%{GREEDYDATA:message}"
	}
)
Return
{
  "level": "info",
  "message": "Hello world",
  "timestamp": "2020-10-02T23:22:12.223222Z"
}

parse_int

fallible pure
Parses the string value representing a number in an optional base/radix to an integer.

Function spec

parse_int(value: <string>, [base: <integer>])
:: <string> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
baseinteger

The base the number is in. Must be between 2 and 36 (inclusive).

If unspecified, the string prefix is used to determine the base: “0b”, 8 for “0” or “0o”, 16 for “0x”, and 10 otherwise.

no

Errors

The parse_int function is fallible, which means that error handling is required for these errors:
The base is not between 2 and 36.
The number cannot be parsed in the base.

Examples

Parse decimal
Source
parse_int!("-42")
Return
-42
Parse binary
Source
parse_int!("0b1001")
Return
9
Parse octal
Source
parse_int!("0o42")
Return
34
Parse hexadecimal
Source
parse_int!("0x2a")
Return
42
Parse explicit base
Source
parse_int!("2a", 17)
Return
44

parse_json

fallible pure
Parses the value as JSON.

Function spec

parse_json(value: <string>, [max_depth: <integer>])
:: <boolean | integer | float | string | object | array | null> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string representation of the JSON to parse.yes
max_depthinteger
Number of layers to parse for nested JSON-formatted documents. The value must be in the range of 1 to 128.no

Notices

This function has special behavior that you should be aware of.
Only JSON types are returned. If you need to convert a string into a timestamp, consider the parse_timestamp function.

Errors

The parse_json function is fallible, which means that error handling is required for these errors:
value is not a valid JSON-formatted payload.

Examples

Parse JSON
Source
parse_json!("{\"key\": \"val\"}")
Return
{
  "key": "val"
}
Parse JSON with max_depth
Source
parse_json!("{\"top_level\":{\"key\": \"val\"}}", max_depth: 1)
Return
{
  "top_level": "{\"key\": \"val\"}"
}

parse_key_value

fallible pure

Parses the value in key-value format. Also known as logfmt.

  • Keys and values can be wrapped with ".
  • " characters can be escaped using \.

Function spec

parse_key_value(value: <string>, [key_value_delimiter: <string>, field_delimiter: <string>, whitespace: <string>, accept_standalone_key: <boolean>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
key_value_delimiterstring
The string that separates the key from the value.=no
field_delimiterstring
The string that separates each key-value pair.no
whitespacestring
Defines the acceptance of unnecessary whitespace surrounding the configured key_value_delimiter.lenientno
accept_standalone_keyboolean
Whether a standalone key should be accepted, the resulting object associates such keys with the boolean value true.trueno

Notices

This function has special behavior that you should be aware of.
All values are returned as strings or as an array of strings for duplicate keys. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_key_value function is fallible, which means that error handling is required for these errors:
value is not a properly formatted key-value string.

Examples

Parse logfmt log
Source
parse_key_value!(
	"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
Return
{
  "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
  "id": "ConsumerFetcherManager-1382721708341",
  "level": "info",
  "module": "kafka.consumer.ConsumerFetcherManager",
  "msg": "Stopping all fetchers",
  "tag#production": "stopping_fetchers"
}
Parse comma delimited log
Source
parse_key_value!(
	"path:\"/cart_link\", host:store.app.com, fwd: \"102.30.171.16\", dyno: web.1, connect:0ms, service:87ms, status:304, bytes:632, protocol:https",
	field_delimiter: ",",
	key_value_delimiter: ":"
)
Return
{
  "bytes": "632",
  "connect": "0ms",
  "dyno": "web.1",
  "fwd": "102.30.171.16",
  "host": "store.app.com",
  "path": "/cart_link",
  "protocol": "https",
  "service": "87ms",
  "status": "304"
}
Parse comma delimited log with standalone keys
Source
parse_key_value!(
	"env:prod,service:backend,region:eu-east1,beta",
	field_delimiter: ",",
	key_value_delimiter: ":",
)
Return
{
  "beta": true,
  "env": "prod",
  "region": "eu-east1",
  "service": "backend"
}
Parse duplicate keys
Source
parse_key_value!(
	"at=info,method=GET,path=\"/index\",status=200,tags=dev,tags=dummy",
	field_delimiter: ",",
	key_value_delimiter: "=",
)
Return
{
  "at": "info",
  "method": "GET",
  "path": "/index",
  "status": "200",
  "tags": [
    "dev",
    "dummy"
  ]
}

parse_klog

fallible pure
Parses the value using the klog format used by Kubernetes components.

Function spec

parse_klog(value: <string>)
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes

Errors

The parse_klog function is fallible, which means that error handling is required for these errors:
value does not match the klog format.

Examples

Parse using klog
Source
parse_klog!("I0505 17:59:40.692994   28133 klog.go:70] hello from klog")
Return
{
  "file": "klog.go",
  "id": 28133,
  "level": "info",
  "line": 70,
  "message": "hello from klog",
  "timestamp": "2024-05-05T17:59:40.692994Z"
}

parse_linux_authorization

fallible pure
Parses Linux authorization logs usually found under either /var/log/auth.log (for Debian-based systems) or /var/log/secure (for RedHat-based systems) according to Syslog format.

Function spec

parse_linux_authorization(value: <string>)
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The text containing the message to parse.yes

Notices

This function has special behavior that you should be aware of.
The function resolves the year for messages that don’t include it. If the current month is January, and the message is for December, it will take the previous year. Otherwise, take the current year.

Errors

The parse_linux_authorization function is fallible, which means that error handling is required for these errors:
value is not a properly formatted Syslog message.

Examples

Parse Linux authorization event
Source
parse_linux_authorization!(
	s'Mar 23 2023 01:49:58 localhost sshd[1111]: Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar'
)
Return
{
  "appname": "sshd",
  "hostname": "localhost",
  "message": "Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar",
  "procid": 1111,
  "timestamp": "2023-03-23T01:49:58Z"
}

parse_logfmt

fallible pure

Parses the value in logfmt.

  • Keys and values can be wrapped using the " character.
  • " characters can be escaped by the \ character.
  • As per this logfmt specification, the parse_logfmt function accepts standalone keys and assigns them a Boolean value of true.

Function spec

parse_logfmt(value: <string>)
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes

Errors

The parse_logfmt function is fallible, which means that error handling is required for these errors:
value is not a properly formatted key-value string

Examples

Parse logfmt log
Source
parse_logfmt!(
	"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
Return
{
  "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
  "id": "ConsumerFetcherManager-1382721708341",
  "level": "info",
  "module": "kafka.consumer.ConsumerFetcherManager",
  "msg": "Stopping all fetchers",
  "tag#production": "stopping_fetchers"
}

parse_nginx_log

fallible pure
Parses Nginx access and error log lines. Lines can be in combined, ingress_upstreaminfo, or error format.

Function spec

parse_nginx_log(value: <string>, format: <string>, [timestamp_format: <string>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes
formatstring
The format to use for parsing the log.yes
timestamp_formatstring
The date/time format to use for encoding the timestamp. The time is parsed in local time if the timestamp doesn’t specify a timezone. The default format is %d/%b/%Y:%T %z for combined logs and %Y/%m/%d %H:%M:%S for error logs.%d/%b/%Y:%T %zno

Notices

This function has special behavior that you should be aware of.
Missing information in the log message may be indicated by -. These fields are omitted in the result.
In case of ingress_upstreaminfo format the following fields may be safely omitted in the log message: remote_addr, remote_user, http_referer, http_user_agent, proxy_alternative_upstream_name, upstream_addr, upstream_response_length, upstream_response_time, upstream_status.

Errors

The parse_nginx_log function is fallible, which means that error handling is required for these errors:
value does not match the specified format.
timestamp_format is not a valid format string.
The timestamp in value fails to parse using the provided timestamp_format.

Examples

Parse via Nginx log format (combined)
Source
parse_nginx_log!(
    s'172.17.0.1 - alice [01/Apr/2021:12:02:31 +0000] "POST /not-found HTTP/1.1" 404 153 "http://localhost/somewhere" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"',
    "combined",
)
Return
{
  "agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36",
  "client": "172.17.0.1",
  "compression": "2.75",
  "referer": "http://localhost/somewhere",
  "request": "POST /not-found HTTP/1.1",
  "size": 153,
  "status": 404,
  "timestamp": "2021-04-01T12:02:31Z",
  "user": "alice"
}
Parse via Nginx log format (error)
Source
parse_nginx_log!(
    s'2021/04/01 13:02:31 [error] 31#31: *1 open() "/usr/share/nginx/html/not-found" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "POST /not-found HTTP/1.1", host: "localhost:8081"',
    "error"
)
Return
{
  "cid": 1,
  "client": "172.17.0.1",
  "host": "localhost:8081",
  "message": "open() \"/usr/share/nginx/html/not-found\" failed (2: No such file or directory)",
  "pid": 31,
  "request": "POST /not-found HTTP/1.1",
  "server": "localhost",
  "severity": "error",
  "tid": 31,
  "timestamp": "2021-04-01T13:02:31Z"
}
Parse via Nginx log format (ingress_upstreaminfo)
Source
parse_nginx_log!(
    s'0.0.0.0 - bob [18/Mar/2023:15:00:00 +0000] "GET /some/path HTTP/2.0" 200 12312 "https://10.0.0.1/some/referer" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 462 0.050 [some-upstream-service-9000] [some-other-upstream-5000] 10.0.50.80:9000 19437 0.049 200 752178adb17130b291aefd8c386279e7',
    "ingress_upstreaminfo"
)
Return
{
  "body_bytes_size": 12312,
  "http_referer": "https://10.0.0.1/some/referer",
  "http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36",
  "proxy_alternative_upstream_name": "some-other-upstream-5000",
  "proxy_upstream_name": "some-upstream-service-9000",
  "remote_addr": "0.0.0.0",
  "remote_user": "bob",
  "req_id": "752178adb17130b291aefd8c386279e7",
  "request": "GET /some/path HTTP/2.0",
  "request_length": 462,
  "request_time": 0.05,
  "status": 200,
  "timestamp": "2023-03-18T15:00:00Z",
  "upstream_addr": "10.0.50.80:9000",
  "upstream_response_length": 19437,
  "upstream_response_time": 0.049,
  "upstream_status": 200
}

parse_query_string

infallible pure
Parses the value as a query string.

Function spec

parse_query_string(value: <string>)
:: <object>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to parse.yes

Notices

This function has special behavior that you should be aware of.
All values are returned as strings. We recommend manually coercing values to desired types as you see fit. Empty keys and values are allowed.

Examples

Parse query string
Source
parse_query_string("foo=%2B1&bar=2&bar=3&xyz")
Return
{
  "bar": [
    "2",
    "3"
  ],
  "foo": "+1",
  "xyz": ""
}
Parse Ruby on Rails' query string
Source
parse_query_string("?foo%5b%5d=1&foo%5b%5d=2")
Return
{
  "foo[]": [
    "1",
    "2"
  ]
}

parse_regex

fallible pure

Parses the value using the provided Regex pattern.

This function differs from the parse_regex_all function in that it returns only the first match.

Function spec

parse_regex(value: <string>, pattern: <regex>, [numeric_groups: <regex>])
:: <object> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to search.yes
patternregex
The regular expression pattern to search against.yes
numeric_groupsregex
If true, the index of each group in the regular expression is also captured. Index 0 contains the whole match.no

Notices

This function has special behavior that you should be aware of.
VRL aims to provide purpose-specific parsing functions for common log formats. Before reaching for the parse_regex function, see if a VRL parse_* function already exists for your format. If not, we recommend opening an issue to request support for the desired format.
All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_regex function is fallible, which means that error handling is required for these errors:
value fails to parse using the provided pattern.

Examples

Parse using Regex (with capture groups)
Source
parse_regex!("first group and second group.", r'(?P<number>.*?) group')
Return
{
  "number": "first"
}
Parse using Regex (without capture groups)
Source
parse_regex!("first group and second group.", r'(\w+) group', numeric_groups: true)
Return
{
  "0": "first group",
  "1": "first"
}

parse_regex_all

fallible pure

Parses the value using the provided Regex pattern.

This function differs from the parse_regex function in that it returns all matches, not just the first.

Function spec

parse_regex_all(value: <string>, pattern: <regex>, [numeric_groups: <regex>])
:: <array> , <error>
required optional <types | ...>
ArgumentTypeDescriptionDefaultRequired?
valuestring
The string to search.yes
patternregex
The regular expression pattern to search against.yes
numeric_groupsregex
If true, the index of each group in the regular expression is also captured. Index 0 contains the whole match.no

Notices

This function has special behavior that you should be aware of.
VRL aims to provide purpose-specific parsing functions for common log formats. Before reaching for the parse_regex function, see if a VRL parse_* function already exists for your format. If not, we recommend opening an issue to request support for the desired format.
All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_regex_all function is fallible, which means that error handling is required for these errors:
value fails to parse using the provided pattern.

Examples

Parse using Regex (all matches)
Source
parse_regex_all!("first group and second group.", r'(?P<number>\w+) group', numeric_groups: true)
Return
[{"0":"first group","1":"first","number":"first"},{"0":"second group","1":"second","number":"second"}]

parse_ruby_hash

fallible pure
Parses the value as ruby hash.

Function spec