VRL function reference

Here you’ll find a comprehensive list of all built-in VRL functions. Functions are categorized by their purpose and sorted alphabetically for easy discovery. To use these functions in Vector, see the documentation on function call expressions and Vector’s remap transform.

Array functions

append

infallible pure

Appends each item in the items array to the end of the value array.

Function spec

append(value: <array> ,items: <array> )

:: <array>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	array	The initial array.		yes
items	array	The items to append.		yes

Examples

Append to an array

Source

append([1, 2], [3, 4])

Return

[1,2,3,4]

chunks

fallible pure

Chunks value into slices of length chunk_size bytes.

Function spec

chunks(value: <array | string> ,chunk_size: <integer> )

:: <array> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	array string	The array of bytes to split.		yes
chunk_size	integer	The desired length of each chunk in bytes. This may be constrained by the host platform architecture.		yes

Errors

The chunks function is fallible, which means that error handling is required for these errors:

chunk_size must be at least 1 byte.

chunk_size is too large.

Examples

Split a string into chunks

Source

chunks("abcdefgh", 4)

Return

["abcd","efgh"]

Chunks do not respect unicode code point boundaries

Source

chunks("ab你好", 4)

Return

["ab�","�好"]

pop

infallible pure

Removes the last item from the value array.

Function spec

pop(value: <array> )

:: <array>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	array	The target array.		yes

Examples

Pop an item from an array

Source

pop([1, 2, 3])

Return

[1,2]

push

infallible pure

Adds the item to the end of the value array.

Function spec

push(value: <array> ,item: <any> )

:: <array>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	array	The target array.		yes
item	any	The item to push.		yes

Examples

Push an item onto an array

Source

push([1, 2], 3)

Return

[1,2,3]

zip

fallible pure

Iterate over several arrays in parallel, producing a new array containing arrays of items from each source. The resulting array will be as long as the shortest input array, with all the remaining elements dropped. This function is modeled from the zip function in Python, but similar methods can be found in Ruby and Rust.

If a single parameter is given, it must contain an array of all the input arrays.

Function spec

zip(array_0: <array> , [array_1: <array> ] )

:: <array> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
array_0	array	The first array of elements, or the array of input arrays if no other parameter is present.		yes
array_1	array	The second array of elements. If not present, the first parameter contains all the arrays.		no

Errors

The zip function is fallible, which means that error handling is required for these errors:

array_0 and array_1 must be arrays.

Examples

Merge two arrays

Source

zip([1, 2, 3], [4, 5, 6, 7])

Return

[[1,4],[2,5],[3,6]]

Merge three arrays

Source

zip([[1, 2], [3, 4], [5, 6]])

Return

[[1,3,5],[2,4,6]]

Codec functions

decode_base16

fallible pure

Decodes the value (a Base16 string) into its original string.

Function spec

decode_base16(value: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The Base16 data to decode.		yes

Errors

The decode_base16 function is fallible, which means that error handling is required for these errors:

value isn’t a valid encoded Base16 string.

Examples

Decode Base16 data

Source

decode_base16!("796f752068617665207375636365737366756c6c79206465636f646564206d65")

Return

you have successfully decoded me

decode_base64

fallible pure

Decodes the value (a Base64 string) into its original string.

Function spec

decode_base64(value: <string> , [charset: <string> ] )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The Base64 data to decode.		yes
charset	string	The character set to use when decoding the data.	`standard`	no

Errors

The decode_base64 function is fallible, which means that error handling is required for these errors:

value isn’t a valid encoded Base64 string.

Examples

Decode Base64 data (default)

Source

decode_base64!("eW91IGhhdmUgc3VjY2Vzc2Z1bGx5IGRlY29kZWQgbWU=")

Return

you have successfully decoded me

Decode Base64 data (URL safe)

Source

decode_base64!("eW91IGNhbid0IG1ha2UgeW91ciBoZWFydCBmZWVsIHNvbWV0aGluZyBpdCB3b24ndA==", charset: "url_safe")

Return

you can't make your heart feel something it won't

decode_charset

fallible pure

Decodes the value (a non-UTF8 string) to a UTF8 string using the specified character set.

Function spec

decode_charset(value: <string> ,from_charset: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The non-UTF8 string to decode.		yes
from_charset	string	The character set to use when decoding the data.		yes

Errors

The decode_charset function is fallible, which means that error handling is required for these errors:

from_charset isn’t a valid character set.

Examples

Decode EUC-KR string

Source

decode_charset!(decode_base64!("vsiz58fPvLy/5A=="), "euc-kr")

Return

안녕하세요

Decode EUC-JP string

Source

decode_charset!(decode_base64!("pLOk86TLpMGkzw=="), "euc-jp")

Return

こんにちは

Decode GB2312 string

Source

decode_charset!(decode_base64!("xOO6ww=="), "gb2312")

Return

你好

decode_gzip

fallible pure

Decodes the value (a Gzip string) into its original string.

Function spec

decode_gzip(value: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The Gzip data to decode.		yes

Errors

The decode_gzip function is fallible, which means that error handling is required for these errors:

value isn’t a valid encoded Gzip string.

Examples

Decode Gzip data

Source

encoded_text = decode_base64!("H4sIAHEAymMAA6vML1XISCxLVSguTU5OLS5OK83JqVRISU3OT0lNUchNBQD7BGDaIAAAAA==")
decode_gzip!(encoded_text)

Return

you have successfully decoded me

decode_lz4

fallible pure

Decodes the value (an lz4 string) into its original string. buf_size is the size of the buffer to decode into, this must be equal to or larger than the uncompressed size. If prepended_size is set to true, it expects the original uncompressed size to be prepended to the compressed data. prepended_size is useful for some implementations of lz4 that require the original size to be known before decoding.

Function spec

decode_lz4(value: <string> , [buf_size: <integer> ,prepended_size: <boolean> ] )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The lz4 block data to decode.		yes
buf_size	integer	The size of the buffer to decode into, this must be equal to or larger than the uncompressed size.	`1.048576e+06`	no
prepended_size	boolean	Some implementations of lz4 require the original uncompressed size to be prepended to the compressed data.		no

Errors

The decode_lz4 function is fallible, which means that error handling is required for these errors:

value unable to decode value with lz4 frame decoder.

value unable to decode value with lz4 block decoder.

value unable to decode because the output is too large for the buffer.

value unable to decode because the prepended size is not a valid integer.

Examples

Decode Lz4 data with prepended size.

Source

encoded_text = decode_base64!("LAAAAPAdVGhlIHF1aWNrIGJyb3duIGZveCBqdW1wcyBvdmVyIDEzIGxhenkgZG9ncy4=")
decode_lz4!(encoded_text, use_prepended_size: true)

Return

The quick brown fox jumps over 13 lazy dogs.

Decode Lz4 data without prepended size.

Source

encoded_text = decode_base64!("8B1UaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg==")
decode_lz4!(encoded_text)

Return

The quick brown fox jumps over 13 lazy dogs.

decode_mime_q

fallible pure

Replaces q-encoded or base64-encoded encoded-word substrings in the value with their original string.

Function spec

decode_mime_q(value: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string with encoded-words to decode.		yes

Errors

The decode_mime_q function is fallible, which means that error handling is required for these errors:

value has invalid encoded encoded-word string.

Examples

Decode single encoded-word

Source

decode_mime_q!("=?utf-8?b?SGVsbG8sIFdvcmxkIQ==?=")

Return

Hello, World!

Embedded

Source

decode_mime_q!("From: =?utf-8?b?SGVsbG8sIFdvcmxkIQ==?= <=?utf-8?q?hello=5Fworld=40example=2ecom?=>")

Return

From: Hello, World! <hello_world@example.com>

Without charset

Source

decode_mime_q!("?b?SGVsbG8sIFdvcmxkIQ==")

Return

Hello, World!

decode_percent

infallible pure

Decodes a percent-encoded value like a URL.

Function spec

decode_percent(value: <string> )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to decode.		yes

Examples

Percent decode a value

Source

decode_percent("foo%20bar%3F")

Return

foo bar?

decode_punycode

fallible pure

Decodes a punycode encoded value, such as an internationalized domain name (IDN). This function assumes that the value passed is meant to be used in IDN context and that it is either a domain name or a part of it.

Function spec

decode_punycode(value: <string> , [validate: <boolean> ] )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to decode.		yes
validate	boolean	If enabled, checks if the input string is a valid domain name.	`true`	no

Errors

The decode_punycode function is fallible, which means that error handling is required for these errors:

value is not valid punycode

Examples

Decode a punycode encoded internationalized domain name

Source

decode_punycode!("www.xn--caf-dma.com")

Return

www.café.com

Decode an ASCII only string

Source

decode_punycode!("www.cafe.com")

Return

www.cafe.com

Ignore validation

Source

decode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false)

Return

١٠.٦٦.٣٠.٥.

decode_snappy

fallible pure

Decodes the value (a Snappy string) into its original string.

Function spec

decode_snappy(value: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The Snappy data to decode.		yes

Errors

The decode_snappy function is fallible, which means that error handling is required for these errors:

value isn’t a valid encoded Snappy string.

Examples

Decode Snappy data

Source

encoded_text = decode_base64!("LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg==")
decode_snappy!(encoded_text)

Return

The quick brown fox jumps over 13 lazy dogs.

decode_zlib

fallible pure

Decodes the value (a Zlib string) into its original string.

Function spec

decode_zlib(value: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The Zlib data to decode.		yes

Errors

The decode_zlib function is fallible, which means that error handling is required for these errors:

value isn’t a valid encoded Zlib string.

Examples

Decode Zlib data

Source

encoded_text = decode_base64!("eJwNy4ENwCAIBMCNXIlQ/KqplUSgCdvXAS41qPMHshCB2R1zJlWIVlR6UURX2+wx2YcuK3kAb9C1wd6dn7Fa+QH9gRxr")
decode_zlib!(encoded_text)

Return

you_have_successfully_decoded_me.congratulations.you_are_breathtaking.

decode_zstd

fallible pure

Decodes the value (a Zstandard string) into its original string.

Function spec

decode_zstd(value: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The Zstandard data to decode.		yes

Errors

The decode_zstd function is fallible, which means that error handling is required for these errors:

value isn’t a valid encoded Zstd string.

Examples

Decode Zstd data

Source

encoded_text = decode_base64!("KLUv/QBY/QEAYsQOFKClbQBedqXsb96EWDax/f/F/z+gNU4ZTInaUeAj82KqPFjUzKqhcfDqAIsLvAsnY1bI/N2mHzDixRQA")
decode_zstd!(encoded_text)

Return

you_have_successfully_decoded_me.congratulations.you_are_breathtaking.

encode_base16

infallible pure

Encodes the value to Base16.

Function spec

encode_base16(value: <string> )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to encode.		yes

Examples

Encode to Base16

Source

encode_base16("please encode me")

Return

706c6561736520656e636f6465206d65

encode_base64

infallible pure

Encodes the value to Base64.

Function spec

encode_base64(value: <string> , [padding: <boolean> ,charset: <string> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to encode.		yes
padding	boolean	Whether the Base64 output is padded.	`true`	no
charset	string	The character set to use when encoding the data.	`standard`	no

Examples

Encode to Base64 (default)

Source

encode_base64("please encode me")

Return

cGxlYXNlIGVuY29kZSBtZQ==

Encode to Base64 (without padding)

Source

encode_base64("please encode me, no padding though", padding: false)

Return

cGxlYXNlIGVuY29kZSBtZSwgbm8gcGFkZGluZyB0aG91Z2g

Encode to Base64 (URL safe)

Source

encode_base64("please encode me, but safe for URLs", charset: "url_safe")

Return

cGxlYXNlIGVuY29kZSBtZSwgYnV0IHNhZmUgZm9yIFVSTHM=

encode_charset

fallible pure

Encodes the value (a UTF8 string) to a non-UTF8 string using the specified character set.

Function spec

encode_charset(value: <string> ,to_charset: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The UTF8 string to encode.		yes
to_charset	string	The character set to use when encoding the data.		yes

Errors

The encode_charset function is fallible, which means that error handling is required for these errors:

to_charset isn’t a valid character set.

Examples

Encode UTF8 string to EUC-KR

Source

encode_base64(encode_charset!("안녕하세요", "euc-kr"))

Return

vsiz58fPvLy/5A==

Encode UTF8 string to EUC-JP

Source

encode_base64(encode_charset!("こんにちは", "euc-jp"))

Return

pLOk86TLpMGkzw==

Encode UTF8 string to GB2312

Source

encode_base64(encode_charset!("你好", "gb2312"))

Return

xOO6ww==

encode_gzip

infallible pure

Encodes the value to Gzip.

Function spec

encode_gzip(value: <string> , [compression_level: <integer> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to encode.		yes
compression_level	integer	The default compression level.	`6`	no

Examples

Encode to Gzip

Source

encoded_text = encode_gzip("please encode me")
encode_base64(encoded_text)

Return

H4sIAAAAAAAA/yvISU0sTlVIzUvOT0lVyE0FAI4R4vcQAAAA

encode_json

infallible pure

Encodes the value to JSON.

Function spec

encode_json(value: <any> , [pretty: <boolean> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to convert to a JSON string.		yes
pretty	boolean	Whether to pretty print the JSON string or not.		no

Examples

Encode to JSON

Source

.payload = encode_json({"hello": "world"})

Return

{"hello":"world"}

encode_key_value

fallible pure

Encodes the value into key-value format with customizable delimiters. Default delimiters match the logfmt format.

Function spec

encode_key_value(value: <object> , [fields_ordering: <array> ,key_value_delimiter: <string> ,field_delimiter: <string> ,flatten_boolean: <boolean> ] )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	object	The value to convert to a string.		yes
fields_ordering	array	The ordering of fields to preserve. Any fields not in this list are listed unordered, after all ordered fields.		no
key_value_delimiter	string	The string that separates the key from the value.	`=`	no
field_delimiter	string	The string that separates each key-value pair.		no
flatten_boolean	boolean	Whether to encode key-value with a boolean value as a standalone key if `true` and nothing if `false`.		no

Notices

This function has special behavior that you should be aware of.

If fields_ordering is specified then the function is fallible else it is infallible.

Errors

The encode_key_value function is fallible, which means that error handling is required for these errors:

fields_ordering contains a non-string element.

Examples

Encode with default delimiters (no ordering)

Source

encode_key_value({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})

Return

lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z

Encode with default delimiters (fields ordering)

Source

encode_key_value!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])

Return

ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345

Encode with default delimiters (nested fields)

Source

encode_key_value({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})

Return

agent.name=foo event=log log.file.path=my.log

Encode with default delimiters (nested fields ordering)

Source

encode_key_value!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])

Return

event=log log.file.path=my.log agent.name=foo

Encode with custom delimiters (no ordering)

Source

encode_key_value(
	{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"},
	field_delimiter: ",",
	key_value_delimiter: ":"
)

Return

lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z

Encode with custom delimiters and flatten boolean

Source

encode_key_value(
	{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "beta": true, "dropped": false},
	field_delimiter: ",",
	key_value_delimiter: ":",
	flatten_boolean: true
)

Return

beta,lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z

encode_logfmt

fallible pure

Encodes the value to logfmt.

Function spec

encode_logfmt(value: <object> , [fields_ordering: <array> ] )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	object	The value to convert to a logfmt string.		yes
fields_ordering	array	The ordering of fields to preserve. Any fields not in this list are listed unordered, after all ordered fields.		no

Notices

This function has special behavior that you should be aware of.

If fields_ordering is specified then the function is fallible else it is infallible.

Errors

The encode_logfmt function is fallible, which means that error handling is required for these errors:

fields_ordering contains a non-string element.

Examples

Encode to logfmt (no ordering)

Source

encode_logfmt({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})

Return

lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z

Encode to logfmt (fields ordering)

Source

encode_logfmt!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])

Return

ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345

Encode to logfmt (nested fields)

Source

encode_logfmt({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})

Return

agent.name=foo event=log log.file.path=my.log

Encode to logfmt (nested fields ordering)

Source

encode_logfmt!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])

Return

event=log log.file.path=my.log agent.name=foo

encode_lz4

infallible pure

Encodes the value to Lz4. This function compresses the input string into an lz4 block. If prepend_size is set to true, it prepends the original uncompressed size to the compressed data. This is useful for some implementations of lz4 that require the original size to be known before decoding.

Function spec

encode_lz4(value: <string> , [prepend_size: <boolean> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to encode.		yes
prepend_size	boolean	Whether to prepend the original size to the compressed data.		no

Examples

Encode to Lz4

Source

encoded_text = encode_lz4!("The quick brown fox jumps over 13 lazy dogs.")
encode_base64(encoded_text)

Return

LAAAAPAdVGhlIHF1aWNrIGJyb3duIGZveCBqdW1wcyBvdmVyIDEzIGxhenkgZG9ncy4=

encode_percent

infallible pure

Encodes a value with percent encoding to safely be used in URLs.

Function spec

encode_percent(value: <string> , [ascii_set: <string> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to encode.		yes
ascii_set	string	The ASCII set to use when encoding the data.	`NON_ALPHANUMERIC`	no

Examples

Percent encode all non-alphanumeric characters (default)

Source

encode_percent("foo bar?")

Return

foo%20bar%3F

Percent encode only control characters

Source

encode_percent("foo 	bar", ascii_set: "CONTROLS")

Return

foo %09bar

encode_proto

fallible pure

Encodes the value into a protocol buffer payload.

Function spec

encode_proto(value: <object> ,desc_file: <string> ,message_type: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Required?
value	object	The object to convert to a protocol buffer payload.	yes
desc_file	string	The path to the protobuf descriptor set file. Must be a literal string. This file is the output of protoc -o …	yes
message_type	string	The name of the message type to use for serializing. Must be a literal string.	yes

Argument

Type

Description

Default

Required?

value

object

The object to convert to a protocol buffer payload.

yes

desc_file

string

The path to the protobuf descriptor set file. Must be a literal string.

This file is the output of protoc -o …

yes

message_type

string

The name of the message type to use for serializing.

Must be a literal string.

yes

Errors

The encode_proto function is fallible, which means that error handling is required for these errors:

desc_file file does not exist.

message_type message type does not exist in the descriptor file.

Examples

Encode to proto

Source

.payload = encode_base64(encode_proto!({"name": "someone", "phones": [{"number": "123456"}]}, "resources/protobuf_descriptor_set.desc", "test_protobuf.Person"))

Return

Cgdzb21lb25lIggKBjEyMzQ1Ng==

encode_punycode

fallible pure

Encodes a value to punycode. Useful for internationalized domain names (IDN). This function assumes that the value passed is meant to be used in IDN context and that it is either a domain name or a part of it.

Function spec

encode_punycode(value: <string> , [validate: <boolean> ] )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to encode.		yes
validate	boolean	Whether to validate the input string to check if it is a valid domain name.	`true`	no

Errors

The encode_punycode function is fallible, which means that error handling is required for these errors:

value can not be encoded to punycode

Examples

Encode an internationalized domain name

Source

encode_punycode!("www.café.com")

Return

www.xn--caf-dma.com

Encode an internationalized domain name with mixed case

Source

encode_punycode!("www.CAFé.com")

Return

www.xn--caf-dma.com

Encode an ASCII only string

Source

encode_punycode!("www.cafe.com")

Return

www.cafe.com

Ignore validation

Source

encode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false)

Return

xn--8hbb.xn--fiba.xn--8hbf.xn--eib.

encode_snappy

fallible pure

Encodes the value to Snappy.

Function spec

encode_snappy(value: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to encode.		yes

Errors

The encode_snappy function is fallible, which means that error handling is required for these errors:

value cannot be encoded into a Snappy string.

Examples

Encode to Snappy

Source

encoded_text = encode_snappy!("The quick brown fox jumps over 13 lazy dogs.")
encode_base64(encoded_text)

Return

LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg==

encode_zlib

infallible pure

Encodes the value to Zlib.

Function spec

encode_zlib(value: <string> , [compression_level: <integer> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to encode.		yes
compression_level	integer	The default compression level.	`6`	no

Examples

Encode to Zlib

Source

encoded_text = encode_zlib("please encode me")
encode_base64(encoded_text)

Return

eJwryElNLE5VSM1Lzk9JVchNBQA0RQX7

encode_zstd

infallible pure

Encodes the value to Zstandard.

Function spec

encode_zstd(value: <string> , [compression_level: <integer> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to encode.		yes
compression_level	integer	The default compression level.	`3`	no

Examples

Encode to Zstd

Source

encoded_text = encode_zstd("please encode me")
encode_base64(encoded_text)

Return

KLUv/QBYgQAAcGxlYXNlIGVuY29kZSBtZQ==

Coerce functions

to_bool

fallible pure

Coerces the value into a boolean.

Function spec

to_bool(value: <boolean | integer | float | null | string> )

:: <boolean> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	boolean integer float null string	The value to convert to a Boolean.		yes

Errors

The to_bool function is fallible, which means that error handling is required for these errors:

value is not a supported boolean representation.

Examples

Coerce to a Boolean (string)

Source

to_bool!("yes")

Return

true

Coerce to a Boolean (float)

Source

to_bool(0.0)

Return

false

Coerce to a Boolean (int)

Source

to_bool(0)

Return

false

Coerce to a Boolean (null)

Source

to_bool(null)

Return

false

Coerce to a Boolean (Boolean)

Source

to_bool(true)

Return

true

to_float

fallible pure

Coerces the value into a float.

Function spec

to_float(value: <integer | float | boolean | string | timestamp> )

:: <float> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	integer float boolean string timestamp	The value to convert to a float. Must be convertible to a float, otherwise an error is raised.		yes

Errors

The to_float function is fallible, which means that error handling is required for these errors:

value is not a supported float representation.

Examples

Coerce to a float

Source

to_float!("3.145")

Return

3.145

Coerce to a float (timestamp)

Source

to_float(t'2020-12-30T22:20:53.824727Z')

Return

1609366853.824727

to_int

fallible pure

Coerces the value into an integer.

Function spec

:: <integer> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	integer float boolean string timestamp null	The value to convert to an integer.		yes

Errors

The to_int function is fallible, which means that error handling is required for these errors:

value is a string but the text is not an integer.

value is not a string, int, or timestamp.

Examples

Coerce to an int (string)

Source

to_int!("2")

Return

Coerce to an int (timestamp)

Source

to_int(t'2020-12-30T22:20:53.824727Z')

Return

1609366853

to_regex

fallible pure

Coerces the value into a regex.

Function spec

to_regex(value: <string> )

:: <regex> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The value to convert to a regex.		yes

Notices

This function has special behavior that you should be aware of.

Compiling a regular expression is an expensive operation and can limit Vector throughput. Don’t use this function unless you are absolutely sure there is no other way!

Errors

The to_regex function is fallible, which means that error handling is required for these errors:

value is not a string.

Examples

Coerce to a regex

Source

to_regex("^foo$") ?? r''

Return

^foo$

to_string

fallible pure

Coerces the value into a string.

Function spec

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	integer float boolean string timestamp null	The value to convert to a string.		yes

Errors

The to_string function is fallible, which means that error handling is required for these errors:

value is not an integer, float, boolean, string, timestamp, or null.

Examples

Coerce to a string (Boolean)

Source

to_string(true)

Return

true

Coerce to a string (int)

Source

to_string(52)

Return

Coerce to a string (float)

Source

to_string(52.2)

Return

52.2

Convert functions

from_unix_timestamp

infallible pure

Converts the value integer from a Unix timestamp to a VRL timestamp.

Converts from the number of seconds since the Unix epoch by default. To convert from milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.

Function spec

from_unix_timestamp(value: <integer> , [unit: <string> ] )

:: <timestamp>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	integer	The Unix timestamp to convert.		yes
unit	string	The time unit.	`seconds`	no

Examples

Convert from a Unix timestamp (seconds)

Source

from_unix_timestamp!(5)

Return

1970-01-01T00:00:05Z

Convert from a Unix timestamp (milliseconds)

Source

from_unix_timestamp!(5000, unit: "milliseconds")

Return

1970-01-01T00:00:05Z

Convert from a Unix timestamp (nanoseconds)

Source

from_unix_timestamp!(5000, unit: "nanoseconds")

Return

1970-01-01T00:00:00.000005Z

to_syslog_facility

fallible pure

Converts the value, a Syslog facility code, into its corresponding Syslog keyword. For example, 0 into "kern", 1 into "user", etc.

Function spec

to_syslog_facility(value: <integer> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	integer	The facility code.		yes

Errors

The to_syslog_facility function is fallible, which means that error handling is required for these errors:

value is not a valid Syslog facility code.

Examples

Coerce to a Syslog facility

Source

to_syslog_facility!(4)

Return

auth

to_syslog_facility_code

fallible pure

Converts the value, a Syslog facility keyword, into a Syslog integer facility code (0 to 23).

Function spec

to_syslog_facility_code(value: <string> )

:: <integer> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The Syslog facility keyword to convert.		yes

Errors

The to_syslog_facility_code function is fallible, which means that error handling is required for these errors:

value is not a valid Syslog facility keyword.

Examples

Coerce to Syslog facility code

Source

to_syslog_facility_code!("authpriv")

Return

to_syslog_level

fallible pure

Converts the value, a Syslog severity level, into its corresponding keyword, i.e. 0 into "emerg", 1 into "alert", etc.

Function spec

to_syslog_level(value: <integer> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	integer	The severity level.		yes

Errors

The to_syslog_level function is fallible, which means that error handling is required for these errors:

value isn’t a valid Syslog severity level.

Examples

Coerce to a Syslog level

Source

to_syslog_level!(5)

Return

notice

to_syslog_severity

fallible pure

Converts the value, a Syslog log level keyword, into a Syslog integer severity level (0 to 7).

Function spec

to_syslog_severity(value: <string> )

:: <integer> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The Syslog level keyword to convert.		yes

Errors

The to_syslog_severity function is fallible, which means that error handling is required for these errors:

value is not a valid Syslog level keyword.

Examples

Coerce to Syslog severity

Source

to_syslog_severity!("alert")

Return

to_unix_timestamp

fallible pure

Converts the value timestamp into a Unix timestamp.

Returns the number of seconds since the Unix epoch by default. To return the number in milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.

Function spec

to_unix_timestamp(value: <timestamp> , [unit: <string> ] )

:: <integer> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	timestamp	The timestamp to convert into a Unix timestamp.		yes
unit	string	The time unit.	`seconds`	no

Errors

The to_unix_timestamp function is fallible, which means that error handling is required for these errors:

value cannot be represented in nanoseconds. Result is too large or too small for a 64 bit integer.

Examples

Convert to a Unix timestamp (seconds)

Source

to_unix_timestamp(t'2021-01-01T00:00:00+00:00')

Return

1609459200

Convert to a Unix timestamp (milliseconds)

Source

to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "milliseconds")

Return

1609459200000

Convert to a Unix timestamp (nanoseconds)

Source

to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "nanoseconds")

Return

1609459200000000000

Debug functions

assert

fallible impure

Asserts the condition, which must be a Boolean expression. The program is aborted with message if the condition evaluates to false.

Function spec

assert(condition: <boolean> , [message: <string> ] )

:: <null> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
condition	boolean	The condition to check.		yes
message	string	An optional custom error message. If the equality assertion fails, `message` is appended to the default message prefix. See the examples below for a fully formed log message sample.		no

Notices

This function has special behavior that you should be aware of.

The assert function should be used in a standalone fashion and only when you want to abort the program. You should avoid it in logical expressions and other situations in which you want the program to continue if the condition evaluates to false.

Errors

The assert function is fallible, which means that error handling is required for these errors:

condition evaluates to false.

Examples

Assertion (true)

Source

assert!("foo" == "foo", message: "\"foo\" must be \"foo\"!")

Return

true

Assertion (false)

Source

assert!("foo" == "bar", message: "\"foo\" must be \"foo\"!")

assert_eq

infallible impure

Asserts that two expressions, left and right, have the same value. The program is aborted with message if they do not have the same value.

Function spec

assert_eq(left: <any> ,right: <any> , [message: <string> ] )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Required?
left	any	The value to check for equality against `right`.	yes
right	any	The value to check for equality against `left`.	yes
message	string	An optional custom error message. If the equality assertion fails, `message` is appended to the default message prefix. See the examples below for a fully formed log message sample.	no

Notices

This function has special behavior that you should be aware of.

The assert_eq function should be used in a standalone fashion and only when you want to abort the program. You should avoid it in logical expressions and other situations in which you want the program to continue if the condition evaluates to false.

Examples

Successful assertion

Source

assert_eq!(1, 1)

Return

true

Unsuccessful assertion

Source

assert_eq!(127, [1, 2, 3])

Unsuccessful assertion with custom log message

Source

 assert_eq!(1, 0, message: "Unequal integers")

log

infallible impure

Logs the value to stdout at the specified level.

Function spec

log(value: <any> , [level: <string> ,rate_limit_secs: <integer> ] )

:: <null>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to log.		yes
level	string	The log level.	`info`	no
rate_limit_secs	integer	Specifies that the log message is output no more than once per the given number of seconds. Use a value of `0` to turn rate limiting off.	`1`	no

Examples

Log a message

Source

log("Hello, World!", level: "info", rate_limit_secs: 60)

Return

Log an error

Source

_, err = to_int(.field)
if err != null {
	log(err, level: "error")
}

Return

Enrichment functions

find_enrichment_table_records

infallible pure

Searches an enrichment table for rows that match the provided condition.

For file enrichment tables, this condition needs to be a VRL object in which the key-value pairs indicate a field to search mapped to a value to search in that field. This function returns the rows that match the provided condition(s). All fields need to match for rows to be returned; if any fields do not match, then no rows are returned.

There are currently three forms of search criteria:

Exact match search. The given field must match the value exactly. Case sensitivity can be specified using the case_sensitive argument. An exact match search can use an index directly into the dataset, which should make this search fairly “cheap” from a performance perspective.
Wildcard match search. The given fields specified by the exact match search may also be matched exactly to the value provided to the wildcard parameter. A wildcard match search can also use an index directly into the dataset.
Date range search. The given field must be greater than or equal to the from date and/or less than or equal to the to date. A date range search involves sequentially scanning through the rows that have been located using any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. Therefore, use date ranges as the only criteria when the enrichment data set is very small.

For geoip and mmdb enrichment tables, this condition needs to be a VRL object with a single key-value pair whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected and without a value, null is used. This table can return the following fields:

ISP databases:
- autonomous_system_number
- autonomous_system_organization
- isp
- organization
City databases:
- city_name
- continent_code
- country_code
- country_name
- region_code
- region_name
- metro_code
- latitude
- longitude
- postal_code
- timezone
Connection-Type databases:
- connection_type

To use this function, you need to update your configuration to include an enrichment_tables parameter.

Function spec

find_enrichment_table_records(table: <string> ,condition: <object> , [select: <array> ,case_sensitive: <boolean> ] )

:: <array>

required optional <types | ...>

Argument	Type	Description	Default	Required?
table	string	The enrichment table to search.		yes
condition	object	The condition to search on. Since the condition is used at boot time to create indices into the data, these conditions must be statically defined.		yes
select	array	A subset of fields from the enrichment table to return. If not specified, all fields are returned.		no
case_sensitive	boolean	Whether text fields need to match cases exactly.	`true`	no

Examples

Exact match

Source

find_enrichment_table_records!("test",
  {
	"surname": "smith",
  },
  case_sensitive: false)

Return

[{"firstname":"Bob","id":1,"surname":"Smith"},{"firstname":"Fred","id":2,"surname":"Smith"}]

Wildcard match

Source

find_enrichment_table_records!("test",
  {
    "firstname": "Bob",
  },
  wildcard: "fred",
  case_sensitive: false)

Return

[{"firstname":"Bob","id":1,"surname":"Smith"},{"firstname":"Fred","id":2,"surname":"Smith"}]

Date range search

Source

find_enrichment_table_records!("test",
  {
	"surname": "Smith",
	"date_of_birth": {
	  "from": t'1985-01-01T00:00:00Z',
	  "to": t'1985-12-31T00:00:00Z'
	}
  })

Return

[{"firstname":"Bob","id":1,"surname":"Smith"},{"firstname":"Fred","id":2,"surname":"Smith"}]

get_enrichment_table_record

fallible pure

Searches an enrichment table for a row that matches the provided condition. A single row must be matched. If no rows are found or more than one row is found, an error is returned.

There are currently three forms of search criteria:

Exact match search. The given field must match the value exactly. Case sensitivity can be specified using the case_sensitive argument. An exact match search can use an index directly into the dataset, which should make this search fairly “cheap” from a performance perspective.
Wildcard match search. The given fields specified by the exact match search may also be matched exactly to the value provided to the wildcard parameter. A wildcard match search can also use an index directly into the dataset.
Date range search. The given field must be greater than or equal to the from date and/or less than or equal to the to date. A date range search involves sequentially scanning through the rows that have been located using any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. Therefore, use date ranges as the only criteria when the enrichment data set is very small.

ISP databases:
- autonomous_system_number
- autonomous_system_organization
- isp
- organization
City databases:
- city_name
- continent_code
- country_code
- country_name
- region_code
- region_name
- metro_code
- latitude
- longitude
- postal_code
- timezone
Connection-Type databases:
- connection_type

To use this function, you need to update your configuration to include an enrichment_tables parameter.

Function spec

get_enrichment_table_record(table: <string> ,condition: <object> , [select: <array> ,case_sensitive: <boolean> ] )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
table	string	The enrichment table to search.		yes
condition	object	The condition to search on. Since the condition is used at boot time to create indices into the data, these conditions must be statically defined.		yes
select	array	A subset of fields from the enrichment table to return. If not specified, all fields are returned.		no
case_sensitive	boolean	Whether the text fields match the case exactly.	`true`	no

Errors

The get_enrichment_table_record function is fallible, which means that error handling is required for these errors:

The row is not found.

Multiple rows are found that match the condition.

Examples

Exact match

Source

get_enrichment_table_record!("test",
  {
    "surname": "bob",
    "firstname": "John"
  },
  case_sensitive: false)

Return

{
  "firstname": "Bob",
  "id": 1,
  "surname": "Smith"
}

Wildcard match

Source

find_enrichment_table_records!("test",
  {
    "firstname": "Bob",
  },
  wildcard: "fred",
  case_sensitive: false)

Return

[{"firstname":"Bob","id":1,"surname":"Smith"},{"firstname":"Fred","id":2,"surname":"Smith"}]

Date range search

Source

get_enrichment_table_record!("test",
  {
    "surname": "Smith",
    "date_of_birth": {
      "from": t'1985-01-01T00:00:00Z',
      "to": t'1985-12-31T00:00:00Z'
    }
  })

Return

{
  "firstname": "Bob",
  "id": 1,
  "surname": "Smith"
}

Enumerate functions

compact

infallible pure

Compacts the value by removing empty values, where empty values are defined using the available parameters.

Function spec

compact(value: <array | object> , [recursive: <boolean> ,null: <boolean> ,string: <boolean> ,object: <boolean> ,array: <boolean> ,nullish: <boolean> ] )

:: <array | object>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	array object	The object or array to compact.		yes
recursive	boolean	Whether the compaction be recursive.	`true`	no
null	boolean	Whether null should be treated as an empty value.	`true`	no
string	boolean	Whether an empty string should be treated as an empty value.	`true`	no
object	boolean	Whether an empty object should be treated as an empty value.	`true`	no
array	boolean	Whether an empty array should be treated as an empty value.	`true`	no
nullish	boolean	Tests whether the value is “nullish” as defined by the `is_nullish` function.		no

Examples

Compact an array

Source

compact(["foo", "bar", "", null, [], "buzz"], string: true, array: true, null: true)

Return

["foo","bar","buzz"]

Compact an object

Source

compact({"field1": 1, "field2": "", "field3": [], "field4": null}, string: true, array: true, null: true)

Return

{
  "field1": 1
}

filter

infallible pure

Filter elements from a collection.

This function currently does not support recursive iteration.

The function uses the function closure syntax to allow reading the key-value or index-value combination for each item in the collection.

The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

See the examples below to learn about the closure syntax.

Function spec

filter(value: <array | object> )

:: <array | object>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	array object	The array or object to filter.		yes

Examples

Filter elements

Source

filter(array!(.tags)) -> |_index, value| {
    # keep any elements that aren't equal to "foo"
    value != "foo"
}

Return

["bar","baz"]

flatten

infallible pure

Flattens the value into a single-level representation.

Function spec

flatten(value: <array | object> , [separator: <string> ] )

:: <array | object>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	array object	The array or object to flatten.		yes
separator	string	The separator to join nested keys	`.`	no

Examples

Flatten array

Source

flatten([1, [2, 3, 4], [5, [6, 7], 8], 9])

Return

[1,2,3,4,5,6,7,8,9]

Flatten object

Source

flatten({
	"parent1": {
		"child1": 1,
		"child2": 2
	},
	"parent2": {
		"child3": 3
	}
})

Return

{
  "parent1.child1": 1,
  "parent1.child2": 2,
  "parent2.child3": 3
}

for_each

infallible pure

Iterate over a collection.

This function currently does not support recursive iteration.

The function uses the “function closure syntax” to allow reading the key/value or index/value combination for each item in the collection.

See the examples below to learn about the closure syntax.

Function spec

for_each(value: <array | object> )

:: <null>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	array object	The array or object to iterate.		yes

Examples

Tally elements

Source

tally = {}
for_each(array!(.tags)) -> |_index, value| {
    # Get the current tally for the `value`, or
    # set to `0`.
    count = int(get!(tally, [value])) ?? 0

    # Increment the tally for the value by `1`.
    tally = set!(tally, [value], count + 1)
}

tally

Return

{
  "bar": 1,
  "baz": 1,
  "foo": 2
}

includes

infallible pure

Determines whether the value array includes the specified item.

Function spec

includes(value: <array> ,item: <any> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	array	The array.		yes
item	any	The item to check.		yes

Examples

Array includes

Source

includes(["apple", "orange", "banana"], "banana")

Return

true

keys

infallible pure

Returns the keys from the object passed into the function.

Function spec

keys(value: <object> )

:: <array>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	object	The object to extract keys from.		yes

Examples

Get keys from the object

Source

keys({"key1": "val1", "key2": "val2"})

Return

["key1","key2"]

length

infallible pure

Returns the length of the value.

If value is an array, returns the number of elements.
If value is an object, returns the number of top-level keys.
If value is a string, returns the number of bytes in the string. If you want the number of characters, see strlen.

Function spec

length(value: <array | object | string> )

:: <integer>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	array object string	The array or object.		yes

Examples

Length (object)

Source

length({
	"portland": "Trail Blazers",
	"seattle":  "Supersonics"
})

Return

Length (nested object)

Source

length({
	"home": {
		"city":  "Portland",
		"state": "Oregon"
	},
	"name": "Trail Blazers",
	"mascot": {
		"name": "Blaze the Trail Cat"
	}
})

Return

Length (array)

Source

length(["Trail Blazers", "Supersonics", "Grizzlies"])

Return

Length (string)

Source

length("The Planet of the Apes Musical")

Return

map_keys

infallible pure

Map the keys within an object.

If recursive is enabled, the function iterates into nested objects, using the following rules:

Iteration starts at the root.
For every nested object type:
- First return the key of the object type itself.
- Then recurse into the object, and loop back to item (1) in this list.
- Any mutation done on a nested object before recursing into it, are preserved.
For every nested array type:
- First return the key of the array type itself.
- Then find all objects within the array, and apply item (2) to each individual object.

The above rules mean that map_keys with recursive enabled finds all keys in the target, regardless of whether nested objects are nested inside arrays.

The function uses the function closure syntax to allow reading the key for each item in the object.

See the examples below to learn about the closure syntax.

Function spec

map_keys(value: <object> , [recursive: <boolean> ] )

:: <object>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	object	The object to iterate.		yes
recursive	boolean	Whether to recursively iterate the collection.		no

Examples

Upcase keys

Source

map_keys(.) -> |key| { upcase(key) }

Return

{
  "BAR": "bar",
  "FOO": "foo"
}

De-dot keys

Source

map_keys(., recursive: true) -> |key| { replace(key, ".", "_") }

Return

{
  "labels": {
    "app_kubernetes_io/name": "mysql"
  }
}

map_values

infallible pure

Map the values within a collection.

If recursive is enabled, the function iterates into nested collections, using the following rules:

Iteration starts at the root.
For every nested collection type:
- First return the collection type itself.
- Then recurse into the collection, and loop back to item (1) in the list
- Any mutation done on a collection before recursing into it, are preserved.

The function uses the function closure syntax to allow mutating the value for each item in the collection.

The same scoping rules apply to closure blocks as they do for regular blocks, meaning, any variable defined in parent scopes are accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

Check out the examples below to learn about the closure syntax.

Function spec

map_values(value: <array | object> , [recursive: <boolean> ] )

:: <array | object>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	array object	The object or array to iterate.		yes
recursive	boolean	Whether to recursively iterate the collection.		no

Examples

Upcase values

Source

map_values(.) -> |value| { upcase!(value) }

Return

{
  "bar": "BAR",
  "foo": "FOO"
}

match_array

infallible pure

Determines whether the elements in the value array matches the pattern. By default, it checks that at least one element matches, but can be set to determine if all the elements match.

Function spec

match_array(value: <array> ,pattern: <regex> , [all: <boolean> ] )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Required?
value	array	The array.	yes
pattern	regex	The regular expression pattern to match against.	yes
all	boolean	Whether to match on all elements of `value`.	no

Examples

Match at least one element

Source

match_array(["foobar", "bazqux"], r'foo')

Return

true

Match all elements

Source

match_array(["foo", "foobar", "barfoo"], r'foo', all: true)

Return

true

No matches

Source

match_array(["bazqux", "xyz"], r'foo')

Return

false

Not all elements match

Source

match_array(["foo", "foobar", "baz"], r'foo', all: true)

Return

false

strlen

infallible pure

Returns the number of UTF-8 characters in value. This differs from length which counts the number of bytes of a string.

Note: This is the count of Unicode scalar values which can sometimes differ from Unicode code points.

Function spec

strlen(value: <string> )

:: <integer>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string.		yes

Examples

strlen

Source

strlen("ñandú")

Return

unflatten

infallible pure

Unflattens the value into a nested representation.

Function spec

unflatten(value: <object> , [separator: <string> ,recursive: <boolean> ] )

:: <object>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	object	The array or object to unflatten.		yes
separator	string	The separator to split flattened keys.	`.`	no
recursive	boolean	Whether to recursively unflatten the object values.	`true`	no

Examples

Unflatten

Source

unflatten({
    "foo.bar.baz": true,
    "foo.bar.qux": false,
	"foo.quux": 42
})

Return

{
  "foo": {
    "bar": {
      "baz": true,
      "qux": false
    },
    "quux": 42
  }
}

Unflatten recursively

Source

unflatten({
    "flattened.parent": {
		"foo.bar": true,
		"foo.baz": false
	}
})

Return

{
  "flattened": {
    "parent": {
      "foo": {
        "bar": true,
        "baz": false
      }
    }
  }
}

Unflatten non-recursively

Source

unflatten({
    "flattened.parent": {
		"foo.bar": true,
		"foo.baz": false
	}
}, recursive: false)

Return

{
  "flattened": {
    "parent": {
      "foo.bar": true,
      "foo.baz": false
    }
  }
}

Ignore inconsistent keys values

Source

unflatten({
	"a": 3,
	"a.b": 2,
	"a.c": 4
})

Return

{
  "a": {
    "b": 2,
    "c": 4
  }
}

unique

infallible pure

Returns the unique values for an array.

The first occurrence of each element is kept.

Function spec

unique(value: <array> )

:: <array>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	array	The array to return unique elements from.		yes

Examples

Unique

Source

unique(["foo", "bar", "foo", "baz"])

Return

["foo","bar","baz"]

values

infallible pure

Returns the values from the object passed into the function.

Function spec

values(value: <object> )

:: <array>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	object	The object to extract values from.		yes

Examples

Get values from the object

Source

values({"key1": "val1", "key2": "val2"})

Return

["val1","val2"]

Event functions

get_secret

infallible pure

Returns the value of the given secret from an event.

Function spec

get_secret(key: <string> )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
key	string	The name of the secret.		yes

Examples

Get the Datadog API key from the event metadata

Source

get_secret("datadog_api_key")

Return

secret value

remove_secret

infallible pure

Removes a secret from an event.

Function spec

remove_secret(key: <string> )

:: <null>

required optional <types | ...>

Argument	Type	Description	Default	Required?
key	string	The name of the secret to remove.		yes

Examples

Removes the Datadog API key from the event

Source

remove_secret("datadog_api_key")

Return

set_secret

infallible pure

Sets the given secret in the event.

Function spec

set_secret(key: <string> ,secret: <string> )

:: <null>

required optional <types | ...>

Argument	Type	Description	Default	Required?
key	string	The name of the secret.		yes
secret	string	The secret value.		yes

Examples

Set the Datadog API key to the given value

Source

set_secret("datadog_api_key", "abc122")

Return

set_semantic_meaning

infallible pure

Sets a semantic meaning for an event. Note: This function assigns meaning at startup, and has no runtime behavior. It is suggested to put all calls to this function at the beginning of a VRL function. The function cannot be conditionally called. For example, using an if statement cannot stop the meaning from being assigned.

Function spec

set_semantic_meaning(target: <path> ,meaning: <string> )

:: <null>

required optional <types | ...>

Argument	Type	Description	Default	Required?
target	path	The path of the value that is assigned a meaning.		yes
meaning	string	The name of the meaning to assign.		yes

Examples

Sets custom field semantic meaning

Source

set_semantic_meaning(.foo, "bar")

Return

Path functions

del

infallible impure

Removes the field specified by the static path from the target.

For dynamic path deletion, see the remove function.

Function spec

del(path: <path> , [compact: <boolean> ] )

:: <any>

required optional <types | ...>

Argument	Type	Description	Default	Required?
path	path	The path of the field to delete.		yes
compact	boolean	After deletion, if `compact` is `true` and there is an empty object or array left, the empty object or array is also removed, cascading up to the root. This only applies to the path being deleted, and any parent paths.		no

Notices

This function has special behavior that you should be aware of.

The del function modifies the current event in place and returns the value of the deleted field.

Examples

Delete a field

Source

del(.field1)

Rename a field

Source

.new_field = del(.old_field)

exists

infallible pure

Checks whether the path exists for the target.

This function distinguishes between a missing path and a path with a null value. A regular path lookup, such as .foo, cannot distinguish between the two cases since it always returns null if the path doesn’t exist.

Function spec

exists(path: <path> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
path	path	The path of the field to check.		yes

Examples

Exists (field)

Source

exists(.field)

Return

true

Exists (array element)

Source

exists(.array[2])

Return

true

get

fallible pure

Dynamically get the value of a given path.

If you know the path you want to look up, use static paths such as .foo.bar[1] to get the value of that path. However, if you do not know the path names, use the dynamic get function to get the requested value.

Function spec

get(value: <object | array> ,path: <array> )

:: <any> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	object array	The object or array to query.		yes
path	array	An array of path segments to look for the value.		yes

Errors

The get function is fallible, which means that error handling is required for these errors:

The path segment must be a string or an integer.

Examples

single-segment top-level field

Source

get!(value: { "foo": "bar" }, path: ["foo"])

Return

bar

multi-segment nested field

Source

get!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])

Return

baz

array indexing

Source

get!(value: ["foo", "bar", "baz"], path: [-2])

Return

bar

remove

fallible pure

Dynamically remove the value for a given path.

If you know the path you want to remove, use the del function and static paths such as del(.foo.bar[1]) to remove the value at that path. The del function returns the deleted value, and is more performant than remove. However, if you do not know the path names, use the dynamic remove function to remove the value at the provided path.

Function spec

remove(value: <object | array> ,path: <array> , [compact: <boolean> ] )

:: <object | array> , <error>

required optional <types | ...>

Argument	Type	Description	Required?
value	object array	The object or array to remove data from.	yes
path	array	An array of path segments to remove the value from.	yes
compact	boolean	After deletion, if `compact` is `true`, any empty objects or arrays left are also removed.	no

Errors

The remove function is fallible, which means that error handling is required for these errors:

The path segment must be a string or an integer.

Examples

single-segment top-level field

Source

remove!(value: { "foo": "bar" }, path: ["foo"])

Return

{}

multi-segment nested field

Source

remove!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])

Return

{
  "foo": {}
}

array indexing

Source

remove!(value: ["foo", "bar", "baz"], path: [-2])

Return

["foo","baz"]

compaction

Source

remove!(value: { "foo": { "bar": [42], "baz": true } }, path: ["foo", "bar", 0], compact: true)

Return

{
  "foo": {
    "baz": true
  }
}

set

fallible pure

Dynamically insert data into the path of a given object or array.

If you know the path you want to assign a value to, use static path assignments such as .foo.bar[1] = true for improved performance and readability. However, if you do not know the path names, use the dynamic set function to insert the data into the object or array.

Function spec

set(value: <object | array> ,path: <array> ,data: <any> )

:: <object | array> , <error>

required optional <types | ...>

Argument	Type	Description	Required?
value	object array	The object or array to insert data into.	yes
path	array	An array of path segments to insert the value into.	yes
data	any	The data to be inserted.	yes

Errors

The set function is fallible, which means that error handling is required for these errors:

The path segment must be a string or an integer.

Examples

single-segment top-level field

Source

set!(value: { "foo": "bar" }, path: ["foo"], data: "baz")

Return

{
  "foo": "baz"
}

multi-segment nested field

Source

set!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"], data: "qux")

Return

{
  "foo": {
    "bar": "qux"
  }
}

array

Source

set!(value: ["foo", "bar", "baz"], path: [-2], data: 42)

Return

["foo",42,"baz"]

Cryptography functions

decrypt

fallible pure

Decrypts a string with a symmetric encryption algorithm.

Supported Algorithms:

AES-256-CFB (key = 32 bytes, iv = 16 bytes)
AES-192-CFB (key = 24 bytes, iv = 16 bytes)
AES-128-CFB (key = 16 bytes, iv = 16 bytes)
AES-256-OFB (key = 32 bytes, iv = 16 bytes)
AES-192-OFB (key = 24 bytes, iv = 16 bytes)
AES-128-OFB (key = 16 bytes, iv = 16 bytes)
AES-128-SIV (key = 32 bytes, iv = 16 bytes)
AES-256-SIV (key = 64 bytes, iv = 16 bytes)
Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)

Function spec

decrypt(ciphertext: <string> ,algorithm: <string> ,key: <string> ,iv: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Required?
ciphertext	string	The string in raw bytes (not encoded) to decrypt.	yes
algorithm	string	The algorithm to use.	yes
key	string	The key in raw bytes (not encoded) for decryption. The length must match the algorithm requested.	yes
iv	string	The IV in raw bytes (not encoded) for decryption. The length must match the algorithm requested. A new IV should be generated for every message. You can use `random_bytes` to generate a cryptographically secure random value. The value should match the one used during encryption.	yes

Errors

The decrypt function is fallible, which means that error handling is required for these errors:

algorithm is not a supported algorithm.

key length does not match the key size required for the algorithm specified.

iv length does not match the iv size required for the algorithm specified.

Examples

Decrypt value

Source

ciphertext = decode_base64!("5fLGcu1VHdzsPcGNDio7asLqE1P43QrVfPfmP4i4zOU=")
iv = decode_base64!("fVEIRkIiczCRWNxaarsyxA==")
key = "16_byte_keyxxxxx"
decrypt!(ciphertext, "AES-128-CBC-PKCS7", key, iv: iv)

Return

super_secret_message

encrypt

fallible pure

Encrypts a string with a symmetric encryption algorithm.

Supported Algorithms:

AES-256-CFB (key = 32 bytes, iv = 16 bytes)
AES-192-CFB (key = 24 bytes, iv = 16 bytes)
AES-128-CFB (key = 16 bytes, iv = 16 bytes)
AES-256-OFB (key = 32 bytes, iv = 16 bytes)
AES-192-OFB (key = 24 bytes, iv = 16 bytes)
AES-128-OFB (key = 16 bytes, iv = 16 bytes)
AES-128-SIV (key = 32 bytes, iv = 16 bytes)
AES-256-SIV (key = 64 bytes, iv = 16 bytes)
Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)

Function spec

encrypt(plaintext: <string> ,algorithm: <string> ,key: <string> ,iv: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Required?
plaintext	string	The string to encrypt.	yes
algorithm	string	The algorithm to use.	yes
key	string	The key in raw bytes (not encoded) for encryption. The length must match the algorithm requested.	yes
iv	string	The IV in raw bytes (not encoded) for encryption. The length must match the algorithm requested. A new IV should be generated for every message. You can use `random_bytes` to generate a cryptographically secure random value.	yes

Errors

The encrypt function is fallible, which means that error handling is required for these errors:

algorithm is not a supported algorithm.

key length does not match the key size required for the algorithm specified.

iv length does not match the iv size required for the algorithm specified.

Examples

Encrypt value

Source

plaintext = "super secret message"
iv = "1234567890123456" # typically you would call random_bytes(16)
key = "16_byte_keyxxxxx"
encrypted_message = encrypt!(plaintext, "AES-128-CBC-PKCS7", key, iv: iv)
encode_base64(encrypted_message)

Return

GBw8Mu00v0Kc38+/PvsVtGgWuUJ+ZNLgF8Opy8ohIYE=

hmac

infallible pure

Calculates a HMAC of the value using the given key. The hashing algorithm used can be optionally specified.

For most use cases, the resulting bytestream should be encoded into a hex or base64 string using either encode_base16 or encode_base64.

This function is infallible if either the default algorithm value or a recognized-valid compile-time algorithm string literal is used. Otherwise, it is fallible.

Function spec

hmac(value: <string> ,key: <string> , [algorithm: <string> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to calculate the HMAC for.		yes
key	string	The string to use as the cryptographic key.		yes
algorithm	string	The hashing algorithm to use.	`SHA-256`	no

Examples

Calculate message HMAC (defaults: SHA-256), encoding to a base64 string

Source

encode_base64(hmac("Hello there", "super-secret-key"))

Return

eLGE8YMviv85NPXgISRUZxstBNSU47JQdcXkUWcClmI=

Calculate message HMAC using SHA-224, encoding to a hex-encoded string

Source

encode_base16(hmac("Hello there", "super-secret-key", algorithm: "SHA-224"))

Return

42fccbc2b7d22a143b92f265a8046187558a94d11ddbb30622207e90

Calculate message HMAC using a variable hash algorithm

Source

.hash_algo = "SHA-256"
hmac_bytes, err = hmac("Hello there", "super-secret-key", algorithm: .hash_algo)
if err == null {
	.hmac = encode_base16(hmac_bytes)
}

Return

78b184f1832f8aff3934f5e0212454671b2d04d494e3b25075c5e45167029662

md5

infallible pure

Calculates an md5 hash of the value.

Function spec

md5(value: <string> )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to calculate the hash for.		yes

Examples

Create md5 hash

Source

md5("foo")

Return

acbd18db4cc2f85cedef654fccc4a4d8

seahash

infallible pure

Calculates a Seahash hash of the value. Note: Due to limitations in the underlying VRL data types, this function converts the unsigned 64-bit integer SeaHash result to a signed 64-bit integer. Results higher than the signed 64-bit integer maximum value wrap around to negative values.

Function spec

seahash(value: <string> )

:: <integer>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to calculate the hash for.		yes

Examples

Calculate seahash

Source

seahash("foobar")

Return

5348458858952426000

Calculate negative seahash

Source

seahash("bar")

Return

-2796170501982571500

sha1

infallible pure

Calculates a SHA-1 hash of the value.

Function spec

sha1(value: <string> )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to calculate the hash for.		yes

Examples

Calculate sha1 hash

Source

sha1("foo")

Return

0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33

sha2

infallible pure

Calculates a SHA-2 hash of the value.

Function spec

sha2(value: <string> , [variant: <string> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to calculate the hash for.		yes
variant	string	The variant of the algorithm to use.	`SHA-512/256`	no

Examples

Calculate sha2 hash

Source

sha2("foo", variant: "SHA-512/224")

Return

d68f258d37d670cfc1ec1001a0394784233f88f056994f9a7e5e99be

sha3

infallible pure

Calculates a SHA-3 hash of the value.

Function spec

sha3(value: <string> , [variant: <string> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to calculate the hash for.		yes
variant	string	The variant of the algorithm to use.	`SHA3-512`	no

Examples

Calculate sha3 hash

Source

sha3("foo", variant: "SHA3-224")

Return

f4f6779e153c391bbd29c95e72b0708e39d9166c7cea51d1f10ef58a

IP functions

decrypt_ip

fallible pure

Decrypts an IP address that was previously encrypted, restoring the original IP address.

Supported Modes:

AES128 - Decrypts an IP address that was scrambled using AES-128 encryption. Can transform between IPv4 and IPv6.
PFX (Prefix-preserving) - Decrypts an IP address that was encrypted with prefix-preserving mode, where network hierarchy was maintained.

Function spec

decrypt_ip(ip: <string> ,key: <string> ,mode: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Required?
ip	string	The encrypted IP address to decrypt (v4 or v6).	yes
key	string	The decryption key in raw bytes (not encoded). Must be the same key that was used for encryption. For AES128 mode, the key must be exactly 16 bytes. For PFX mode, the key must be exactly 32 bytes.	yes
mode	string	The decryption mode to use. Must match the mode used for encryption: either `aes128` or `pfx`.	yes

Notices

This function has special behavior that you should be aware of.

The aes128 mode implements the ipcrypt-deterministic algorithm from the IPCrypt specification, while the pfx mode implements the ipcrypt-pfx algorithm. This function reverses the encryption performed by encrypt_ip - the same key and algorithm that were used for encryption must be used for decryption.

Errors

The decrypt_ip function is fallible, which means that error handling is required for these errors:

ip is not a valid IP address.

mode is not a supported mode (must be aes128 or pfx).

key length does not match the requirements for the specified mode (16 bytes for aes128, 32 bytes for pfx).

Examples

Decrypt IPv4 address with AES128

Source

decrypted_ip = decrypt_ip!("72b9:a747:f2e9:72af:76ca:5866:6dcf:c3b0", "sixteen byte key", "aes128")
decrypted_ip

Return

192.168.1.1

Decrypt IPv6 address with AES128

Source

decrypted_ip = decrypt_ip!("d09e:a5ea:585a:2547:dc6d:65ea:d9f1:d09d", "sixteen byte key", "aes128")
decrypted_ip

Return

2001:db8::1

Decrypt IPv4 address with prefix-preserving mode

Source

decrypted_ip = decrypt_ip!("b51c:3c43:4e89:819e:64ce:225f:d6d1:bf01", "thirty-two bytes key for pfx use", "pfx")
decrypted_ip

Return

192.168.1.1

Decrypt IPv6 address with prefix-preserving mode

Source

decrypted_ip = decrypt_ip!("88bd:d2bf:8865:8c4d:84b:44f6:6077:72c9", "thirty-two bytes key for ipv6pfx", "pfx")
decrypted_ip

Return

2001:db8::1

Round-trip encryption and decryption

Source

original_ip = "192.168.1.100"
key = "sixteen byte key"

encrypted = encrypt_ip!(original_ip, key, "aes128")
decrypted = decrypt_ip!(encrypted, key, "aes128")

decrypted == original_ip

Return

true

encrypt_ip

fallible pure

Encrypts an IP address, transforming it into a different valid IP address.

Supported Modes:

AES128 - Scrambles the entire IP address using AES-128 encryption. Can transform between IPv4 and IPv6.
PFX (Prefix-preserving) - Maintains network hierarchy by ensuring that IP addresses within the same network are encrypted to addresses that also share a common network. This preserves prefix relationships while providing confidentiality.

Function spec

encrypt_ip(ip: <string> ,key: <string> ,mode: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Required?
ip	string	The IP address to encrypt (v4 or v6).	yes
key	string	The encryption key in raw bytes (not encoded). For AES128 mode, the key must be exactly 16 bytes. For PFX mode, the key must be exactly 32 bytes.	yes
mode	string	The encryption mode to use. Must be either `aes128` or `pfx`.	yes

Notices

This function has special behavior that you should be aware of.

The aes128 mode implements the ipcrypt-deterministic algorithm from the IPCrypt specification, while the pfx mode implements the ipcrypt-pfx algorithm. Both modes provide deterministic encryption where the same input IP address encrypted with the same key will always produce the same encrypted output.

Errors

The encrypt_ip function is fallible, which means that error handling is required for these errors:

ip is not a valid IP address.

mode is not a supported mode (must be aes128 or pfx).

key length does not match the requirements for the specified mode (16 bytes for aes128, 32 bytes for pfx).

Examples

Encrypt IPv4 address with AES128

Source

encrypted_ip = encrypt_ip!("192.168.1.1", "sixteen byte key", "aes128")
encrypted_ip

Return

72b9:a747:f2e9:72af:76ca:5866:6dcf:c3b0

Encrypt IPv6 address with AES128

Source

encrypted_ip = encrypt_ip!("2001:db8::1", "sixteen byte key", "aes128")
encrypted_ip

Return

d09e:a5ea:585a:2547:dc6d:65ea:d9f1:d09d

Encrypt IPv4 address with prefix-preserving mode

Source

encrypted_ip = encrypt_ip!("192.168.1.1", "thirty-two bytes key for pfx use", "pfx")
encrypted_ip

Return

b51c:3c43:4e89:819e:64ce:225f:d6d1:bf01

Encrypt IPv6 address with prefix-preserving mode

Source

encrypted_ip = encrypt_ip!("2001:db8::1", "thirty-two bytes key for ipv6pfx", "pfx")
encrypted_ip

Return

88bd:d2bf:8865:8c4d:84b:44f6:6077:72c9

ip_aton

fallible pure

Converts IPv4 address in numbers-and-dots notation into network-order bytes represented as an integer.

This behavior mimics inet_aton.

Function spec

ip_aton(value: <string> )

:: <integer> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The IP address to convert to binary.		yes

Errors

The ip_aton function is fallible, which means that error handling is required for these errors:

value is not a valid IPv4 address.

Examples

IPv4 to integer

Source

ip_aton!("1.2.3.4")

Return

16909060

ip_cidr_contains

fallible pure

Determines whether the ip is contained in the block referenced by the cidr.

Function spec

ip_cidr_contains(cidr: <string | array> ,ip: <string> )

:: <boolean> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
cidr	string array	The CIDR mask (v4 or v6).		yes
ip	string	The IP address (v4 or v6).		yes

Errors

The ip_cidr_contains function is fallible, which means that error handling is required for these errors:

cidr is not a valid CIDR.

ip is not a valid IP address.

Examples

IPv4 contains CIDR

Source

ip_cidr_contains!("192.168.0.0/16", "192.168.10.32")

Return

true

IPv4 is private

Source

ip_cidr_contains!(["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"], "192.168.10.32")

Return

true

IPv6 contains CIDR

Source

ip_cidr_contains!("2001:4f8:4:ba::/64", "2001:4f8:4:ba:2e0:81ff:fe22:d1f1")

Return

true

ip_ntoa

fallible pure

Converts numeric representation of IPv4 address in network-order bytes to numbers-and-dots notation.

This behavior mimics inet_ntoa.

Function spec

ip_ntoa(value: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The integer representation of an IPv4 address.		yes

Errors

The ip_ntoa function is fallible, which means that error handling is required for these errors:

value cannot fit in an unsigned 32-bit integer.

Examples

Integer to IPv4

Source

ip_ntoa!(16909060)

Return

1.2.3.4

ip_ntop

fallible pure

Converts IPv4 and IPv6 addresses from binary to text form.

This behavior mimics inet_ntop.

Function spec

ip_ntop(value: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The binary data to convert from. For IPv4 addresses, it must be 4 bytes (32 bits) long. For IPv6 addresses, it must be 16 bytes (128 bits) long.		yes

Notices

This function has special behavior that you should be aware of.

The binary data for this function is not easily printable. However, the results from functions such as decode_base64 or decode_percent can still be used correctly.

Errors

The ip_ntop function is fallible, which means that error handling is required for these errors:

value must be of length 4 or 16 bytes.

Examples

Convert IPv4 address from bytes after decoding from Base64

Source

ip_ntop!(decode_base64!("wKgAAQ=="))

Return

192.168.0.1

Convert IPv6 address from bytes after decoding from Base64

Source

ip_ntop!(decode_base64!("IAENuIWjAAAAAIouA3BzNA=="))

Return

2001:db8:85a3::8a2e:370:7334

ip_pton

fallible pure

Converts IPv4 and IPv6 addresses from text to binary form.

The binary form of IPv4 addresses is 4 bytes (32 bits) long.
The binary form of IPv6 addresses is 16 bytes (128 bits) long.

This behavior mimics inet_pton.

Function spec

ip_pton(value: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The IP address (v4 or v6) to convert to binary form.		yes

Notices

This function has special behavior that you should be aware of.

The binary data from this function is not easily printable. However, functions such as encode_base64 or encode_percent can still process it correctly.

Errors

The ip_pton function is fallible, which means that error handling is required for these errors:

value is not a valid IP (v4 or v6) address in text form.

Examples

Convert IPv4 address to bytes and encode to Base64

Source

encode_base64(ip_pton!("192.168.0.1"))

Return

wKgAAQ==

Convert IPv6 address to bytes and encode to Base64

Source

encode_base64(ip_pton!("2001:db8:85a3::8a2e:370:7334"))

Return

IAENuIWjAAAAAIouA3BzNA==

ip_subnet

fallible pure

Extracts the subnet address from the ip using the supplied subnet.

Function spec

ip_subnet(ip: <string> ,subnet: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
ip	string	The IP address (v4 or v6).		yes
subnet	string	The subnet to extract from the IP address. This can be either a prefix length like `/8` or a net mask like `255.255.0.0`. The net mask can be either an IPv4 or IPv6 address.		yes

Notices

This function has special behavior that you should be aware of.

Works with both IPv4 and IPv6 addresses. The IP version for the mask must be the same as the supplied address.

Errors

The ip_subnet function is fallible, which means that error handling is required for these errors:

ip is not a valid IP address.

subnet is not a valid subnet.

Examples

IPv4 subnet

Source

ip_subnet!("192.168.10.32", "255.255.255.0")

Return

192.168.10.0

IPv6 subnet

Source

ip_subnet!("2404:6800:4003:c02::64", "/32")

Return

2404:6800::

ip_to_ipv6

fallible pure

Converts the ip to an IPv6 address.

Function spec

ip_to_ipv6(ip: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
ip	string	The IP address to convert to IPv6.		yes

Errors

The ip_to_ipv6 function is fallible, which means that error handling is required for these errors:

ip is not a valid IP address.

Examples

IPv4 to IPv6

Source

ip_to_ipv6!("192.168.10.32")

Return

::ffff:192.168.10.32

ipv6_to_ipv4

fallible pure

Converts the ip to an IPv4 address. ip is returned unchanged if it’s already an IPv4 address. If ip is currently an IPv6 address then it needs to be IPv4 compatible, otherwise an error is thrown.

Function spec

ipv6_to_ipv4(ip: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
ip	string	The IPv4-mapped IPv6 address to convert.		yes

Errors

The ipv6_to_ipv4 function is fallible, which means that error handling is required for these errors:

ip is not a valid IP address.

ip is an IPv6 address that is not compatible with IPv4.

Examples

IPv6 to IPv4

Source

ipv6_to_ipv4!("::ffff:192.168.0.1")

Return

192.168.0.1

is_ipv4

infallible pure

Check if the string is a valid IPv4 address or not.

An [IPv4-mapped][https://datatracker.ietf.org/doc/html/rfc6890] or [IPv4-compatible][https://datatracker.ietf.org/doc/html/rfc6890] IPv6 address is not considered valid for the purpose of this function.

Function spec

is_ipv4(value: <string> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The IP address to check		yes

Examples

Valid IPv4 address

Source

is_ipv4("10.0.102.37")

Return

true

Valid IPv6 address

Source

is_ipv4("2001:0db8:85a3:0000:0000:8a2e:0370:7334")

Return

false

Arbitrary string

Source

is_ipv4("foobar")

Return

false

is_ipv6

infallible pure

Check if the string is a valid IPv6 address or not.

Function spec

is_ipv6(value: <string> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The IP address to check		yes

Examples

Valid IPv6 address

Source

is_ipv6("2001:0db8:85a3:0000:0000:8a2e:0370:7334")

Return

true

Valid IPv4 address

Source

is_ipv6("10.0.102.37")

Return

false

Arbitrary string

Source

is_ipv6("foobar")

Return

false

Map functions

haversine

infallible pure

Calculates haversine distance and bearing between two points. Results are available in kilometers or miles.

Function spec

haversine(latitude1: <float> ,longitude1: <float> ,latitude2: <float> ,longitude2: <float> , [measurement: <string> ] )

:: <object>

required optional <types | ...>

Argument	Type	Description	Default	Required?
latitude1	float	Latitude of the first point.		yes
longitude1	float	Longitude of the first point.		yes
latitude2	float	Latitude of the second point.		yes
longitude2	float	Longitude of the second point.		yes
measurement	string	Measurement system to use for resulting distance.	`kilometers`	no

Examples

Haversine in kilometers

Source

haversine(0.0, 0.0, 10.0, 10.0)

Return

{
  "bearing": 44.561,
  "distance": 1568.5227233
}

Haversine in miles

Source

haversine(0.0, 0.0, 10.0, 10.0, "miles")

Return

{
  "bearing": 44.561,
  "distance": 974.6348468
}

Number functions

abs

infallible pure

Computes the absolute value of value.

Function spec

abs(value: <integer | float> )

:: <integer | float>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	integer float	The number to calculate the absolute value.		yes

Examples

Computes the absolute value of the integer

Source

abs(-42)

Return

Computes the absolute value of the float

Source

abs(-42.2)

Return

42.2

ceil

infallible pure

Rounds the value up to the specified precision.

Function spec

ceil(value: <integer | float> , [precision: <integer> ] )

:: <integer | float>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	integer float	The number to round up.		yes
precision	integer	The number of decimal places to round to.		no

Examples

Round a number up (without precision)

Source

ceil(4.345)

Return

Round a number up (with precision)

Source

ceil(4.345, precision: 2)

Return

4.35

floor

infallible pure

Rounds the value down to the specified precision.

Function spec

floor(value: <integer | float> , [precision: <integer> ] )

:: <integer | float>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	integer float	The number to round down.		yes
precision	integer	The number of decimal places to round to.		no

Examples

Round a number down (without precision)

Source

floor(4.345)

Return

Round a number down (with precision)

Source

floor(4.345, precision: 2)

Return

4.34

format_int

fallible pure

Formats the integer value into a string representation using the given base/radix.

Function spec

format_int(value: <integer> , [base: <integer> ] )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	integer	The number to format.		yes
base	integer	The base to format the number in. Must be between 2 and 36 (inclusive).	`10`	no

Errors

The format_int function is fallible, which means that error handling is required for these errors:

The base is not between 2 and 36.

Examples

Format as a hexadecimal integer

Source

format_int!(42, 16)

Return

2a

Format as a negative hexadecimal integer

Source

format_int!(-42, 16)

Return

-2a

format_number

infallible pure

Formats the value into a string representation of the number.

Function spec

format_number(value: <integer | float> , [scale: <integer> ,decimal_separator: <string> ,grouping_separator: <string> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	integer float	The number to format as a string.		yes
scale	integer	The number of decimal places to display.		no
decimal_separator	string	The character to use between the whole and decimal parts of the number.	`.`	no
grouping_separator	string	The character to use between each thousands part of the number.	`,`	no

Examples

Format a number (3 decimals)

Source

format_number(1234567.89, 3, decimal_separator: ".", grouping_separator: ",")

Return

1,234,567.890

mod

fallible pure

Calculates the remainder of value divided by modulus.

Function spec

mod(value: <integer | float> ,modulus: <integer | float> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	integer float	The value the `modulus` is applied to.		yes
modulus	integer float	The `modulus` value.		yes

Errors

The mod function is fallible, which means that error handling is required for these errors:

value is not an integer or float.

modulus is not an integer or float.

modulus is equal to 0.

Examples

Calculate the remainder of two integers

Source

mod(5, 2)

Return

round

infallible pure

Rounds the value to the specified precision.

Function spec

round(value: <integer | float> , [precision: <integer> ] )

:: <integer | float>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	integer float	The number to round.		yes
precision	integer	The number of decimal places to round to.		no

Examples

Round a number (without precision)

Source

round(4.345)

Return

Round a number (with precision)

Source

round(4.345, precision: 2)

Return

4.35

Object functions

match_datadog_query

infallible pure

Matches an object against a Datadog Search Syntax query.

Function spec

match_datadog_query(value: <object> ,query: <string> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	object	The object.		yes
query	string	The Datadog Search Syntax query.		yes

Examples

OR query

Source

match_datadog_query({"message": "contains this and that"}, "this OR that")

Return

true

AND query

Source

match_datadog_query({"message": "contains only this"}, "this AND that")

Return

false

Attribute wildcard

Source

match_datadog_query({"name": "foobar"}, "@name:foo*")

Return

true

Tag range

Source

match_datadog_query({"tags": ["a:x", "b:y", "c:z"]}, s'b:["x" TO "z"]')

Return

true

merge

infallible pure

Merges the from object into the to object.

Function spec

merge(to: <object> ,from: <object> , [deep: <boolean> ] )

:: <object>

required optional <types | ...>

Argument	Type	Description	Required?
to	object	The object to merge into.	yes
from	object	The object to merge from.	yes
deep	boolean	A deep merge is performed if `true`, otherwise only top-level fields are merged.	no

Examples

Object merge (shallow)

Source

merge(
	{
		"parent1": {
			"child1": 1,
			"child2": 2
		},
		"parent2": {
			"child3": 3
		}
	},
	{
		"parent1": {
			"child2": 4,
			"child5": 5
		}
	}
)

Return

{
  "parent1": {
    "child2": 4,
    "child5": 5
  },
  "parent2": {
    "child3": 3
  }
}

Object merge (deep)

Source

merge(
	{
		"parent1": {
			"child1": 1,
			"child2": 2
		},
		"parent2": {
			"child3": 3
		}
	},
	{
		"parent1": {
			"child2": 4,
			"child5": 5
		}
	},
	deep: true
)

Return

{
  "parent1": {
    "child1": 1,
    "child2": 4,
    "child5": 5
  },
  "parent2": {
    "child3": 3
  }
}

object_from_array

fallible pure

Iterate over either one array of arrays or a pair of arrays and create an object out of all the key-value pairs contained in them. With one array of arrays, any entries with no value use null instead. Any keys that are null skip the corresponding value.

If a single parameter is given, it must contain an array of all the input arrays.

Function spec

object_from_array(values: <array> , [keys: <array> ] )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
values	array	The first array of elements, or the array of input arrays if no other parameter is present.		yes
keys	array	The second array of elements. If not present, the first parameter must contain all the arrays.		no

Errors

The object_from_array function is fallible, which means that error handling is required for these errors:

values and keys must be arrays.

If keys is not present, values must contain only arrays.

Examples

Create an object from one array

Source

object_from_array([["one", 1], [null, 2], ["two", 3]])

Return

{
  "one": 1,
  "two": 3
}

Create an object from separate key and value arrays

Source

object_from_array([1, 2, 3], keys: ["one", null, "two"])

Return

{
  "one": 1,
  "two": 3
}

unnest

fallible pure

Unnest an array field from an object to create an array of objects using that field; keeping all other fields.

Assigning the array result of this to . results in multiple events being emitted from remap. See the remap transform docs for more details.

This is also referred to as explode in some languages.

Function spec

unnest(path: <path> )

:: <array> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
path	path	The path of the field to unnest.		yes

Errors

The unnest function is fallible, which means that error handling is required for these errors:

The field path referred to is not an array.

Examples

Unnest an array field

Source

. = unnest!(.messages)

Unnest nested an array field

Source

. = unnest!(.event.messages)

Parse functions

parse_apache_log

fallible pure

Parses Apache access and error log lines. Lines can be in common, combined, or the default error format.

Function spec

parse_apache_log(value: <string> ,format: <string> , [timestamp_format: <string> ] )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to parse.		yes
format	string	The format to use for parsing the log.		yes
timestamp_format	string	The date/time format to use for encoding the timestamp. The time is parsed in local time if the timestamp does not specify a timezone.	`%d/%b/%Y:%T %z`	no

Notices

This function has special behavior that you should be aware of.

Missing information in the log message may be indicated by -. These fields are omitted in the result.

Errors

The parse_apache_log function is fallible, which means that error handling is required for these errors:

value does not match the specified format.

timestamp_format is not a valid format string.

The timestamp in value fails to parse using the provided timestamp_format.

Examples

Parse using Apache log format (common)

Source

parse_apache_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326", format: "common")

Return

{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}

Parse using Apache log format (combined)

Source

parse_apache_log!(
	s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth" "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"',
	"combined",
)

Return

{
  "agent": "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0",
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "referrer": "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}

Parse using Apache log format (error)

Source

parse_apache_log!(
	s'[01/Mar/2021:12:00:19 +0000] [ab:alert] [pid 4803:tid 3814] [client 147.159.108.175:24259] I will bypass the haptic COM bandwidth, that should matrix the CSS driver!',
	"error"
)

Return

{
  "client": "147.159.108.175",
  "message": "I will bypass the haptic COM bandwidth, that should matrix the CSS driver!",
  "module": "ab",
  "pid": 4803,
  "port": 24259,
  "severity": "alert",
  "thread": "3814",
  "timestamp": "2021-03-01T12:00:19Z"
}

parse_aws_alb_log

fallible pure

Parses value in the Elastic Load Balancer Access format.

Function spec

parse_aws_alb_log(value: <string> , [strict_mode: <boolean> ] )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	Access log of the Application Load Balancer.		yes
strict_mode	boolean	When set to `false`, the parser ignores any newly added or trailing fields in AWS ALB logs instead of failing. Defaults to `true` to preserve strict parsing behavior.	`true`	no

Errors

The parse_aws_alb_log function is fallible, which means that error handling is required for these errors:

value is not a properly formatted AWS ALB log.

Examples

Parse AWS ALB log

Source

parse_aws_alb_log!(
	"http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\""
)

Return

{
  "actions_executed": "forward",
  "chosen_cert_arn": null,
  "classification": null,
  "classification_reason": null,
  "client_host": "192.168.131.39:2817",
  "domain_name": null,
  "elb": "app/my-loadbalancer/50dc6c495c0c9188",
  "elb_status_code": "200",
  "error_reason": null,
  "matched_rule_priority": "0",
  "received_bytes": 34,
  "redirect_url": null,
  "request_creation_time": "2018-11-30T22:22:48.364000Z",
  "request_method": "GET",
  "request_processing_time": 0,
  "request_protocol": "HTTP/1.1",
  "request_url": "http://www.example.com:80/",
  "response_processing_time": 0,
  "sent_bytes": 366,
  "ssl_cipher": null,
  "ssl_protocol": null,
  "target_group_arn": "arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067",
  "target_host": null,
  "target_port_list": [],
  "target_processing_time": 0.001,
  "target_status_code": "200",
  "target_status_code_list": [],
  "timestamp": "2018-11-30T22:23:00.186641Z",
  "trace_id": "Root=1-58337364-23a8c76965a2ef7629b185e3",
  "traceability_id": null,
  "type": "http",
  "user_agent": "curl/7.46.0"
}

Parse AWS ALB log with trailing fields (non-strict mode)

Source

parse_aws_alb_log!(
	"http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\" TID_12345 \"-\" \"-\" \"-\"",
	strict_mode: false
)

Return

{
  "actions_executed": "forward",
  "chosen_cert_arn": null,
  "classification": null,
  "classification_reason": null,
  "client_host": "192.168.131.39:2817",
  "domain_name": null,
  "elb": "app/my-loadbalancer/50dc6c495c0c9188",
  "elb_status_code": "200",
  "error_reason": null,
  "matched_rule_priority": "0",
  "received_bytes": 34,
  "redirect_url": null,
  "request_creation_time": "2018-11-30T22:22:48.364000Z",
  "request_method": "GET",
  "request_processing_time": 0,
  "request_protocol": "HTTP/1.1",
  "request_url": "http://www.example.com:80/",
  "response_processing_time": 0,
  "sent_bytes": 366,
  "ssl_cipher": null,
  "ssl_protocol": null,
  "target_group_arn": "arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067",
  "target_host": null,
  "target_port_list": [],
  "target_processing_time": 0.001,
  "target_status_code": "200",
  "target_status_code_list": [],
  "timestamp": "2018-11-30T22:23:00.186641Z",
  "trace_id": "Root=1-58337364-23a8c76965a2ef7629b185e3",
  "traceability_id": null,
  "type": "http",
  "user_agent": "curl/7.46.0"
}

parse_aws_cloudwatch_log_subscription_message

fallible pure

Parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) from the aws_kinesis_firehose source.

Function spec

parse_aws_cloudwatch_log_subscription_message(value: <string> )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string representation of the message to parse.		yes

Errors

The parse_aws_cloudwatch_log_subscription_message function is fallible, which means that error handling is required for these errors:

value is not a properly formatted AWS CloudWatch Log subscription message.

Examples

Parse AWS Cloudwatch Log subscription message

Source

parse_aws_cloudwatch_log_subscription_message!(.message)

Return

{
  "log_events": [
    {
      "id": "35683658089614582423604394983260738922885519999578275840",
      "message": "{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}",
      "timestamp": "2020-09-14T19:09:29.039Z"
    }
  ],
  "log_group": "test",
  "log_stream": "test",
  "message_type": "DATA_MESSAGE",
  "owner": "111111111111",
  "subscription_filters": [
    "Destination"
  ]
}

parse_aws_vpc_flow_log

fallible pure

Parses value in the VPC Flow Logs format.

Function spec

parse_aws_vpc_flow_log(value: <string> , [format: <string> ] )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	VPC Flow Log.		yes
format	string	VPC Flow Log format.		no

Errors

The parse_aws_vpc_flow_log function is fallible, which means that error handling is required for these errors:

value is not a properly formatted AWS VPC Flow log.

Examples

Parse AWS VPC Flow log (default format)

Source

parse_aws_vpc_flow_log!("2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA")

Return

{
  "account_id": "123456789010",
  "action": null,
  "bytes": null,
  "dstaddr": null,
  "dstport": null,
  "end": 1431280934,
  "interface_id": "eni-1235b8ca123456789",
  "log_status": "NODATA",
  "packets": null,
  "protocol": null,
  "srcaddr": null,
  "srcport": null,
  "start": 1431280876,
  "version": 2
}

Parse AWS VPC Flow log (custom format)

Source

parse_aws_vpc_flow_log!(
	"- eni-1235b8ca123456789 10.0.1.5 10.0.0.220 10.0.1.5 203.0.113.5",
	"instance_id interface_id srcaddr dstaddr pkt_srcaddr pkt_dstaddr"
)

Return

{
  "dstaddr": "10.0.0.220",
  "instance_id": null,
  "interface_id": "eni-1235b8ca123456789",
  "pkt_dstaddr": "203.0.113.5",
  "pkt_srcaddr": "10.0.1.5",
  "srcaddr": "10.0.1.5"
}

Parse AWS VPC Flow log including v5 fields

Source

parse_aws_vpc_flow_log!("5 52.95.128.179 10.0.0.71 80 34210 6 1616729292 1616729349 IPv4 14 15044 123456789012 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-0c50d5961bcb2d47b eni-1235b8ca123456789 ap-southeast-2 apse2-az3 - - ACCEPT 19 52.95.128.179 10.0.0.71 S3 - - ingress OK",
format: "version srcaddr dstaddr srcport dstport protocol start end type packets bytes account_id vpc_id subnet_id instance_id interface_id region az_id sublocation_type sublocation_id action tcp_flags pkt_srcaddr pkt_dstaddr pkt_src_aws_service pkt_dst_aws_service traffic_path flow_direction log_status")

Return

{
  "account_id": "123456789012",
  "action": "ACCEPT",
  "az_id": "apse2-az3",
  "bytes": 15044,
  "dstaddr": "10.0.0.71",
  "dstport": 34210,
  "end": 1616729349,
  "flow_direction": "ingress",
  "instance_id": "i-0c50d5961bcb2d47b",
  "interface_id": "eni-1235b8ca123456789",
  "log_status": "OK",
  "packets": 14,
  "pkt_dst_aws_service": null,
  "pkt_dstaddr": "10.0.0.71",
  "pkt_src_aws_service": "S3",
  "pkt_srcaddr": "52.95.128.179",
  "protocol": 6,
  "region": "ap-southeast-2",
  "srcaddr": "52.95.128.179",
  "srcport": 80,
  "start": 1616729292,
  "sublocation_id": null,
  "sublocation_type": null,
  "subnet_id": "subnet-aaaaaaaa012345678",
  "tcp_flags": 19,
  "traffic_path": null,
  "type": "IPv4",
  "version": 5,
  "vpc_id": "vpc-abcdefab012345678"
}

parse_bytes

fallible pure

Parses the value into a human-readable bytes format specified by unit and base.

Function spec

parse_bytes(value: <string> ,unit: <string> , [base: <string> ] )

:: <float> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string of the duration with either binary or SI unit.		yes
unit	string	The output units for the byte.		yes
base	string	The base for the byte, either 2 or 10.	`2`	no

Errors

The parse_bytes function is fallible, which means that error handling is required for these errors:

value is not a properly formatted bytes.

Examples

Parse bytes (kilobytes)

Source

parse_bytes!("1024KiB", unit: "MiB")

Return

Parse bytes in SI unit (terabytes)

Source

parse_bytes!("4TB", unit: "MB", base: "10")

Return

Parse bytes in ambiguous unit (gigabytes)

Source

parse_bytes!("1GB", unit: "B", base: "2")

Return

1073741824

parse_cbor

fallible pure

Parses the value as CBOR.

Function spec

parse_cbor(value: <string> )

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The CBOR payload to parse.		yes

Notices

This function has special behavior that you should be aware of.

Only CBOR types are returned.

Errors

The parse_cbor function is fallible, which means that error handling is required for these errors:

value is not a valid CBOR-formatted payload.

Examples

Parse CBOR

Source

parse_cbor!(decode_base64!("oWVmaWVsZGV2YWx1ZQ=="))

Return

{
  "field": "value"
}

parse_cef

fallible pure

Parses the value in CEF (Common Event Format) format. Ignores everything up to CEF header. Empty values are returned as empty strings. Surrounding quotes are removed from values.

Function spec

parse_cef(value: <string> , [translate_custom_fields: <boolean> ] )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to parse.		yes
translate_custom_fields	boolean	Toggles translation of custom field pairs to `key:value`.		no

Notices

This function has special behavior that you should be aware of.

All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_cef function is fallible, which means that error handling is required for these errors:

value is not a properly formatted CEF string.

Examples

Parse output generated by PTA

Source

parse_cef!(
	"CEF:0|CyberArk|PTA|12.6|1|Suspected credentials theft|8|suser=mike2@prod1.domain.com shost=prod1.domain.com src=1.1.1.1 duser=andy@dev1.domain.com dhost=dev1.domain.com dst=2.2.2.2 cs1Label=ExtraData cs1=None cs2Label=EventID cs2=52b06812ec3500ed864c461e deviceCustomDate1Label=detectionDate deviceCustomDate1=1388577900000 cs3Label=PTAlink cs3=https://1.1.1.1/incidents/52b06812ec3500ed864c461e cs4Label=ExternalLink cs4=None"
)

Return

{
  "cefVersion": "0",
  "cs1": "None",
  "cs1Label": "ExtraData",
  "cs2": "52b06812ec3500ed864c461e",
  "cs2Label": "EventID",
  "cs3": "https://1.1.1.1/incidents/52b06812ec3500ed864c461e",
  "cs3Label": "PTAlink",
  "cs4": "None",
  "cs4Label": "ExternalLink",
  "deviceCustomDate1": "1388577900000",
  "deviceCustomDate1Label": "detectionDate",
  "deviceEventClassId": "1",
  "deviceProduct": "PTA",
  "deviceVendor": "CyberArk",
  "deviceVersion": "12.6",
  "dhost": "dev1.domain.com",
  "dst": "2.2.2.2",
  "duser": "andy@dev1.domain.com",
  "name": "Suspected credentials theft",
  "severity": "8",
  "shost": "prod1.domain.com",
  "src": "1.1.1.1",
  "suser": "mike2@prod1.domain.com"
}

Ignore syslog header

Source

parse_cef!(
	"Sep 29 08:26:10 host CEF:1|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232"
)

Return

{
  "cefVersion": "1",
  "deviceEventClassId": "100",
  "deviceProduct": "threatmanager",
  "deviceVendor": "Security",
  "deviceVersion": "1.0",
  "dst": "2.1.2.2",
  "name": "worm successfully stopped",
  "severity": "10",
  "spt": "1232",
  "src": "10.0.0.1"
}

Translate custom fields

Source

parse_cef!(
	"CEF:0|Dev|firewall|2.2|1|Connection denied|5|c6a1=2345:0425:2CA1:0000:0000:0567:5673:23b5 c6a1Label=Device IPv6 Address",
	translate_custom_fields: true
)

Return

{
  "Device IPv6 Address": "2345:0425:2CA1:0000:0000:0567:5673:23b5",
  "cefVersion": "0",
  "deviceEventClassId": "1",
  "deviceProduct": "firewall",
  "deviceVendor": "Dev",
  "deviceVersion": "2.2",
  "name": "Connection denied",
  "severity": "5"
}

parse_common_log

fallible pure

Parses the value using the Common Log Format (CLF).

Function spec

parse_common_log(value: <string> , [timestamp_format: <string> ] )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to parse.		yes
timestamp_format	string	The date/time format to use for encoding the timestamp.	`%d/%b/%Y:%T %z`	no

Notices

This function has special behavior that you should be aware of.

Missing information in the log message may be indicated by -. These fields are omitted in the result.

Errors

The parse_common_log function is fallible, which means that error handling is required for these errors:

value does not match the Common Log Format.

timestamp_format is not a valid format string.

The timestamp in value fails to parse using the provided timestamp_format.

Examples

Parse using Common Log Format (with default timestamp format)

Source

parse_common_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326")

Return

{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}

Parse using Common Log Format (with custom timestamp format)

Source

parse_common_log!(
	"127.0.0.1 bob frank [2000-10-10T20:55:36Z] \"GET /apache_pb.gif HTTP/1.0\" 200 2326",
	"%+"
)

Return

{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}

parse_csv

fallible pure

Parses a single CSV formatted row. Only the first row is parsed in case of multiline input value.

Function spec

parse_csv(value: <string> , [delimiter: <string> ] )

:: <array> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to parse.		yes
delimiter	string	The field delimiter to use when parsing. Must be a single-byte utf8 character.	`,`	no

Notices

This function has special behavior that you should be aware of.

All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_csv function is fallible, which means that error handling is required for these errors:

The delimiter must be a single-byte UTF-8 character.

value is not a valid CSV string.

Examples

Parse a single CSV formatted row

Source

parse_csv!("foo,bar,\"foo \"\", bar\"")

Return

["foo","bar","foo \", bar"]

Parse a single CSV formatted row with custom delimiter

Source

parse_csv!("foo bar", delimiter: " ")

Return

["foo","bar"]

parse_dnstap

fallible pure

Parses the value as base64 encoded DNSTAP data.

Function spec

parse_dnstap(value: <string> , [lowercase_hostnames: <boolean> ] )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The base64 encoded representation of the DNSTAP data to parse.		yes
lowercase_hostnames	boolean	Whether to turn all hostnames found in resulting data lowercase, for consistency.		no

Errors

The parse_dnstap function is fallible, which means that error handling is required for these errors:

value is not a valid base64 encoded string.

dnstap parsing failed for value

Examples

Parse dnstap query message

Source

parse_dnstap!("ChVqYW1lcy1WaXJ0dWFsLU1hY2hpbmUSC0JJTkQgOS4xNi4zGgBy5wEIAxACGAEiEAAAAAAAAAAAAAAAAAAAAAAqECABBQJwlAAAAAAAAAAAADAw8+0CODVA7+zq9wVNMU3WNlI2kwIAAAABAAAAAAABCWZhY2Vib29rMQNjb20AAAEAAQAAKQIAAACAAAAMAAoACOxjCAG9zVgzWgUDY29tAGAAbQAAAAByZLM4AAAAAQAAAAAAAQJoNQdleGFtcGxlA2NvbQAABgABAAApBNABAUAAADkADwA1AAlubyBTRVAgbWF0Y2hpbmcgdGhlIERTIGZvdW5kIGZvciBkbnNzZWMtZmFpbGVkLm9yZy54AQ==")

Return

{
  "dataType": "Message",
  "dataTypeId": 1,
  "extraInfo": "",
  "messageType": "ResolverQuery",
  "messageTypeId": 3,
  "queryZone": "com.",
  "requestData": {
    "fullRcode": 0,
    "header": {
      "aa": false,
      "ad": false,
      "anCount": 0,
      "arCount": 1,
      "cd": false,
      "id": 37634,
      "nsCount": 0,
      "opcode": 0,
      "qdCount": 1,
      "qr": 0,
      "ra": false,
      "rcode": 0,
      "rd": false,
      "tc": false
    },
    "opt": {
      "do": true,
      "ednsVersion": 0,
      "extendedRcode": 0,
      "options": [
        {
          "optCode": 10,
          "optName": "Cookie",
          "optValue": "7GMIAb3NWDM="
        }
      ],
      "udpPayloadSize": 512
    },
    "question": [
      {
        "class": "IN",
        "domainName": "facebook1.com.",
        "questionType": "A",
        "questionTypeId": 1
      }
    ],
    "rcodeName": "NoError"
  },
  "responseAddress": "2001:502:7094::30",
  "responseData": {
    "fullRcode": 16,
    "header": {
      "aa": false,
      "ad": false,
      "anCount": 0,
      "arCount": 1,
      "cd": false,
      "id": 45880,
      "nsCount": 0,
      "opcode": 0,
      "qdCount": 1,
      "qr": 0,
      "ra": false,
      "rcode": 16,
      "rd": false,
      "tc": false
    },
    "opt": {
      "do": false,
      "ede": [
        {
          "extraText": "no SEP matching the DS found for dnssec-failed.org.",
          "infoCode": 9,
          "purpose": "DNSKEY Missing"
        }
      ],
      "ednsVersion": 1,
      "extendedRcode": 1,
      "udpPayloadSize": 1232
    },
    "question": [
      {
        "class": "IN",
        "domainName": "h5.example.com.",
        "questionType": "SOA",
        "questionTypeId": 6
      }
    ],
    "rcodeName": "BADVERS"
  },
  "responsePort": 53,
  "serverId": "james-Virtual-Machine",
  "serverVersion": "BIND 9.16.3",
  "socketFamily": "INET6",
  "socketProtocol": "UDP",
  "sourceAddress": "::",
  "sourcePort": 46835,
  "time": 1593489007920014000,
  "timePrecision": "ns",
  "timestamp": "2020-06-30T03:50:07.920014129Z"
}

parse_duration

fallible pure

Parses the value into a human-readable duration format specified by unit.

Function spec

parse_duration(value: <string> ,unit: <string> )

:: <float> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string of the duration.		yes
unit	string	The output units for the duration.		yes

Errors

The parse_duration function is fallible, which means that error handling is required for these errors:

value is not a properly formatted duration.

Examples

Parse duration (milliseconds)

Source

parse_duration!("1005ms", unit: "s")

Return

1.005

Parse multiple durations (seconds & milliseconds)

Source

parse_duration!("1s 1ms", unit: "ms")

Return

parse_etld

fallible pure

Parses the eTLD from value representing domain name.

Function spec

parse_etld(value: <string> , [plus_parts: <integer> ,psl: <string> ] )

:: <object> , <error>

required optional <types | ...>

Argument

Type

Description

Default

Required?

value

string

The domain string.

yes

plus_parts

integer

Can be provided to get additional parts of the domain name. When 1 is passed, eTLD+1 will be returned, which represents a domain registrable by a single organization. Higher numbers will return subdomains.

psl

string

Can be provided to use a different public suffix list.

By default, https://publicsuffix.org/list/public_suffix_list.dat is used.

Errors

The parse_etld function is fallible, which means that error handling is required for these errors:

unable to determine eTLD for value

Examples

Parse eTLD

Source

parse_etld!("sub.sussex.ac.uk")

Return

{
  "etld": "ac.uk",
  "etld_plus": "ac.uk",
  "known_suffix": true
}

Parse eTLD+1

Source

parse_etld!("sub.sussex.ac.uk", plus_parts: 1)

Return

{
  "etld": "ac.uk",
  "etld_plus": "sussex.ac.uk",
  "known_suffix": true
}

Parse eTLD with unknown suffix

Source

parse_etld!("vector.acmecorp")

Return

{
  "etld": "acmecorp",
  "etld_plus": "acmecorp",
  "known_suffix": false
}

Parse eTLD with custom PSL

Source

parse_etld!("vector.acmecorp", psl: "resources/public_suffix_list.dat")

Return

{
  "etld": "acmecorp",
  "etld_plus": "acmecorp",
  "known_suffix": false
}

parse_glog

fallible pure

Parses the value using the glog (Google Logging Library) format.

Function spec

parse_glog(value: <string> )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to parse.		yes

Errors

The parse_glog function is fallible, which means that error handling is required for these errors:

value does not match the glog format.

Examples

Parse using glog

Source

parse_glog!("I20210131 14:48:54.411655 15520 main.c++:9] Hello world!")

Return

{
  "file": "main.c++",
  "id": 15520,
  "level": "info",
  "line": 9,
  "message": "Hello world!",
  "timestamp": "2021-01-31T14:48:54.411655Z"
}

parse_grok

fallible pure

Parses the value using the grok format. All patterns listed here are supported.

Function spec

parse_grok(value: <string> ,pattern: <string> )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to parse.		yes
pattern	string	The Grok pattern.		yes

Notices

This function has special behavior that you should be aware of.

We recommend using community-maintained Grok patterns when possible, as they’re more likely to be properly vetted and improved over time than bespoke patterns.

Errors

The parse_grok function is fallible, which means that error handling is required for these errors:

value fails to parse using the provided pattern.

Examples

Parse using Grok

Source

parse_grok!(
	"2020-10-02T23:22:12.223222Z info Hello world",
	"%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
)

Return

{
  "level": "info",
  "message": "Hello world",
  "timestamp": "2020-10-02T23:22:12.223222Z"
}

parse_groks

fallible pure

Parses the value using multiple grok patterns. All patterns listed here are supported.

Function spec

parse_groks(value: <string> ,patterns: <array> , [aliases: <object> ,alias_sources: <string> ] )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to parse.		yes
patterns	array	The Grok patterns, which are tried in order until the first match.		yes
aliases	object	The shared set of grok aliases that can be referenced in the patterns to simplify them.	`true`	no
alias_sources	string	Path to the file containing aliases in a JSON format.		no

Notices

This function has special behavior that you should be aware of.

We recommend using community-maintained Grok patterns when possible, as they’re more likely to be properly vetted and improved over time than bespoke patterns.

Errors

The parse_groks function is fallible, which means that error handling is required for these errors:

value fails to parse using the provided pattern.

patterns is not an array.

aliases is not an object.

alias_sources is not a string or doesn’t point to a valid file.

Examples

Parse using multiple Grok patterns

Source

parse_groks!(
	"2020-10-02T23:22:12.223222Z info Hello world",
	patterns: [
		"%{common_prefix} %{_status} %{_message}",
		"%{common_prefix} %{_message}",
	],
	aliases: {
		"common_prefix": "%{_timestamp} %{_loglevel}",
		"_timestamp": "%{TIMESTAMP_ISO8601:timestamp}",
		"_loglevel": "%{LOGLEVEL:level}",
		"_status": "%{POSINT:status}",
		"_message": "%{GREEDYDATA:message}"
	}
)

Return

{
  "level": "info",
  "message": "Hello world",
  "timestamp": "2020-10-02T23:22:12.223222Z"
}

Parse using aliases from file

Source

parse_groks!(
  "username=foo",
  patterns: [ "%{PATTERN_A}" ],
  alias_sources: [ "path/to/aliases.json" ]
)
# aliases.json contents:
# {
#   "PATTERN_A": "%{PATTERN_B}",
#   "PATTERN_B": "username=%{USERNAME:username}"
# }

parse_influxdb

fallible pure

Parses the value as an InfluxDB line protocol string, producing a list of Vector-compatible metrics.

Function spec

parse_influxdb(value: <string> )

:: <array> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string representation of the InfluxDB line protocol to parse.		yes

Notices

This function has special behavior that you should be aware of.

This function will return a log event with the shape of a Vector-compatible metric, but not a metric event itself. You will likely want to pipe the output of this function through a log_to_metric transform with the option all_metrics set to true to convert the metric-shaped log events to metric events so real metrics are produced.

The only metric type that is produced is a gauge. Each metric name is prefixed with the measurement field, followed by an underscore (_), and then the field key field.

string is the only type that is not supported as a field value, due to limitations of Vector’s metric model.

Errors

The parse_influxdb function is fallible, which means that error handling is required for these errors:

value is not a valid InfluxDB line protocol string.

field set contains a field value of type string.

field set contains a NaN field value.

Examples

Parse InfluxDB line protocol

Source

parse_influxdb!("cpu,host=A,region=us-west usage_system=64i,usage_user=10u,temperature=50.5,on=true,sleep=false 1590488773254420000")

Return

[{"gauge":{"value":64},"kind":"absolute","name":"cpu_usage_system","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":10},"kind":"absolute","name":"cpu_usage_user","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":50.5},"kind":"absolute","name":"cpu_temperature","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":1},"kind":"absolute","name":"cpu_on","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":0},"kind":"absolute","name":"cpu_sleep","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"}]

parse_int

fallible pure

Parses the string value representing a number in an optional base/radix to an integer.

Function spec

parse_int(value: <string> , [base: <integer> ] )

:: <string> , <error>

required optional <types | ...>

Argument

Type

Description

Default

Required?

value

string

The string to parse.

yes

base

integer

The base the number is in. Must be between 2 and 36 (inclusive).

If unspecified, the string prefix is used to determine the base: “0b”, 8 for “0” or “0o”, 16 for “0x”, and 10 otherwise.

Errors

The parse_int function is fallible, which means that error handling is required for these errors:

The base is not between 2 and 36.

The number cannot be parsed in the base.

Examples

Parse decimal

Source

parse_int!("-42")

Return

-42

Parse binary

Source

parse_int!("0b1001")

Return

Parse octal

Source

parse_int!("0o42")

Return

Parse hexadecimal

Source

parse_int!("0x2a")

Return

Parse explicit base

Source

parse_int!("2a", 17)

Return

parse_json

fallible pure

Parses the value as JSON.

Function spec

parse_json(value: <string> , [max_depth: <integer> ,lossy: <boolean> ] )

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string representation of the JSON to parse.		yes
max_depth	integer	Number of layers to parse for nested JSON-formatted documents. The value must be in the range of 1 to 128.		no
lossy	boolean	Whether to parse the JSON in a lossy manner. Replaces invalid UTF-8 characters with the Unicode character `�` (U+FFFD) if set to true, otherwise returns an error if there are any invalid UTF-8 characters present.	`true`	no

Notices

This function has special behavior that you should be aware of.

Only JSON types are returned. If you need to convert a string into a timestamp, consider the parse_timestamp function.

Errors

The parse_json function is fallible, which means that error handling is required for these errors:

value is not a valid JSON-formatted payload.

Examples

Parse JSON

Source

parse_json!("{\"key\": \"val\"}")

Return

{
  "key": "val"
}

Parse JSON with max_depth

Source

parse_json!("{\"top_level\":{\"key\": \"val\"}}", max_depth: 1)

Return

{
  "top_level": "{\"key\": \"val\"}"
}

parse_key_value

fallible pure

Parses the value in key-value format. Also known as logfmt.

Keys and values can be wrapped with ".
" characters can be escaped using \.

Function spec

parse_key_value(value: <string> , [key_value_delimiter: <string> ,field_delimiter: <string> ,whitespace: <string> ,accept_standalone_key: <boolean> ] )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to parse.		yes
key_value_delimiter	string	The string that separates the key from the value.	`=`	no
field_delimiter	string	The string that separates each key-value pair.		no
whitespace	string	Defines the acceptance of unnecessary whitespace surrounding the configured `key_value_delimiter`.	`lenient`	no
accept_standalone_key	boolean	Whether a standalone key should be accepted, the resulting object associates such keys with the boolean value `true`.	`true`	no

Notices

This function has special behavior that you should be aware of.

All values are returned as strings or as an array of strings for duplicate keys. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_key_value function is fallible, which means that error handling is required for these errors:

value is not a properly formatted key-value string.

Examples

Parse logfmt log

Source

parse_key_value!(
	"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)

Return

{
  "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
  "id": "ConsumerFetcherManager-1382721708341",
  "level": "info",
  "module": "kafka.consumer.ConsumerFetcherManager",
  "msg": "Stopping all fetchers",
  "tag#production": "stopping_fetchers"
}

Parse comma delimited log

Source

parse_key_value!(
	"path:\"/cart_link\", host:store.app.com, fwd: \"102.30.171.16\", dyno: web.1, connect:0ms, service:87ms, status:304, bytes:632, protocol:https",
	field_delimiter: ",",
	key_value_delimiter: ":"
)

Return

{
  "bytes": "632",
  "connect": "0ms",
  "dyno": "web.1",
  "fwd": "102.30.171.16",
  "host": "store.app.com",
  "path": "/cart_link",
  "protocol": "https",
  "service": "87ms",
  "status": "304"
}

Parse comma delimited log with standalone keys

Source

parse_key_value!(
	"env:prod,service:backend,region:eu-east1,beta",
	field_delimiter: ",",
	key_value_delimiter: ":",
)

Return

{
  "beta": true,
  "env": "prod",
  "region": "eu-east1",
  "service": "backend"
}

Parse duplicate keys

Source

parse_key_value!(
	"at=info,method=GET,path=\"/index\",status=200,tags=dev,tags=dummy",
	field_delimiter: ",",
	key_value_delimiter: "=",
)

Return

{
  "at": "info",
  "method": "GET",
  "path": "/index",
  "status": "200",
  "tags": [
    "dev",
    "dummy"
  ]
}

parse_klog

fallible pure

Parses the value using the klog format used by Kubernetes components.

Function spec

parse_klog(value: <string> )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to parse.		yes

Notices

This function has special behavior that you should be aware of.

This function resolves the year for messages. If the current month is January and the provided month is December, it sets the year to the previous year. Otherwise, it sets the year to the current year.

Errors

The parse_klog function is fallible, which means that error handling is required for these errors:

value does not match the klog format.

Examples

Parse using klog

Source

parse_klog!("I0505 17:59:40.692994   28133 klog.go:70] hello from klog")

Return

{
  "file": "klog.go",
  "id": 28133,
  "level": "info",
  "line": 70,
  "message": "hello from klog",
  "timestamp": "2025-05-05T17:59:40.692994Z"
}

parse_linux_authorization

fallible pure

Parses Linux authorization logs usually found under either /var/log/auth.log (for Debian-based systems) or /var/log/secure (for RedHat-based systems) according to Syslog format.

Function spec

parse_linux_authorization(value: <string> )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The text containing the message to parse.		yes

Notices

This function has special behavior that you should be aware of.

The function resolves the year for messages that don’t include it. If the current month is January, and the message is for December, it will take the previous year. Otherwise, take the current year.

Errors

The parse_linux_authorization function is fallible, which means that error handling is required for these errors:

value is not a properly formatted Syslog message.

Examples

Parse Linux authorization event

Source

parse_linux_authorization!(
	s'Mar 23 01:49:58 localhost sshd[1111]: Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar'
)

Return

{
  "appname": "sshd",
  "hostname": "localhost",
  "message": "Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar",
  "procid": 1111,
  "timestamp": "2025-03-23T01:49:58Z"
}

parse_logfmt

fallible pure

Parses the value in logfmt.

Keys and values can be wrapped using the " character.
" characters can be escaped by the \ character.
As per this logfmt specification, the parse_logfmt function accepts standalone keys and assigns them a Boolean value of true.

Function spec

parse_logfmt(value: <string> )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to parse.		yes

Errors

The parse_logfmt function is fallible, which means that error handling is required for these errors:

value is not a properly formatted key-value string

Examples

Parse logfmt log

Source

parse_logfmt!(
	"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)

Return

{
  "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
  "id": "ConsumerFetcherManager-1382721708341",
  "level": "info",
  "module": "kafka.consumer.ConsumerFetcherManager",
  "msg": "Stopping all fetchers",
  "tag#production": "stopping_fetchers"
}

parse_nginx_log

fallible pure

Parses Nginx access and error log lines. Lines can be in combined, ingress_upstreaminfo, main or error format.

Function spec

parse_nginx_log(value: <string> ,format: <string> , [timestamp_format: <string> ] )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to parse.		yes
format	string	The format to use for parsing the log.		yes
timestamp_format	string	The date/time format to use for encoding the timestamp. The time is parsed in local time if the timestamp doesn’t specify a timezone. The default format is `%d/%b/%Y:%T %z` for combined logs and `%Y/%m/%d %H:%M:%S` for error logs.	`%d/%b/%Y:%T %z`	no

Notices

This function has special behavior that you should be aware of.

Missing information in the log message may be indicated by -. These fields are omitted in the result.

In case of ingress_upstreaminfo format the following fields may be safely omitted in the log message: remote_addr, remote_user, http_referer, http_user_agent, proxy_alternative_upstream_name, upstream_addr, upstream_response_length, upstream_response_time, upstream_status.

Errors

The parse_nginx_log function is fallible, which means that error handling is required for these errors:

value does not match the specified format.

timestamp_format is not a valid format string.

The timestamp in value fails to parse using the provided timestamp_format.

Examples

Parse via Nginx log format (combined)

Source

parse_nginx_log!(
    s'172.17.0.1 - alice [01/Apr/2021:12:02:31 +0000] "POST /not-found HTTP/1.1" 404 153 "http://localhost/somewhere" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"',
    "combined",
)

Return

{
  "agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36",
  "client": "172.17.0.1",
  "compression": "2.75",
  "referer": "http://localhost/somewhere",
  "request": "POST /not-found HTTP/1.1",
  "size": 153,
  "status": 404,
  "timestamp": "2021-04-01T12:02:31Z",
  "user": "alice"
}

Parse via Nginx log format (error)

Source

parse_nginx_log!(
    s'2021/04/01 13:02:31 [error] 31#31: *1 open() "/usr/share/nginx/html/not-found" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "POST /not-found HTTP/1.1", host: "localhost:8081"',
    "error"
)

Return

{
  "cid": 1,
  "client": "172.17.0.1",
  "host": "localhost:8081",
  "message": "open() \"/usr/share/nginx/html/not-found\" failed (2: No such file or directory)",
  "pid": 31,
  "request": "POST /not-found HTTP/1.1",
  "server": "localhost",
  "severity": "error",
  "tid": 31,
  "timestamp": "2021-04-01T13:02:31Z"
}

Parse via Nginx log format (ingress_upstreaminfo)

Source

parse_nginx_log!(
    s'0.0.0.0 - bob [18/Mar/2023:15:00:00 +0000] "GET /some/path HTTP/2.0" 200 12312 "https://10.0.0.1/some/referer" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 462 0.050 [some-upstream-service-9000] [some-other-upstream-5000] 10.0.50.80:9000 19437 0.049 200 752178adb17130b291aefd8c386279e7',
    "ingress_upstreaminfo"
)

Return

{
  "body_bytes_size": 12312,
  "http_referer": "https://10.0.0.1/some/referer",
  "http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36",
  "proxy_alternative_upstream_name": "some-other-upstream-5000",
  "proxy_upstream_name": "some-upstream-service-9000",
  "remote_addr": "0.0.0.0",
  "remote_user": "bob",
  "req_id": "752178adb17130b291aefd8c386279e7",
  "request": "GET /some/path HTTP/2.0",
  "request_length": 462,
  "request_time": 0.05,
  "status": 200,
  "timestamp": "2023-03-18T15:00:00Z",
  "upstream_addr": "10.0.50.80:9000",
  "upstream_response_length": 19437,
  "upstream_response_time": 0.049,
  "upstream_status": 200
}

Parse via Nginx log format (main)

Source

parse_nginx_log!(
    s'172.24.0.3 - alice [31/Dec/2024:17:32:06 +0000] "GET / HTTP/1.1" 200 615 "https://domain.tld/path" "curl/8.11.1" "1.2.3.4, 10.10.1.1"',
    "main"
)

Return

{
  "body_bytes_size": 615,
  "http_referer": "https://domain.tld/path",
  "http_user_agent": "curl/8.11.1",
  "http_x_forwarded_for": "1.2.3.4, 10.10.1.1",
  "remote_addr": "172.24.0.3",
  "remote_user": "alice",
  "request": "GET / HTTP/1.1",
  "status": 200,
  "timestamp": "2024-12-31T17:32:06Z"
}

parse_proto

fallible pure

Parses the value as a protocol buffer payload.

Function spec

parse_proto(value: <string> ,desc_file: <string> ,message_type: <string> )

:: <object> , <error>

required optional <types | ...>

Argument

Type

Description

Default

Required?

value

string

The protocol buffer payload to parse.

yes

desc_file

string

The path to the protobuf descriptor set file. Must be a literal string.

This file is the output of protoc -o …

yes

message_type

string

The name of the message type to use for serializing.

Must be a literal string.

yes

Notices

This function has special behavior that you should be aware of.

Only proto messages are parsed and returned.

Errors

The parse_proto function is fallible, which means that error handling is required for these errors:

value is not a valid proto payload.

desc_file file does not exist.

message_type message type does not exist in the descriptor file.

Examples

Parse proto

Source

parse_proto!(decode_base64!("Cgdzb21lb25lIggKBjEyMzQ1Ng=="), "resources/protobuf_descriptor_set.desc", "test_protobuf.Person")

Return

{
  "name": "someone",
  "phones": [
    {
      "number": "123456"
    }
  ]
}

parse_query_string

infallible pure

Parses the value as a query string.

Function spec

parse_query_string(value: <string> )

:: <object>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to parse.		yes

Notices

This function has special behavior that you should be aware of.

All values are returned as strings. We recommend manually coercing values to desired types as you see fit. Empty keys and values are allowed.

Examples

Parse query string

Source

parse_query_string("foo=%2B1&bar=2&bar=3&xyz")

Return

{
  "bar": [
    "2",
    "3"
  ],
  "foo": "+1",
  "xyz": ""
}

Parse Ruby on Rails’ query string

Source

parse_query_string("?foo%5b%5d=1&foo%5b%5d=2")

Return

{
  "foo[]": [
    "1",
    "2"
  ]
}

parse_regex

fallible pure

Parses the value using the provided Regex pattern.

This function differs from the parse_regex_all function in that it returns only the first match.

Function spec

parse_regex(value: <string> ,pattern: <regex> , [numeric_groups: <regex> ] )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Required?
value	string	The string to search.	yes
pattern	regex	The regular expression pattern to search against.	yes
numeric_groups	regex	If true, the index of each group in the regular expression is also captured. Index `0` contains the whole match.	no

Notices

This function has special behavior that you should be aware of.

VRL aims to provide purpose-specific parsing functions for common log formats. Before reaching for the parse_regex function, see if a VRL parse_* function already exists for your format. If not, we recommend opening an issue to request support for the desired format.

All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_regex function is fallible, which means that error handling is required for these errors:

value fails to parse using the provided pattern.

Examples

Parse using Regex (with capture groups)

Source

parse_regex!("first group and second group.", r'(?P<number>.*?) group')

Return

{
  "number": "first"
}

Parse using Regex (without capture groups)

Source

parse_regex!("first group and second group.", r'(\w+) group', numeric_groups: true)

Return

{
  "0": "first group",
  "1": "first"
}

parse_regex_all

fallible pure

Parses the value using the provided Regex pattern.

This function differs from the parse_regex function in that it returns all matches, not just the first.

Function spec

parse_regex_all(value: <string> ,pattern: <regex> , [numeric_groups: <regex> ] )

:: <array> , <error>

required optional <types | ...>

Argument	Type	Description	Required?
value	string	The string to search.	yes
pattern	regex	The regular expression pattern to search against.	yes
numeric_groups	regex	If `true`, the index of each group in the regular expression is also captured. Index `0` contains the whole match.	no

Notices

This function has special behavior that you should be aware of.

All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_regex_all function is fallible, which means that error handling is required for these errors:

value is not a string.

pattern is not a regex.

Examples

Parse using Regex (all matches)

Source

parse_regex_all!("first group and second group.", r'(?P<number>\w+) group', numeric_groups: true)

Return

[{"0":"first group","1":"first","number":"first"},{"0":"second group","1":"second","number":"second"}]

parse_ruby_hash

fallible pure

Parses the value as ruby hash.

Function spec

parse_ruby_hash(value: <string> )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string representation of the ruby hash to parse.		yes

Notices

This function has special behavior that you should be aware of.

Only ruby types are returned. If you need to convert a string into a timestamp, consider the parse_timestamp function.

Errors

The parse_ruby_hash function is fallible, which means that error handling is required for these errors:

value is not a valid ruby hash formatted payload.

Examples

Parse ruby hash

Source

parse_ruby_hash!(s'{ "test" => "value", "testNum" => 0.2, "testObj" => { "testBool" => true, "testNull" => nil } }')

Return

{
  "test": "value",
  "testNum": 0.2,
  "testObj": {
    "testBool": true,
    "testNull": null
  }
}

parse_syslog

fallible pure

Parses the value in Syslog format.

Function spec

parse_syslog(value: <string> )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The text containing the Syslog message to parse.		yes

Notices

This function has special behavior that you should be aware of.

The function makes a best effort to parse the various Syslog formats that exists out in the wild. This includes RFC 6587, RFC 5424, RFC 3164, and other common variations (such as the Nginx Syslog style).

All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_syslog function is fallible, which means that error handling is required for these errors:

value is not a properly formatted Syslog message.

Examples

Parse Syslog log (5424)

Source

parse_syslog!(
	s'<13>1 2020-03-13T20:45:38.119Z dynamicwireless.name non 2426 ID931 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"] Try to override the THX port, maybe it will reboot the neural interface!'
)

Return

{
  "appname": "non",
  "exampleSDID@32473": {
    "eventID": "1011",
    "eventSource": "Application",
    "iut": "3"
  },
  "facility": "user",
  "hostname": "dynamicwireless.name",
  "message": "Try to override the THX port, maybe it will reboot the neural interface!",
  "msgid": "ID931",
  "procid": 2426,
  "severity": "notice",
  "timestamp": "2020-03-13T20:45:38.119Z",
  "version": 1
}

parse_timestamp

fallible pure

Parses the value in strptime format.

Function spec

parse_timestamp(value: <string> ,format: <string> , [timezone: <string> ] )

:: <timestamp> , <error>

required optional <types | ...>

Argument	Type	Description	Required?
value	string	The text of the timestamp.	yes
format	string	The strptime format.	yes
timezone	string	The TZ database format. By default, this function parses the timestamp by global `timezone` option. This argument overwrites the setting and is useful for parsing timestamps without a specified timezone, such as `16/10/2019 12:00:00`.	no

Errors

The parse_timestamp function is fallible, which means that error handling is required for these errors:

value fails to parse using the provided format.

value fails to parse using the provided timezone.

Examples

Parse timestamp

Source

parse_timestamp!("10-Oct-2020 16:00+00:00", format: "%v %R %:z")

Return

2020-10-10T16:00:00Z

Parse timestamp with timezone

Source

parse_timestamp!("16/10/2019 12:00:00", format: "%d/%m/%Y %H:%M:%S", timezone: "Asia/Taipei")

Return

2019-10-16T04:00:00Z

parse_tokens

fallible pure

Parses the value in token format. A token is considered to be one of the following:

A word surrounded by whitespace.
Text delimited by double quotes: "..". Quotes can be included in the token if they are escaped by a backslash (\).
Text delimited by square brackets: [..]. Closing square brackets can be included in the token if they are escaped by a backslash (\).

Function spec

parse_tokens(value: <string> )

:: <array> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to tokenize.		yes

Notices

This function has special behavior that you should be aware of.

All token values are returned as strings. We recommend manually coercing values to desired types as you see fit.

Errors

The parse_tokens function is fallible, which means that error handling is required for these errors:

value is not a properly formatted tokenized string.

Examples

Parse tokens

Source

parse_tokens(
	"A sentence \"with \\\"a\\\" sentence inside\" and [some brackets]"
)

Return

["A","sentence","with \\\"a\\\" sentence inside","and","some brackets"]

parse_url

fallible pure

Parses the value in URL format.

Function spec

parse_url(value: <string> , [default_known_ports: <boolean> ] )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The text of the URL.		yes
default_known_ports	boolean	If true and the port number is not specified in the input URL string (or matches the default port for the scheme), it is populated from well-known ports for the following schemes: `http`, `https`, `ws`, `wss`, and `ftp`.		no

Errors

The parse_url function is fallible, which means that error handling is required for these errors:

value is not a properly formatted URL.

Examples

Parse URL

Source

parse_url!("ftp://foo:bar@example.com:4343/foobar?hello=world#123")

Return

{
  "fragment": "123",
  "host": "example.com",
  "password": "bar",
  "path": "/foobar",
  "port": 4343,
  "query": {
    "hello": "world"
  },
  "scheme": "ftp",
  "username": "foo"
}

Parse URL with default port

Source

parse_url!("https://example.com", default_known_ports: true)

Return

{
  "fragment": null,
  "host": "example.com",
  "password": "",
  "path": "/",
  "port": 443,
  "query": {},
  "scheme": "https",
  "username": ""
}

Parse URL with internationalized domain name

Source

parse_url!("https://www.café.com")

Return

{
  "fragment": null,
  "host": "www.xn--caf-dma.com",
  "password": "",
  "path": "/",
  "port": null,
  "query": {},
  "scheme": "https",
  "username": ""
}

Parse URL with mixed case internationalized domain name

Source

parse_url!("https://www.CAFé.com")

Return

{
  "fragment": null,
  "host": "www.xn--caf-dma.com",
  "password": "",
  "path": "/",
  "port": null,
  "query": {},
  "scheme": "https",
  "username": ""
}

parse_user_agent

infallible pure

Parses the value as a user agent string, which has a loosely defined format so this parser only provides best effort guarantee.

Function spec

parse_user_agent(value: <string> , [mode: <string> ] )

:: <object>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to parse.		yes
mode	string	Determines performance and reliability characteristics.	`fast`	no

Notices

This function has special behavior that you should be aware of.

All values are returned as strings or as null. We recommend manually coercing values to desired types as you see fit.

Different modes return different schema.

Field which were not parsed out are set as null.

Examples

Fast mode

Source

parse_user_agent(
	"Mozilla Firefox 1.0.1 Mozilla/5.0 (X11; U; Linux i686; de-DE; rv:1.7.6) Gecko/20050223 Firefox/1.0.1"
)

Return

{
  "browser": {
    "family": "Firefox",
    "version": "1.0.1"
  },
  "device": {
    "category": "pc"
  },
  "os": {
    "family": "Linux",
    "version": null
  }
}

Reliable mode

Source

parse_user_agent(
	"Mozilla/4.0 (compatible; MSIE 7.66; Windows NT 5.1; SV1; .NET CLR 1.1.4322)",
	mode: "reliable"
)

Return

{
  "browser": {
    "family": "Internet Explorer",
    "version": "7.66"
  },
  "device": {
    "category": "pc"
  },
  "os": {
    "family": "Windows XP",
    "version": "NT 5.1"
  }
}

Enriched mode

Source

parse_user_agent(
	"Opera/9.80 (J2ME/MIDP; Opera Mini/4.3.24214; iPhone; CPU iPhone OS 4_2_1 like Mac OS X; AppleWebKit/24.783; U; en) Presto/2.5.25 Version/10.54",
	mode: "enriched"
)

Return

{
  "browser": {
    "family": "Opera Mini",
    "major": "4",
    "minor": "3",
    "patch": "24214",
    "version": "10.54"
  },
  "device": {
    "brand": "Apple",
    "category": "smartphone",
    "family": "iPhone",
    "model": "iPhone"
  },
  "os": {
    "family": "iOS",
    "major": "4",
    "minor": "2",
    "patch": "1",
    "patch_minor": null,
    "version": "4.2.1"
  }
}

parse_xml

fallible pure

Parses the value as XML.

Function spec

parse_xml(value: <string> , [include_attr: <boolean> ,attr_prefix: <string> ,text_key: <string> ,always_use_text_key: <boolean> ,parse_bool: <boolean> ,parse_null: <boolean> ,parse_number: <boolean> ] )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string representation of the XML document to parse.		yes
include_attr	boolean	Include XML tag attributes in the returned object.	`true`	no
attr_prefix	string	String prefix to use for XML tag attribute keys.	`@`	no
text_key	string	Key name to use for expanded text nodes.	`text`	no
always_use_text_key	boolean	Always return text nodes as `{"<text_key>": "value"}.`		no
parse_bool	boolean	Parse “true” and “false” as boolean.	`true`	no
parse_null	boolean	Parse “null” as null.	`true`	no
parse_number	boolean	Parse numbers as integers/floats.	`true`	no

Notices

This function has special behavior that you should be aware of.

Valid XML must contain exactly one root node. Always returns an object.

Errors

The parse_xml function is fallible, which means that error handling is required for these errors:

value is not a valid XML document.

Examples

Parse XML

Source

value = s'<book category="CHILDREN"><title lang="en">Harry Potter</title><author>J K. Rowling</author><year>2005</year></book>';

parse_xml!(value, text_key: "value", parse_number: false)

Return

{
  "book": {
    "@category": "CHILDREN",
    "author": "J K. Rowling",
    "title": {
      "@lang": "en",
      "value": "Harry Potter"
    },
    "year": "2005"
  }
}

Random functions

random_bool

infallible pure

Returns a random boolean.

Function spec

random_bool()

:: <boolean>

required optional <types | ...>

Examples

Random boolean

Source

is_boolean(random_bool())

Return

true

random_bytes

fallible pure

A cryptographically secure random number generator. Returns a string value containing the number of random bytes requested.

Function spec

random_bytes(length: <integer> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
length	integer	The number of bytes to generate. Must not be larger than 64k.		yes

Errors

The random_bytes function is fallible, which means that error handling is required for these errors:

length is negative.

length is larger than the maximum value (64k).

Examples

Generate random base 64 encoded bytes

Source

encode_base64(random_bytes(16))

Return

LNu0BBgUbh7XAlXbjSOomQ==

random_float

fallible pure

Returns a random float between [min, max).

Function spec

random_float(min: <float> ,max: <float> )

:: <float> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
min	float	Minimum value (inclusive).		yes
max	float	Maximum value (exclusive).		yes

Errors

The random_float function is fallible, which means that error handling is required for these errors:

max is not greater than min.

Examples

Random float from 0.0 to 10.0, not including 10.0

Source

f = random_float(0.0, 10.0)
f >= 0 && f < 10

Return

true

random_int

fallible pure

Returns a random integer between [min, max).

Function spec

random_int(min: <integer> ,max: <integer> )

:: <integer> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
min	integer	Minimum value (inclusive).		yes
max	integer	Maximum value (exclusive).		yes

Errors

The random_int function is fallible, which means that error handling is required for these errors:

max is not greater than min.

Examples

Random integer from 0 to 10, not including 10

Source

i = random_int(0, 10)
i >= 0 && i < 10

Return

true

uuid_from_friendly_id

fallible pure

Convert a Friendly ID (base62 encoding a 128-bit word) to a UUID.

Function spec

uuid_from_friendly_id(value: <timestamp> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	timestamp	A string that is a Friendly ID		yes

Errors

The uuid_from_friendly_id function is fallible, which means that error handling is required for these errors:

value is a string but the text uses characters outside of class [0-9A-Za-z].

value is a base62 encoding of an integer, but the integer is greater than or equal to 2^128.

Examples

Convert a Friendly ID to a UUID

Source

uuid_from_friendly_id!("3s87yEvnmkiPBMHsj8bwwc")

Return

7f41deed-d5e2-8b5e-7a13-ab4ff93cfad2

uuid_v4

infallible pure

Generates a random UUIDv4 string.

Function spec

uuid_v4()

:: <string>

required optional <types | ...>

Examples

Create a UUIDv4

Source

uuid_v4()

Return

1d262f4f-199b-458d-879f-05fd0a5f0683

uuid_v7

infallible pure

Generates a random UUIDv7 string.

Function spec

uuid_v7(, [timestamp: <timestamp> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
timestamp	timestamp	The timestamp used to generate the UUIDv7.	`now()`	no

Examples

Create a UUIDv7 with implicit `now()`

Source

uuid_v7()

Return

06338364-8305-7b74-8000-de4963503139

Create a UUIDv7 with explicit `now()`

Source

uuid_v7(now())

Return

018e29b3-0bea-7f78-8af3-d32ccb1b93c1

Create a UUIDv7 with custom timestamp

Source

uuid_v7(t'2020-12-30T22:20:53.824727Z')

Return

0176b5bd-5d19-7394-bb60-c21028c6152b

String functions

basename

fallible pure

Returns the filename component of the given path. This is similar to the Unix basename command. If the path ends in a directory separator, the function returns the name of the directory.

Function spec

basename(value: <string> )

:: <string | null> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The path from which to extract the basename.		yes

Errors

The basename function is fallible, which means that error handling is required for these errors:

value is not a valid string.

Examples

Extract basename from file path

Source

basename!("/usr/local/bin/vrl")

Return

vrl

Extract basename from file path with extension

Source

basename!("/home/user/file.txt")

Return

file.txt

Extract basename from directory path

Source

basename!("/home/user/")

Return

user

Root directory has no basename

Source

basename!("/")

Return

camelcase

infallible pure

Takes the value string, and turns it into camelCase. Optionally, you can pass in the existing case of the function, or else an attempt is made to determine the case automatically.

Function spec

camelcase(value: <string> , [original_case: <string> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to convert to camelCase.		yes
original_case	string	Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case		no

Examples

camelCase a string

Source

camelcase("input-string")

Return

inputString

camelCase a string

Source

camelcase("input-string", "kebab-case")

Return

inputString

community_id

infallible pure

Generates an ID based on the Community ID Spec.

Function spec

community_id(source_ip: <string> ,destination_ip: <string> ,protocol: <integer> , [source_port: <integer> ,destination_port: <integer> ,seed: <integer> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Required?
source_ip	string	The source IP address.	yes
destination_ip	string	The destination IP address.	yes
protocol	integer	The protocol number.	yes
source_port	integer	The source port or ICMP type.	no
destination_port	integer	The destination port or ICMP code.	no
seed	integer	The custom seed number.	no

Examples

TCP

Source

community_id!(source_ip: "1.2.3.4", destination_ip: "5.6.7.8", source_port: 1122, destination_port: 3344, protocol: 6)

Return

1:wCb3OG7yAFWelaUydu0D+125CLM=

contains

infallible pure

Determines whether the value string contains the specified substring.

Function spec

contains(value: <string> ,substring: <string> , [case_sensitive: <boolean> ] )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The text to search.		yes
substring	string	The substring to search for in `value`.		yes
case_sensitive	boolean	Whether the match should be case sensitive.	`true`	no

Examples

String contains (case sensitive)

Source

contains("The Needle In The Haystack", "Needle")

Return

true

String contains (case insensitive)

Source

contains("The Needle In The Haystack", "needle", case_sensitive: false)

Return

true

contains_all

infallible pure

Determines whether the value string contains all the specified substrings.

Function spec

contains_all(value: <string> ,substrings: <array> , [case_sensitive: <boolean> ] )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Required?
value	string	The text to search.	yes
substrings	array	An array of substrings to search for in `value`.	yes
case_sensitive	boolean	Whether the match should be case sensitive.	no

Examples

String contains all

Source

contains_all("The Needle In The Haystack", ["Needle", "Haystack"])

Return

true

String contains all (case sensitive)

Source

contains_all("the NEEDLE in the haystack", ["needle", "haystack"])

Return

false

dirname

fallible pure

Returns the directory component of the given path. This is similar to the Unix dirname command. The directory component is the path with the final component removed.

Function spec

dirname(value: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The path from which to extract the directory name.		yes

Errors

The dirname function is fallible, which means that error handling is required for these errors:

value is not a valid string.

Examples

Extract dirname from file path

Source

dirname!("/usr/local/bin/vrl")

Return

/usr/local/bin

Extract dirname from file path with extension

Source

dirname!("/home/user/file.txt")

Return

/home/user

Extract dirname from directory path

Source

dirname!("/home/user/")

Return

/home

Root directory dirname is itself

Source

dirname!("/")

Return

Relative files have current directory as dirname

Source

dirname!("file.txt")

Return

downcase

infallible pure

Downcases the value string, where downcase is defined according to the Unicode Derived Core Property Lowercase.

Function spec

downcase(value: <string> )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to convert to lowercase.		yes

Examples

Downcase a string

Source

downcase("Hello, World!")

Return

hello, world!

ends_with

infallible pure

Determines whether the value string ends with the specified substring.

Function spec

ends_with(value: <string> ,substring: <string> , [case_sensitive: <boolean> ] )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to search.		yes
substring	string	The substring with which `value` must end.		yes
case_sensitive	boolean	Whether the match should be case sensitive.	`true`	no

Examples

String ends with (case sensitive)

Source

ends_with("The Needle In The Haystack", "The Haystack")

Return

true

String ends with (case insensitive)

Source

ends_with("The Needle In The Haystack", "the haystack", case_sensitive: false)

Return

true

find

infallible pure

Determines from left to right the start position of the first found element in value that matches pattern. Returns -1 if not found.

Function spec

find(value: <string> ,pattern: <regex | string> , [from: <integer> ] )

:: <integer>

required optional <types | ...>

Argument	Type	Description	Required?
value	string	The string to find the pattern in.	yes
pattern	regex string	The regular expression or string pattern to match against.	yes
from	integer	Offset to start searching.	no

Examples

Match text

Source

find("foobar", "foo")

Return

Match regex

Source

find("foobar", r'b.r')

Return

No matches

Source

find("foobar", "baz")

Return

-1

With an offset

Source

find("foobarfoobarfoo", "bar", 4)

Return

join

infallible pure

Joins each string in the value array into a single string, with items optionally separated from one another by a separator.

Function spec

join(value: <array> , [separator: <string> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	array	The array of strings to join together.		yes
separator	string	The string separating each original element when joined.		no

Examples

Join array (no separator)

Source

join!(["bring", "us", "together"])

Return

bringustogether

Join array (comma separator)

Source

join!(["sources", "transforms", "sinks"], separator: ", ")

Return

sources, transforms, sinks

kebabcase

infallible pure

Takes the value string, and turns it into kebab-case. Optionally, you can pass in the existing case of the function, or else we will try to figure out the case automatically.

Function spec

kebabcase(value: <string> , [original_case: <string> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to convert to kebab-case.		yes
original_case	string	Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case		no

Examples

kebab-case a string

Source

kebabcase("InputString")

Return

input-string

kebab-case a string

Source

kebabcase("InputString", "PascalCase")

Return

input-string

match

infallible pure

Determines whether the value matches the pattern.

Function spec

match(value: <string> ,pattern: <regex> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The value to match.		yes
pattern	regex	The regular expression pattern to match against.		yes

Examples

Regex match on a string

Source

match("I'm a little teapot", r'teapot')

Return

true

String does not match the regular expression

Source

match("I'm a little teapot", r'.*balloon')

Return

false

match_any

infallible pure

Determines whether value matches any of the given patterns. All patterns are checked in a single pass over the target string, giving this function a potential performance advantage over the multiple calls in the match function.

Function spec

match_any(value: <string> ,patterns: <array> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The value to match.		yes
patterns	array	The array of regular expression patterns to match against.		yes

Examples

Regex match on a string

Source

match_any("I'm a little teapot", [r'frying pan', r'teapot'])

Return

true

parse_float

fallible pure

Parses the string value representing a floating point number in base 10 to a float.

Function spec

parse_float(value: <string> )

:: <float> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to parse.		yes

Errors

The parse_float function is fallible, which means that error handling is required for these errors:

value is not a string.

Examples

Parse negative integer

Source

parse_float!("-42")

Return

-42

Parse negative integer

Source

parse_float!("42.38")

Return

42.38

Scientific notation

Source

parse_float!("2.5e3")

Return

pascalcase

infallible pure

Takes the value string, and turns it into PascalCase. Optionally, you can pass in the existing case of the function, or else we will try to figure out the case automatically.

Function spec

pascalcase(value: <string> , [original_case: <string> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to convert to PascalCase.		yes
original_case	string	Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case		no

Examples

PascalCase a string

Source

pascalcase("input-string")

Return

InputString

PascalCase a string

Source

pascalcase("input-string", "kebab-case")

Return

InputString

redact

infallible pure

Redact sensitive data in value such as:

US social security card numbers
Other forms of personally identifiable information with custom patterns

This can help achieve compliance by ensuring sensitive data does not leave your network.

Function spec

redact(value: <string | object | array> ,filters: <array> , [redactor: <string | object> ] )

:: <string | object | array>

required optional <types | ...>

Argument Type Description Default Required?

value

string
object
array

The value to redact sensitive data from.

The function’s behavior depends on value’s type:

For strings, the sensitive data is redacted and a new string is returned.
For arrays, the sensitive data is redacted in each string element.
For objects, the sensitive data in each string value is masked, but the keys are not masked.

For arrays and objects, the function recurses into any nested arrays or objects. Any non-string elements are skipped.

Redacted text is replaced with [REDACTED].

yes

filters

array

List of filters applied to value.

Each filter can be specified in the following ways:

As a regular expression, which is used to redact text that match it.
As an object with a type key that corresponds to a named filter and additional keys for customizing that filter.
As a named filter, if it has no required parameters.

Named filters can be a:

pattern: Redacts text matching any regular expressions specified in the patterns key, which is required. This is the expanded version of just passing a regular expression as a filter.
us_social_security_number: Redacts US social security card numbers.

See examples for more details.

This parameter must be a static expression so that the argument can be validated at compile-time to avoid runtime errors. You cannot use variables or other dynamic expressions with it.

yes

redactor

string
object

Specifies what to replace the redacted strings with.

It is given as an object with a “type” key specifying the type of redactor to use and additional keys depending on the type. The following types are supported:

full: The default. Replace with the string “[REDACTED]”.
text: Replace with a custom string. The replacement key is required, and must contain the string that is used as a replacement.
sha2: Hash the redacted text with SHA-2 as with sha2. Supports two optional parameters:
- variant: The variant of the algorithm to use. Defaults to SHA-512/256.
- encoding: How to encode the hash as text. Can be base16 or base64. Defaults to base64.
sha3: Hash the redacted text with SHA-3 as with sha3. Supports two optional parameters:
- variant: The variant of the algorithm to use. Defaults to SHA3-512.
- encoding: How to encode the hash as text. Can be base16 or base64. Defaults to base64.

As a convenience you can use a string as a shorthand for common redactor patterns:

"full" is equivalent to {"type": "full"}
"sha2" is equivalent to {"type": "sha2", "variant": "SHA-512/256", "encoding": "base64"}
"sha3" is equivalent to {"type": "sha3", "variant": "SHA3-512", "encoding": "base64"}

This parameter must be a static expression so that the argument can be validated at compile-time to avoid runtime errors. You cannot use variables or other dynamic expressions with it.

Examples

Replace text using a regex

Source

redact("my id is 123456", filters: [r'\d+'])

Return

my id is [REDACTED]

Source

redact({ "name": "John Doe", "ssn": "123-12-1234"}, filters: ["us_social_security_number"])

Return

{
  "name": "John Doe",
  "ssn": "[REDACTED]"
}

Replace with custom text

Source

redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "text", "replacement": "***"})

Return

my id is ***

Replace with SHA-2 hash

Source

redact("my id is 123456", filters: [r'\d+'], redactor: "sha2")

Return

my id is GEtTedW1p6tC094dDKH+3B8P+xSnZz69AmpjaXRd63I=

Replace with SHA-3 hash

Source

redact("my id is 123456", filters: [r'\d+'], redactor: "sha3")

Return

my id is ZNCdmTDI7PeeUTFnpYjLdUObdizo+bIupZdl8yqnTKGdLx6X3JIqPUlUWUoFBikX+yTR+OcvLtAqWO11NPlNJw==

Replace with SHA-256 hash using hex encoding

Source

redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "sha2", "variant": "SHA-256", "encoding": "base16"})

Return

my id is 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92

replace

infallible pure

Replaces all matching instances of pattern in value.

The pattern argument accepts regular expression capture groups.

Note when using capture groups:

You will need to escape the $ by using $$ to avoid Vector interpreting it as an environment variable when loading configuration
If you want a literal $ in the replacement pattern, you will also need to escape this with $$. When combined with environment variable interpolation in config files this means you will need to use $$$$ to have a literal $ in the replacement pattern.

Function spec

replace(value: <string> ,pattern: <regex | string> ,with: <string> , [count: <integer> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The original string.		yes
pattern	regex string	Replace all matches of this pattern. Can be a static string or a regular expression.		yes
with	string	The string that the matches are replaced with.		yes
count	integer	The maximum number of replacements to perform. `-1` means replace all matches.	`-1`	no

Examples

Replace literal text

Source

replace("Apples and Bananas", "and", "not")

Return

Apples not Bananas

Replace using regular expression

Source

replace("Apples and Bananas", r'(?i)bananas', "Pineapples")

Return

Apples and Pineapples

Replace first instance

Source

replace("Bananas and Bananas", "Bananas", "Pineapples", count: 1)

Return

Pineapples and Bananas

Replace with capture groups when not set in the configuration file (use `$$num` in config files)

Source

# Note that in the context of Vector configuration files, an extra `$` escape character is required
# (i.e. `$$num`) to avoid interpreting `num` as an environment variable.
replace("foo123bar", r'foo(?P<num>\d+)bar', "$num")

Return

replace_with

infallible pure

Replaces all matching instances of pattern using a closure.

The pattern argument accepts a regular expression that can use capture groups.

The function uses the function closure syntax to compute the replacement values.

The closure takes a single parameter, which is an array, where the first item is always present and contains the entire string that matched pattern. The items from index one on contain the capture groups of the corresponding index. If a capture group is optional, the value may be null if it didn’t match.

The value returned by the closure must be a string and will replace the section of the input that was matched.

This returns a new string with the replacements, the original string is not mutated.

Function spec

replace_with(value: <string> ,pattern: <regex> , [count: <integer> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The original string.		yes
pattern	regex	Replace all matches of this pattern. Must be a regular expression.		yes
count	integer	The maximum number of replacements to perform. `-1` means replace all matches.	`-1`	no

Examples

Capitalize words

Source

	replace_with("apples and bananas", r'\b(\w)(\w*)') -> |match| {
		upcase!(match.captures[0]) + string!(match.captures[1])
	}

Return

Apples And Bananas

Replace with hash

Source

	replace_with("email from test@example.com", r'\w+@example.com') -> |match| {
		sha2(match.string, variant: "SHA-512/224")
	}

Return

email from adf6e1bc4415d24912bd93072ad34ef825a7b6eb3bf53f68def1fc17

Replace first instance

Source

	replace_with("Apples and Apples", r'(?i)apples|cones', count: 1) -> |match| {
		"Pine" + downcase(match.string)
	}

Return

Pineapples and Apples

Named capture group

Source

	replace_with("level=error A message", r'level=(?P<level>\w+)') -> |match| {
		lvl = upcase!(match.level)
		"[{{lvl}}]"
	}

Return

[ERROR] A message

screamingsnakecase

infallible pure

Takes the value string, and turns it into SCREAMING_SNAKE case. Optionally, you can pass in the existing case of the function, or else we will try to figure out the case automatically.

Function spec

screamingsnakecase(value: <string> , [original_case: <string> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to convert to SCREAMING_SNAKE case.		yes
original_case	string	Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case		no

Examples

SCREAMING_SNAKE a string

Source

screamingsnakecase("input-string")

Return

INPUT_STRING

SCREAMING_SNAKE a string

Source

screamingsnakecase("input-string", "kebab-case")

Return

INPUT_STRING

shannon_entropy

infallible pure

Generates Shannon entropy from given string. It can generate it based on string bytes, codepoints, or graphemes.

Function spec

shannon_entropy(value: <string> , [segmentation: <string> ] )

:: <float>

required optional <types | ...>

Argument Type Description Default Required?

value string The input string. yes

segmentation

string

Defines how to split the string to calculate entropy, based on occurrences of segments.

Byte segmentation is the fastest, but it might give undesired results when handling UTF-8 strings, while grapheme segmentation is the slowest, but most correct in these cases.

byte

Examples

Simple byte segmentation example

Source

floor(shannon_entropy("vector.dev"), precision: 4)

Return

2.9219

UTF-8 string with bytes segmentation

Source

floor(shannon_entropy("test123%456.فوائد.net."), precision: 4)

Return

4.0784

UTF-8 string with grapheme segmentation

Source

floor(shannon_entropy("test123%456.فوائد.net.", segmentation: "grapheme"), precision: 4)

Return

3.9362

sieve

infallible pure

Keeps only matches of pattern in value.

This can be used to define patterns that are allowed in the string and remove everything else.

Function spec

sieve(value: <string> ,pattern: <regex> , [replace_single: <string> ,replace_repeated: <string> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Required?
value	string	The original string.	yes
pattern	regex	Keep all matches of this pattern.	yes
replace_single	string	The string to use to replace single rejected characters.	no
replace_repeated	string	The string to use to replace multiple sequential instances of rejected characters.	no

Examples

Sieve with regex

Source

sieve("test123%456.فوائد.net.", r'[a-z0-9.]')

Return

test123456..net.

Custom replacements

Source

sieve("test123%456.فوائد.net.", r'[a-z.0-9]', replace_single: "X", replace_repeated: "<REMOVED>")

Return

test123X456.<REMOVED>.net.

slice

infallible pure

Returns a slice of value between the start and end positions.

If the start and end parameters are negative, they refer to positions counting from the right of the string or array. If end refers to a position that is greater than the length of the string or array, a slice up to the end of the string or array is returned.

Function spec

slice(value: <array | string> ,start: <integer> , [end: <integer> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	array string	The string or array to slice.		yes
start	integer	The inclusive start position. A zero-based index that can be negative.		yes
end	integer	The exclusive end position. A zero-based index that can be negative.	`String length`	no

Examples

Slice a string (positive index)

Source

slice!("Supercalifragilisticexpialidocious", start: 5, end: 13)

Return

califrag

Slice a string (negative index)

Source

slice!("Supercalifragilisticexpialidocious", start: 5, end: -14)

Return

califragilistic

snakecase

infallible pure

Takes the value string, and turns it into snake-case. Optionally, you can pass in the existing case of the function, or else we will try to figure out the case automatically.

Function spec

snakecase(value: <string> , [original_case: <string> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to convert to snake-case.		yes
original_case	string	Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case		no

Examples

snake-case a string

Source

snakecase("input-string")

Return

input_string

snake-case a string

Source

snakecase("input-string", "kebab-case")

Return

input_string

split

infallible pure

Splits the value string using pattern.

Function spec

split(value: <string> ,pattern: <string | regex> , [limit: <integer> ] )

:: <array>

required optional <types | ...>

Argument	Type	Description	Required?
value	string	The string to split.	yes
pattern	string regex	The string is split whenever this pattern is matched.	yes
limit	integer	The maximum number of substrings to return.	no

Examples

Split a string (no limit)

Source

split("apples and pears and bananas", " and ")

Return

["apples","pears","bananas"]

Split a string (with a limit)

Source

split("apples and pears and bananas", " and ", limit: 2)

Return

["apples","pears and bananas"]

split_path

fallible pure

Splits the given path into its constituent components, returning an array of strings. Each component represents a part of the file system path hierarchy.

Function spec

split_path(value: <string> )

:: <array> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The path to split into components.		yes

Errors

The split_path function is fallible, which means that error handling is required for these errors:

value is not a valid string.

Examples

Split path with trailing slash

Source

split_path!("/home/user/")

Return

["/","home","user"]

Split path from file path

Source

split_path!("/home/user")

Return

["/","home","user"]

Split path from root

Source

split_path!("/")

Return

["/"]

Empty path returns empty array

Source

split_path!("")

Return

[]

starts_with

infallible pure

Determines whether value begins with substring.

Function spec

starts_with(value: <string> ,substring: <string> , [case_sensitive: <boolean> ] )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to search.		yes
substring	string	The substring that the `value` must start with.		yes
case_sensitive	boolean	Whether the match should be case sensitive.	`true`	no

Examples

String starts with (case sensitive)

Source

starts_with("The Needle In The Haystack", "The Needle")

Return

true

String starts with (case insensitive)

Source

starts_with("The Needle In The Haystack", "the needle", case_sensitive: false)

Return

true

strip_ansi_escape_codes

infallible pure

Strips ANSI escape codes from value.

Function spec

strip_ansi_escape_codes(value: <string> )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to strip.		yes

Examples

Strip ANSI escape codes

Source

strip_ansi_escape_codes("\e[46mfoo\e[0m bar")

Return

foo bar

strip_whitespace

infallible pure

Strips whitespace from the start and end of value, where whitespace is defined by the Unicode White_Space property.

Function spec

strip_whitespace(value: <string> )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to trim.		yes

Examples

Strip whitespace

Source

strip_whitespace("  A sentence.  ")

Return

A sentence.

truncate

infallible pure

Truncates the value string up to the limit number of characters.

Function spec

truncate(value: <string> ,limit: <integer | float> , [ellipsis: <boolean> ,suffix: <string> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Required?
value	string	The string to truncate.	yes
limit	integer float	The number of characters to truncate the string after.	yes
ellipsis	boolean	This argument is deprecated. An ellipsis (`...`) is appended if the parameter is set to `true` and the `value` string is truncated because it exceeded the `limit`.	no
suffix	string	A custom suffix (`...`) is appended to truncated strings. If `ellipsis` is set to `true`, this parameter is ignored for backwards compatibility.	no

Examples

Truncate a string

Source

truncate("A rather long sentence.", limit: 11, suffix: "...")

Return

A rather lo...

Truncate a string

Source

truncate("A rather long sentence.", limit: 11, suffix: "[TRUNCATED]")

Return

A rather lo[TRUNCATED]

upcase

infallible pure

Upcases value, where upcase is defined according to the Unicode Derived Core Property Uppercase.

Function spec

upcase(value: <string> )

:: <string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to convert to uppercase.		yes

Examples

Upcase a string

Source

upcase("Hello, World!")

Return

HELLO, WORLD!

System functions

get_env_var

fallible pure

Returns the value of the environment variable specified by name.

Function spec

get_env_var(name: <string> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
name	string	The name of the environment variable.		yes

Errors

The get_env_var function is fallible, which means that error handling is required for these errors:

Environment variable name does not exist.

The value of environment variable name is not valid Unicode

Examples

Get an environment variable

Source

get_env_var!("HOME")

Return

/root

get_hostname

fallible pure

Returns the local system’s hostname.

Function spec

get_hostname()

:: <string> , <error>

required optional <types | ...>

Errors

The get_hostname function is fallible, which means that error handling is required for these errors:

Internal hostname resolution failed.

Examples

Get hostname

Source

.hostname = get_hostname!()

get_timezone_name

fallible pure

Returns the name of the timezone in the Vector configuration (see global configuration options). If the configuration is set to local, then it attempts to determine the name of the timezone from the host OS. If this is not possible, then it returns the fixed offset of the local timezone for the current time in the format "[+-]HH:MM", for example, "+02:00".

Function spec

get_timezone_name()

:: <string> , <error>

required optional <types | ...>

Errors

The get_timezone_name function is fallible, which means that error handling is required for these errors:

Retrieval of local timezone information failed.

Examples

Get the IANA name of Vector’s timezone

Source

.vector_timezone = get_timezone_name!()

Timestamp functions

format_timestamp

infallible pure

Formats value into a string representation of the timestamp.

Function spec

format_timestamp(value: <timestamp> ,format: <string> , [timezone: <string> ] )

:: <string>

required optional <types | ...>

Argument	Type	Description	Required?
value	timestamp	The timestamp to format as text.	yes
format	string	The format string as described by the Chrono library.	yes
timezone	string	The timezone to use when formatting the timestamp. The parameter uses the TZ identifier or `local`.	no

Examples

Format a timestamp (ISO8601/RFC 3339)

Source

format_timestamp!(t'2020-10-21T16:00:00Z', format: "%+")

Return

2020-10-21T16:00:00+00:00

Format a timestamp (custom)

Source

format_timestamp!(t'2020-10-21T16:00:00Z', format: "%v %R")

Return

21-Oct-2020 16:00

now

infallible pure

Returns the current timestamp in the UTC timezone with nanosecond precision.

Function spec

now()

:: <timestamp>

required optional <types | ...>

Examples

Generate a current timestamp

Source

now()

Return

2021-03-04T10:51:15.928937Z

Type functions

array

fallible pure

Returns value if it is an array, otherwise returns an error. This enables the type checker to guarantee that the returned value is an array and can be used in any function that expects an array.

Function spec

array(value: <any> )

:: <array> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check if it is an array.		yes

Errors

The array function is fallible, which means that error handling is required for these errors:

value is not an array.

Examples

Declare an array type

Source

array!(.value)

Return

[1,2,3]

bool

fallible pure

Returns value if it is a Boolean, otherwise returns an error. This enables the type checker to guarantee that the returned value is a Boolean and can be used in any function that expects a Boolean.

Function spec

bool(value: <any> )

:: <boolean> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check if it is a Boolean.		yes

Errors

The bool function is fallible, which means that error handling is required for these errors:

value is not a Boolean.

Examples

Declare a Boolean type

Source

bool!(.value)

Return

false

float

fallible pure

Returns value if it is a float, otherwise returns an error. This enables the type checker to guarantee that the returned value is a float and can be used in any function that expects a float.

Function spec

float(value: <any> )

:: <float> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check if it is a float.		yes

Errors

The float function is fallible, which means that error handling is required for these errors:

value is not a float.

Examples

Declare a float type

Source

float!(.value)

Return

int

fallible pure

Returns value if it is an integer, otherwise returns an error. This enables the type checker to guarantee that the returned value is an integer and can be used in any function that expects an integer.

Function spec

int(value: <any> )

:: <integer> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check if it is an integer.		yes

Errors

The int function is fallible, which means that error handling is required for these errors:

value is not an integer.

Examples

Declare an integer type

Source

int!(.value)

Return

is_array

infallible pure

Check if the value’s type is an array.

Function spec

is_array(value: <any> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check if it is an array.		yes

Examples

Valid array

Source

is_array([1, 2, 3])

Return

true

Non-matching type

Source

is_array("a string")

Return

false

is_boolean

infallible pure

Check if the value’s type is a boolean.

Function spec

is_boolean(value: <any> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check if it is a Boolean.		yes

Examples

Valid boolean

Source

is_boolean(false)

Return

true

Non-matching type

Source

is_boolean("a string")

Return

false

is_empty

infallible pure

Check if the object, array, or string has a length of 0.

Function spec

is_empty(value: <object | array | string> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	object array string	The value to check.		yes

Examples

Empty array

Source

is_empty([])

Return

true

Non-empty string

Source

is_empty("a string")

Return

false

Non-empty object

Source

is_empty({"foo": "bar"})

Return

false

is_float

infallible pure

Check if the value’s type is a float.

Function spec

is_float(value: <any> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check if it is a float.		yes

Examples

Valid float

Source

is_float(0.577)

Return

true

Non-matching type

Source

is_float("a string")

Return

false

is_integer

infallible pure

Check if the value`’s type is an integer.

Function spec

is_integer(value: <any> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check if it is an integer.		yes

Examples

Valid integer

Source

is_integer(1)

Return

true

Non-matching type

Source

is_integer("a string")

Return

false

is_json

infallible pure

Check if the string is a valid JSON document.

Function spec

is_json(value: <string> , [variant: <string> ] )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The value to check if it is a valid JSON document.		yes
variant	string	The variant of the JSON type to explicitly check for.		no

Examples

Valid JSON object

Source

is_json("{}")

Return

true

Non-valid value

Source

is_json("{")

Return

false

Exact variant

Source

is_json("{}", variant: "object")

Return

true

Non-valid exact variant

Source

is_json("{}", variant: "array")

Return

false

is_null

infallible pure

Check if value’s type is null. For a more relaxed function, see is_nullish.

Function spec

is_null(value: <any> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check if it is `null`.		yes

Examples

Null value

Source

is_null(null)

Return

true

Non-matching type

Source

is_null("a string")

Return

false

is_nullish

infallible pure

Determines whether value is nullish. Returns true if the specified value is null, an empty string, a string containing only whitespace, or the string "-". Returns false otherwise.

Function spec

is_nullish(value: <any> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check for nullishness, for example, a useless value.		yes

Notices

This function has special behavior that you should be aware of.

This function behaves inconsistently: it returns false for empty arrays ([]) and objects ({}), but true for empty strings ("") and null.

Examples

Null detection (blank string)

Source

is_nullish("")

Return

true

Null detection (dash string)

Source

is_nullish("-")

Return

true

Null detection (whitespace)

Source

is_nullish("
  
")

Return

true

is_object

infallible pure

Check if value’s type is an object.

Function spec

is_object(value: <any> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check if it is an object.		yes

Examples

Valid object

Source

is_object({"foo": "bar"})

Return

true

Non-matching type

Source

is_object("a string")

Return

false

is_regex

infallible pure

Check if value’s type is a regex.

Function spec

is_regex(value: <any> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check if it is a regex.		yes

Examples

Valid regex

Source

is_regex(r'pattern')

Return

true

Non-matching type

Source

is_regex("a string")

Return

false

is_string

infallible pure

Check if value’s type is a string.

Function spec

is_string(value: <any> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check if it is a string.		yes

Examples

Valid string

Source

is_string("a string")

Return

true

Non-matching type

Source

is_string([1, 2, 3])

Return

false

is_timestamp

infallible pure

Check if value’s type is a timestamp.

Function spec

is_timestamp(value: <any> )

:: <boolean>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check if it is a timestamp.		yes

Examples

Valid timestamp

Source

is_timestamp(t'2021-03-26T16:00:00Z')

Return

true

Non-matching type

Source

is_timestamp("a string")

Return

false

object

fallible pure

Returns value if it is an object, otherwise returns an error. This enables the type checker to guarantee that the returned value is an object and can be used in any function that expects an object.

Function spec

object(value: <any> )

:: <object> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check if it is an object.		yes

Errors

The object function is fallible, which means that error handling is required for these errors:

value is not an object.

Examples

Declare an object type

Source

object!(.value)

Return

{
  "field1": "value1",
  "field2": "value2"
}

string

fallible pure

Returns value if it is a string, otherwise returns an error. This enables the type checker to guarantee that the returned value is a string and can be used in any function that expects a string.

Function spec

string(value: <any> )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check if it is a string.		yes

Errors

The string function is fallible, which means that error handling is required for these errors:

value is not a string.

Examples

Declare a string type

Source

string!(.message)

Return

{"field": "value"}

tag_types_externally

infallible pure

Adds type information to all (nested) scalar values in the provided value.

The type information is added externally, meaning that value has the form of "type": value after this transformation.

Function spec

tag_types_externally(value: <any> )

:: <object | array | null>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to tag with types.		yes

Examples

Tag types externally (scalar)

Source

tag_types_externally(123)

Return

{
  "integer": 123
}

Tag types externally (object)

Source

tag_types_externally({
	"message": "Hello world",
	"request": {
		"duration_ms": 67.9
	}
})

Return

{
  "message": {
    "string": "Hello world"
  },
  "request": {
    "duration_ms": {
      "float": 67.9
    }
  }
}

Tag types externally (array)

Source

tag_types_externally(["foo", "bar"])

Return

[{"string":"foo"},{"string":"bar"}]

Tag types externally (null)

Source

tag_types_externally(null)

Return

timestamp

fallible pure

Returns value if it is a timestamp, otherwise returns an error. This enables the type checker to guarantee that the returned value is a timestamp and can be used in any function that expects a timestamp.

Function spec

timestamp(value: <any> )

:: <timestamp> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	any	The value to check if it is a timestamp.		yes

Errors

The timestamp function is fallible, which means that error handling is required for these errors:

value is not a timestamp.

Examples

Declare a timestamp type

Source

timestamp(t'2020-10-10T16:00:00Z')

Return

2020-10-10T16:00:00Z

validate_json_schema

fallible pure

Check if value conforms to a JSON Schema definition. This function validates a JSON payload against a JSON Schema definition. It can be used to ensure that the data structure and types in value match the expectations defined in schema_definition.

Function spec

validate_json_schema(value: <any> ,schema_definition: <any> , [ignore_unknown_formats: <boolean> ] )

:: <boolean> , <error>

required optional <types | ...>

Argument	Type	Description	Required?
value	any	The value to check if it conforms to the JSON schema definition.	yes
schema_definition	any	The location (path) of the JSON Schema definition.	yes
ignore_unknown_formats	boolean	Unknown formats can be silently ignored by setting this to `true` and validation continues without failing due to those fields.	no

Notices

This function has special behavior that you should be aware of.

This function uses a compiled schema cache. The first time it is called with a specific schema_definition, it will compile the schema and cache it for subsequent calls. This improves performance when validating multiple values against the same schema. The cache implementation is fairly naive and does not support refreshing the schema if it changes. If you update the schema definition file, you must restart Vector to clear the cache.

Errors

The validate_json_schema function is fallible, which means that error handling is required for these errors:

value is not a valid JSON Schema payload.

value contains custom format declarations and ignore_unknown_formats has not been set to true.

schema_definition is not a valid JSON Schema definition.

schema_definition file does not exist.

Examples

Payload contains a valid email.

Source

validate_json_schema!(s'{ "productUser": "valid@email.com" }', "resources/json-schema_definition.json", false)

Return

true

Payload contains an invalid email.

Source

validate_json_schema!(s'{ "productUser": "invalidEmail" }', "resources/json-schema_definition.json", false)

Return

false

Payload contains a custom format declaration.

Source

validate_json_schema!(s'{ "productUser": "a-custom-formatted-string" }', "resources/json-schema_definition.json", false)

Return

false

Payload contains a custom format declaration, with ignore_unknown_formats set to true.

Source

validate_json_schema!(s'{ "productUser": "a-custom-formatted-string" }', "resources/json-schema_definition.json", true)

Return

true

Checksum functions

crc

fallible pure

Calculates a CRC of the value. The CRC algorithm used can be optionally specified.

This function is infallible if either the default algorithm value or a recognized-valid compile-time algorithm string literal is used. Otherwise, it is fallible.

Function spec

crc(value: <string> , [algorithm: <string> ] )

:: <string> , <error>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to calculate the checksum for.		yes
algorithm	string	The CRC algorithm to use.	`CRC_32_ISO_HDLC`	no

Errors

The crc function is fallible, which means that error handling is required for these errors:

value is not a string.

algorithm is not a supported algorithm.

Examples

Create CRC checksum using the default algorithm

Source

crc("foo")

Return

2356372769

Create CRC checksum using the CRC_32_CKSUM algorithm

Source

crc("foo", algorithm: "CRC_32_CKSUM")

Return

4271552933

xxhash

infallible pure

Calculates a xxHash hash of the value. Note: Due to limitations in the underlying VRL data types, this function converts the unsigned 64-bit integer hash result to a signed 64-bit integer. Results higher than the signed 64-bit integer maximum value wrap around to negative values. For the XXH3-128 hash algorithm, values are returned as a string.

Function spec

xxhash(value: <string> , [variant: <string> ] )

:: <integer | string>

required optional <types | ...>

Argument	Type	Description	Default	Required?
value	string	The string to calculate the hash for.		yes
variant	string	The xxHash hashing algorithm to use.	`XXH32`	no

Examples

Calculate a hash using the default (XXH32) algorithm

Source

xxhash("foo")

Return

3792637401

Calculate a hash using the XXH32 algorithm

Source

xxhash("foo", "XXH32")

Return

3792637401

Calculate a hash using the XXH64 algorithm

Source

xxhash("foo", "XXH64")

Return

-3728699739546630700

Calculate a hash using the XXH3-64 algorithm

Source

xxhash("foo", "XXH3-64)

Return

-6093828362558604000

Calculate a hash using the XXH3-128 algorithm

Source

xxhash("foo", "XXH3-128)

Return

161745101148472925293886522910304009610