VRL function reference

Here you’ll find a comprehensive list of all built-in VRL functions. Functions are categorized by their purpose and sorted alphabetically for easy discovery. To use these functions in Vector, see the documentation on function call expressions and Vector’s remap transform.

Array functions

append

infallible

Appends each item in the items array to the end of the value array.

Function spec

append(value: <array>, items: <array>)

:: <array>

Argument	Type	Description
value	array	The initial array
items	array	The items to append

required optional <types | ...>

Examples

Append to an array

Source

append([1, 2], [3, 4])

Return

[
  1,
  2,
  3,
  4
]

push

infallible

Adds the item to the end of the value array.

Function spec

push(value: <array>, item: <any>)

:: <array>

Argument	Type	Description
value	array	The target array.
item	any	The item to push.

required optional <types | ...>

Examples

Push an item onto an array

Source

push([1, 2], 3)

Return

[
  1,
  2,
  3
]

Codec functions

decode_base64

fallible

Decodes the value (a Base64 string) into its original string.

Function spec

decode_base64(value: <string>, charset: <string>)

:: <string>

Argument	Type	Description
value	string	The Base64 data to decode.
charset	string	The character set to use when decoding the data.

required optional <types | ...>

Examples

Decode Base64 data (default)

Source

decode_base64!("eW91IGhhdmUgc3VjY2Vzc2Z1bGx5IGRlY29kZWQgbWU=")

Return

you have successfully decoded me

Decode Base64 data (URL safe)

Source

decode_base64!("eW91IGNhbid0IG1ha2UgeW91ciBoZWFydCBmZWVsIHNvbWV0aGluZyBpdCB3b24ndA==", charset: "url_safe")

Return

you can't make your heart feel something it won't

decode_percent

infallible

Decodes a percent-encoded value like a URL.

Function spec

decode_percent(value: <string>)

:: <string>

Argument	Type	Description
value	string	The string to decode.

required optional <types | ...>

Examples

Percent decode a value

Source

decode_percent("foo%20bar%3F")

Return

foo bar?

encode_base64

infallible

Encodes the value to Base64.

Function spec

encode_base64(value: <string>, padding: <boolean>, charset: <string>)

:: <string>

Argument	Type	Description
value	string	The string to encode.
padding	boolean	Whether the Base64 output is padded.
charset	string	The character set to use when encoding the data.

required optional <types | ...>

Examples

Encode to Base64 (default)

Source

encode_base64("please encode me")

Return

cGxlYXNlIGVuY29kZSBtZQ==

Encode to Base64 (without padding)

Source

encode_base64("please encode me, no padding though", padding: false)

Return

cGxlYXNlIGVuY29kZSBtZSwgbm8gcGFkZGluZyB0aG91Z2g

Encode to Base64 (URL safe)

Source

encode_base64("please encode me, but safe for URLs", charset: "url_safe")

Return

cGxlYXNlIGVuY29kZSBtZSwgYnV0IHNhZmUgZm9yIFVSTHM=

encode_json

infallible

Encodes the value to JSON.

Function spec

encode_json(value: <any>)

:: <string>

Argument	Type	Description
value	any	The value to convert to a JSON string.

required optional <types | ...>

Examples

Encode to JSON

Source

.payload = encode_json({"hello": "world"})

Return

{"hello":"world"}

encode_key_value

fallible

Encodes the value to in key/value format with customizable delimiters. Default delimiters match the logfmt format.

Function spec

encode_key_value(value: <object>, fields_ordering: <array>, key_value_delimiter: <string>, field_delimiter: <string>, flatten_boolean: <boolean>)

:: <string>

Argument	Type	Description
value	object	The value to convert to a string.
fields_ordering	array	The ordering of fields to preserve. Any fields not in this list will appear unordered, after any ordered fields.
key_value_delimiter	string	The string that separates the key from the value.
field_delimiter	string	The string that separates each key/value pair.
flatten_boolean	boolean	Whether to encode key/value with a boolean value as a standalone key if `true` and nothing if `false`.

required optional <types | ...>

Examples

Encode with default delimiters (no ordering)

Source

encode_key_value({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})

Return

lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z

Encode with default delimiters (fields ordering)

Source

encode_key_value!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])

Return

ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345

Encode with default delimiters (nested fields)

Source

encode_key_value({"agent": {"name": "vector"}, "log": {"file": {"path": "my.log"}}, "event": "log"})

Return

agent.name=vector event=log log.file.path=my.log

Encode with default delimiters (nested fields ordering)

Source

encode_key_value!({"agent": {"name": "vector"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])

Return

event=log log.file.path=my.log agent.name=vector

Encode with custom delimiters (no ordering)

Source

encode_key_value(
	{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"},
	field_delimiter: ",",
	key_value_delimiter: ":"
)

Return

lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z

Encode with custom delimiters and flatten boolean

Source

encode_key_value(
	{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "beta": true, "dropped": false},
	field_delimiter: ",",
	key_value_delimiter: ":",
	flatten_boolean: true
)

Return

beta,lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z

Notices This function has special behavior that you should be aware of.

If fields_ordering is specified then the function is fallible else it is infallible.

encode_logfmt

fallible

Encodes the value to logfmt.

Function spec

encode_logfmt(value: <object>, fields_ordering: <array>)

:: <string>

Argument	Type	Description
value	object	The value to convert to a logfmt string.
fields_ordering	array	The ordering of fields to preserve. Any fields not in this list will appear unordered, after any ordered fields.

required optional <types | ...>

Examples

Encode to logfmt (no ordering)

Source

encode_logfmt({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})

Return

lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z

Encode to logfmt (fields ordering)

Source

encode_logfmt!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])

Return

ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345

Encode to logfmt (nested fields)

Source

encode_logfmt({"agent": {"name": "vector"}, "log": {"file": {"path": "my.log"}}, "event": "log"})

Return

agent.name=vector event=log log.file.path=my.log

Encode to logfmt (nested fields ordering)

Source

encode_logfmt!({"agent": {"name": "vector"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])

Return

event=log log.file.path=my.log agent.name=vector

Notices This function has special behavior that you should be aware of.

If fields_ordering is specified then the function is fallible else it is infallible.

encode_percent

infallible

Encodes a value with percent encoding to safely be used in URLs.

Function spec

encode_percent(value: <string>, ascii_set: <string>)

:: <string>

Argument	Type	Description
value	string	The string to encode.
ascii_set	string	The ascii set to use when encoding the data.

required optional <types | ...>

Examples

Percent encode all non-alphanumeric characters (default)

Source

encode_percent("foo bar?")

Return

foo%20bar%3F

Percent encode only control characters

Source

encode_percent("foo 	bar", ascii_set: "CONTROLS")

Return

foo %09bar

Coerce functions

to_bool

fallible

Coerces the value into a boolean.

Function spec

to_bool(value: <boolean | integer | float | null | string>)

:: <boolean>

Argument	Type	Description
value	boolean integer float null string	The value to convert to a Boolean.

required optional <types | ...>

Examples

Coerce to a Boolean (string)

Source

to_bool!("yes")

Return

true

Coerce to a Boolean (float)

Source

to_bool(0.0)

Coerce to a Boolean (int)

Source

to_bool(0)

Coerce to a Boolean (null)

Source

to_bool(null)

Coerce to a Boolean (Boolean)

Source

to_bool(true)

Return

true

to_float

fallible

Coerces the value into a float.

Function spec

to_float(value: <float | integer | boolean | string>)

:: <float>

Argument	Type	Description
value	float integer boolean string	The value to convert to a float. Must be convertible to a float, otherwise an error is raised.

required optional <types | ...>

Examples

Coerce to a float

Source

to_float!("3.145")

Return

3.145

to_int

fallible

Coerces the value into an integer.

Function spec

to_int(value: <integer | float | boolean | string | timestamp>)

:: <integer>

Argument	Type	Description
value	integer float boolean string timestamp	The value to convert to an integer.

required optional <types | ...>

Examples

Coerce to an int (string)

Source

to_int!("2")

Return

Coerce to an int (timestamp)

Source

to_int(t'2020-12-30T22:20:53.824727Z')

Return

1609366853

to_regex

fallible

Coerces the value into a regex.

Function spec

to_regex(value: <string>)

:: <regex>

Argument	Type	Description
value	string	The value to convert to a regex.

required optional <types | ...>

Examples

Coerce to a regex

Source

to_regex("^foo$") ?? r''

Return

^foo$

Notices This function has special behavior that you should be aware of.

Compiling a regular expression is an expensive operation and can limit Vector throughput. Don’t use this function unless you are absolutely sure there is no other way!

to_string

fallible

Coerces the value into a string.

Function spec

:: <string>

Argument	Type	Description
value	integer float boolean string timestamp null	The value to convert to a string.

required optional <types | ...>

Examples

Coerce to a string (Boolean)

Source

to_string(true)

Return

true

Coerce to a string (int)

Source

to_string(52)

Return

Coerce to a string (float)

Source

to_string(52.2)

Return

52.2

to_timestamp

fallible

Coerces the value into a timestamp.

Function spec

to_timestamp(value: <string | integer | timestamp>)

:: <timestamp>

Argument	Type	Description
value	string integer timestamp	The value that is to be converted to a timestamp. If a string, must be a valid representation of a `timestamp`, and no `default` exists, an `ArgumentError` will be raised.

required optional <types | ...>

Examples

Coerce to a timestamp

Source

to_timestamp!("2020-10-21T16:00:00Z")

Return

2020-10-21T16:00:00Z

Convert functions

to_syslog_facility

fallible

Converts the value, a Syslog facility code, into its corresponding Syslog keyword. i.e. 0 into "kern", 1 into "user", etc.

Function spec

to_syslog_facility(value: <integer>)

:: <string>

Argument	Type	Description
value	integer	The facility code.

required optional <types | ...>

Examples

Coerce to a Syslog facility

Source

to_syslog_facility!(4)

Return

auth

to_syslog_level

fallible

Converts the value, a Syslog severity level, into its corresponding keyword, i.e. 0 into "emerg", 1 into "alert", etc.

Function spec

to_syslog_level(value: <integer>)

:: <string>

Argument	Type	Description
value	integer	The severity level.

required optional <types | ...>

Examples

Coerce to a Syslog level

Source

to_syslog_level!(5)

Return

notice

to_syslog_severity

fallible

Converts the value, a Syslog log level keyword, into a Syslog integer severity level (0 to 7).

Function spec

to_syslog_severity(value: <string>)

:: <integer>

Argument	Type	Description
value	string	The Syslog level keyword to convert.

required optional <types | ...>

Examples

Coerce to Syslog severity

Source

to_syslog_severity!("alert")

Return

to_unix_timestamp

infallible

Converts the value timestamp into a Unix timestamp.

Returns the number of seconds since the Unix epoch by default, but milliseconds or nanoseconds can also be specified by unit.

Function spec

to_unix_timestamp(value: <timestamp>, unit: <string>)

:: <integer>

Argument	Type	Description
value	timestamp	The timestamp to convert to Unix.
unit	string	The time unit.

required optional <types | ...>

Examples

Convert to a Unix timestamp (seconds)

Source

to_unix_timestamp(t'2021-01-01T00:00:00+00:00')

Return

1609459200

Convert to a Unix timestamp (milliseconds)

Source

to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "milliseconds")

Return

1609459200000

Convert to a Unix timestamp (nanoseconds)

Source

to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "nanoseconds")

Return

1609459200000000000

Debug functions

assert

fallible

Asserts the condition, which must be a Boolean expression. The program is aborted with message if the condition evaluates to false.

Function spec

assert(condition: <boolean>, message: <string>)

:: <null>

Argument	Type	Description
condition	boolean	The condition to check.
message	string	The failure message that’s reported if `condition` evaluates to `false`.

required optional <types | ...>

Examples

Assertion (true)

Source

ok, err = assert("foo" == "foo", message: "\"foo\" must be \"foo\"!")

Return

true

Assertion (false)

Source

assert!("foo" == "bar", message: "\"foo\" must be \"foo\"!")

Notices This function has special behavior that you should be aware of.

The assert function should be used in a standalone fashion and only when you want to abort the program. You should avoid it in logical expressions and other situations in which you want the program to continue if the condition evaluates to false.

assert_eq

infallible

Asserts that two expressions, left and right, have the same value. The program is aborted with the message if they are unequal.

Function spec

assert_eq(left: <any>, right: <any>, message: <string>)

:: <boolean>

Argument	Type	Description
left	any	The value to check for equality against `right`.
right	any	The value to check for equality against `left`.
message	string	An optional custom error message. If the equality assertion fails, `message` is appended to the default message prefix. See the examples below for a sample fully formed log message.

required optional <types | ...>

Examples

Successful assertion

Source

assert_eq!(1, 1)

Return

true

Unsuccessful assertion

Source

assert_eq!(127, [1, 2, 3])

Unsuccessful assertion with custom log message

Source

 assert_eq!(1, 0, message: "Unequal integers")

Notices This function has special behavior that you should be aware of.

The assert_eq function should be used in a standalone fashion and only when you want to abort the program. You should avoid it in logical expressions and other situations in which you want the program to continue if the condition evaluates to false.

log

infallible

Logs the value to Vector’s stdout at the specified level.

Function spec

log(value: <any>, level: <string>, rate_limit_secs: <integer>)

:: <null>

Argument	Type	Description
value	any	The value to log.
level	string	The log level.
rate_limit_secs	integer	Specifies that the log message is output no more than once per the given number of seconds. Use a value of `0` to turn rate limiting off.

required optional <types | ...>

Examples

Log a message

Source

log("Hello, World!", level: "info", rate_limit_secs: 60)

Log an error

Source

ts, err = to_int(.field)
if err != null {
	log(err, level: "error")
}

Enumerate functions

compact

infallible

Compacts the value by removing “empty” values, where emptiness is defined using the available parameters.

Function spec

compact(value: <array | object>, recursive: <boolean>, null: <boolean>, string: <boolean>, object: <boolean>, array: <boolean>, nullish: <boolean>)

:: <array | object>

Argument	Type	Description
value	array object	The object or array to compact.
recursive	boolean	Whether the compaction be recursive.
null	boolean	Whether null should be treated as an empty value.
string	boolean	Whether an empty string should be treated as an empty value.
object	boolean	Whether an empty object should be treated as an empty value.
array	boolean	Whether an empty array should be treated as an empty value.
nullish	boolean	Tests whether the value is “nullish” as defined by the `is_nullish` function.

required optional <types | ...>

Examples

Compact an array

Source

compact(["foo", "bar", "", null, [], "buzz"], string: true, array: true, null: true)

Return

[
  "foo",
  "bar",
  "buzz"
]

Compact an object

Source

compact({"field1": 1, "field2": "", "field3": [], "field4": null}, string: true, array: true, null: true)

Return

{
  "field1": 1
}

flatten

infallible

Flattens the value into a single-level representation.

Function spec

flatten(value: <array | object>)

:: <array | object>

Argument	Type	Description
value	array object	The array or object to flatten.

required optional <types | ...>

Examples

Flatten array

Source

flatten([1, [2, 3, 4], [5, [6, 7], 8], 9])

Return

[
  1,
  2,
  3,
  4,
  5,
  6,
  7,
  8,
  9
]

Flatten object

Source

flatten({
	"parent1": {
		"child1": 1,
		"child2": 2
	},
	"parent2": {
		"child3": 3
	}
})

Return

{
  "parent1.child1": 1,
  "parent1.child2": 2,
  "parent2.child3": 3
}

includes

infallible

Determines whether the value array includes the specified item.

Function spec

includes(value: <array>, item: <any>)

:: <boolean>

Argument	Type	Description
value	array	The array.
item	any	The item to check.

required optional <types | ...>

Examples

Array includes

Source

includes(["apple", "orange", "banana"], "banana")

Return

true

length

infallible

Returns the length of the value.

Function spec

length(value: <array | object | string>)

:: <integer>

Argument	Type	Description
value	array object string	The array or object

required optional <types | ...>

Examples

Length (object)

Source

length({
	"portland": "Trail Blazers",
	"seattle":  "Supersonics"
})

Return

Length (nested object)

Source

length({
	"home": {
		"city":  "Portland",
		"state": "Oregon"
	},
	"name": "Trail Blazers",
	"mascot": {
		"name": "Blaze the Trail Cat"
	}
})

Return

Length (array)

Source

length(["Trail Blazers", "Supersonics", "Grizzlies"])

Return

Length (string)

Source

length("The Planet of the Apes Musical")

Return

match_array

infallible

Determines whether the elements in the value array matches the pattern - by default it checks at least one element matches, but can be set to determine if all the elements match.

Function spec

match_array(value: <array>, pattern: <regex>, all: <boolean>)

:: <boolean>

Argument	Type	Description
value	array	The array.
pattern	regex	The regular expression pattern to match against.
all	boolean	Whether to match on all elements of `value`.

required optional <types | ...>

Examples

Match at least one element

Source

	match_array(["foobar", "bazqux"], r'foo')

Return

true

Match all elements

Source

	match_array(["foo", "foobar", "barfoo"], r'foo', all: true)

Return

true

No matches

Source

	match_array(["bazqux", "xyz"], r'foo')

Not all elements match

Source

	match_array(["foo", "foobar", "baz"], r'foo', all: true)

Event functions

del

infallible

Removes the field specified by the path from the current event object.

Function spec

del(path: <path>)

:: <any>

Argument	Type	Description
path	path	The path of the field to delete.

required optional <types | ...>

Examples

Delete a field

Source

del(.field1)

Rename a field

Source

.new_field = del(.old_field)

Notices This function has special behavior that you should be aware of.

The del function modifies the current event in place and returns the value of the deleted field.

exists

infallible

Checks whether the path exists for the current event.

Function spec

exists(path: <path>)

:: <boolean>

Argument	Type	Description
path	path	The path of the field to check.

required optional <types | ...>

Examples

Exists (field)

Source

exists(.field)

Return

true

Exists (array element)

Source

exists(.array[2])

Return

true

Hash functions

md5

infallible

Calculates an md5 hash of the value.

Function spec

md5(value: <string>)

:: <string>

Argument	Type	Description
value	string	The string to calculate the hash for.

required optional <types | ...>

Examples

Create md5 hash

Source

md5("foo")

Return

acbd18db4cc2f85cedef654fccc4a4d8

sha1

infallible

Calculates a SHA-1 hash of the value.

Function spec

sha1(value: <string>)

:: <string>

Argument	Type	Description
value	string	The string to calculate the hash for.

required optional <types | ...>

Examples

Calculate sha1 hash

Source

sha1("foo")

Return

0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33

sha2

infallible

Calculates a SHA-2 hash of the value.

Function spec

sha2(value: <string>, variant: <string>)

:: <string>

Argument	Type	Description
value	string	The string to calculate the hash for.
variant	string	The variant of the algorithm to use.

required optional <types | ...>

Examples

Calculate sha2 hash

Source

sha2("foo", variant: "SHA-512/224")

Return

d68f258d37d670cfc1ec1001a0394784233f88f056994f9a7e5e99be

sha3

infallible

Calculates a SHA-3 hash of the value.

Function spec

sha3(value: <string>, variant: <string>)

:: <string>

Argument	Type	Description
value	string	The string to calculate the hash for.
variant	string	The variant of the algorithm to use.

required optional <types | ...>

Examples

Calaculate sha3 hash

Source

sha3("foo", variant: "SHA3-224")

Return

f4f6779e153c391bbd29c95e72b0708e39d9166c7cea51d1f10ef58a

IP functions

ip_aton

fallible

Converts IPv4 address in numbers-and-dots notation into network-order bytes represented as an integer.

This behavior mimics inet_aton.

Function spec

ip_aton(value: <string>)

:: <integer>

Argument	Type	Description
value	string	The IP address to convert to binary.

required optional <types | ...>

Examples

IPv4 to integer

Source

ip_aton!("1.2.3.4")

Return

67305985

ip_cidr_contains

fallible

Determines whether the ip is contained in the block referenced by the cidr.

Function spec

ip_cidr_contains(cidr: <string>, ip: <string>)

:: <boolean>

Argument	Type	Description
cidr	string	The CIDR mask (v4 or v6).
ip	string	The IP address (v4 or v6).

required optional <types | ...>

Examples

IPv4 contains CIDR

Source

ip_cidr_contains!("192.168.0.0/16", "192.168.10.32")

Return

true

IPv6 contains CIDR

Source

ip_cidr_contains!("2001:4f8:4:ba::/64", "2001:4f8:4:ba:2e0:81ff:fe22:d1f1")

Return

true

ip_ntoa

fallible

Converts numeric representation of IPv4 address in network-order bytes to numbers-and-dots notation..

This behavior mimics inet_ntoa.

Function spec

ip_ntoa(value: <string>)

:: <string>

Argument	Type	Description
value	string	The integer representation of an IPv4 address.

required optional <types | ...>

Examples

Integer to IPv4

Source

ip_ntoa!(67305985)

Return

1.2.3.4

ip_subnet

fallible

Extracts the subnet address from the ip using the supplied subnet.

Function spec

ip_subnet(ip: <string>, subnet: <string>)

:: <string>

Argument	Type	Description
ip	string	The IP address (v4 or v6).
subnet	string	The subnet to extract from the IP address. This can be either a prefix length like `/8` or a net mask like `255.255.0.0`. The net mask can be either an IPv4 or IPv6 address.

required optional <types | ...>

Examples

IPv4 subnet

Source

ip_subnet!("192.168.10.32", "255.255.255.0")

Return

192.168.10.0

IPv6 subnet

Source

ip_subnet!("2404:6800:4003:c02::64", "/32")

Return

2404:6800::

Notices This function has special behavior that you should be aware of.

Works with both IPv4 and IPv6 addresses. The IP version for the mask must be the same as the supplied address.

ip_to_ipv6

fallible

Converts the ip to an IPv6 address.

Function spec

ip_to_ipv6(ip: <string>)

:: <string>

Argument	Type	Description
ip	string	The IP address to convert to IPv6.

required optional <types | ...>

Examples

IPv4 to IPv6

Source

ip_to_ipv6!("192.168.10.32")

Return

::ffff:192.168.10.32

ipv6_to_ipv4

fallible

Converts the ip to an IPv4 address. ip is returned unchanged if it’s already an IPv4 address. If ip is currently an IPv6 address then it needs to be IPv4 compatible, otherwise an error is thrown.

Function spec

ipv6_to_ipv4(ip: <string>)

:: <string>

Argument	Type	Description
ip	string	The IPv4-mapped IPv6 address to convert.

required optional <types | ...>

Examples

IPv6 to IPv4

Source

ipv6_to_ipv4!("::ffff:192.168.0.1")

Return

192.168.0.1

Number functions

ceil

infallible

Rounds the value up to the specified precision.

Function spec

ceil(value: <integer | float>, precision: <integer>)

:: <integer | float>

Argument	Type	Description
value	integer float	The number to round up.
precision	integer	The number of decimal places to round to.

required optional <types | ...>

Examples

Round a number up (without precision)

Source

ceil(4.345)

Return

Round a number up (with precision)

Source

ceil(4.345, precision: 2)

Return

4.35

floor

infallible

Rounds the value down to the specified precision.

Function spec

floor(value: <integer | float>, precision: <integer>)

:: <integer | float>

Argument	Type	Description
value	integer float	The number to round down.
precision	integer	The number of decimal places to round to.

required optional <types | ...>

Examples

Round a number down (without precision)

Source

floor(4.345)

Return

Round a number down (with precision)

Source

floor(4.345, precision: 2)

Return

4.34

format_int

fallible

Formats the integer value into a string representation using the given base/radix.

Function spec

format_int(value: <integer>, base: <integer>)

:: <string>

Argument	Type	Description
value	integer	The number to format.
base	integer	The base to format the number in. Must be between 2 and 36 (inclusive).

required optional <types | ...>

Examples

Format as a hexidecimal integer

Source

format_int!(42, 16)

Return

2a

Format as a negative hexidecimal integer

Source

format_int!(-42, 16)

Return

-2a

format_number

infallible

Formats the value into a string representation of the number.

Function spec

format_number(value: <integer | float>, scale: <integer>, decimal_separator: <string>, grouping_separator: <string>)

:: <string>

Argument	Type	Description
value	integer float	The number to format as a string.
scale	integer	The number of decimal places to display.
decimal_separator	string	The character to use between the whole and decimal parts of the number.
grouping_separator	string	The character to use between each thousands part of the number.

required optional <types | ...>

Examples

Format a number (3 decimals)

Source

format_number(1234567.89, 3, decimal_separator: ".", grouping_separator: ",")

Return

1,234,567.890

round

infallible

Rounds the value to the specified precision.

Function spec

round(value: <integer | float>, precision: <integer>)

:: <integer | float>

Argument	Type	Description
value	integer float	The number to round.
precision	integer	The number of decimal places to round to.

required optional <types | ...>

Examples

Round a number (without precision)

Source

round(4.345)

Return

Round a number (with precision)

Source

round(4.345, precision: 2)

Return

4.35

Object functions

merge

infallible

Merges the from object into the to object.

Function spec

merge(to: <string>, from: <object>, deep: <boolean>)

:: <object>

Argument	Type	Description
to	string	The object to merge into.
from	object	The object to merge from.
deep	boolean	A deep merge is performed if `true`, otherwise only top-level fields are merged.

required optional <types | ...>

Examples

Object merge (shallow)

Source

merge(
	{
		"parent1": {
			"child1": 1,
			"child2": 2
		},
		"parent2": {
			"child3": 3
		}
	},
	{
		"parent1": {
			"child2": 4,
			"child5": 5
		}
	}
)

Return

{
  "parent1": {
    "child2": 4,
    "child5": 5
  },
  "parent2": {
    "child3": 3
  }
}

Object merge (deep)

Source

merge(
	{
		"parent1": {
			"child1": 1,
			"child2": 2
		},
		"parent2": {
			"child3": 3
		}
	},
	{
		"parent1": {
			"child2": 4,
			"child5": 5
		}
	},
	deep: true
)

Return

{
  "parent1": {
    "child1": 1,
    "child2": 4,
    "child5": 5
  },
  "parent2": {
    "child3": 3
  }
}

unnest

fallible

Unnest an array field from an object to create an array of objects using that field; keeping all other fields.

Assigning the array result of this to . will result in multiple events being emitted from remap. See the remap transform docs for more details.

This is also referred to as explodeing in some languages.

Function spec

unnest(path: <path>)

:: <array>

Argument	Type	Description
path	path	The path of the field to unnest.

required optional <types | ...>

Examples

Unnest an array field

Source

. = unnest!(.messages)

Unnest nested an array field

Source

. = unnest!(.event.messages)

Parse functions

parse_apache_log

fallible

Parses Apache access and error log lines. Lines can be in common, combined, or default error format.

Function spec

parse_apache_log(value: <string>, timestamp_format: <string>, format: <string>)

:: <object>

Argument	Type	Description
value	string	The string to parse.
timestamp_format	string	The date/time format to use for encoding the timestamp. The time is parsed in local time if the timestamp doesn’t specify a timezone.
format	string	The format to use for parsing the log.

required optional <types | ...>

Examples

Parse via Apache log format (common)

Source

parse_apache_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326", format: "common")

Return

{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}

Parse via Apache log format (combined)

Source

parse_apache_log!(
	s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth" "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"',
	"combined",
)

Return

{
  "agent": "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0",
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "referrer": "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}

Parse via Apache log format (error)

Source

parse_apache_log!(
	s'[01/Mar/2021:12:00:19 +0000] [ab:alert] [pid 4803:tid 3814] [client 147.159.108.175:24259] I will bypass the haptic COM bandwidth, that should matrix the CSS driver!',
	"error"
)

Return

{
  "client": "147.159.108.175",
  "message": "I will bypass the haptic COM bandwidth, that should matrix the CSS driver!",
  "module": "ab",
  "pid": 4803,
  "port": 24259,
  "severity": "alert",
  "thread": "3814",
  "timestamp": "2021-03-01T12:00:19Z"
}

Notices This function has special behavior that you should be aware of.

Missing information in the log message may be indicated by -. These fields are omitted in the result.

parse_aws_alb_log

fallible

Parses value in the Elastic Load Balancer Access format.

Function spec

parse_aws_alb_log(value: <string>)

:: <object>

Argument	Type	Description
value	string	Access log of the Application Load Balancer.

required optional <types | ...>

Examples

Parse AWS ALB log

Source

parse_aws_alb_log!(
	"http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\""
)

Return

{
  "actions_executed": "forward",
  "chosen_cert_arn": null,
  "classification": null,
  "classification_reason": null,
  "client_host": "192.168.131.39:2817",
  "domain_name": null,
  "elb": "app/my-loadbalancer/50dc6c495c0c9188",
  "elb_status_code": "200",
  "error_reason": null,
  "matched_rule_priority": "0",
  "received_bytes": 34,
  "redirect_url": null,
  "request_creation_time": "2018-11-30T22:22:48.364000Z",
  "request_method": "GET",
  "request_processing_time": 0,
  "request_protocol": "HTTP/1.1",
  "request_url": "http://www.example.com:80/",
  "response_processing_time": 0,
  "sent_bytes": 366,
  "ssl_cipher": null,
  "ssl_protocol": null,
  "target_group_arn": "arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067",
  "target_host": null,
  "target_port_list": [],
  "target_processing_time": 0.001,
  "target_status_code": "200",
  "target_status_code_list": [],
  "timestamp": "2018-11-30T22:23:00.186641Z",
  "trace_id": "Root=1-58337364-23a8c76965a2ef7629b185e3",
  "type": "http",
  "user_agent": "curl/7.46.0"
}

parse_aws_cloudwatch_log_subscription_message

fallible

Parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) from the aws_kinesis_firehose source.

Function spec

parse_aws_cloudwatch_log_subscription_message(value: <string>)

:: <object>

Argument	Type	Description
value	string	The string representation of the message to parse.

required optional <types | ...>

Examples

Parse AWS Cloudwatch Log subscription message

Source

parse_aws_cloudwatch_log_subscription_message!(.message)

Return

{
  "log_events": [
    {
      "id": "35683658089614582423604394983260738922885519999578275840",
      "message": "{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}",
      "timestamp": "2020-09-14T19:09:29.039Z"
    }
  ],
  "log_group": "test",
  "log_stream": "test",
  "message_type": "DATA_MESSAGE",
  "owner": "111111111111",
  "subscription_filters": [
    "Destination"
  ]
}

parse_aws_vpc_flow_log

fallible

Parses value in the VPC Flow Logs format.

Function spec

parse_aws_vpc_flow_log(value: <string>, format: <string>)

:: <object>

Argument	Type	Description
value	string	VPC Flow Log.
format	string	VPC Flow Log format.

required optional <types | ...>

Examples

Parse AWS VPC Flow log (default format)

Source

parse_aws_vpc_flow_log!("2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA")

Return

{
  "account_id": 123456789010,
  "action": null,
  "bytes": null,
  "dstaddr": null,
  "dstport": null,
  "end": 1431280934,
  "interface_id": "eni-1235b8ca123456789",
  "log_status": "NODATA",
  "packets": null,
  "protocol": null,
  "srcaddr": null,
  "srcport": null,
  "start": 1431280876,
  "version": 2
}

Parse AWS VPC Flow log (custom format)

Source

parse_aws_vpc_flow_log!(
	"- eni-1235b8ca123456789 10.0.1.5 10.0.0.220 10.0.1.5 203.0.113.5",
	"instance_id interface_id srcaddr dstaddr pkt_srcaddr pkt_dstaddr"
)

Return

{
  "dstaddr": "10.0.0.220",
  "instance_id": null,
  "interface_id": "eni-1235b8ca123456789",
  "pkt_dstaddr": "203.0.113.5",
  "pkt_srcaddr": "10.0.1.5",
  "srcaddr": "10.0.1.5"
}

parse_common_log

fallible

Parses the value using the Common Log Format (CLF).

Function spec

parse_common_log(value: <string>, timestamp_format: <string>)

:: <object>

Argument	Type	Description
value	string	The string to parse.
timestamp_format	string	The date/time format to use for encoding the timestamp.

required optional <types | ...>

Examples

Parse via Common Log Format (with default timestamp format)

Source

parse_common_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326")

Return

{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}

Parse via Common Log Format (with custom timestamp format)

Source

parse_common_log!(
	"127.0.0.1 bob frank [2000-10-10T20:55:36Z] \"GET /apache_pb.gif HTTP/1.0\" 200 2326",
	"%+"
)

Return

{
  "host": "127.0.0.1",
  "identity": "bob",
  "message": "GET /apache_pb.gif HTTP/1.0",
  "method": "GET",
  "path": "/apache_pb.gif",
  "protocol": "HTTP/1.0",
  "size": 2326,
  "status": 200,
  "timestamp": "2000-10-10T20:55:36Z",
  "user": "frank"
}

Notices This function has special behavior that you should be aware of.

Missing information in the log message may be indicated by -. These fields are omitted in the result.

parse_csv

fallible

Parses a single CSV formatted row. Only the first row is parsed in case of multiline input value.

Function spec

parse_csv(value: <string>, delimiter: <string>)

:: <array>

Argument	Type	Description
value	string	The string to parse.
delimiter	string	The field delimiter to use when parsing. Must be a single-byte utf8 character.

required optional <types | ...>

Examples

Parse a single CSV formatted row

Source

parse_csv!("foo,bar,\"foo \"\", bar\"")

Return

[
  "foo",
  "bar",
  "foo \", bar"
]

Parse a single CSV formatted row with custom delimiter

Source

parse_csv!("foo bar", delimiter: " ")

Return

[
  "foo",
  "bar"
]

Notices This function has special behavior that you should be aware of.

All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

parse_duration

fallible

Parses the value into a human-readable duration format specified by unit.

Function spec

parse_duration(value: <string>, unit: <string>)

:: <float>

Argument	Type	Description
value	string	The string of the duration.
unit	string	The output units for the duration.

required optional <types | ...>

Examples

Parse duration (milliseconds)

Source

parse_duration!("1005ms", unit: "s")

Return

1.005

parse_glog

fallible

Parses the value using the glog (Google Logging Library) format.

Function spec

parse_glog(value: <string>)

:: <object>

Argument	Type	Description
value	string	The string to parse.

required optional <types | ...>

Examples

Parse via glog

Source

parse_glog!("I20210131 14:48:54.411655 15520 main.c++:9] Hello world!")

Return

{
  "file": "main.c++",
  "id": 15520,
  "level": "info",
  "line": 9,
  "message": "Hello world!",
  "timestamp": "2021-01-31T14:48:54.411655Z"
}

parse_grok

fallible

Parses the value using the grok format. All patterns listed here are supported.

Function spec

parse_grok(value: <string>, pattern: <string>, remove_empty: <boolean>)

:: <object>

Argument	Type	Description
value	string	The string to parse.
pattern	string	The Grok pattern.
remove_empty	boolean	If set to `true`, any patterns that resolve to an empty value are removed from the result.

required optional <types | ...>

Examples

Parse using Grok

Source

parse_grok!(
	"2020-10-02T23:22:12.223222Z info Hello world",
	"%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
)

Return

{
  "level": "info",
  "message": "Hello world",
  "timestamp": "2020-10-02T23:22:12.223222Z"
}

Notices This function has special behavior that you should be aware of.

We recommend using community-maintained Grok patterns when possible, as they’re more likely to be properly vetted and improved over time than bespoke patterns.

parse_int

fallible

Parses the string value representing a number in an optional base/radix to an integer.

Function spec

parse_int(value: <string>, base: <integer>)

:: <string>

Argument

Type

Description

value

string

The string to parse.

base

integer

The base the number is in. Must be between 2 and 36 (inclusive).

If unspecified, will use the string prefix to try to determine the base: “0b”, 8 for “0” or “0o”, 16 for “0x”, and 10 otherwise

required optional <types | ...>

Examples

Parse decimal

Source

parse_int!("-42")

Return

-42

Parse binary

Source

parse_int!("0b1001")

Return

Parse octal

Source

parse_int!("0o42")

Return

Parse hexdecimal

Source

parse_int!("0x2a")

Return

Parse explicit base

Source

parse_int!("2a", 17)

Return

parse_json

fallible

Parses the value as JSON.

Function spec

parse_json(value: <string>)

Argument	Type	Description
value	string	The string representation of the JSON to parse.

required optional <types | ...>

Examples

Parse JSON

Source

parse_json!("{\"key\": \"val\"}")

Return

{
  "key": "val"
}

Notices This function has special behavior that you should be aware of.

Only JSON types are returned. If you need to convert a string into a timestamp, consider the parse_timestamp function.

parse_key_value

fallible

Parses the value in key/value format. Also known as logfmt.

Keys and values can be wrapped with ".
" characters can be escaped using \.

Function spec

parse_key_value(value: <string>, key_value_delimiter: <string>, field_delimiter: <string>, whitespace: <string>, accept_standalone_key: <boolean>)

:: <object>

Argument	Type	Description
value	string	The string to parse.
key_value_delimiter	string	The string that separates the key from the value.
field_delimiter	string	The string that separates each key/value pair.
whitespace	string	Defines the acceptance of unnecessary whitespace surrounding the configured `key_value_delimiter`.
accept_standalone_key	boolean	Whether a standalone key should be accepted, the resulting object will associate such keys with boolean value `true`

required optional <types | ...>

Examples

Parse logfmt log

Source

parse_key_value!(
	"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)

Return

{
  "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
  "id": "ConsumerFetcherManager-1382721708341",
  "level": "info",
  "module": "kafka.consumer.ConsumerFetcherManager",
  "msg": "Stopping all fetchers",
  "tag#production": "stopping_fetchers"
}

Parse comma delimited log

Source

parse_key_value!(
	"path:\"/cart_link\", host:store.app.com, fwd: \"102.30.171.16\", dyno: web.1, connect:0ms, service:87ms, status:304, bytes:632, protocol:https",
	field_delimiter: ",",
	key_value_delimiter: ":"
)

Return

{
  "bytes": "632",
  "connect": "0ms",
  "dyno": "web.1",
  "fwd": "102.30.171.16",
  "host": "store.app.com",
  "path": "/cart_link",
  "protocol": "https",
  "service": "87ms",
  "status": "304"
}

Parse comma delimited log with standalone keys

Source

parse_key_value!(
	"env:prod,service:backend,region:eu-east1,beta",
	field_delimiter: ",",
	key_value_delimiter: ":",
)

Return

{
  "beta": true,
  "env": "prod",
  "region": "eu-east1",
  "service": "backend"
}

Notices This function has special behavior that you should be aware of.

All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

parse_klog

fallible

Parses the value using the klog format used by Kubernetes components.

Function spec

parse_klog(value: <string>)

:: <object>

Argument	Type	Description
value	string	The string to parse.

required optional <types | ...>

Examples

Parse via klog

Source

parse_klog!("I0505 17:59:40.692994   28133 klog.go:70] hello from klog")

Return

{
  "file": "klog.go",
  "id": 28133,
  "level": "info",
  "line": 70,
  "message": "hello from klog",
  "timestamp": "2021-05-05T17:59:40.692994Z"
}

parse_linux_authorization

fallible

Parses Linux authorization logs usually found under either /var/log/auth.log (for Debian-based systems) or /var/log/secure (for RedHat-based systems) according to Syslog format.

Function spec

parse_linux_authorization(value: <string>)

:: <object>

Argument	Type	Description
value	string	The text containing the message to parse.

required optional <types | ...>

Examples

Parse Linux authorization event

Source

parse_linux_authorization!(
	s'Mar 23 01:49:58 localhost sshd[1111]: Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar'
)

Return

{
  "appname": "sshd",
  "hostname": "localhost",
  "message": "Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar",
  "procid": 1111,
  "timestamp": "2021-03-23T01:49:58Z"
}

Notices This function has special behavior that you should be aware of.

The function resolves the year for messages that don’t include it. If the current month is January, and the message is for December, it will take the previous year. Otherwise, take the current year.

parse_logfmt

fallible

Parses the value in logfmt.

Keys and values can be wrapped using the " character.
" characters can be escaped by the \ character.
As per this logfmt specification, the parse_logfmt function accepts standalone keys and assigns them a Boolean value of true.

Function spec

parse_logfmt(value: <string>)

:: <object>

Argument	Type	Description
value	string	The string to parse.

required optional <types | ...>

Examples

Parse logfmt log

Source

parse_logfmt!(
	"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)

Return

{
  "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
  "id": "ConsumerFetcherManager-1382721708341",
  "level": "info",
  "module": "kafka.consumer.ConsumerFetcherManager",
  "msg": "Stopping all fetchers",
  "tag#production": "stopping_fetchers"
}

Notices This function has special behavior that you should be aware of.

If fields_ordering is specified then the function is fallible else it is infallible.

parse_nginx_log

fallible

Parses Nginx access and error log lines. Lines can be in combined, or error format.

Function spec

parse_nginx_log(value: <string>, timestamp_format: <string>, format: <string>)

:: <object>

Argument	Type	Description
value	string	The string to parse.
timestamp_format	string	The date/time format to use for encoding the timestamp. The time is parsed in local time if the timestamp doesn’t specify a timezone. The default format is `%d/%b/%Y:%T %z` for combined logs and `%Y/%m/%d %H:%M:%S` for error logs.
format	string	The format to use for parsing the log.

required optional <types | ...>

Examples

Parse via Nginx log format (combined)

Source

parse_nginx_log!(
    s'172.17.0.1 alice - [01/Apr/2021:12:02:31 +0000] "POST /not-found HTTP/1.1" 404 153 "http://localhost/somewhere" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"',
    "combined",
)

Return

{
  "agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36",
  "client": "172.17.0.1",
  "compression": "2.75",
  "method": "POST",
  "path": "/not-found",
  "protocol": "HTTP/1.1",
  "referer": "http://localhost/somewhere",
  "request": "POST /not-found HTTP/1.1",
  "size": 153,
  "status": 404,
  "timestamp": "2021-04-01T12:02:31Z",
  "user": "alice"
}

Parse via Nginx log format (error)

Source

parse_nginx_log!(
    s'2021/04/01 13:02:31 [error] 31#31: *1 open() "/usr/share/nginx/html/not-found" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "POST /not-found HTTP/1.1", host: "localhost:8081"',
    "error"
)

Return

{
  "cid": 1,
  "client": "172.17.0.1",
  "host": "localhost:8081",
  "message": "open() \"/usr/share/nginx/html/not-found\" failed (2: No such file or directory)",
  "pid": 31,
  "request": "POST /not-found HTTP/1.1",
  "server": "localhost",
  "severity": "error",
  "tid": 31,
  "timestamp": "2021-04-01T13:02:31Z"
}

Notices This function has special behavior that you should be aware of.

Missing information in the log message may be indicated by -. These fields are omitted in the result.

parse_query_string

infallible

Parses the value as a query string.

Function spec

parse_query_string(value: <string>)

:: <object>

Argument	Type	Description
value	string	The string to parse.

required optional <types | ...>

Examples

Parse query string

Source

parse_query_string("foo=%2B1&bar=2&bar=3&xyz")

Return

{
  "bar": [
    "2",
    "3"
  ],
  "foo": "+1",
  "xyz": ""
}

Parse Ruby on Rails' query string

Source

parse_query_string("?foo%5b%5d=1&foo%5b%5d=2")

Return

{
  "foo[]": [
    "1",
    "2"
  ]
}

Notices This function has special behavior that you should be aware of.

All values are returned as strings. We recommend manually coercing values to desired types as you see fit. Empty keys and values are allowed.

parse_regex

fallible

Parses the value via the provided Regex pattern.

This function differs from the parse_regex_all function in that it returns only the first match.

Function spec

parse_regex(value: <string>, pattern: <regex>, numeric_groups: <regex>)

:: <object>

Argument	Type	Description
value	string	The string to search.
pattern	regex	The regular expression pattern to search against.
numeric_groups	regex	If true, the index of each group in the regular expression is also captured. The 0th index will contain the whole match.

required optional <types | ...>

Examples

Parse using Regex (with capture groups)

Source

parse_regex!("first group and second group.", r'(?P<number>.*?) group')

Return

{
  "number": "first"
}

Parse using Regex (without capture groups)

Source

parse_regex!("first group and second group.", r'(\w+) group', numeric_groups: true)

Return

{
  "0": "first group",
  "1": "first"
}

Notices This function has special behavior that you should be aware of.

VRL aims to provide purpose-specific parsing functions for common log formats. Before reaching for the parse_regex function, see if a VRL parse_* function already exists for your format. If not, we recommend opening an issue to request support for the desired format.

All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

parse_regex_all

fallible

Parses the value via the provided Regex pattern.

This function differs from the parse_regex function in that it returns all matches, not just the first.

Function spec

parse_regex_all(value: <string>, pattern: <regex>, numeric_groups: <regex>)

:: <array>

Argument	Type	Description
value	string	The string to search.
pattern	regex	The regular expression pattern to search against.
numeric_groups	regex	If `true`, the index of each group in the regular expression is also captured. The 0th index contains the whole match.

required optional <types | ...>

Examples

Parse using Regex (all matches)

Source

parse_regex_all!("first group and second group.", r'(?P<number>\w+) group', numeric_groups: true)

Return

[
  {
    "0": "first group",
    "1": "first",
    "number": "first"
  },
  {
    "0": "second group",
    "1": "second",
    "number": "second"
  }
]

Notices This function has special behavior that you should be aware of.

All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

parse_ruby_hash

fallible

Parses the value as ruby hash.

Function spec

parse_ruby_hash(value: <string>)

:: <object>

Argument	Type	Description
value	string	The string representation of the ruby hash to parse.

required optional <types | ...>

Examples

Parse ruby hash

Source

parse_ruby_hash!(s'{ "test" => "value", "testNum" => 0.2, "testObj" => { "testBool" => true, "testNull" => nil } }')

Return

{
  "test": "value",
  "testNum": 0.2,
  "testObj": {
    "testBool": true,
    "testNull": null
  }
}

Notices This function has special behavior that you should be aware of.

Only ruby types are returned. If you need to convert a string into a timestamp, consider the parse_timestamp function.

parse_syslog

fallible

Parses the value in Syslog format.

Function spec

parse_syslog(value: <string>)

:: <object>

Argument	Type	Description
value	string	The text containing the Syslog message to parse.

required optional <types | ...>

Examples

Parse Syslog log (5424)

Source

parse_syslog!(
	s'<13>1 2020-03-13T20:45:38.119Z dynamicwireless.name non 2426 ID931 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"] Try to override the THX port, maybe it will reboot the neural interface!'
)

Return

{
  "appname": "non",
  "exampleSDID@32473.eventID": "1011",
  "exampleSDID@32473.eventSource": "Application",
  "exampleSDID@32473.iut": "3",
  "facility": "user",
  "hostname": "dynamicwireless.name",
  "message": "Try to override the THX port, maybe it will reboot the neural interface!",
  "msgid": "ID931",
  "procid": 2426,
  "severity": "notice",
  "timestamp": "2020-03-13T20:45:38.119Z",
  "version": 1
}

Notices This function has special behavior that you should be aware of.

The function makes a best effort to parse the various Syslog formats that exists out in the wild. This includes RFC 6587, RFC 5424, RFC 3164, and other common variations (such as the Nginx Syslog style).

All values are returned as strings. We recommend manually coercing values to desired types as you see fit.

parse_timestamp

fallible

Parses the value in strptime format.

Function spec

parse_timestamp(value: <string>, format: <string>)

:: <timestamp>

Argument	Type	Description
value	string	The text of the timestamp.
format	string	The strptime format.

required optional <types | ...>

Examples

Parse timestamp

Source

parse_timestamp!("10-Oct-2020 16:00+00:00", format: "%v %R %:z")

Return

2020-10-10T16:00:00Z

parse_tokens

fallible

Parses the value in “token” format. A token is considered to be one of the following:

A word surrounded by whitespace.
Text delimited by double quotes: "..". Quotes can be included in the token if they are escaped by a backslash (\).
Text delimited by square brackets: [..]. Closing square brackets can be included in the token if they are escaped by a backslash (\).

Function spec

parse_tokens(value: <string>)

:: <array>

Argument	Type	Description
value	string	The string to tokenize.

required optional <types | ...>

Examples

Parse tokens

Source

parse_tokens(
	"A sentence \"with \\\"a\\\" sentence inside\" and [some brackets]"
)

Return

[
  "A",
  "sentence",
  "with \\\"a\\\" sentence inside",
  "and",
  "some brackets"
]

Notices This function has special behavior that you should be aware of.

All token values are returned as strings. We recommend manually coercing values to desired types as you see fit.

parse_url

fallible

Parses the value in URL format.

Function spec

parse_url(value: <string>)

:: <object>

Argument	Type	Description
value	string	The text of the URL.

required optional <types | ...>

Examples

Parse URL

Source

parse_url!("ftp://foo:bar@vector.dev:4343/foobar?hello=world#123")

Return

{
  "fragment": "123",
  "host": "vector.dev",
  "password": "bar",
  "path": "/foobar",
  "port": 4343,
  "query": {
    "hello": "world"
  },
  "scheme": "ftp",
  "username": "foo"
}

parse_xml

fallible

Parses the value as XML.

Function spec

parse_xml(value: <string>, include_attr: <boolean>, attr_prefix: <string>, text_key: <string>, always_use_text_key: <boolean>, parse_bool: <boolean>, parse_null: <boolean>, parse_number: <boolean>)

:: <object>

Argument	Type	Description
value	string	The string representation of the XML document to parse.
include_attr	boolean	Include XML tag attributes in the returned object.
attr_prefix	string	String prefix to use for XML tag attribute keys.
text_key	string	Key name to use for expanded text nodes.
always_use_text_key	boolean	Always return text nodes as `{"<text_key>": "value"}.`
parse_bool	boolean	Parse “true” and “false” as boolean.
parse_null	boolean	Parse “null” as null.
parse_number	boolean	Parse numbers as integers/floats.

required optional <types | ...>

Examples

Parse XML

Source

value = s'<book category="CHILDREN"><title lang="en">Harry Potter</title><author>J K. Rowling</author><year>2005</year></book>';

parse_xml!(value, text_key: "value", parse_number: false)

Return

{
  "book": {
    "@category": "CHILDREN",
    "author": "J K. Rowling",
    "title": {
      "@lang": "en",
      "value": "Harry Potter"
    },
    "year": "2005"
  }
}

Notices This function has special behavior that you should be aware of.

Valid XML must contain exactly one root node. Always returns an object.

Random functions

uuid_v4

infallible

Generates a random UUIDv4 string.

Function spec

uuid_v4()

:: <string>

required optional <types | ...>

Examples

Create a UUIDv4

Source

uuid_v4()

Return

1d262f4f-199b-458d-879f-05fd0a5f0683

String functions

contains

infallible

Determines whether the value string contains the specified substring.

Function spec

contains(value: <string>, substring: <string>, case_sensitive: <boolean>)

:: <boolean>

Argument	Type	Description
value	string	The text to search.
substring	string	The substring to search for in `value`.
case_sensitive	boolean	Whether the match should be case sensitive.

required optional <types | ...>

Examples

String contains (case sensitive)

Source

contains("The Needle In The Haystack", "Needle")

Return

true

String contains (case insensitive)

Source

contains("The Needle In The Haystack", "needle", case_sensitive: false)

Return

true

downcase

infallible

Downcases the value string, where “downcase” is defined according to the terms of the Unicode Derived Core Property Lowercase.

Function spec

downcase(value: <string>)

:: <string>

Argument	Type	Description
value	string	The string to convert to lowercase.

required optional <types | ...>

Examples

Downcase a string

Source

downcase("Hello, World!")

Return

hello, world!

ends_with

infallible

Determines whether the value string ends with the specified substring.

Function spec

ends_with(value: <string>, substring: <string>, case_sensitive: <boolean>)

:: <boolean>

Argument	Type	Description
value	string	The string to search.
substring	string	The substring with which `value` must end.
case_sensitive	boolean	Whether the match should be case sensitive.

required optional <types | ...>

Examples

String ends with (case sensitive)

Source

ends_with("The Needle In The Haystack", "The Haystack")

Return

true

String ends with (case insensitive)

Source

ends_with("The Needle In The Haystack", "the haystack", case_sensitive: false)

Return

true

join

infallible

Joins each string in the value array into a single string, with items optionally separated from one another by a separator.

Function spec

join(value: <array>, separator: <string>)

:: <string>

Argument	Type	Description
value	array	The array of strings to join together.
separator	string	The string separating each original element when joined.

required optional <types | ...>

Examples

Join array (no separator)

Source

join!(["bring", "us", "together"])

Return

bringustogether

Join array (comma separator)

Source

join!(["sources", "transforms", "sinks"], separator: ", ")

Return

sources, transforms, sinks

match

infallible

Determines whether the value matches the pattern.

Function spec

match(value: <string>, pattern: <regex>)

:: <boolean>

Argument	Type	Description
value	string	The value to match.
pattern	regex	The regular expression pattern to match against.

required optional <types | ...>

Examples

Regex match on a string

Source

match("I'm a little teapot", r'teapot')

Return

true

match_any

infallible

Determines whether the value matches any the given patterns. All patterns are checked in a single pass over the target string, giving this function a potentially significant performance advantage over multiple calls to match.

Function spec

match_any(value: <string>, patterns: <array>)

:: <boolean>

Argument	Type	Description
value	string	The value to match.
patterns	array	The array of regular expression patterns to match against.

required optional <types | ...>

Examples

Regex match on a string

Source

match_any("I'm a little teapot", [r'frying pan', r'teapot'])

Return

true

redact

infallible

Redact sensitive data in value such as:

US social security card numbers
and other forms of personally identifiable information via custom patterns
(more to come!)

This can help achieve compliance by ensuring sensitive data never leaves your network.

Function spec

redact(value: <string | object | array>, filters: <array>)

:: <string | object | array>

Argument Type Description

value

string
object
array

The value to redact sensitive data from.

Its behavior differs depending on the type of value:

For strings, it simply redacts the sensitive data and returns a new string
For arrays, it redacts the sensitive data in each string element
For objects, it masks the sensitive data in each string value, but not keys

For arrays and objects it will recurse into any nested arrays or objects. Any non-string elements will be skipped.

Any redacted text will be replaced with [REDACTED].

filters

array

List of filters to be applied to the value.

Each filter can be specified in one of three ways:

As a regular expression directly, which will be used to redact text matching it
As an object with a type key that corresponds to a named filter and additional keys for customizing that filter
As a named filter, if it has no required parameters

Named filters are:

pattern: Redact text matching any regular expressions specified in the, required, patterns key. This is the expanded form of just passing a regular expression as a filter.
us_social_security_number: Redact US social security card numbers.

See examples for more details.

This parameter must be a static expression. You cannot use variables or other dynamic expressions with it. This allows us to validate the argument at compile-time to avoid runtime errors.

required optional <types | ...>

Examples

Replace text using a regex

Source

redact("my id is 123456", filters: [r'\d+'])

Return

my id is [REDACTED]

Replace us social security numbers in any field

Source

redact({ "name": "John Doe", "ssn": "123-12-1234"}, filters: ["us_social_security_number"])

Return

{
  "name": "John Doe",
  "ssn": "[REDACTED]"
}

replace

infallible

Replaces all matching instances of pattern in the value.

Function spec

replace(value: <string>, pattern: <regex | string>, with: <string>, count: <integer>)

:: <string>

Argument	Type	Description
value	string	The original string.
pattern	regex string	Replace all matches of this pattern. Can be a static string or a regular expression.
with	string	The string that the matches are replaced with.
count	integer	The maximum number of replacements to perform. -1 means replace all matches.

required optional <types | ...>

Examples

Replace literal text

Source

replace("Apples and Bananas", "and", "not")

Return

Apples not Bananas

Replace via regular expression

Source

replace("Apples and Bananas", r'(?i)bananas', "Pineapples")

Return

Apples and Pineapples

Replace first instance

Source

replace("Bananas and Bananas", "Bananas", "Pineapples", count: 1)

Return

Pineapples and Bananas

slice

infallible

Returns a slice of the value between the start and end positions.

If the start and end parameters are negative, they refer to positions counting from the right of the string or array. If end refers to a position that is greater than the length of the string or array a slice up to the end of the string or array is returned.

Function spec

slice(value: <array | string>, start: <integer>, end: <integer>)

:: <string>

Argument	Type	Description
value	array string	The string or array to slice.
start	integer	The inclusive start position. A zero-based index that can be negative.
end	integer	The inclusive end position. A zero-based index that can be negative.

required optional <types | ...>

Examples

Slice a string (positve index)

Source

slice!("Supercalifragilisticexpialidocious", start: 5, end: 13)

Return

califrag

Slice a string (negative index)

Source

slice!("Supercalifragilisticexpialidocious", start: 5, end: -14)

Return

califragilistic

split

infallible

Splits the value string via the pattern.

Function spec

split(value: <string>, pattern: <string | regex>, limit: <integer>)

:: <string>

Argument	Type	Description
value	string	The string to split.
pattern	string regex	The string is split whenever this pattern is matched.
limit	integer	The maximum number of substrings to return.

required optional <types | ...>

Examples

Split a string (no limit)

Source

split("apples and pears and bananas", " and ")

Return

[
  "apples",
  "pears",
  "bananas"
]

Split a string (with a limit)

Source

split("apples and pears and bananas", " and ", limit: 2)

Return

[
  "apples",
  "pears and bananas"
]

starts_with

infallible

Determines whether the value begins with the substring.

Function spec

starts_with(value: <string>, substring: <string>, case_sensitive: <boolean>)

:: <boolean>

Argument	Type	Description
value	string	The string to search.
substring	string	The substring that the `value` must start with.
case_sensitive	boolean	Whether the match should be case sensitive.

required optional <types | ...>

Examples

String starts with (case sensitive)

Source

starts_with("The Needle In The Haystack", "The Needle")

Return

true

String starts with (case insensitive)

Source

starts_with("The Needle In The Haystack", "the needle", case_sensitive: false)

Return

true

strip_ansi_escape_codes

infallible

Strips ANSI escape codes from the value.

Function spec

strip_ansi_escape_codes(value: <string>)

:: <string>

Argument	Type	Description
value	string	The string to strip.

required optional <types | ...>

Examples

Strip ANSI escape codes

Source

strip_ansi_escape_codes("\e[46mfoo\e[0m bar")

Return

foo bar

strip_whitespace

infallible

Strips whitespace from the start and end of the value, where whitespace is defined by the Unicode White_Space property

Function spec

strip_whitespace(value: <string>)

:: <string>

Argument	Type	Description
value	string	The string to trim.

required optional <types | ...>

Examples

Strip whitespace

Source

strip_whitespace("  A sentence.  ")

Return

A sentence.

truncate

infallible

Truncates the value string up to the limit number of characters.

Function spec

truncate(value: <string>, limit: <integer | float>, ellipsis: <boolean>)

:: <string>

Argument	Type	Description
value	string	The string to truncate.
limit	integer float	The number of characters to truncate the string after.
ellipsis	boolean	An ellipsis (`...`) is appended if this is set to `true` and the `value` string ends up being truncated because it’s exceeded the `limit`.

required optional <types | ...>

Examples

Truncate a string

Source

truncate("A rather long sentence.", limit: 11, ellipsis: true)

Return

A rather lo...

upcase

infallible

Upcases the value, where “upcase” is defined according to the terms of the Unicode Derived Core Property Uppercase.

Function spec

upcase(value: <string>)

:: <string>

Argument	Type	Description
value	string	The string to convert to uppercase.

required optional <types | ...>

Examples

Upcase a string

Source

upcase("Hello, World!")

Return

HELLO, WORLD!

System functions

get_env_var

fallible

Returns the value of the environment variable specifed by name.

Function spec

get_env_var(name: <string>)

:: <string>

Argument	Type	Description
name	string	The name of the environment variable.

required optional <types | ...>

Examples

Get an environment variable

Source

get_env_var!("HOME")

Return

/root

get_hostname

fallible

Returns the local system’s hostname.

Function spec

get_hostname()

:: <string>

required optional <types | ...>

Examples

Get hostname

Source

.hostname = get_hostname!()

Timestamp functions

format_timestamp

infallible

Formats the value into a string representation of the timestamp.

Function spec

format_timestamp(value: <timestamp>, format: <string>)

:: <string>

Argument	Type	Description
value	timestamp	The timestamp to format as text.
format	string	The format string as decribed by the Chrono library.

required optional <types | ...>

Examples

Format a timestamp (ISO8601/RFC 3339)

Source

format_timestamp!(t'2020-10-21T16:00:00Z', format: "%+")

Return

2020-10-21T16:00:00+00:00

Format a timestamp (custom)

Source

format_timestamp!(t'2020-10-21T16:00:00Z', format: "%v %R")

Return

21-Oct-2020 16:00

now

infallible

Returns the current timestamp in the UTC timezone with nanosecond precision.

Function spec

now()

:: <timestamp>

required optional <types | ...>

Examples

Generate a current timestamp

Source

now()

Return

2021-03-04T10:51:15.928937Z

Type functions

array

fallible

Returns the value if it’s an array and errors otherwise. This enables the type checker to guarantee that the returned value is an array and can be used in any function that expects one.

Function spec

array(value: <any>)

:: <array>

Argument	Type	Description
value	any	The value that you need to ensure is an array.

required optional <types | ...>

Examples

Declare an array type

Source

array!(.value)

Return

[
  1,
  2,
  3
]

bool

fallible

Returns the value if it’s a Boolean and errors otherwise. This enables the type checker to guarantee that the returned value is a Boolean and can be used in any function that expects one.

Function spec

bool(value: <any>)

:: <boolean>

Argument	Type	Description
value	any	The value that you need to ensure is a Boolean.

required optional <types | ...>

Examples

Declare a Boolean type

Source

bool!(.value)

float

fallible

Returns the value if it’s a float and errors otherwise. This enables the type checker to guarantee that the returned value is a float and can be used in any function that expects one.

Function spec

float(value: <any>)

:: <float>

Argument	Type	Description
value	any	The value that you need to ensure is a float.

required optional <types | ...>

Examples

Declare a float type

Source

float!(.value)

Return

int

fallible

Returns the value if it’s an integer and errors otherwise. This enables the type checker to guarantee that the returned value is an integer and can be used in any function that expects one.

Function spec

int(value: <any>)

:: <integer>

Argument	Type	Description
value	any	The value that you need to ensure is an integer.

required optional <types | ...>

Examples

Declare an integer type

Source

int!(.value)

Return

is_array

infallible

Check if the type of a value is an array or not.

Function spec

is_array(value: <any>)

:: <boolean>

Argument	Type	Description
value	any	The value to check

required optional <types | ...>

Examples

Valid array

Source

is_array([1, 2, 3])

Return

true

Non-matching type

Source

is_array("a string")

is_boolean

infallible

Check if the type of a value is a boolean or not.

Function spec

is_boolean(value: <any>)

:: <boolean>

Argument	Type	Description
value	any	The value to check

required optional <types | ...>

Examples

Valid boolean

Source

is_boolean(false)

Return

true

Non-matching type

Source

is_boolean("a string")

is_float

infallible

Check if the type of a value is a float or not.

Function spec

is_float(value: <any>)

:: <boolean>

Argument	Type	Description
value	any	The value to check

required optional <types | ...>

Examples

Valid float

Source

is_float(0.577)

Return

true

Non-matching type

Source

is_float("a string")

is_integer

infallible

Check if the type of a value is an integer or not.

Function spec

is_integer(value: <any>)

:: <boolean>

Argument	Type	Description
value	any	The value to check

required optional <types | ...>

Examples

Valid integer

Source

is_integer(1)

Return

true

Non-matching type

Source

is_integer("a string")

is_null

infallible

Check if the type of a value is null or not. For a more relaxed function please check is_nullish.

Function spec

is_null(value: <any>)

:: <boolean>

Argument	Type	Description
value	any	The value to check

required optional <types | ...>

Examples

Null value

Source

is_null(null)

Return

true

Non-matching type

Source

is_null("a string")

is_nullish

infallible

Determines whether the value is “nullish,” where nullish denotes the absence of a meaningful value.

Function spec

is_nullish(value: <any>)

:: <boolean>

Argument	Type	Description
value	any	The value to check for “nullishness,” i.e. a useless value.

required optional <types | ...>

Examples

Null detection (blank string)

Source

is_nullish("")

Return

true

Null detection (dash string)

Source

is_nullish("-")

Return

true

Null detection (whitespace)

Source

is_nullish("
  
")

Return

true

is_object

infallible

Check if the type of a value is an object or not.

Function spec

is_object(value: <any>)

:: <boolean>

Argument	Type	Description
value	any	The value to check

required optional <types | ...>

Examples

Valid object

Source

is_object({"foo": "bar"})

Return

true

Non-matching type

Source

is_object("a string")

is_regex

infallible

Check if the type of a value is a regex or not.

Function spec

is_regex(value: <any>)

:: <boolean>

Argument	Type	Description
value	any	The value to check

required optional <types | ...>

Examples

Valid regex

Source

is_regex(r'pattern')

Return

true

Non-matching type

Source

is_regex("a string")

is_string

infallible

Check if the type of a value is a string or not.

Function spec

is_string(value: <any>)

:: <boolean>

Argument	Type	Description
value	any	The value to check

required optional <types | ...>

Examples

Valid string

Source

is_string("a string")

Return

true

Non-matching type

Source

is_string([1, 2, 3])

is_timestamp

infallible

Check if the type of a value is a timestamp or not.

Function spec

is_timestamp(value: <any>)

:: <boolean>

Argument	Type	Description
value	any	The value to check

required optional <types | ...>

Examples

Valid timestamp

Source

is_timestamp(t'2021-03-26T16:00:00Z')

Return

true

Non-matching type

Source

is_timestamp("a string")

object

fallible

Returns the value if it’s an object and errors otherwise. This enables the type checker to guarantee that the returned value is an object and can be used in any function that expects one.

Function spec

object(value: <any>)

:: <object>

Argument	Type	Description
value	any	The value that you need to ensure is an object.

required optional <types | ...>

Examples

Declare an object type

Source

object!(.value)

Return

{
  "field1": "value1",
  "field2": "value2"
}

string

fallible

Returns the value if it’s a string and errors otherwise. This enables the type checker to guarantee that the returned value is a string and can be used in any function that expects one.

Function spec

string(value: <any>)

:: <string>

Argument	Type	Description
value	any	The value that you need to ensure is a string.

required optional <types | ...>

Examples

Declare a string type

Source

string!(.message)

Return

{"field": "value"}

tag_types_externally

infallible

Adds type information to all (nested) scalar values in the provided value.

The type information is added externally, meaning that value has the shape of "type": value after this transformation.

Function spec

tag_types_externally(value: <any>)

:: <object | array | null>

Argument	Type	Description
value	any	The value to tag with types.

required optional <types | ...>

Examples

Tag types externally (scalar)

Source

tag_types_externally(123)

Return

{
  "integer": 123
}

Tag types externally (object)

Source

tag_types_externally({
	"message": "Hello world",
	"request": {
		"duration_ms": 67.9
	}
})

Return

{
  "message": {
    "string": "Hello world"
  },
  "request": {
    "duration_ms": {
      "float": 67.9
    }
  }
}

Tag types externally (array)

Source

tag_types_externally(["foo", "bar"])

Return

[
  {
    "string": "foo"
  },
  {
    "string": "bar"
  }
]

Tag types externally (null)

Source

tag_types_externally(null)

timestamp

fallible

Returns the value if it’s a timestamp and errors otherwise. This enables the type checker to guarantee that the returned value is a timestamp and can be used in any function that expects one.

Function spec

timestamp(value: <any>)

:: <timestamp>

Argument	Type	Description
value	any	The value that you need to ensure is a timestamp.

required optional <types | ...>

Examples

Declare a timestamp type

Source

ok, err = timestamp(.timestamp)

Return

function call error for "timestamp" at (10:31): expected "timestamp", got "string"