VRL function reference
Here you’ll find a comprehensive list of all built-in VRL functions. Functions are categorized by their purpose and sorted alphabetically for easy discovery. To use these functions in Vector, see the documentation on function call expressions and Vector’s remap
transform.
Array functions
append
infallible pureitems
array to the end of the value
array.chunks
fallible purevalue
into slices of length chunk_size
bytes.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | array string | The array of bytes to split. | yes | |
chunk_size | integer | The desired length of each chunk in bytes. This may be constrained by the host platform architecture. | yes |
Errors
Thechunks
function is fallible, which means that
error handling is required for these errors:chunk_size
must be at least 1 byte.chunk_size
is too large.push
infallible pureitem
to the end of the value
array.Codec functions
decode_base16
fallible purevalue
(a Base16 string) into its original string.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The Base16 data to decode. | yes |
Errors
Thedecode_base16
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid encoded Base16 string.decode_base64
fallible purevalue
(a Base64 string) into its original string.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The Base64 data to decode. | yes | |
charset | string | The character set to use when decoding the data. | standard | no |
Errors
Thedecode_base64
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid encoded Base64 string.decode_gzip
fallible purevalue
(a Gzip string) into its original string.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The Gzip data to decode. | yes |
Errors
Thedecode_gzip
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid encoded Gzip string.decode_mime_q
fallible purevalue
with their original string.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string with encoded-words to decode. | yes |
Errors
Thedecode_mime_q
function is fallible, which means that
error handling is required for these errors:value
has invalid encoded encoded-word string.decode_percent
infallible purevalue
like a URL.decode_punycode
fallible purevalue
, such as an internationalized domain name (IDN). This function assumes that the value passed is meant to be used in IDN context and that it is either a domain name or a part of it.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to decode. | yes | |
validate | boolean | If enabled, checks if the input string is a valid domain name. | true | no |
Errors
Thedecode_punycode
function is fallible, which means that
error handling is required for these errors:value
is not valid punycode
decode_snappy
fallible purevalue
(a Snappy string) into its original string.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The Snappy data to decode. | yes |
Errors
Thedecode_snappy
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid encoded Snappy string.decode_zlib
fallible purevalue
(a Zlib string) into its original string.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The Zlib data to decode. | yes |
Errors
Thedecode_zlib
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid encoded Zlib string.decode_zstd
fallible purevalue
(a Zstandard string) into its original string.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The Zstandard data to decode. | yes |
Errors
Thedecode_zstd
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid encoded Zstd string.encode_base16
infallible purevalue
to Base16.encode_base64
infallible purevalue
to Base64.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to encode. | yes | |
padding | boolean | Whether the Base64 output is padded. | true | no |
charset | string | The character set to use when encoding the data. | standard | no |
encode_gzip
infallible purevalue
to Gzip.encode_json
infallible purevalue
to JSON.encode_key_value
fallible purevalue
into key-value format with customizable delimiters. Default delimiters match
the logfmt format.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | object | The value to convert to a string. | yes | |
fields_ordering | array | The ordering of fields to preserve. Any fields not in this list are listed unordered, after all ordered fields. | no | |
key_value_delimiter | string | The string that separates the key from the value. | = | no |
field_delimiter | string | The string that separates each key-value pair. |
| no |
flatten_boolean | boolean | Whether to encode key-value with a boolean value as a standalone key if true and nothing if false . | no |
Notices
This function has special behavior that you should be aware of.fields_ordering
is specified then the function is fallible else it is infallible.Errors
Theencode_key_value
function is fallible, which means that
error handling is required for these errors:fields_ordering
contains a non-string element.Examples
Encode with default delimiters (no ordering)
encode_key_value({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z
Encode with default delimiters (fields ordering)
encode_key_value!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345
Encode with default delimiters (nested fields)
encode_key_value({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
agent.name=foo event=log log.file.path=my.log
Encode with default delimiters (nested fields ordering)
encode_key_value!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
event=log log.file.path=my.log agent.name=foo
Encode with custom delimiters (no ordering)
encode_key_value(
{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"},
field_delimiter: ",",
key_value_delimiter: ":"
)
lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z
Encode with custom delimiters and flatten boolean
encode_key_value(
{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "beta": true, "dropped": false},
field_delimiter: ",",
key_value_delimiter: ":",
flatten_boolean: true
)
beta,lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z
encode_logfmt
fallible purevalue
to logfmt.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | object | The value to convert to a logfmt string. | yes | |
fields_ordering | array | The ordering of fields to preserve. Any fields not in this list are listed unordered, after all ordered fields. | no |
Notices
This function has special behavior that you should be aware of.fields_ordering
is specified then the function is fallible else it is infallible.Errors
Theencode_logfmt
function is fallible, which means that
error handling is required for these errors:fields_ordering
contains a non-string element.Examples
Encode to logfmt (no ordering)
encode_logfmt({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z
Encode to logfmt (fields ordering)
encode_logfmt!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345
encode_percent
infallible purevalue
with percent encoding to safely be used in URLs.encode_proto
fallible purevalue
into a protocol buffer payload.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | object | The object to convert to a protocol buffer payload. | yes | |
desc_file | string | The path to the protobuf descriptor set file. Must be a literal string. This file is the output of protoc -o | yes | |
message_type | string | The name of the message type to use for serializing. Must be a literal string. | yes |
Errors
Theencode_proto
function is fallible, which means that
error handling is required for these errors:desc_file
file does not exist.message_type
message type does not exist in the descriptor file.encode_punycode
fallible purevalue
to punycode. Useful for internationalized domain names (IDN). This function assumes that the value passed is meant to be used in IDN context and that it is either a domain name or a part of it.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to encode. | yes | |
validate | boolean | Whether to validate the input string to check if it is a valid domain name. | true | no |
Errors
Theencode_punycode
function is fallible, which means that
error handling is required for these errors:value
can not be encoded to punycode
encode_snappy
fallible purevalue
to Snappy.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to encode. | yes |
Errors
Theencode_snappy
function is fallible, which means that
error handling is required for these errors:value
cannot be encoded into a Snappy string.encode_zlib
infallible purevalue
to Zlib.encode_zstd
infallible purevalue
to Zstandard.Coerce functions
to_bool
fallible purevalue
into a boolean.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | boolean integer float null string | The value to convert to a Boolean. | yes |
Errors
Theto_bool
function is fallible, which means that
error handling is required for these errors:value
is not a supported boolean representation.to_float
fallible purevalue
into a float.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer float boolean string timestamp | The value to convert to a float. Must be convertible to a float, otherwise an error is raised. | yes |
Notices
This function has special behavior that you should be aware of.to_float
will return an error for non-normal floats whereas parse_float can return inf
which might result in unsafe arithmetic operations that panic. This will be addressed in the future.Errors
Theto_float
function is fallible, which means that
error handling is required for these errors:value
is not a supported float representation.to_int
fallible purevalue
into an integer.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer float boolean string timestamp null | The value to convert to an integer. | yes |
Errors
Theto_int
function is fallible, which means that
error handling is required for these errors:value
is a string but the text is not an integer.value
is not a string, int, or timestamp.to_regex
fallible purevalue
into a regex.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The value to convert to a regex. | yes |
Notices
This function has special behavior that you should be aware of.Errors
Theto_regex
function is fallible, which means that
error handling is required for these errors:value
is not a string.to_string
fallible purevalue
into a string.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer float boolean string timestamp null | The value to convert to a string. | yes |
Errors
Theto_string
function is fallible, which means that
error handling is required for these errors:value
is not an integer, float, boolean, string, timestamp, or null.Convert functions
from_unix_timestamp
infallible pureConverts the value
integer from a Unix timestamp to a VRL timestamp
.
Converts from the number of seconds since the Unix epoch by default. To convert from milliseconds or nanoseconds, set the unit
argument to milliseconds
or nanoseconds
.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer | The Unix timestamp to convert. | yes | |
unit | string | The time unit. | seconds | no |
to_syslog_facility
fallible purevalue
, a Syslog facility code, into its corresponding
Syslog keyword. For example, 0
into "kern"
, 1
into "user"
, etc.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer | The facility code. | yes |
Errors
Theto_syslog_facility
function is fallible, which means that
error handling is required for these errors:value
is not a valid Syslog facility code.to_syslog_level
fallible purevalue
, a Syslog severity level, into its corresponding keyword,
i.e. 0 into "emerg"
, 1 into "alert"
, etc.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer | The severity level. | yes |
Errors
Theto_syslog_level
function is fallible, which means that
error handling is required for these errors:value
isn’t a valid Syslog severity level.to_syslog_severity
fallible pureFunction spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The Syslog level keyword to convert. | yes |
Errors
Theto_syslog_severity
function is fallible, which means that
error handling is required for these errors:value
is not a valid Syslog level keyword.to_unix_timestamp
infallible pureConverts the value
timestamp into a Unix timestamp.
Returns the number of seconds since the Unix epoch by default. To return the number in milliseconds or nanoseconds, set the unit
argument to milliseconds
or nanoseconds
.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | timestamp | The timestamp to convert into a Unix timestamp. | yes | |
unit | string | The time unit. | seconds | no |
Debug functions
assert
fallible impurecondition
, which must be a Boolean expression. The program is aborted with
message
if the condition evaluates to false
.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
condition | boolean | The condition to check. | yes | |
message | string | An optional custom error message. If the equality assertion fails, message is
appended to the default message prefix. See the examples below
for a fully formed log message sample. | no |
Notices
This function has special behavior that you should be aware of.assert
function should be used in a standalone fashion and only when you want to abort the program. You
should avoid it in logical expressions and other situations in which you want the program to continue if the
condition evaluates to false
.Errors
Theassert
function is fallible, which means that
error handling is required for these errors:condition
evaluates to false
.assert_eq
infallible impureleft
and right
, have the same value. The program is
aborted with message
if they do not have the same value.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
left | any | The value to check for equality against right . | yes | |
right | any | The value to check for equality against left . | yes | |
message | string | An optional custom error message. If the equality assertion fails, message is
appended to the default message prefix. See the examples
below for a fully formed log message sample. | no |
Notices
This function has special behavior that you should be aware of.assert_eq
function should be used in a standalone fashion and only when you want to
abort the program. You should avoid it in logical expressions and other situations in which
you want the program to continue if the condition evaluates to false
.log
infallible impureFunction spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | any | The value to log. | yes | |
level | string | The log level. | info | no |
rate_limit_secs | integer | Specifies that the log message is output no more than once per the given number of seconds.
Use a value of 0 to turn rate limiting off. | 1 | no |
Enrichment functions
find_enrichment_table_records
infallible pureSearches an enrichment table for rows that match the provided condition.
For file
enrichment tables, this condition needs to be a VRL object in which
the key-value pairs indicate a field to search mapped to a value to search in that field.
This function returns the rows that match the provided condition(s). All fields need to
match for rows to be returned; if any fields do not match, then no rows are returned.
There are currently two forms of search criteria:
Exact match search. The given field must match the value exactly. Case sensitivity can be specified using the
case_sensitive
argument. An exact match search can use an index directly into the dataset, which should make this search fairly “cheap” from a performance perspective.Date range search. The given field must be greater than or equal to the
from
date and less than or equal to theto
date. A date range search involves sequentially scanning through the rows that have been located using any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. Therefore, use date ranges as the only criteria when the enrichment data set is very small.
For geoip
and mmdb
enrichment tables, this condition needs to be a VRL object with a single key-value pair
whose value needs to be a valid IP address. Example: {"ip": .ip }
. If a return field is expected
and without a value, null
is used. This table can return the following fields:
ISP databases:
autonomous_system_number
autonomous_system_organization
isp
organization
City databases:
city_name
continent_code
country_code
country_name
region_code
region_name
metro_code
latitude
longitude
postal_code
timezone
Connection-Type databases:
connection_type
To use this function, you need to update your configuration to
include an
enrichment_tables
parameter.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
table | string | The enrichment table to search. | yes | |
condition | object | The condition to search on. Since the condition is used at boot time to create indices into the data, these conditions must be statically defined. | yes | |
select | array | A subset of fields from the enrichment table to return. If not specified, all fields are returned. | no | |
case_sensitive | boolean | Whether text fields need to match cases exactly. | true | no |
get_enrichment_table_record
fallible pureSearches an enrichment table for a row that matches the provided condition. A single row must be matched. If no rows are found or more than one row is found, an error is returned.
For file
enrichment tables, this condition needs to be a VRL object in which
the key-value pairs indicate a field to search mapped to a value to search in that field.
This function returns the rows that match the provided condition(s). All fields need to
match for rows to be returned; if any fields do not match, then no rows are returned.
There are currently two forms of search criteria:
Exact match search. The given field must match the value exactly. Case sensitivity can be specified using the
case_sensitive
argument. An exact match search can use an index directly into the dataset, which should make this search fairly “cheap” from a performance perspective.Date range search. The given field must be greater than or equal to the
from
date and less than or equal to theto
date. A date range search involves sequentially scanning through the rows that have been located using any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. Therefore, use date ranges as the only criteria when the enrichment data set is very small.
For geoip
and mmdb
enrichment tables, this condition needs to be a VRL object with a single key-value pair
whose value needs to be a valid IP address. Example: {"ip": .ip }
. If a return field is expected
and without a value, null
is used. This table can return the following fields:
ISP databases:
autonomous_system_number
autonomous_system_organization
isp
organization
City databases:
city_name
continent_code
country_code
country_name
region_code
region_name
metro_code
latitude
longitude
postal_code
timezone
Connection-Type databases:
connection_type
To use this function, you need to update your configuration to
include an
enrichment_tables
parameter.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
table | string | The enrichment table to search. | yes | |
condition | object | The condition to search on. Since the condition is used at boot time to create indices into the data, these conditions must be statically defined. | yes | |
select | array | A subset of fields from the enrichment table to return. If not specified, all fields are returned. | no | |
case_sensitive | boolean | Whether the text fields match the case exactly. | true | no |
Errors
Theget_enrichment_table_record
function is fallible, which means that
error handling is required for these errors:Enumerate functions
compact
infallible purevalue
by removing empty values, where empty values are defined using the
available parameters.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | array object | The object or array to compact. | yes | |
recursive | boolean | Whether the compaction be recursive. | true | no |
null | boolean | Whether null should be treated as an empty value. | true | no |
string | boolean | Whether an empty string should be treated as an empty value. | true | no |
object | boolean | Whether an empty object should be treated as an empty value. | true | no |
array | boolean | Whether an empty array should be treated as an empty value. | true | no |
nullish | boolean | Tests whether the value is “nullish” as defined by the is_nullish function. | no |
filter
infallible pureFilter elements from a collection.
This function currently does not support recursive iteration.
The function uses the function closure syntax to allow reading the key-value or index-value combination for each item in the collection.
The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.
See the examples below to learn about the closure syntax.
flatten
infallible purevalue
into a single-level representation.for_each
infallible pureIterate over a collection.
This function currently does not support recursive iteration.
The function uses the “function closure syntax” to allow reading the key/value or index/value combination for each item in the collection.
The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.
See the examples below to learn about the closure syntax.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | array object | The array or object to iterate. | yes |
Examples
Tally elements
tally = {}
for_each(array!(.tags)) -> |_index, value| {
# Get the current tally for the `value`, or
# set to `0`.
count = int(get!(tally, [value])) ?? 0
# Increment the tally for the value by `1`.
tally = set!(tally, [value], count + 1)
}
tally
{
"bar": 1,
"baz": 1,
"foo": 2
}
includes
infallible purevalue
array includes the specified item
.keys
infallible purelength
infallible pureReturns the length of the value
.
- If
value
is an array, returns the number of elements. - If
value
is an object, returns the number of top-level keys. - If
value
is a string, returns the number of bytes in the string. If you want the number of characters, seestrlen
.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | array object string | The array or object. | yes |
map_keys
infallible pureMap the keys within an object.
If recursive
is enabled, the function iterates into nested
objects, using the following rules:
- Iteration starts at the root.
- For every nested object type:
- First return the key of the object type itself.
- Then recurse into the object, and loop back to item (1) in this list.
- Any mutation done on a nested object before recursing into it, are preserved.
- For every nested array type:
- First return the key of the array type itself.
- Then find all objects within the array, and apply item (2) to each individual object.
The above rules mean that map_keys
with
recursive
enabled finds all keys in the target,
regardless of whether nested objects are nested inside arrays.
The function uses the function closure syntax to allow reading the key for each item in the object.
The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.
See the examples below to learn about the closure syntax.
map_values
infallible pureMap the values within a collection.
If recursive
is enabled, the function iterates into nested
collections, using the following rules:
- Iteration starts at the root.
- For every nested collection type:
- First return the collection type itself.
- Then recurse into the collection, and loop back to item (1) in the list
- Any mutation done on a collection before recursing into it, are preserved.
The function uses the function closure syntax to allow mutating the value for each item in the collection.
The same scoping rules apply to closure blocks as they do for regular blocks, meaning, any variable defined in parent scopes are accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.
Check out the examples below to learn about the closure syntax.
match_array
infallible purevalue
array matches the pattern
. By default, it checks that at least one element matches, but can be set to determine if all the elements match.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | array | The array. | yes | |
pattern | regex | The regular expression pattern to match against. | yes | |
all | boolean | Whether to match on all elements of value . | no |
strlen
infallible pureReturns the number of UTF-8 characters in value
. This differs from
length
which counts the number of bytes of a string.
Note: This is the count of Unicode scalar values which can sometimes differ from Unicode code points.
unflatten
infallible purevalue
into a nested representation.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | object | The array or object to unflatten. | yes | |
separator | string | The separator to split flattened keys. | . | no |
recursive | boolean | Whether to recursively unflatten the object values. | true | no |
Examples
Unflatten
unflatten({
"foo.bar.baz": true,
"foo.bar.qux": false,
"foo.quux": 42
})
{
"foo": {
"bar": {
"baz": true,
"qux": false
},
"quux": 42
}
}
Unflatten recursively
unflatten({
"flattened.parent": {
"foo.bar": true,
"foo.baz": false
}
})
{
"flattened": {
"parent": {
"foo": {
"bar": true,
"baz": false
}
}
}
}
unique
infallible pureReturns the unique values for an array.
The first occurrence of each element is kept.
values
infallible pureEvent functions
get_secret
infallible pureremove_secret
infallible pureset_secret
infallible pureset_semantic_meaning
infallible purePath functions
del
infallible impureRemoves the field specified by the static path
from the target.
For dynamic path deletion, see the remove
function.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
path | path | The path of the field to delete. | yes | |
compact | boolean | After deletion, if compact is true and there is an empty object or array left,
the empty object or array is also removed, cascading up to the root. This only
applies to the path being deleted, and any parent paths. | no |
exists
infallible pureChecks whether the path
exists for the target.
This function distinguishes between a missing path
and a path with a null
value. A regular path lookup,
such as .foo
, cannot distinguish between the two cases
since it always returns null
if the path doesn’t exist.
get
fallible pureDynamically get the value of a given path.
If you know the path you want to look up, use
static paths such as .foo.bar[1]
to get the value of that
path. However, if you do not know the path names,
use the dynamic get
function to get the requested
value.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | object array | The object or array to query. | yes | |
path | array | An array of path segments to look for the value. | yes |
Errors
Theget
function is fallible, which means that
error handling is required for these errors:path
segment must be a string or an integer.remove
fallible pureDynamically remove the value for a given path.
If you know the path you want to remove, use
the del
function and static paths such as del(.foo.bar[1])
to remove the value at that path. The del
function returns the
deleted value, and is more performant than remove
.
However, if you do not know the path names, use the dynamic
remove
function to remove the value at the provided path.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | object array | The object or array to remove data from. | yes | |
path | array | An array of path segments to remove the value from. | yes | |
compact | boolean | After deletion, if compact is true , any empty objects or
arrays left are also removed. | no |
Errors
Theremove
function is fallible, which means that
error handling is required for these errors:path
segment must be a string or an integer.set
fallible pureDynamically insert data into the path of a given object or array.
If you know the path you want to assign a value to,
use static path assignments such as .foo.bar[1] = true
for
improved performance and readability. However, if you do not
know the path names, use the dynamic set
function to
insert the data into the object or array.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | object array | The object or array to insert data into. | yes | |
path | array | An array of path segments to insert the value into. | yes | |
data | any | The data to be inserted. | yes |
Errors
Theset
function is fallible, which means that
error handling is required for these errors:path
segment must be a string or an integer.Cryptography functions
decrypt
fallible pureDecrypts a string with a symmetric encryption algorithm.
Supported Algorithms:
- AES-256-CFB (key = 32 bytes, iv = 16 bytes)
- AES-192-CFB (key = 24 bytes, iv = 16 bytes)
- AES-128-CFB (key = 16 bytes, iv = 16 bytes)
- AES-256-OFB (key = 32 bytes, iv = 16 bytes)
- AES-192-OFB (key = 24 bytes, iv = 16 bytes)
- AES-128-OFB (key = 16 bytes, iv = 16 bytes)
- AES-128-SIV (key = 32 bytes, iv = 16 bytes)
- AES-256-SIV (key = 64 bytes, iv = 16 bytes)
- Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
- Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
- Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
- CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
- XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
- XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
ciphertext | string | The string in raw bytes (not encoded) to decrypt. | yes | |
algorithm | string | The algorithm to use. | yes | |
key | string | The key in raw bytes (not encoded) for decryption. The length must match the algorithm requested. | yes | |
iv | string | The IV in raw bytes (not encoded) for decryption. The length must match the algorithm requested.
A new IV should be generated for every message. You can use random_bytes to generate a cryptographically secure random value.
The value should match the one used during encryption. | yes |
Errors
Thedecrypt
function is fallible, which means that
error handling is required for these errors:algorithm
is not a supported algorithm.key
length does not match the key size required for the algorithm specified.iv
length does not match the iv
size required for the algorithm specified.encrypt
fallible pureEncrypts a string with a symmetric encryption algorithm.
Supported Algorithms:
- AES-256-CFB (key = 32 bytes, iv = 16 bytes)
- AES-192-CFB (key = 24 bytes, iv = 16 bytes)
- AES-128-CFB (key = 16 bytes, iv = 16 bytes)
- AES-256-OFB (key = 32 bytes, iv = 16 bytes)
- AES-192-OFB (key = 24 bytes, iv = 16 bytes)
- AES-128-OFB (key = 16 bytes, iv = 16 bytes)
- AES-128-SIV (key = 32 bytes, iv = 16 bytes)
- AES-256-SIV (key = 64 bytes, iv = 16 bytes)
- Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
- Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
- Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
- CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
- XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
- XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
plaintext | string | The string to encrypt. | yes | |
algorithm | string | The algorithm to use. | yes | |
key | string | The key in raw bytes (not encoded) for encryption. The length must match the algorithm requested. | yes | |
iv | string | The IV in raw bytes (not encoded) for encryption. The length must match the algorithm requested.
A new IV should be generated for every message. You can use random_bytes to generate a cryptographically secure random value. | yes |
Errors
Theencrypt
function is fallible, which means that
error handling is required for these errors:algorithm
is not a supported algorithm.key
length does not match the key size required for the algorithm specified.iv
length does not match the iv
size required for the algorithm specified.Examples
Encrypt value
plaintext = "super secret message"
iv = "1234567890123456" # typically you would call random_bytes(16)
key = "16_byte_keyxxxxx"
encrypted_message = encrypt!(plaintext, "AES-128-CBC-PKCS7", key, iv: iv)
encode_base64(encrypted_message)
GBw8Mu00v0Kc38+/PvsVtGgWuUJ+ZNLgF8Opy8ohIYE=
hmac
infallible pureCalculates a HMAC of the value
using the given key
.
The hashing algorithm
used can be optionally specified.
For most use cases, the resulting bytestream should be encoded into a hex or base64 string using either encode_base16 or encode_base64.
This function is infallible if either the default algorithm
value or a recognized-valid compile-time
algorithm
string literal is used. Otherwise, it is fallible.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to calculate the HMAC for. | yes | |
key | string | The string to use as the cryptographic key. | yes | |
algorithm | string | The hashing algorithm to use. | SHA-256 | no |
Examples
Calculate message HMAC (defaults: SHA-256), encoding to a base64 string
encode_base64(hmac("Hello there", "super-secret-key"))
eLGE8YMviv85NPXgISRUZxstBNSU47JQdcXkUWcClmI=
md5
infallible purevalue
.seahash
infallible purevalue
.
Note: Due to limitations in the underlying VRL data types, this function converts the unsigned 64-bit integer SeaHash result to a signed 64-bit integer. Results higher than the signed 64-bit integer maximum value wrap around to negative values.sha1
infallible purevalue
.sha2
infallible purevalue
.sha3
infallible purevalue
.IP functions
ip_aton
fallible pureConverts IPv4 address in numbers-and-dots notation into network-order bytes represented as an integer.
This behavior mimics inet_aton.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The IP address to convert to binary. | yes |
Errors
Theip_aton
function is fallible, which means that
error handling is required for these errors:value
is not a valid IPv4 address.ip_cidr_contains
fallible pureip
is contained in the block referenced by the cidr
.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
cidr | string | The CIDR mask (v4 or v6). | yes | |
ip | string | The IP address (v4 or v6). | yes |
Errors
Theip_cidr_contains
function is fallible, which means that
error handling is required for these errors:cidr
is not a valid CIDR.ip
is not a valid IP address.ip_ntoa
fallible pureConverts numeric representation of IPv4 address in network-order bytes to numbers-and-dots notation.
This behavior mimics inet_ntoa.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The integer representation of an IPv4 address. | yes |
Errors
Theip_ntoa
function is fallible, which means that
error handling is required for these errors:value
cannot fit in an unsigned 32-bit integer.ip_ntop
fallible pureConverts IPv4 and IPv6 addresses from binary to text form.
This behavior mimics inet_ntop.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The binary data to convert from. For IPv4 addresses, it must be 4 bytes (32 bits) long. For IPv6 addresses, it must be 16 bytes (128 bits) long. | yes |
Notices
This function has special behavior that you should be aware of.decode_base64
or
decode_percent
can still be used correctly.Errors
Theip_ntop
function is fallible, which means that
error handling is required for these errors:value
must be of length 4 or 16 bytes.ip_pton
fallible pureConverts IPv4 and IPv6 addresses from text to binary form.
- The binary form of IPv4 addresses is 4 bytes (32 bits) long.
- The binary form of IPv6 addresses is 16 bytes (128 bits) long.
This behavior mimics inet_pton.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The IP address (v4 or v6) to convert to binary form. | yes |
Notices
This function has special behavior that you should be aware of.encode_base64
or encode_percent
can
still process it correctly.Errors
Theip_pton
function is fallible, which means that
error handling is required for these errors:value
is not a valid IP (v4 or v6) address in text form.ip_subnet
fallible pureip
using the supplied subnet
.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
ip | string | The IP address (v4 or v6). | yes | |
subnet | string | The subnet to extract from the IP address. This can be either a prefix length like /8 or a net mask
like 255.255.0.0 . The net mask can be either an IPv4 or IPv6 address. | yes |
Notices
This function has special behavior that you should be aware of.Errors
Theip_subnet
function is fallible, which means that
error handling is required for these errors:ip
is not a valid IP address.subnet
is not a valid subnet.ip_to_ipv6
fallible pureip
to an IPv6 address.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
ip | string | The IP address to convert to IPv6. | yes |
Errors
Theip_to_ipv6
function is fallible, which means that
error handling is required for these errors:ip
is not a valid IP address.ipv6_to_ipv4
fallible pureip
to an IPv4 address. ip
is returned unchanged if it’s already an IPv4 address. If ip
is
currently an IPv6 address then it needs to be IPv4 compatible, otherwise an error is thrown.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
ip | string | The IPv4-mapped IPv6 address to convert. | yes |
Errors
Theipv6_to_ipv4
function is fallible, which means that
error handling is required for these errors:ip
is not a valid IP address.ip
is an IPv6 address that is not compatible with IPv4.is_ipv4
infallible pureCheck if the string is a valid IPv4 address or not.
An [IPv4-mapped][https://datatracker.ietf.org/doc/html/rfc6890] or [IPv4-compatible][https://datatracker.ietf.org/doc/html/rfc6890] IPv6 address is not considered valid for the purpose of this function.
is_ipv6
infallible pureNumber functions
abs
infallible purevalue
.ceil
infallible purevalue
up to the specified precision
.floor
infallible purevalue
down to the specified precision
.format_int
fallible purevalue
into a string representation using the given base/radix.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer | The number to format. | yes | |
base | integer | The base to format the number in. Must be between 2 and 36 (inclusive). | 10 | no |
Errors
Theformat_int
function is fallible, which means that
error handling is required for these errors:format_number
infallible purevalue
into a string representation of the number.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer float | The number to format as a string. | yes | |
scale | integer | The number of decimal places to display. | no | |
decimal_separator | string | The character to use between the whole and decimal parts of the number. | . | no |
grouping_separator | string | The character to use between each thousands part of the number. | , | no |
mod
fallible purevalue
divided by modulus
.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | integer float | The value the modulus is applied to. | yes | |
modulus | integer float | The modulus value. | yes |
Errors
Themod
function is fallible, which means that
error handling is required for these errors:value
is not an integer or float.modulus
is not an integer or float.modulus
is equal to 0.round
infallible purevalue
to the specified precision
.Object functions
match_datadog_query
infallible puremerge
infallible purefrom
object into the to
object.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
to | object | The object to merge into. | yes | |
from | object | The object to merge from. | yes | |
deep | boolean | A deep merge is performed if true , otherwise only top-level fields are merged. | no |
unnest
fallible pureUnnest an array field from an object to create an array of objects using that field; keeping all other fields.
Assigning the array result of this to .
results in multiple events being emitted from remap
. See the
remap
transform docs for more details.
This is also referred to as explode
in some languages.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
path | path | The path of the field to unnest. | yes |
Errors
Theunnest
function is fallible, which means that
error handling is required for these errors:Parse functions
parse_apache_log
fallible purecommon
,
combined
, or the default error
format.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
format | string | The format to use for parsing the log. | yes | |
timestamp_format | string | The date/time format to use for encoding the timestamp. The time is parsed in local time if the timestamp does not specify a timezone. | %d/%b/%Y:%T %z | no |
Notices
This function has special behavior that you should be aware of.-
. These fields are omitted in the result.Errors
Theparse_apache_log
function is fallible, which means that
error handling is required for these errors:value
does not match the specified format.timestamp_format
is not a valid format string.value
fails to parse using the provided timestamp_format
.Examples
Parse using Apache log format (common)
parse_apache_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326", format: "common")
{
"host": "127.0.0.1",
"identity": "bob",
"message": "GET /apache_pb.gif HTTP/1.0",
"method": "GET",
"path": "/apache_pb.gif",
"protocol": "HTTP/1.0",
"size": 2326,
"status": 200,
"timestamp": "2000-10-10T20:55:36Z",
"user": "frank"
}
Parse using Apache log format (combined)
parse_apache_log!(
s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth" "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"',
"combined",
)
{
"agent": "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0",
"host": "127.0.0.1",
"identity": "bob",
"message": "GET /apache_pb.gif HTTP/1.0",
"method": "GET",
"path": "/apache_pb.gif",
"protocol": "HTTP/1.0",
"referrer": "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth",
"size": 2326,
"status": 200,
"timestamp": "2000-10-10T20:55:36Z",
"user": "frank"
}
Parse using Apache log format (error)
parse_apache_log!(
s'[01/Mar/2021:12:00:19 +0000] [ab:alert] [pid 4803:tid 3814] [client 147.159.108.175:24259] I will bypass the haptic COM bandwidth, that should matrix the CSS driver!',
"error"
)
{
"client": "147.159.108.175",
"message": "I will bypass the haptic COM bandwidth, that should matrix the CSS driver!",
"module": "ab",
"pid": 4803,
"port": 24259,
"severity": "alert",
"thread": "3814",
"timestamp": "2021-03-01T12:00:19Z"
}
parse_aws_alb_log
fallible purevalue
in the Elastic Load Balancer Access format.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | Access log of the Application Load Balancer. | yes |
Errors
Theparse_aws_alb_log
function is fallible, which means that
error handling is required for these errors:value
is not a properly formatted AWS ALB log.Examples
Parse AWS ALB log
parse_aws_alb_log!(
"http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\""
)
{
"actions_executed": "forward",
"chosen_cert_arn": null,
"classification": null,
"classification_reason": null,
"client_host": "192.168.131.39:2817",
"domain_name": null,
"elb": "app/my-loadbalancer/50dc6c495c0c9188",
"elb_status_code": "200",
"error_reason": null,
"matched_rule_priority": "0",
"received_bytes": 34,
"redirect_url": null,
"request_creation_time": "2018-11-30T22:22:48.364000Z",
"request_method": "GET",
"request_processing_time": 0,
"request_protocol": "HTTP/1.1",
"request_url": "http://www.example.com:80/",
"response_processing_time": 0,
"sent_bytes": 366,
"ssl_cipher": null,
"ssl_protocol": null,
"target_group_arn": "arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067",
"target_host": null,
"target_port_list": [],
"target_processing_time": 0.001,
"target_status_code": "200",
"target_status_code_list": [],
"timestamp": "2018-11-30T22:23:00.186641Z",
"trace_id": "Root=1-58337364-23a8c76965a2ef7629b185e3",
"traceability_id": null,
"type": "http",
"user_agent": "curl/7.46.0"
}
parse_aws_cloudwatch_log_subscription_message
fallible pureaws_kinesis_firehose
source.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string representation of the message to parse. | yes |
Errors
Theparse_aws_cloudwatch_log_subscription_message
function is fallible, which means that
error handling is required for these errors:value
is not a properly formatted AWS CloudWatch Log subscription message.Examples
Parse AWS Cloudwatch Log subscription message
parse_aws_cloudwatch_log_subscription_message!(.message)
{
"log_events": [
{
"id": "35683658089614582423604394983260738922885519999578275840",
"message": "{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}",
"timestamp": "2020-09-14T19:09:29.039Z"
}
],
"log_group": "test",
"log_stream": "test",
"message_type": "DATA_MESSAGE",
"owner": "111111111111",
"subscription_filters": [
"Destination"
]
}
parse_aws_vpc_flow_log
fallible purevalue
in the VPC Flow Logs format.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | VPC Flow Log. | yes | |
format | string | VPC Flow Log format. | no |
Errors
Theparse_aws_vpc_flow_log
function is fallible, which means that
error handling is required for these errors:value
is not a properly formatted AWS VPC Flow log.Examples
Parse AWS VPC Flow log (default format)
parse_aws_vpc_flow_log!("2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA")
{
"account_id": "123456789010",
"action": null,
"bytes": null,
"dstaddr": null,
"dstport": null,
"end": 1431280934,
"interface_id": "eni-1235b8ca123456789",
"log_status": "NODATA",
"packets": null,
"protocol": null,
"srcaddr": null,
"srcport": null,
"start": 1431280876,
"version": 2
}
Parse AWS VPC Flow log (custom format)
parse_aws_vpc_flow_log!(
"- eni-1235b8ca123456789 10.0.1.5 10.0.0.220 10.0.1.5 203.0.113.5",
"instance_id interface_id srcaddr dstaddr pkt_srcaddr pkt_dstaddr"
)
{
"dstaddr": "10.0.0.220",
"instance_id": null,
"interface_id": "eni-1235b8ca123456789",
"pkt_dstaddr": "203.0.113.5",
"pkt_srcaddr": "10.0.1.5",
"srcaddr": "10.0.1.5"
}
Parse AWS VPC Flow log including v5 fields
parse_aws_vpc_flow_log!("5 52.95.128.179 10.0.0.71 80 34210 6 1616729292 1616729349 IPv4 14 15044 123456789012 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-0c50d5961bcb2d47b eni-1235b8ca123456789 ap-southeast-2 apse2-az3 - - ACCEPT 19 52.95.128.179 10.0.0.71 S3 - - ingress OK",
format: "version srcaddr dstaddr srcport dstport protocol start end type packets bytes account_id vpc_id subnet_id instance_id interface_id region az_id sublocation_type sublocation_id action tcp_flags pkt_srcaddr pkt_dstaddr pkt_src_aws_service pkt_dst_aws_service traffic_path flow_direction log_status")
{
"account_id": "123456789012",
"action": "ACCEPT",
"az_id": "apse2-az3",
"bytes": 15044,
"dstaddr": "10.0.0.71",
"dstport": 34210,
"end": 1616729349,
"flow_direction": "ingress",
"instance_id": "i-0c50d5961bcb2d47b",
"interface_id": "eni-1235b8ca123456789",
"log_status": "OK",
"packets": 14,
"pkt_dst_aws_service": null,
"pkt_dstaddr": "10.0.0.71",
"pkt_src_aws_service": "S3",
"pkt_srcaddr": "52.95.128.179",
"protocol": 6,
"region": "ap-southeast-2",
"srcaddr": "52.95.128.179",
"srcport": 80,
"start": 1616729292,
"sublocation_id": null,
"sublocation_type": null,
"subnet_id": "subnet-aaaaaaaa012345678",
"tcp_flags": 19,
"traffic_path": null,
"type": "IPv4",
"version": 5,
"vpc_id": "vpc-abcdefab012345678"
}
parse_cef
fallible purevalue
in CEF (Common Event Format) format. Ignores everything up to CEF header. Empty values are returned as empty strings. Surrounding quotes are removed from values.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
translate_custom_fields | boolean | Toggles translation of custom field pairs to key:value . | no |
Notices
This function has special behavior that you should be aware of.Errors
Theparse_cef
function is fallible, which means that
error handling is required for these errors:value
is not a properly formatted CEF string.Examples
Parse output generated by PTA
parse_cef!(
"CEF:0|CyberArk|PTA|12.6|1|Suspected credentials theft|8|suser=mike2@prod1.domain.com shost=prod1.domain.com src=1.1.1.1 duser=andy@dev1.domain.com dhost=dev1.domain.com dst=2.2.2.2 cs1Label=ExtraData cs1=None cs2Label=EventID cs2=52b06812ec3500ed864c461e deviceCustomDate1Label=detectionDate deviceCustomDate1=1388577900000 cs3Label=PTAlink cs3=https://1.1.1.1/incidents/52b06812ec3500ed864c461e cs4Label=ExternalLink cs4=None"
)
{
"cefVersion": "0",
"cs1": "None",
"cs1Label": "ExtraData",
"cs2": "52b06812ec3500ed864c461e",
"cs2Label": "EventID",
"cs3": "https://1.1.1.1/incidents/52b06812ec3500ed864c461e",
"cs3Label": "PTAlink",
"cs4": "None",
"cs4Label": "ExternalLink",
"deviceCustomDate1": "1388577900000",
"deviceCustomDate1Label": "detectionDate",
"deviceEventClassId": "1",
"deviceProduct": "PTA",
"deviceVendor": "CyberArk",
"deviceVersion": "12.6",
"dhost": "dev1.domain.com",
"dst": "2.2.2.2",
"duser": "andy@dev1.domain.com",
"name": "Suspected credentials theft",
"severity": "8",
"shost": "prod1.domain.com",
"src": "1.1.1.1",
"suser": "mike2@prod1.domain.com"
}
Ignore syslog header
parse_cef!(
"Sep 29 08:26:10 host CEF:1|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232"
)
{
"cefVersion": "1",
"deviceEventClassId": "100",
"deviceProduct": "threatmanager",
"deviceVendor": "Security",
"deviceVersion": "1.0",
"dst": "2.1.2.2",
"name": "worm successfully stopped",
"severity": "10",
"spt": "1232",
"src": "10.0.0.1"
}
Translate custom fields
parse_cef!(
"CEF:0|Dev|firewall|2.2|1|Connection denied|5|c6a1=2345:0425:2CA1:0000:0000:0567:5673:23b5 c6a1Label=Device IPv6 Address",
translate_custom_fields: true
)
{
"Device IPv6 Address": "2345:0425:2CA1:0000:0000:0567:5673:23b5",
"cefVersion": "0",
"deviceEventClassId": "1",
"deviceProduct": "firewall",
"deviceVendor": "Dev",
"deviceVersion": "2.2",
"name": "Connection denied",
"severity": "5"
}
parse_common_log
fallible purevalue
using the Common Log Format (CLF).Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
timestamp_format | string | The date/time format to use for encoding the timestamp. | %d/%b/%Y:%T %z | no |
Notices
This function has special behavior that you should be aware of.-
. These fields are omitted in the result.Errors
Theparse_common_log
function is fallible, which means that
error handling is required for these errors:value
does not match the Common Log Format.timestamp_format
is not a valid format string.value
fails to parse using the provided timestamp_format
.Examples
Parse using Common Log Format (with default timestamp format)
parse_common_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326")
{
"host": "127.0.0.1",
"identity": "bob",
"message": "GET /apache_pb.gif HTTP/1.0",
"method": "GET",
"path": "/apache_pb.gif",
"protocol": "HTTP/1.0",
"size": 2326,
"status": 200,
"timestamp": "2000-10-10T20:55:36Z",
"user": "frank"
}
Parse using Common Log Format (with custom timestamp format)
parse_common_log!(
"127.0.0.1 bob frank [2000-10-10T20:55:36Z] \"GET /apache_pb.gif HTTP/1.0\" 200 2326",
"%+"
)
{
"host": "127.0.0.1",
"identity": "bob",
"message": "GET /apache_pb.gif HTTP/1.0",
"method": "GET",
"path": "/apache_pb.gif",
"protocol": "HTTP/1.0",
"size": 2326,
"status": 200,
"timestamp": "2000-10-10T20:55:36Z",
"user": "frank"
}
parse_csv
fallible pureFunction spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
delimiter | string | The field delimiter to use when parsing. Must be a single-byte utf8 character. | , | no |
Notices
This function has special behavior that you should be aware of.Errors
Theparse_csv
function is fallible, which means that
error handling is required for these errors:value
is not a valid CSV string.parse_duration
fallible purevalue
into a human-readable duration format specified by unit
.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string of the duration. | yes | |
unit | string | The output units for the duration. | yes |
Errors
Theparse_duration
function is fallible, which means that
error handling is required for these errors:value
is not a properly formatted duration.parse_etld
fallible purevalue
representing domain name.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The domain string. | yes | |
plus_parts | integer | Can be provided to get additional parts of the domain name. When 1 is passed, eTLD+1 will be returned, which represents a domain registrable by a single organization. Higher numbers will return subdomains. | no | |
psl | string | Can be provided to use a different public suffix list. By default, https://publicsuffix.org/list/public_suffix_list.dat is used. | no |
Errors
Theparse_etld
function is fallible, which means that
error handling is required for these errors:value
Examples
Parse eTLD
parse_etld!("sub.sussex.ac.uk")
{
"etld": "ac.uk",
"etld_plus": "ac.uk",
"known_suffix": true
}
Parse eTLD+1
parse_etld!("sub.sussex.ac.uk", plus_parts: 1)
{
"etld": "ac.uk",
"etld_plus": "sussex.ac.uk",
"known_suffix": true
}
parse_glog
fallible purevalue
using the glog (Google Logging Library) format.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes |
Errors
Theparse_glog
function is fallible, which means that
error handling is required for these errors:value
does not match the glog
format.parse_grok
fallible pureFunction spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
pattern | string | The Grok pattern. | yes |
Notices
This function has special behavior that you should be aware of.Errors
Theparse_grok
function is fallible, which means that
error handling is required for these errors:value
fails to parse using the provided pattern
.parse_groks
fallible pureFunction spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
patterns | array | The Grok patterns, which are tried in order until the first match. | yes | |
aliases | object | The shared set of grok aliases that can be referenced in the patterns to simplify them. | true | no |
Notices
This function has special behavior that you should be aware of.Errors
Theparse_groks
function is fallible, which means that
error handling is required for these errors:value
fails to parse using the provided pattern
.Examples
Parse using multiple Grok patterns
parse_groks!(
"2020-10-02T23:22:12.223222Z info Hello world",
patterns: [
"%{common_prefix} %{_status} %{_message}",
"%{common_prefix} %{_message}",
],
aliases: {
"common_prefix": "%{_timestamp} %{_loglevel}",
"_timestamp": "%{TIMESTAMP_ISO8601:timestamp}",
"_loglevel": "%{LOGLEVEL:level}",
"_status": "%{POSINT:status}",
"_message": "%{GREEDYDATA:message}"
}
)
{
"level": "info",
"message": "Hello world",
"timestamp": "2020-10-02T23:22:12.223222Z"
}
parse_influxdb
fallible purevalue
as an InfluxDB line protocol
string, producing a list of Vector-compatible metrics.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string representation of the InfluxDB line protocol to parse. | yes |
Notices
This function has special behavior that you should be aware of.log_to_metric
transform with the option all_metrics
set to true
to convert the metric-shaped log events to metric events so real metrics are produced.gauge
. Each metric name is prefixed with the measurement
field, followed
by an underscore (_
), and then the field key
field.string
is the only type that is not supported as a field value,
due to limitations of Vector’s metric model.Errors
Theparse_influxdb
function is fallible, which means that
error handling is required for these errors:value
is not a valid InfluxDB line protocol string.string
.NaN
field value.Examples
Parse InfluxDB line protocol
parse_influxdb!("cpu,host=A,region=us-west usage_system=64i,usage_user=10u,temperature=50.5,on=true,sleep=false 1590488773254420000")
[{"gauge":{"value":64},"kind":"absolute","name":"cpu_usage_system","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":10},"kind":"absolute","name":"cpu_usage_user","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":50.5},"kind":"absolute","name":"cpu_temperature","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":1},"kind":"absolute","name":"cpu_on","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"},{"gauge":{"value":0},"kind":"absolute","name":"cpu_sleep","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z"}]
parse_int
fallible purevalue
representing a number in an optional base/radix to an integer.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
base | integer | The base the number is in. Must be between 2 and 36 (inclusive). If unspecified, the string prefix is used to determine the base: “0b”, 8 for “0” or “0o”, 16 for “0x”, and 10 otherwise. | no |
Errors
Theparse_int
function is fallible, which means that
error handling is required for these errors:parse_json
fallible purevalue
as JSON.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string representation of the JSON to parse. | yes | |
max_depth | integer | Number of layers to parse for nested JSON-formatted documents. The value must be in the range of 1 to 128. | no | |
lossy | boolean | Whether to parse the JSON in a lossy manner. Replaces invalid UTF-8 characters
with the Unicode character � (U+FFFD) if set to true, otherwise returns an error
if there are any invalid UTF-8 characters present. | true | no |
Notices
This function has special behavior that you should be aware of.string
into a timestamp
, consider the
parse_timestamp
function.Errors
Theparse_json
function is fallible, which means that
error handling is required for these errors:value
is not a valid JSON-formatted payload.parse_key_value
fallible pureParses the value
in key-value format. Also known as logfmt.
- Keys and values can be wrapped with
"
. "
characters can be escaped using\
.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
key_value_delimiter | string | The string that separates the key from the value. | = | no |
field_delimiter | string | The string that separates each key-value pair. |
| no |
whitespace | string | Defines the acceptance of unnecessary whitespace surrounding the configured key_value_delimiter . | lenient | no |
accept_standalone_key | boolean | Whether a standalone key should be accepted, the resulting object associates such keys with the boolean value true . | true | no |
Notices
This function has special behavior that you should be aware of.Errors
Theparse_key_value
function is fallible, which means that
error handling is required for these errors:value
is not a properly formatted key-value string.Examples
Parse logfmt log
parse_key_value!(
"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
{
"@timestamp": "Sun Jan 10 16:47:39 EST 2021",
"id": "ConsumerFetcherManager-1382721708341",
"level": "info",
"module": "kafka.consumer.ConsumerFetcherManager",
"msg": "Stopping all fetchers",
"tag#production": "stopping_fetchers"
}
Parse comma delimited log
parse_key_value!(
"path:\"/cart_link\", host:store.app.com, fwd: \"102.30.171.16\", dyno: web.1, connect:0ms, service:87ms, status:304, bytes:632, protocol:https",
field_delimiter: ",",
key_value_delimiter: ":"
)
{
"bytes": "632",
"connect": "0ms",
"dyno": "web.1",
"fwd": "102.30.171.16",
"host": "store.app.com",
"path": "/cart_link",
"protocol": "https",
"service": "87ms",
"status": "304"
}
parse_klog
fallible purevalue
using the klog format used by Kubernetes components.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes |
Errors
Theparse_klog
function is fallible, which means that
error handling is required for these errors:value
does not match the klog
format.parse_linux_authorization
fallible pure/var/log/auth.log
(for Debian-based systems) or
/var/log/secure
(for RedHat-based systems) according to Syslog format.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The text containing the message to parse. | yes |
Notices
This function has special behavior that you should be aware of.Errors
Theparse_linux_authorization
function is fallible, which means that
error handling is required for these errors:value
is not a properly formatted Syslog message.Examples
Parse Linux authorization event
parse_linux_authorization!(
s'Mar 23 2023 01:49:58 localhost sshd[1111]: Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar'
)
{
"appname": "sshd",
"hostname": "localhost",
"message": "Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar",
"procid": 1111,
"timestamp": "2023-03-23T01:49:58Z"
}
parse_logfmt
fallible pureParses the value
in logfmt.
- Keys and values can be wrapped using the
"
character. "
characters can be escaped by the\
character.- As per this logfmt specification, the
parse_logfmt
function accepts standalone keys and assigns them a Boolean value oftrue
.
Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes |
Errors
Theparse_logfmt
function is fallible, which means that
error handling is required for these errors:value
is not a properly formatted key-value stringExamples
Parse logfmt log
parse_logfmt!(
"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
{
"@timestamp": "Sun Jan 10 16:47:39 EST 2021",
"id": "ConsumerFetcherManager-1382721708341",
"level": "info",
"module": "kafka.consumer.ConsumerFetcherManager",
"msg": "Stopping all fetchers",
"tag#production": "stopping_fetchers"
}
parse_nginx_log
fallible purecombined
,
ingress_upstreaminfo
, or error
format.Function spec
Argument | Type | Description | Default | Required? |
---|---|---|---|---|
value | string | The string to parse. | yes | |
format | string | The format to use for parsing the log. | yes | |
timestamp_format | string | The date/time format to use for encoding the timestamp. The time is parsed
in local time if the timestamp doesn’t specify a timezone. The default format is %d/%b/%Y:%T %z for
combined logs and %Y/%m/%d %H:%M:%S for error logs. | %d/%b/%Y:%T %z | no |