Coercer Transform

The Vector coercer transform coerces log fields into typed values.

Warnings

Configuration

[transforms.my_transform_id]
# General
type = "coercer" # required
inputs = ["my-source-or-transform-id", "prefix-*"] # required
# Types
types.status = "int" # example
types.duration = "float" # example
types.success = "bool" # example
types.timestamp_iso8601 = "timestamp|%F" # example
types.timestamp_custom = "timestamp|%a %b %e %T %Y" # example
types.timestamp_unix = "timestamp|%F %T" # example
types.parent.child = "int" # example
  • optionalbool

    drop_unspecified

    Set to true to drop all fields that are not specified in the types table. Make sure both message and timestamp are specified in the types table as their absense will cause the original message data to be dropped along with other extraneous fields.

    • Default: false
  • optionalstring

    timezone

    The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone. This overrides the global timezone option. The time zone name may be any name in the TZ database, or local to indicate system local time.

    • Syntax: literal
    • Default: "local"
  • commonoptionaltable

    types

    Key/value pairs representing mapped log field names and types. This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.

    Timestamp coercions need to be prefaced with timestamp|, for example "timestamp|%F". Timestamp specifiers can use either of the following:

    1. One of the built-in-formats listed in the Timestamp Formats table below.
    2. The time format specifiers from Rust's chrono library.

    Types

    • array
    • bool
    • bytes
    • float
    • int
    • map
    • null
    • timestamp (see the table below for formats)

    Timestamp Formats

    FormatDescriptionExample
    %F %TYYYY-MM-DD HH:MM:SS2020-12-01 02:37:54
    %v %TDD-Mmm-YYYY HH:MM:SS01-Dec-2020 02:37:54
    %FT%TISO 8601[RFC
    3339](https://tools.ietf.org/html/rfc3339) format without time zone

    2020-12-01T02:37:54 %a, %d %b %Y %T | RFC 822/2822 without time zone | Tue, 01 Dec 2020 02:37:54 %a %d %b %T %Y | date command output without time zone | Tue 01 Dec 02:37:54 2020 %a %b %e %T %Y | ctime format | Tue Dec 1 02:37:54 2020 %s | UNIX timestamp | 1606790274 %FT%TZ | ISO 8601/RFC 3339 UTC | 2020-12-01T09:37:54Z %+ | ISO 8601/RFC 3339 UTC with time zone | 2020-12-01T02:37:54-07:00 %a %d %b %T %Z %Y | date command output with time zone | Tue 01 Dec 02:37:54 PST 2020 %a %d %b %T %z %Y| date command output with numeric time zone | Tue 01 Dec 02:37:54 -0700 2020 %a %d %b %T %#z %Y | date command output with numeric time zone (minutes can be missing or present) | Tue 01 Dec 02:37:54 -07 2020

    Note: the examples in this table are for 54 seconds after 2:37 am on December 1st, 2020 in Pacific Standard Time.

Telemetry

This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

  • counter

    processing_errors_total

    The total number of processing errors encountered by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • error_type - The type of the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    events_in_total

    The total number of events accepted by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    processed_events_total

    The total number of events processed by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • file - The file that produced the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    events_out_total

    The total number of events emitted by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    processed_bytes_total

    The total number of bytes processed by the component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

Examples

Given the following Vector log event:

{
"bytes_in": "5667",
"bytes_out": "20574",
"host": "5.86.210.12",
"message": "GET /embrace/supply-chains/dynamic/vertical",
"status": "201",
"timestamp": "19/06/2019:17:20:49 -0400",
"user_id": "zieme4647"
}

And the following configuration:

vector.toml
[transforms.coercer]
type = "coercer"
types.bytes_in = "int"
types.bytes_out = "int"
types.status = "int"
types.timestamp = "timestamp|%d/%m/%Y:%H:%M:%S %z"

The following Vector log event will be output:

{
"bytes_in": 5667,
"bytes_out": 20574,
"host": "5.86.210.12",
"message": "GET /embrace/supply-chains/dynamic/vertical",
"status": 201,
"timestamp": "19/06/2019:17:20:49 -0400",
"user_id": "zieme4647"
}

How It Works

State

This component is stateless, meaning its behavior is consistent across each input.