AWS CloudWatch Logs Subscription Parser Transform

The Vector aws_cloudwatch_logs_subscription_parser transform parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) coming from the aws_kinesis_firehose source.

Warnings

This transform has been deprecated in favor of the remap transform, which enables you to use Vector Remap Language (VRL for short) to create transform logic of any degree of complexity. The examples below show how you can use VRL to replace this transform's functionality. coffee .message = parse_aws_cloudwatch_log_subscription_message(.message)

Configuration

Common
Advanced

vector.toml
vector.yaml
vector.json

[transforms.my_transform_id]
  type = "aws_cloudwatch_logs_subscription_parser" # required
  inputs = ["my-source-or-transform-id", "prefix-*"] # required
  field = "message" # optional, default

commonoptionalstring
field
The log field to decode as an AWS CloudWatch Logs Subscription JSON event. The field must hold a string value.
- Syntax: literal
- Default: "message"

Output

This component outputs log events with the following fields:

{
  "id" : "35683658089614582423604394983260738922885519999578275840",
  "log_group" : "/lambda/test",
  "log_stream" : "2020/03/24/[$LATEST]794dbaf40a7846c4984ad80ebf110544",
  "message" : "hello",
  "owner" : "111111111111",
  "subscription_filters" : [
    "Destination"
  ],
  "timestamp" : "2020-10-10T17:07:36+00:00"
}

7 items

commonrequired

commonrequiredstring
id
The CloudWatch Logs event id.
- Syntax: literal
- View examples
commonrequiredstring
log_group
The log group the event came from.
- Syntax: literal
- View examples
commonrequiredstring
log_stream
The log stream the event came from.
- Syntax: literal
- View examples
commonrequiredstring
message
The body of the log event.
- Syntax: literal
- View examples
commonrequiredstring
owner
The ID of the AWS account the logs came from.
- Syntax: literal
- View examples
commonrequired[string]
subscription_filters
The list of subscription filter names that the logs were sent by.
- View examples
commonrequiredtimestamp
timestamp
The timestamp of the log event.
- View examples

Telemetry

This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

counter
processing_errors_total
The total number of processing errors encountered by this component. This metric includes the following tags:
- component_kind - The Vector component kind.
- component_name - The Vector component ID.
- component_type - The Vector component type.
- error_type - The type of the error
- instance - The Vector instance identified by host and port.
- job - The name of the job producing Vector metrics.
counter
events_in_total
The number of events accepted by this component either from tagged origin like file and uri, or cumulatively from other origins. This metric includes the following tags:
- component_kind - The Vector component kind.
- component_name - The Vector component ID.
- component_type - The Vector component type.
- container_name - The name of the container from which the event originates.
- file - The file from which the event originates.
- instance - The Vector instance identified by host and port.
- job - The name of the job producing Vector metrics.
- mode - The connection mode used by the component.
- peer_addr - The IP from which the event originates.
- peer_path - The pathname from which the event originates.
- pod_name - The name of the pod from which the event originates.
- uri - The sanitized uri from which the event originates.
counter
processed_events_total
The total number of events processed by this component. This metric includes the following tags:
- component_kind - The Vector component kind.
- component_name - The Vector component ID.
- component_type - The Vector component type.
- file - The file that produced the error
- instance - The Vector instance identified by host and port.
- job - The name of the job producing Vector metrics.
counter
events_out_total
The total number of events emitted by this component. This metric includes the following tags:
- component_kind - The Vector component kind.
- component_name - The Vector component ID.
- component_type - The Vector component type.
- instance - The Vector instance identified by host and port.
- job - The name of the job producing Vector metrics.
counter
processed_bytes_total
The number of bytes processed by the component. This metric includes the following tags:
- component_kind - The Vector component kind.
- component_name - The Vector component ID.
- component_type - The Vector component type.
- container_name - The name of the container from which the bytes originate.
- file - The file from which the bytes originate.
- instance - The Vector instance identified by host and port.
- job - The name of the job producing Vector metrics.
- mode - The connection mode used by the component.
- peer_addr - The IP from which the bytes originate.
- peer_path - The pathname from which the bytes originate.
- pod_name - The name of the pod from which the bytes originate.
- uri - The sanitized uri from which the bytes originate.

Examples

Given the following Vector log event:

{
  "message": "\t{\n\t  \"messageType\": \"DATA_MESSAGE\",\n\t  \"owner\": \"111111111111\",\n\t  \"logGroup\": \"test\",\n\t  \"logStream\": \"test\",\n\t  \"subscriptionFilters\": [\n\t\t\"Destination\"\n\t  ],\n\t  \"logEvents\": [\n\t\t{\n\t\t  \"id\": \"35683658089614582423604394983260738922885519999578275840\",\n\t\t  \"timestamp\": 1600110569039,\n\t\t  \"message\": \"{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}\"\n\t\t},\n\t\t{\n\t\t  \"id\": \"35683658089659183914001456229543810359430816722590236673\",\n\t\t  \"timestamp\": 1600110569041,\n\t\t  \"message\": \"{\"bytes\":17707,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"109.81.244.252\",\"method\":\"GET\",\"protocol\":\"HTTP/2.0\",\"referer\":\"http://www.investormission-critical.io/24/7/vortals\",\"request\":\"/scale/functionalities/optimize\",\"source_type\":\"stdin\",\"status\":502,\"user-identifier\":\"feeney1708\"}\"\n\t\t}\n\t  ]\n\t}"
}

And the following configuration:

vector.toml

[transforms.aws_cloudwatch_logs_subscription_parser]
type = "aws_cloudwatch_logs_subscription_parser"
field = "message"

The following Vector log event will be output:

{
  "id": "35683658089614582423604394983260738922885519999578275840",
  "log_group": "test",
  "log_stream": "test",
  "message": "{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-latform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}",
  "owner": "111111111111",
  "timestamp": "2020-09-14T19:09:29.039Z",
  "subscription_filters": [
    "Destination"
  ]
}

How It Works

State

This component is stateless, meaning its behavior is consistent across each input.

Structured Log Events

Note that the events themselves are not parsed. If they are structured data, you will typically want to pass them through a parsing transform.

#Warnings

#Configuration

#field

#Output

#id

#log_group

#log_stream

#message

#owner

#subscription_filters

#timestamp

#Telemetry

#processing_errors_total

#events_in_total

#processed_events_total

#events_out_total

#processed_bytes_total

#Examples

#How It Works

#State

#Structured Log Events

Warnings

Configuration

field

Output

id

log_group

log_stream

message

owner

subscription_filters

timestamp

Telemetry

processing_errors_total

events_in_total

processed_events_total

events_out_total

processed_bytes_total

Examples

How It Works

State

Structured Log Events