AWS CloudWatch Logs Subscription Parser Transform

The Vector aws_cloudwatch_logs_subscription_parser transform parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) coming from the aws_kinesis_firehose source.

Warnings

Configuration

[transforms.my_transform_id]
type = "aws_cloudwatch_logs_subscription_parser" # required
inputs = ["my-source-or-transform-id", "prefix-*"] # required
field = "message" # optional, default
  • commonoptionalstring

    field

    The log field to decode as an AWS CloudWatch Logs Subscription JSON event. The field must hold a string value.

    • Syntax: literal
    • Default: "message"

Output

This component outputs log events with the following fields:

{
"id" : "35683658089614582423604394983260738922885519999578275840",
"log_group" : "/lambda/test",
"log_stream" : "2020/03/24/[$LATEST]794dbaf40a7846c4984ad80ebf110544",
"message" : "hello",
"owner" : "111111111111",
"subscription_filters" : [
"Destination"
],
"timestamp" : "2020-10-10T17:07:36+00:00"
}
  • commonrequiredstring

    id

    The CloudWatch Logs event id.

    • Syntax: literal
  • commonrequiredstring

    log_group

    The log group the event came from.

    • Syntax: literal
  • commonrequiredstring

    log_stream

    The log stream the event came from.

    • Syntax: literal
  • commonrequiredstring

    message

    The body of the log event.

    • Syntax: literal
  • commonrequiredstring

    owner

    The ID of the AWS account the logs came from.

    • Syntax: literal
  • commonrequired[string]

    subscription_filters

    The list of subscription filter names that the logs were sent by.

  • commonrequiredtimestamp

    timestamp

    The timestamp of the log event.

Telemetry

This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

  • counter

    processing_errors_total

    The total number of processing errors encountered by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • error_type - The type of the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    events_in_total

    The number of events accepted by this component either from tagged origin like file and uri, or cumulatively from other origins. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • container_name - The name of the container from which the event originates.

    • file - The file from which the event originates.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

    • mode - The connection mode used by the component.

    • peer_addr - The IP from which the event originates.

    • peer_path - The pathname from which the event originates.

    • pod_name - The name of the pod from which the event originates.

    • uri - The sanitized uri from which the event originates.

  • counter

    processed_events_total

    The total number of events processed by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • file - The file that produced the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    events_out_total

    The total number of events emitted by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    processed_bytes_total

    The number of bytes processed by the component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • container_name - The name of the container from which the bytes originate.

    • file - The file from which the bytes originate.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

    • mode - The connection mode used by the component.

    • peer_addr - The IP from which the bytes originate.

    • peer_path - The pathname from which the bytes originate.

    • pod_name - The name of the pod from which the bytes originate.

    • uri - The sanitized uri from which the bytes originate.

Examples

Given the following Vector log event:

{
"message": "\t{\n\t \"messageType\": \"DATA_MESSAGE\",\n\t \"owner\": \"111111111111\",\n\t \"logGroup\": \"test\",\n\t \"logStream\": \"test\",\n\t \"subscriptionFilters\": [\n\t\t\"Destination\"\n\t ],\n\t \"logEvents\": [\n\t\t{\n\t\t \"id\": \"35683658089614582423604394983260738922885519999578275840\",\n\t\t \"timestamp\": 1600110569039,\n\t\t \"message\": \"{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}\"\n\t\t},\n\t\t{\n\t\t \"id\": \"35683658089659183914001456229543810359430816722590236673\",\n\t\t \"timestamp\": 1600110569041,\n\t\t \"message\": \"{\"bytes\":17707,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"109.81.244.252\",\"method\":\"GET\",\"protocol\":\"HTTP/2.0\",\"referer\":\"http://www.investormission-critical.io/24/7/vortals\",\"request\":\"/scale/functionalities/optimize\",\"source_type\":\"stdin\",\"status\":502,\"user-identifier\":\"feeney1708\"}\"\n\t\t}\n\t ]\n\t}"
}

And the following configuration:

vector.toml
[transforms.aws_cloudwatch_logs_subscription_parser]
type = "aws_cloudwatch_logs_subscription_parser"
field = "message"

The following Vector log event will be output:

{
"id": "35683658089614582423604394983260738922885519999578275840",
"log_group": "test",
"log_stream": "test",
"message": "{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-latform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}",
"owner": "111111111111",
"timestamp": "2020-09-14T19:09:29.039Z",
"subscription_filters": [
"Destination"
]
}

How It Works

State

This component is stateless, meaning its behavior is consistent across each input.

Structured Log Events

Note that the events themselves are not parsed. If they are structured data, you will typically want to pass them through a parsing transform.