parse

AWS CloudWatch Logs Subscription Parser Transform

The Vector aws_cloudwatch_logs_subscription_parser transform parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) coming from the aws_kinesis_firehose source.

Configuration

[transforms.my_transform_id]
  type = "aws_cloudwatch_logs_subscription_parser" # required
  inputs = ["my-source-or-transform-id"] # required
  field = "message" # optional, default
  • commonoptionalstring

    field

    The log field to decode as an AWS CloudWatch Logs Subscription JSON event. The field must hold a string value.

    • Default: "message"

Output

This component outputs log events with the following fields:

{
  "id" : "35683658089614582423604394983260738922885519999578275840",
  "log_group" : "/lambda/test",
  "log_stream" : "2020/03/24/[$LATEST]794dbaf40a7846c4984ad80ebf110544",
  "message" : "hello",
  "owner" : "111111111111",
  "subscription_filters" : [
    "Destination"
  ],
  "timestamp" : "2020-10-10T17:07:36+00:00"
}
7 items
  • commonrequiredstring

    id

    The CloudWatch Logs event id.

    • View examples
  • commonrequiredstring

    log_group

    The log group the event came from.

    • View examples
  • commonrequiredstring

    log_stream

    The log stream the event came from.

    • View examples
  • commonrequiredstring

    message

    The body of the log event.

    • View examples
  • commonrequiredstring

    owner

    The ID of the AWS account the logs came from.

    • View examples
  • commonrequired[string]

    subscription_filters

    The list of subscription filter names that the logs were sent by.

    • View examples
  • commonrequiredtimestamp

    timestamp

    The timestamp of the log event.

    • View examples

Telemetry

This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

  • counter

    processing_errors_total

    The total number of processing errors encountered by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • error_type - The type of the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    processed_events_total

    The total number of events processed by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • file - The file that produced the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    processed_bytes_total

    The total number of bytes processed by the component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

Examples

Given the following Vector event:

{
  "log": {
    "message": "\t{\n\t  \"messageType\": \"DATA_MESSAGE\",\n\t  \"owner\": \"111111111111\",\n\t  \"logGroup\": \"test\",\n\t  \"logStream\": \"test\",\n\t  \"subscriptionFilters\": [\n\t\t\"Destination\"\n\t  ],\n\t  \"logEvents\": [\n\t\t{\n\t\t  \"id\": \"35683658089614582423604394983260738922885519999578275840\",\n\t\t  \"timestamp\": 1600110569039,\n\t\t  \"message\": \"{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}\"\n\t\t},\n\t\t{\n\t\t  \"id\": \"35683658089659183914001456229543810359430816722590236673\",\n\t\t  \"timestamp\": 1600110569041,\n\t\t  \"message\": \"{\"bytes\":17707,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"109.81.244.252\",\"method\":\"GET\",\"protocol\":\"HTTP/2.0\",\"referer\":\"http://www.investormission-critical.io/24/7/vortals\",\"request\":\"/scale/functionalities/optimize\",\"source_type\":\"stdin\",\"status\":502,\"user-identifier\":\"feeney1708\"}\"\n\t\t}\n\t  ]\n\t}"
  }
}

And the following configuration:

[transforms.aws_cloudwatch_logs_subscription_parser]
type = "aws_cloudwatch_logs_subscription_parser"
field = "message"

The following Vector log event will be output:

{
  "id": "35683658089614582423604394983260738922885519999578275840",
  "log_group": "test",
  "log_stream": "test",
  "message": "{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-latform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}",
  "owner": "111111111111",
  "timestamp": "2020-09-14T19:09:29.039Z",
  "subscription_filters": [
    "Destination"
  ]
}

How It Works

Structured Log Events

Note that the events themselves are not parsed. If they are structured data, you will typically want to pass them through a parsing transform.

Support
Contents
Resources