AWS CloudWatch Logs Subscription Parser Transform
The Vector aws_cloudwatch_logs_subscription_parser
transform
parses AWS CloudWatch Logs events (configured through AWS Cloudwatch
subscriptions) coming from the aws_kinesis_firehose
source.
Configuration
- Common
- Advanced
- vector.toml
- vector.yaml
- vector.json
[transforms.my_transform_id]type = "aws_cloudwatch_logs_subscription_parser" # requiredinputs = ["my-source-or-transform-id"] # requiredfield = "message" # optional, default
- optionalstring
field
The log field to decode as an AWS CloudWatch Logs Subscription JSON event. The field must hold a string value.
- Default:
"message"
- Default:
Output
This component outputs log events with the following fields:
{"id" : "35683658089614582423604394983260738922885519999578275840","log_group" : "/lambda/test","log_stream" : "2020/03/24/[$LATEST]794dbaf40a7846c4984ad80ebf110544","message" : "hello","owner" : "111111111111","subscription_filters" : ["Destination"],"timestamp" : "2020-10-10T17:07:36+00:00"}
- requiredstring
id
The CloudWatch Logs event id.
- View examples
- requiredstring
log_group
The log group the event came from.
- View examples
- requiredstring
log_stream
The log stream the event came from.
- View examples
- requiredstring
message
The body of the log event.
- View examples
- requiredstring
owner
The ID of the AWS account the logs came from.
- View examples
- required[string]
subscription_filters
The list of subscription filter names that the logs were sent by.
- View examples
- requiredtimestamp
timestamp
The timestamp of the log event.
- View examples
Telemetry
This component provides the following metrics that can be retrieved through
the internal_metrics
source. See the
metrics section in the
monitoring page for more info.
- counter
processing_errors_total
The total number of processing errors encountered by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.error_type
- The type of the errorinstance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_events_total
The total number of events processed by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.file
- The file that produced the errorinstance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_bytes_total
The total number of bytes processed by the component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
Examples
Given the following Vector event:
{"log": {"message": "\t{\n\t \"messageType\": \"DATA_MESSAGE\",\n\t \"owner\": \"111111111111\",\n\t \"logGroup\": \"test\",\n\t \"logStream\": \"test\",\n\t \"subscriptionFilters\": [\n\t\t\"Destination\"\n\t ],\n\t \"logEvents\": [\n\t\t{\n\t\t \"id\": \"35683658089614582423604394983260738922885519999578275840\",\n\t\t \"timestamp\": 1600110569039,\n\t\t \"message\": \"{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}\"\n\t\t},\n\t\t{\n\t\t \"id\": \"35683658089659183914001456229543810359430816722590236673\",\n\t\t \"timestamp\": 1600110569041,\n\t\t \"message\": \"{\"bytes\":17707,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"109.81.244.252\",\"method\":\"GET\",\"protocol\":\"HTTP/2.0\",\"referer\":\"http://www.investormission-critical.io/24/7/vortals\",\"request\":\"/scale/functionalities/optimize\",\"source_type\":\"stdin\",\"status\":502,\"user-identifier\":\"feeney1708\"}\"\n\t\t}\n\t ]\n\t}"}}
And the following configuration:
[transforms.aws_cloudwatch_logs_subscription_parser]type = "aws_cloudwatch_logs_subscription_parser"field = "message"
The following Vector log event will be output:
{"id": "35683658089614582423604394983260738922885519999578275840","log_group": "test","log_stream": "test","message": "{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-latform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}","owner": "111111111111","timestamp": "2020-09-14T19:09:29.039Z","subscription_filters": ["Destination"]}
How It Works
Structured Log Events
Note that the events themselves are not parsed. If they are structured data, you will typically want to pass them through a parsing transform.