AWS S3
Collect logs from AWS S3
Requirements
Configuration
Example configurations
{
"sources": {
"my_source_id": {
"type": "aws_s3",
"acknowledgements": null,
"region": "us-east-1",
"sqs": null
}
}
}
[sources.my_source_id]
type = "aws_s3"
region = "us-east-1"
---
sources:
my_source_id:
type: aws_s3
acknowledgements: null
region: us-east-1
sqs: null
{
"sources": {
"my_source_id": {
"type": "aws_s3",
"auth": null,
"endpoint": "http://127.0.0.0:5000/path/to/service",
"acknowledgements": null,
"strategy": "sqs",
"compression": "text",
"multiline": null,
"proxy": null,
"tls": null,
"region": "us-east-1",
"sqs": null
}
}
}
[sources.my_source_id]
type = "aws_s3"
endpoint = "http://127.0.0.0:5000/path/to/service"
strategy = "sqs"
compression = "text"
region = "us-east-1"
---
sources:
my_source_id:
type: aws_s3
auth: null
endpoint: http://127.0.0.0:5000/path/to/service
acknowledgements: null
strategy: sqs
compression: text
multiline: null
proxy: null
tls: null
region: us-east-1
sqs: null
acknowledgements
common optional objectacknowledgement
settings. This setting is deprecated in favor of enabling acknowledgements
in the destination sink.acknowledgements.enabled
optional boolfalse
auth
optional objectauth.access_key_id
optional string literalauth.assume_role
optional string literalauth.load_timeout_secs
optional uintassume_role
.5
(seconds)auth.profile
optional string literaldefault
auth.secret_access_key
optional string literalcompression
optional string literal enumOption | Description |
---|---|
auto | Vector will try to determine the compression format of the object from its: Content-Encoding metadata, Content-Type metadata, and key suffix (e.g. .gz ). It will fallback to ‘none’ if it cannot determine the compression. |
gzip | GZIP format. |
none | Uncompressed. |
zstd | ZSTD format. |
text
endpoint
optional string literalmultiline
optional objectmultiline.condition_pattern
optional string regexmode
.multiline.mode
optional string literal enumcondition_pattern
is interpreted.Option | Description |
---|---|
continue_past | All consecutive lines matching this pattern, plus one additional line, are included in the group. This is useful in cases where a log message ends with a continuation marker, such as a backslash, indicating that the following line is part of the same message. |
continue_through | All consecutive lines matching this pattern are included in the group. The first line (the line that matched the start pattern) does not need to match the ContinueThrough pattern. This is useful in cases such as a Java stack trace, where some indicator in the line (such as leading whitespace) indicates that it is an extension of the preceding line. |
halt_before | All consecutive lines not matching this pattern are included in the group. This is useful where a log line contains a marker indicating that it begins a new message. |
halt_with | All consecutive lines, up to and including the first line matching this pattern, are included in the group. This is useful where a log line ends with a termination marker, such as a semicolon. |
multiline.start_pattern
optional string regexmultiline.timeout_ms
optional uintproxy
optional objectproxy.http
optional string literalproxy.https
optional string literalproxy.no_proxy
optional [string]A list of hosts to avoid proxying. Allowed patterns here include:
Pattern | Example match |
---|---|
Domain names | example.com matches requests to example.com |
Wildcard domains | .example.com matches requests to example.com and its subdomains |
IP addresses | 127.0.0.1 matches requests to 127.0.0.1 |
CIDR blocks | 192.168.0.0./16 matches requests to any IP addresses in this range |
Splat | * matches all hosts |
sqs
common optional objectsqs
.sqs.delete_message
optional boolfalse
to debug or during initial Vector setup.true
sqs.queue_url
optional string literalsqs.visibility_timeout_secs
optional uintvector
does not delete the message before the timeout expires, it will be made reavailable for another consumer; this can happen if, for example, the vector
process crashes.300
(seconds)strategy
optional string literal enumOption | Description |
---|---|
sqs | Consume S3 objects by polling for bucket notifications sent to an AWS SQS queue. |
sqs
tls
optional objecttls.ca_file
optional string literaltls.crt_file
optional string literalkey_file
must also be set.tls.key_file
optional string literalcrt_file
must also be set.tls.key_pass
optional string literalkey_file
is set.tls.verify_certificate
optional booltrue
(the default), Vector will validate the TLS certificate of the remote host.true
tls.verify_hostname
optional booltrue
(the default), Vector will validate the configured remote host name against the remote host’s TLS certificate. Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.true
Environment variables
AWS_ACCESS_KEY_ID
common optional string literalAWS_CONFIG_FILE
common optional string literal~/.aws/config
AWS_CREDENTIAL_EXPIRATION
common optional string literalAWS_DEFAULT_REGION
common optional string literalAWS_PROFILE
common optional string literaldefault
AWS_ROLE_SESSION_NAME
common optional string literalAWS_SECRET_ACCESS_KEY
common optional string literalAWS_SESSION_TOKEN
common optional string literalAWS_SHARED_CREDENTIALS_FILE
common optional string literal~/.aws/credentials
HTTPS_PROXY
common optional string literalThe global URL to proxy HTTPS requests through.
If another HTTPS proxy is set in the configuration file or at a component level, this one will be overridden.
The lowercase variant has priority over the uppercase one.
HTTP_PROXY
common optional string literalThe global URL to proxy HTTP requests through.
If another HTTP proxy is set in the configuration file or at a component level, this one will be overridden.
The lowercase variant has priority over the uppercase one.
NO_PROXY
common optional string literalList of hosts to avoid proxying globally.
Allowed patterns here include:
Pattern | Example match |
---|---|
Domain names | example.com matches requests to example.com |
Wildcard domains | .example.come matches requests to example.com and its subdomains |
IP addresses | 127.0.0.1 matches requests to 127.0.0.1 |
CIDR blocks | 192.168.0.0./16 matches requests to any IP addresses in this range |
Splat | * matches all hosts |
If another no_proxy
value is set in the configuration file or at a component level, this
one is overridden.
The lowercase variant has priority over the uppercase one.
http_proxy
common optional string literalThe global URL to proxy HTTP requests through.
If another HTTP proxy is set in the configuration file or at a component level, this one will be overridden.
The lowercase variant has priority over the uppercase one.
https_proxy
common optional string literalThe global URL to proxy HTTPS requests through.
If another HTTPS proxy is set in the configuration file or at a component level, this one will be overridden.
The lowercase variant has priority over the uppercase one.
no_proxy
common optional string literalList of hosts to avoid proxying globally.
Allowed patterns here include:
Pattern | Example match |
---|---|
Domain names | example.com matches requests to example.com |
Wildcard domains | .example.come matches requests to example.com and its subdomains |
IP addresses | 127.0.0.1 matches requests to 127.0.0.1 |
CIDR blocks | 192.168.0.0./16 matches requests to any IP addresses in this range |
Splat | * matches all hosts |
If another no_proxy
value is set in the configuration file or at a component level, this
one is overridden.
The lowercase variant has priority over the uppercase one.
Outputs
<component_id>
Output Data
Logs
Object
my-bucket
53.126.150.246 - - [01/Oct/2020:11:25:58 -0400] "GET /disintermediate HTTP/2.0" 401 20308
AWSLogs/111111111111/vpcflowlogs/us-east-1/2020/10/26/111111111111_vpcflowlogs_us-east-1_fl-0c5605d9f1baf680d_20201026T1950Z_b1ea4a7a.log.gz
us-east-1
2020-10-10T17:07:36.452332Z
Telemetry
Metrics
linkcomponent_discarded_events_total
countercomponent_id
instead. The value is the same as component_id
.component_errors_total
countercomponent_id
instead. The value is the same as component_id
.component_received_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_received_event_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_received_events_total
countercomponent_id
instead. The value is the same as component_id
.component_sent_event_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_sent_events_total
countercomponent_id
instead. The value is the same as component_id
.events_in_total
countercomponent_received_events_total
instead.component_id
instead. The value is the same as component_id
.events_out_total
countercomponent_sent_events_total
instead.component_id
instead. The value is the same as component_id
.processed_bytes_total
countercomponent_id
instead. The value is the same as component_id
.sqs_message_delete_failed_total
countercomponent_id
instead. The value is the same as component_id
.sqs_message_delete_succeeded_total
countercomponent_id
instead. The value is the same as component_id
.sqs_message_processing_failed_total
countercomponent_id
instead. The value is the same as component_id
.sqs_message_processing_succeeded_total
countercomponent_id
instead. The value is the same as component_id
.sqs_message_receive_failed_total
countercomponent_id
instead. The value is the same as component_id
.sqs_message_receive_succeeded_total
countercomponent_id
instead. The value is the same as component_id
.sqs_message_received_messages_total
countercomponent_id
instead. The value is the same as component_id
.sqs_s3_event_record_ignored_total
counterObjectCreated
).component_id
instead. The value is the same as component_id
.Permissions
Policy | Required for | Required when |
---|---|---|
s3:GetObject |
|
Policy | Required for | Required when |
---|---|---|
sqs:ReceiveMessage |
| strategy is set to sqs |
sqs:DeleteMessage |
| strategy is set to sqs and delete_message is set to true |
How it works
AWS authentication
Vector checks for AWS credentials in the following order:
- The
access_key_id
andsecret_access_key
options. - The
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
environment variables. - The AWS credentials file (usually located at
~/.aws/credentials
). - The IAM instance profile (only works if running on an EC2 instance with an instance profile/role). Requires IMDSv2 to be enabled. For EKS, you may need to increase the metadata token response hop limit to 2.
Note that use of
credentials_process
in AWS credentials files is not supported as the underlying AWS SDK currently lacks
support.
If no credentials are found, Vector’s health check fails and an error is logged. If your AWS credentials expire, Vector will automatically search for up-to-date credentials in the places (and order) described above.
Obtaining an access key
access_key_id
and secret_access_key
options.Assuming roles
assume_role
option. This is an
optional setting that is helpful for a variety of use cases, such as cross
account access.Handling events from the aws_s3
source
This source behaves very similarly to the file
source in that
it will output one event per line (unless the multiline
configuration option is used).
You will commonly want to use transforms to
parse the data. For example, to parse VPC flow logs sent to S3 you can
chain the tokenizer
transform:
[transforms.flow_logs]
type = "tokenizer" # required
inputs = ["s3"]
field_names = ["version", "account_id", "interface_id", "srcaddr", "dstaddr", "srcport", "dstport", "protocol", "packets", "bytes", "start", "end", "action", "log_status"]
types.srcport = "int"
types.dstport = "int"
types.packets = "int"
types.bytes = "int"
types.start = "timestamp|%s"
types.end = "timestamp|%s"
To parse AWS load balancer logs, the regex_parser
transform can be used:
[transforms.elasticloadbalancing_fields_parsed]
type = "regex_parser"
inputs = ["s3"]
regex = '''
(?x)^
(?P<type>[\w]+)[ ]
(?P<timestamp>[\w:.-]+)[ ]
(?P<elb>[^\s]+)[ ]
(?P<client_host>[\d.:-]+)[ ]
(?P<target_host>[\d.:-]+)[ ]
(?P<request_processing_time>[\d.-]+)[ ]
(?P<target_processing_time>[\d.-]+)[ ]
(?P<response_processing_time>[\d.-]+)[ ]
(?P<elb_status_code>[\d-]+)[ ]
(?P<target_status_code>[\d-]+)[ ]
(?P<received_bytes>[\d-]+)[ ]
(?P<sent_bytes>[\d-]+)[ ]
"(?P<request_method>[\w-]+)[ ]
(?P<request_url>[^\s]+)[ ]
(?P<request_protocol>[^"\s]+)"[ ]
"(?P<user_agent>[^"]+)"[ ]
(?P<ssl_cipher>[^\s]+)[ ]
(?P<ssl_protocol>[^\s]+)[ ]
(?P<target_group_arn>[\w.:/-]+)[ ]
"(?P<trace_id>[^\s"]+)"[ ]
"(?P<domain_name>[^\s"]+)"[ ]
"(?P<chosen_cert_arn>[\w:./-]+)"[ ]
(?P<matched_rule_priority>[\d-]+)[ ]
(?P<request_creation_time>[\w.:-]+)[ ]
"(?P<actions_executed>[\w,-]+)"[ ]
"(?P<redirect_url>[^"]+)"[ ]
"(?P<error_reason>[^"]+)"
'''
field = "message"
drop_failed = false
types.received_bytes = "int"
types.request_processing_time = "float"
types.sent_bytes = "int"
types.target_processing_time = "float"
types.response_processing_time = "float"
[transforms.elasticloadbalancing_url_parsed]
type = "regex_parser"
inputs = ["elasticloadbalancing_fields_parsed"]
regex = '^(?P<url_scheme>[\w]+)://(?P<url_hostname>[^\s:/?#]+)(?::(?P<request_port>[\d-]+))?-?(?:/(?P<url_path>[^\s?#]*))?(?P<request_url_query>\?[^\s#]+)?'
field = "request_url"
drop_failed = false
Transport Layer Security (TLS)
tls.*
options.