AWS EC2 Metadata Transform

The Vector aws_ec2_metadata transform accepts and outputs log events allowing you to enrich logs with AWS EC2 instance metadata.

Requirements

Configuration

vector.toml
[transforms.my_transform_id]
type = "aws_ec2_metadata" # required
inputs = ["my-source-id"] # required
fields = ["instance-id", "local-hostname", "local-ipv4", "public-hostname", "public-ipv4", "ami-id", "availability-zone", "vpc-id", "subnet-id", "region"] # optional, default
host = "http://169.254.169.254" # optional, default
namespace = "" # optional, default
refresh_interval_secs = 10 # optional, default
4 items
[string]commonoptional

fields

A list of fields to include in each event.

Default: ["instance-id","local-hostname","local-ipv4","public-hostname","public-ipv4","ami-id","availability-zone","vpc-id","subnet-id","region"]
View examples
stringcommonoptional

host

Override the default EC2 Metadata host.

Default: "http://169.254.169.254"
View examples
stringcommonoptional

namespace

Prepend a namespace to each field's key.

No default
View examples
intcommonoptional

refresh_interval_secs

The interval in seconds at which the EC2 Metadata api will be called.

Default: 10
View examples

Output

The aws_ec2_metadata transform accepts and outputs log events allowing you to enrich logs with AWS EC2 instance metadata. For example:

{
"ami-id": "ami-00068cd7555f543d5",
"availability-zone": "54.234.246.107",
"instance-id": "i-096fba6d03d36d262",
"local-hostname": "ip-172-31-93-227.ec2.internal",
"local-ipv4": "172.31.93.227",
"public-hostname": "ec2-54-234-246-107.compute-1.amazonaws.com",
"public-ipv4": "54.234.246.107",
"region": "us-east-1",
"role-name": "some_iam_role",
"subnet-id": "subnet-9d6713b9",
"vpc-id": "vpc-a51da4dc"
}

More detail on the output schema is below.

11 items
stringoptional

ami-id

The ami-id that the current EC2 instance is using.

No default
View examples
stringoptional

availability-zone

The availability-zone that the current EC2 instance is running in.

No default
View examples
stringoptional

instance-id

The instance-id of the current EC2 instance.

No default
View examples
stringoptional

local-hostname

The local-hostname of the current EC2 instance.

No default
View examples
stringoptional

local-ipv4

The local-ipv4 of the current EC2 instance.

No default
View examples
stringoptional

public-hostname

The public-hostname of the current EC2 instance.

No default
View examples
stringoptional

public-ipv4

The public-ipv4 of the current EC2 instance.

No default
View examples
stringoptional

region

The region that the current EC2 instance is running in.

No default
View examples
stringoptional

role-name

The role-name that the current EC2 instance is using.

No default
View examples
stringoptional

subnet-id

The subnet-id of the current EC2 instance's default network interface.

No default
View examples
stringoptional

vpc-id

The vpc-id of the current EC2 instance's default network interface.

No default
View examples

How It Works

AWS IMDS v2

v2 of the AWS IMDS service addresses a number of very serious security issues with v1. As part of tighening security, Amazon limited the number of network hops allowed to communicate with this service to 1. Unfortunately, when running Vector within Docker this introduces an additional hop. Therefore, you must configure your AWS instances to allow for 2 hops:

aws ec2 modify-instance-metadata-options --instance-id <ID> --http-endpoint enabled --http-put-response-hop-limit 2

If you do not raise this limit the aws_ec2_metadata transform will not work.

Complex Processing

If you encounter limitations with the aws_ec2_metadata transform then we recommend using a runtime transform. These transforms are designed for complex processing and give you the power of full programming runtime.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.