AWS EC2 Metadata Transform

The Vector aws_ec2_metadata transform accepts and outputs log events allowing you to enrich logs with AWS EC2 instance metadata.

Requirements

• AWS IMDS v2 is required. This is available by default on EC2.
• Running this transform within Docker on EC2 requires 2 network hops. Users must raise this limit. See the "AWS imDS v2" section for more info.

Configuration

vector.toml

[transforms.my_transform_id]
  type = "aws_ec2_metadata" # required
  inputs = ["my-source-id"] # required
  fields = ["instance-id", "local-hostname", "local-ipv4", "public-hostname", "public-ipv4", "ami-id", "availability-zone", "vpc-id", "subnet-id", "region"] # optional, default
  host = "http://169.254.169.254" # optional, default
  namespace = "" # optional, default
  refresh_interval_secs = 10 # optional, default

4 items

common only

[string]commonoptional

fields

A list of fields to include in each event.

Default: ["instance-id","local-hostname","local-ipv4","public-hostname","public-ipv4","ami-id","availability-zone","vpc-id","subnet-id","region"]

View examples

stringcommonoptional

host

Override the default EC2 Metadata host.

Default: "http://169.254.169.254"

View examples

stringcommonoptional

namespace

Prepend a namespace to each field's key.

No default

View examples

intcommonoptional

refresh_interval_secs

The interval in seconds at which the EC2 Metadata api will be called.

Default: 10

View examples

Output

The aws_ec2_metadata transform accepts and outputs log events allowing you to enrich logs with AWS EC2 instance metadata. For example:

{
  "ami-id": "ami-00068cd7555f543d5",
  "availability-zone": "54.234.246.107",
  "instance-id": "i-096fba6d03d36d262",
  "local-hostname": "ip-172-31-93-227.ec2.internal",
  "local-ipv4": "172.31.93.227",
  "public-hostname": "ec2-54-234-246-107.compute-1.amazonaws.com",
  "public-ipv4": "54.234.246.107",
  "region": "us-east-1",
  "role-name": "some_iam_role",
  "subnet-id": "subnet-9d6713b9",
  "vpc-id": "vpc-a51da4dc"
}

More detail on the output schema is below.

11 items

stringoptional

ami-id

The ami-id that the current EC2 instance is using.

No default

View examples

stringoptional

availability-zone

The availability-zone that the current EC2 instance is running in.

No default

View examples

stringoptional

instance-id

The instance-id of the current EC2 instance.

No default

View examples

stringoptional

local-hostname

The local-hostname of the current EC2 instance.

No default

View examples

stringoptional

local-ipv4

The local-ipv4 of the current EC2 instance.

No default

View examples

stringoptional

public-hostname

The public-hostname of the current EC2 instance.

No default

View examples

stringoptional

public-ipv4

The public-ipv4 of the current EC2 instance.

No default

View examples

stringoptional

region

The region that the current EC2 instance is running in.

No default

View examples

stringoptional

role-name

The role-name that the current EC2 instance is using.

No default

View examples

stringoptional

subnet-id

The subnet-id of the current EC2 instance's default network interface.

No default

View examples

stringoptional

vpc-id

The vpc-id of the current EC2 instance's default network interface.

No default

View examples

How It Works

AWS IMDS v2

v2 of the AWS IMDS service addresses a number of very serious security issues with v1. As part of tighening security, Amazon limited the number of network hops allowed to communicate with this service to 1. Unfortunately, when running Vector within Docker this introduces an additional hop. Therefore, you must configure your AWS instances to allow for 2 hops:

aws ec2 modify-instance-metadata-options --instance-id <ID> --http-endpoint enabled --http-put-response-hop-limit 2

If you do not raise this limit the aws_ec2_metadata transform will not work.

Complex Processing

If you encounter limitations with the aws_ec2_metadata transform then we recommend using a runtime transform. These transforms are designed for complex processing and give you the power of full programming runtime.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.