AWS EC2 Metadata Transform

The Vector aws_ec2_metadata transform accepts and outputs log events, allowing you to enrich logs with AWS EC2 instance metadata.

Requirements

Configuration

vector.toml
[transforms.my_transform_id]
type = "aws_ec2_metadata" # required
inputs = ["my-source-or-transform-id"] # required
fields = ["instance-id", "local-hostname", "local-ipv4", "public-hostname", "public-ipv4", "ami-id", "availability-zone", "vpc-id", "subnet-id", "region"] # optional, default
host = "http://169.254.169.254" # optional, default
namespace = "" # optional, default
refresh_interval_secs = 10 # optional, default
  • [string]commonoptional

    fields

    A list of fields to include in each event.

    • Default: ["instance-id","local-hostname","local-ipv4","public-hostname","public-ipv4","ami-id","availability-zone","vpc-id","subnet-id","region"]
    • View examples
  • stringcommonoptional

    host

    Override the default EC2 Metadata host.

    • Default: "http://169.254.169.254"
  • stringcommonoptional

    namespace

    Prepend a namespace to each field's key.

    • Default: ""
    • View examples
  • uintcommonoptional

    refresh_interval_secs

    The interval in seconds at which the EC2 Metadata api will be called.

    • Default: 10

Fields

example log event
{
// ...
"ami-id": "ami-00068cd7555f543d5",
"availability-zone": "54.234.246.107",
"instance-id": "i-096fba6d03d36d262",
"local-hostname": "ip-172-31-93-227.ec2.internal",
"local-ipv4": "172.31.93.227",
"public-hostname": "ec2-54-234-246-107.compute-1.amazonaws.com",
"public-ipv4": "54.234.246.107",
"region": "us-east-1",
"role-name": "some_iam_role",
"subnet-id": "subnet-9d6713b9",
"vpc-id": "vpc-a51da4dc"
// ...
}
  • stringoptional

    ami-id

    The ami-id that the current EC2 instance is using.

    • No default
    • View examples
  • stringoptional

    availability-zone

    The availability-zone that the current EC2 instance is running in.

    • No default
    • View examples
  • stringoptional

    instance-id

    The instance-id of the current EC2 instance.

    • No default
    • View examples
  • stringoptional

    local-hostname

    The local-hostname of the current EC2 instance.

    • No default
    • View examples
  • stringoptional

    local-ipv4

    The local-ipv4 of the current EC2 instance.

    • No default
    • View examples
  • stringoptional

    public-hostname

    The public-hostname of the current EC2 instance.

    • No default
    • View examples
  • stringoptional

    public-ipv4

    The public-ipv4 of the current EC2 instance.

    • No default
    • View examples
  • stringoptional

    region

    The region that the current EC2 instance is running in.

    • No default
    • View examples
  • stringoptional

    role-name

    The role-name that the current EC2 instance is using.

    • No default
    • View examples
  • stringoptional

    subnet-id

    The subnet-id of the current EC2 instance's default network interface.

    • No default
    • View examples
  • stringoptional

    vpc-id

    The vpc-id of the current EC2 instance's default network interface.

    • No default
    • View examples

How It Works

AWS IMDS v2

v2 of the AWS IMDS service addresses a number of very serious security issues with v1. As part of tighening security, Amazon limited the number of network hops allowed to communicate with this service to 1. Unfortunately, when running Vector within Docker this introduces an additional hop. Therefore, you must configure your AWS instances to allow for 2 hops:

aws ec2 modify-instance-metadata-options --instance-id <ID> --http-endpoint enabled --http-put-response-hop-limit 2

If you do not raise this limit the aws_ec2_metadata transform will not work.

Complex Processing

If you encounter limitations with the aws_ec2_metadata transform then we recommend using a runtime transform. These transforms are designed for complex processing and give you the power of full programming runtime.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.