AWS EC2 Metadata Transform

The Vector aws_ec2_metadata transform accepts and outputs log events allowing you to enrich logs with AWS EC2 instance metadata.

Requirements

• AWS IMDS v2 is required. This is available by default on EC2.
• Running this transform within Docker on EC2 requires 2 network hops. Users must raise this limit. See the "AWS imDS v2" section for more info.

Configuration

• Common
• Advanced

vector.toml

[transforms.my_transform_id]
  type = "aws_ec2_metadata" # required
  inputs = ["my-source-id"] # required
  fields = ["instance-id", "local-hostname", "local-ipv4", "public-hostname", "public-ipv4", "ami-id", "availability-zone", "vpc-id", "subnet-id", "region"] # optional, default
  host = "http://169.254.169.254" # optional, default
  namespace = "" # optional, default
  refresh_interval_secs = 10 # optional, default

4 items

common

• [string]commonoptional

  fields

  A list of fields to include in each event.

  • Default: ["instance-id","local-hostname","local-ipv4","public-hostname","public-ipv4","ami-id","availability-zone","vpc-id","subnet-id","region"]
  • View examples
• stringcommonoptional

  host

  Override the default EC2 Metadata host.

  • Default: "http://169.254.169.254"
  • View examples
• stringcommonoptional

  namespace

  Prepend a namespace to each field's key.

  • Default: ""
  • View examples
• intcommonoptional

  refresh_interval_secs

  The interval in seconds at which the EC2 Metadata api will be called.

  • Default: 10
  • View examples

Fields

example log event

{
  // ...
  "ami-id": "ami-00068cd7555f543d5",
  "availability-zone": "54.234.246.107",
  "instance-id": "i-096fba6d03d36d262",
  "local-hostname": "ip-172-31-93-227.ec2.internal",
  "local-ipv4": "172.31.93.227",
  "public-hostname": "ec2-54-234-246-107.compute-1.amazonaws.com",
  "public-ipv4": "54.234.246.107",
  "region": "us-east-1",
  "role-name": "some_iam_role",
  "subnet-id": "subnet-9d6713b9",
  "vpc-id": "vpc-a51da4dc"
  // ...
}

11 items

• stringoptional

  ami-id

  The ami-id that the current EC2 instance is using.

  • No default
  • View examples
• stringoptional

  availability-zone

  The availability-zone that the current EC2 instance is running in.

  • No default
  • View examples
• stringoptional

  instance-id

  The instance-id of the current EC2 instance.

  • No default
  • View examples
• stringoptional

  local-hostname

  The local-hostname of the current EC2 instance.

  • No default
  • View examples
• stringoptional

  local-ipv4

  The local-ipv4 of the current EC2 instance.

  • No default
  • View examples
• stringoptional

  public-hostname

  The public-hostname of the current EC2 instance.

  • No default
  • View examples
• stringoptional

  public-ipv4

  The public-ipv4 of the current EC2 instance.

  • No default
  • View examples
• stringoptional

  region

  The region that the current EC2 instance is running in.

  • No default
  • View examples
• stringoptional

  role-name

  The role-name that the current EC2 instance is using.

  • No default
  • View examples
• stringoptional

  subnet-id

  The subnet-id of the current EC2 instance's default network interface.

  • No default
  • View examples
• stringoptional

  vpc-id

  The vpc-id of the current EC2 instance's default network interface.

  • No default
  • View examples

How It Works

AWS IMDS v2

v2 of the AWS IMDS service addresses a number of very serious security issues with v1. As part of tighening security, Amazon limited the number of network hops allowed to communicate with this service to 1. Unfortunately, when running Vector within Docker this introduces an additional hop. Therefore, you must configure your AWS instances to allow for 2 hops:

aws ec2 modify-instance-metadata-options --instance-id <ID> --http-endpoint enabled --http-put-response-hop-limit 2

If you do not raise this limit the aws_ec2_metadata transform will not work.

Complex Processing

If you encounter limitations with the aws_ec2_metadata transform then we recommend using a runtime transform. These transforms are designed for complex processing and give you the power of full programming runtime.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.