AWS EC2 Metadata Transform

The Vector aws_ec2_metadata transform enriches log events with AWS EC2 environment metadata.

Requirements

• Running this transform within Docker on EC2 requires 2 network hops. Users must raise this limit: bash aws ec2 modify-instance-metadata-options --instance-id <ID> --http-endpoint enabled --http-put-response-hop-limit 2
• AWS EC2 instance metadata >= 2 is required.

Configuration

• Common
• Advanced

• vector.toml
• vector.yaml
• vector.json

[transforms.my_transform_id]
  type = "aws_ec2_metadata" # required
  inputs = ["my-source-or-transform-id", "prefix-*"] # required
  fields = ["instance-id", "local-hostname", "local-ipv4", "public-hostname", "public-ipv4", "ami-id", "availability-zone", "vpc-id", "subnet-id", "region"] # optional, default
  namespace = "" # optional, default
  refresh_interval_secs = 10 # optional, default

• optionalstring

  endpoint

  Override the default EC2 Metadata endpoint.

  • Syntax: literal
  • Default: "http://169.254.169.254"
• commonoptional[string]

  fields

  A list of fields to include in each event.

  • Default: ["instance-id","local-hostname","local-ipv4","public-hostname","public-ipv4","ami-id","availability-zone","vpc-id","subnet-id","region"]
  • View examples
• commonoptionalstring

  namespace

  Prepend a namespace to each field's key.

  • Syntax: literal
  • Default: ""
  • View examples
• commonoptionaluint

  refresh_interval_secs

  The interval in seconds at which the EC2 Metadata api will be called.

  • Default: 10

Output

This component outputs log events with the following fields:

{
  "ami-id" : "ami-00068cd7555f543d5",
  "availability-zone" : "54.234.246.107",
  "instance-id" : "i-096fba6d03d36d262",
  "local-hostname" : "ip-172-31-93-227.ec2.internal",
  "local-ipv4" : "172.31.93.227",
  "public-hostname" : "ec2-54-234-246-107.compute-1.amazonaws.com",
  "public-ipv4" : "54.234.246.107",
  "region" : "us-east-1",
  "role-name" : "some_iam_role",
  "subnet-id" : "subnet-9d6713b9",
  "vpc-id" : "vpc-a51da4dc"
}

11 items

commonrequired

• commonrequiredstring

  ami-id

  The ami-id that the current EC2 instance is using.

  • Syntax: literal
  • View examples
• commonrequiredstring

  availability-zone

  The availability-zone that the current EC2 instance is running in.

  • Syntax: literal
  • View examples
• commonrequiredstring

  instance-id

  The instance-id of the current EC2 instance.

  • Syntax: literal
  • View examples
• commonrequiredstring

  local-hostname

  The local-hostname of the current EC2 instance.

  • Syntax: literal
  • View examples
• commonrequiredstring

  local-ipv4

  The local-ipv4 of the current EC2 instance.

  • Syntax: literal
  • View examples
• commonrequiredstring

  public-hostname

  The public-hostname of the current EC2 instance.

  • Syntax: literal
  • View examples
• commonrequiredstring

  public-ipv4

  The public-ipv4 of the current EC2 instance.

  • Syntax: literal
  • View examples
• commonrequiredstring

  region

  The region that the current EC2 instance is running in.

  • Syntax: literal
  • View examples
• commonrequiredstring

  role-name

  The role-name that the current EC2 instance is using.

  • Syntax: literal
  • View examples
• commonrequiredstring

  subnet-id

  The subnet-id of the current EC2 instance's default network interface.

  • Syntax: literal
  • View examples
• commonrequiredstring

  vpc-id

  The vpc-id of the current EC2 instance's default network interface.

  • Syntax: literal
  • View examples

Telemetry

This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

• counter

  metadata_refresh_failed_total

  The total number of failed efforts to refresh AWS EC2 metadata. This metric includes the following tags:

  • component_kind - The Vector component kind.

  • component_name - The Vector component ID.

  • component_type - The Vector component type.

  • instance - The Vector instance identified by host and port.

  • job - The name of the job producing Vector metrics.

• counter

  events_in_total

  The number of events accepted by this component either from tagged origin like file and uri, or cumulatively from other origins. This metric includes the following tags:

  • component_kind - The Vector component kind.

  • component_name - The Vector component ID.

  • component_type - The Vector component type.

  • container_name - The name of the container from which the event originates.

  • file - The file from which the event originates.

  • instance - The Vector instance identified by host and port.

  • job - The name of the job producing Vector metrics.

  • mode - The connection mode used by the component.

  • peer_addr - The IP from which the event originates.

  • peer_path - The pathname from which the event originates.

  • pod_name - The name of the pod from which the event originates.

  • uri - The sanitized uri from which the event originates.

• counter

  processed_events_total

  The total number of events processed by this component. This metric includes the following tags:

  • component_kind - The Vector component kind.

  • component_name - The Vector component ID.

  • component_type - The Vector component type.

  • file - The file that produced the error

  • instance - The Vector instance identified by host and port.

  • job - The name of the job producing Vector metrics.

• counter

  metadata_refresh_successful_total

  The total number of AWS EC2 metadata refreshes. This metric includes the following tags:

  • component_kind - The Vector component kind.

  • component_name - The Vector component ID.

  • component_type - The Vector component type.

  • instance - The Vector instance identified by host and port.

  • job - The name of the job producing Vector metrics.

• counter

  events_out_total

  The total number of events emitted by this component. This metric includes the following tags:

  • component_kind - The Vector component kind.

  • component_name - The Vector component ID.

  • component_type - The Vector component type.

  • instance - The Vector instance identified by host and port.

  • job - The name of the job producing Vector metrics.

• counter

  processed_bytes_total

  The number of bytes processed by the component. This metric includes the following tags:

  • component_kind - The Vector component kind.

  • component_name - The Vector component ID.

  • component_type - The Vector component type.

  • container_name - The name of the container from which the bytes originate.

  • file - The file from which the bytes originate.

  • instance - The Vector instance identified by host and port.

  • job - The name of the job producing Vector metrics.

  • mode - The connection mode used by the component.

  • peer_addr - The IP from which the bytes originate.

  • peer_path - The pathname from which the bytes originate.

  • pod_name - The name of the pod from which the bytes originate.

  • uri - The sanitized uri from which the bytes originate.

How It Works

State

This component is stateless, meaning its behavior is consistent across each input.