Docker Source

The Vector docker source ingests data through the Docker engine daemon and outputs log events.

Requirements

Configuration

vector.toml
[sources.my_source_id]
type = "docker" # required
include_containers = ["serene_", "serene_leakey", "ad08cc418cf9"] # optional, no default
include_images = ["httpd", "redis"] # optional, no default
include_labels = ["com.example.vendor=Timber Inc.", "com.example.name=Vector"] # optional, no default
  • booloptional

    auto_partial_merge

    Setting this to false will disable the automatic merging of partial events.

    See Message Splitting & Merging for more info.

    • Default: true
    • View examples
  • [string]commonoptional

    include_containers

    A list of container IDs or names to match against. Prefix matches are supported, meaning you can supply just the first few characters of the container ID or name. If not provided, all containers will be included.

    • No default
    • View examples
  • [string]commonoptional

    include_images

    A list of image names to match against. If not provided, all images will be included.

    • No default
    • View examples
  • [string]commonoptional

    include_labels

    A list of container object labels to match against when filtering running containers. This should follow the described label's synatx in docker object labels docs.

    • No default
    • View examples
  • stringoptional

    partial_event_marker_field

    The field name to be added to events that are detected to contain an incomplete message (i.e. partial events). If set to "", no field will be added to partial event. This allows to opt-out of partial event detection.

    See Message Splitting & Merging for more info.

    • Default: "_partial"
    • View examples

Env Vars

  • stringoptional

    DOCKER_HOST

    The docker host to connect to.

    See Connecting To The Docker Daemon for more info.

    • Default: "unix:///var/run/docker.sock"
    • View examples
  • booloptional

    DOCKER_VERIFY_TLS

    If true (the default), Vector will validate the TLS certificate of the remote host. Do NOT set this to false unless you understand the risks of not verifying the remote certificate.

    See Connecting To The Docker Daemon for more info.

    • Default: true
    • View examples

Fields

example log event
{
// ...
"container_created_at": "2019-11-01T21:15:47+00:00",
"container_id": "9b6247364a03",
"container_name": "evil_ptolemy",
"image": "ubuntu:latest",
"com.example.vendor": "Timber Inc.",
"message": "Started GET / for 127.0.0.1 at 2012-03-10 14:28:14 +0100",
"stream": "stdout",
"timestamp": "2019-11-01T21:15:47+00:00"
// ...
}
  • timestampcommonrequired

    container_created_at

    A UTC timestamp representing when the container was created.

    • No default
    • View examples
  • stringcommonrequired

    container_id

    The Docker container ID that the log was collected from.

    • No default
    • View examples
  • stringcommonrequired

    container_name

    The Docker container name that the log was collected from.

    • No default
    • View examples
  • stringcommonrequired

    image

    The image name that the container is based on.

    • No default
    • View examples
  • stringcommonrequired

    [label-key]

    Docker object labels. Each label is inserted with it's exact key/value pair.

    • No default
    • View examples
  • stringcommonrequired

    message

    The raw log message, unaltered.

    • No default
    • View examples
  • stringenumcommonrequired

    stream

    The standard stream that the log was collected from.

    • No default
    • Enum, must be one of: "stdout" "stderr"
    • View examples
  • timestampcommonrequired

    timestamp

    The UTC timestamp extracted from the Docker log event.

    • No default
    • View examples

How It Works

Connecting To The Docker Daemon

Vector will automatically attempt to connect to the docker daemon for you. If the user that Vector is running under can run docker ps then Vector will be able to connect. Vector will also respect if DOCKER_HOST and DOCKER_VERIFY_TLS are set (as well as other Docker environment variables). See the Docker daemon docs.

Docker Integration Strategy

There are two primary ways through which you can integrate with Docker to receive its logs:

  1. Interact with the Docker daemon directly via the docker logs command. (simplest)
  2. Configure a compatible Docker logging driver with a matching Vector source. (advanced)

The Vector docker source implements option 1. This is the simplest option, but it is prone to performance and stability issues with large deployments. If you experience this, please see the Alternate Strategies section below.

Alternate Strategies

First, it's worth mentioning that Vector strives to guide you towards the optimal observability setup without presenting you with unncessary details or questions. Unfortunately, there are circumstances where tradeoffs must be made and you must determine which tradeoffs are appropriate. Docker is one of these circumstances.

Second, if you have a large container-based deployment you should consider using a platform Kubernetes. These platforms provide alternate log collection means that side-step the Docker logging problems. For supported platforms see Vector's Platforms installation section.

Finally, if you cannot use a container orchestrator then you can configure a compatible Docker logging driver with a matching Vector source. For example:

  1. The Docker syslog driver with the Vector syslog source.
  2. The Docker journald driver with the Vector journald source.
  3. The Docker splunk driver with the Vector splunk_hec source.

To our knowledge there is no discernable difference in performance or stability between any of these. If we had to recommend one, we would recommend the syslog combination.

Docker Logging Drivers

In order for the Vector docker source to work properly, you must configure the json-file (default) or journald Docker logging drivers. This is a requirement of the Docker daemon, which Vector uses to integrate. See the Docker Integration Strategy section for more info.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.

Message Splitting & Merging

Docker, by default, will split log messages that exceed 16kb. This can be a rather frustrating problem because it produces malformed log messages that are difficult to work with. Vector's docker source solves this by default, automatically merging these messages into a single message. You can turn this off via the auto_partial_merge option. Furthermore, you can adjust the marker that we use to determine if an event is partial via the partial_event_marker_field option.