Docker Source

The Vector docker_logs source collects logs from Docker.

Requirements

Warnings

Setup

This component is part of a larger setup strategy for the Docker platform.

Docker Installation

Configuration

[sources.my_source_id]
type = "docker_logs" # required
docker_host = "http://localhost:2375" # optional, no default
include_containers = ["include_", "include_me_0", "ad08cc418cf9"] # optional, no default
include_images = ["httpd", "redis"] # optional, no default
include_labels = ["com.example.vendor=Timber Inc.", "com.example.name=Vector"] # optional, no default
  • optionalbool

    auto_partial_merge

    Setting this to false will disable the automatic merging of partial events. See Merging Split Messages for more info.

    • Default: true
  • commonoptionalstring

    docker_host

    The Docker host to connect to. Use an HTTPS URL to enable TLS encryption. If absent, Vector will try to use DOCKER_HOST enviroment variable. If DOCKER_HOST is also absent, Vector will use default Docker local socket (/var/run/docker.sock on Unix flatforms, //./pipe/docker_engine on Windows).

    • Syntax: literal
  • optional[string]

    exclude_containers

    A list of container IDs or names to match against for containers you don't want to collect logs from. Prefix matches are supported, so you can supply just the first few characters of the ID or name of containers you want to exclude. This can be used in conjunction with include_containers.

  • optionalstring

    host_key

    The key name added to each event representing the current host. This can also be globally set via the global [host_key](#host_key) option.

    • Syntax: literal
    • Default: "host"
  • commonoptional[string]

    include_containers

    A list of container IDs or names to match against for containers you want to collect logs from. Prefix matches are supported, so you can supply just the first few characters of the ID or name of containers you want to include. This can be used in conjunction with exclude_containers.

  • commonoptional[string]

    include_images

    A list of image names to match against. If not provided, all images will be included.

  • commonoptional[string]

    include_labels

    A list of container object labels to match against when filtering running containers. This should follow the described label's syntax in docker object labels docs.

  • optionaltable

    multiline

    Multiline parsing configuration. If not specified, multiline parsing is disabled.

    • commonrequiredstring

      condition_pattern

      Condition regex pattern to look for. Exact behavior is configured via mode.

      This field accepts a valid [Rust regular expression]urls.rustregex_syntax. Wrapping / characters are _not required or permitted.

      • Syntax: regex
    • enumcommonrequiredstring

      mode

      Mode of operation, specifies how the condition_pattern is interpreted.

      • Syntax: literal
      • Enum, must be one of: "continue_through" "continue_past" "halt_before" "halt_with"
    • commonrequiredstring

      start_pattern

      Start regex pattern to look for as a beginning of the message.

      This field accepts a valid [Rust regular expression]urls.rustregex_syntax. Wrapping / characters are _not required or permitted.

      • Syntax: regex
    • commonrequireduint

      timeout_ms

      The maximum time to wait for the continuation. Once this timeout is reached, the buffered message is guaranteed to be flushed, even if incomplete.

  • optionaluint

    retry_backoff_secs

    The amount of time to wait before retrying after an error.

    • Default: 1 (seconds)
  • optionaltable

    tls

    TLS options to connect to the Docker deamon. This has no effect unless docker_host is an HTTPS URL. If absent, Vector will try to use environment variable DOCKER_CERT_PATH and then DOCKER_CONFIG. If both environment variables are absent, Vector will try to read certificates in ~/.docker/.

    • commonrequiredstring

      ca_file

      Path to CA certificate file.

      • Syntax: literal
    • commonrequiredstring

      crt_file

      Path to TLS certificate file.

      • Syntax: literal
    • commonrequiredstring

      key_file

      Path to TLS key file.

      • Syntax: literal

Env Vars

  • commonoptionalstring

    DOCKER_CERT_PATH

    Path to look for TLS certificates when tls configuration is absent. Vector will use:

    • $DOCKER_CERT_PATH/ca.pem: CA certificate.
    • $DOCKER_CERT_PATH/cert.pem: TLS certificate.
    • $DOCKER_CERT_PATH/key.pem: TLS key.
    • Syntax: literal
  • commonoptionalstring

    DOCKER_CONFIG

    Path to look for TLS certificates when both tls configuration and DOCKER_CERT_PATH are absent.

    • Syntax: literal
  • commonoptionalstring

    DOCKER_HOST

    The Docker host to connect to when docker_host configuration is absent.

    • Syntax: literal

Output

This component outputs log events with the following fields:

{
"*" : "Started GET / for 127.0.0.1 at 2012-03-10 14:28:14 +0100",
"container_created_at" : "2020-10-10T17:07:36+00:00",
"container_id" : "9b6247364a03",
"container_name" : "evil_ptolemy",
"host" : "my-host.local",
"image" : "ubuntu:latest",
"message" : "Started GET / for 127.0.0.1 at 2012-03-10 14:28:14 +0100",
"stream" : "stdout",
"timestamp" : "2020-10-10T17:07:36+00:00"
}
  • commonrequiredtimestamp

    container_created_at

    A UTC timestamp representing when the container was created.

  • commonrequiredstring

    container_id

    The Docker container ID that the log was collected from.

    • Syntax: literal
  • commonrequiredstring

    container_name

    The Docker container name that the log was collected from.

    • Syntax: literal
  • commonrequiredstring

    host

    The local hostname, equivalent to the gethostname command.

    • Syntax: literal
  • commonrequiredstring

    image

    The image name that the container is based on.

    • Syntax: literal
  • commonrequiredstring

    *

    Each container label is inserted with it's exact key/value pair.

    • Syntax: literal
  • commonrequiredstring

    message

    The raw log message.

    • Syntax: literal
  • enumcommonrequiredstring

    stream

    The standard stream that the log was collected from.

    • Syntax: literal
    • Enum, must be one of: "stdout" "stderr"
  • commonrequiredtimestamp

    timestamp

    The UTC timestamp extracted from the Docker log event.

Telemetry

This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

  • counter

    events_in_total

    The number of events accepted by this component either from tagged origin like file and uri, or cumulatively from other origins. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • container_name - The name of the container from which the event originates.

    • file - The file from which the event originates.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

    • mode - The connection mode used by the component.

    • peer_addr - The IP from which the event originates.

    • peer_path - The pathname from which the event originates.

    • pod_name - The name of the pod from which the event originates.

    • uri - The sanitized uri from which the event originates.

  • counter

    communication_errors_total

    The total number of errors stemming from communication with the Docker daemon. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    container_metadata_fetch_errors_total

    The total number of errors encountered when fetching container metadata. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    container_processed_events_total

    The total number of container events processed. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    containers_unwatched_total

    The total number of times Vector stopped watching for container logs. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    containers_watched_total

    The total number of times Vector started watching for container logs. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    logging_driver_errors_total

    The total number of logging driver errors encountered caused by not using either the jsonfile or journald driver. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    processed_bytes_total

    The number of bytes processed by the component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • container_name - The name of the container from which the bytes originate.

    • file - The file from which the bytes originate.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

    • mode - The connection mode used by the component.

    • peer_addr - The IP from which the bytes originate.

    • peer_path - The pathname from which the bytes originate.

    • pod_name - The name of the pod from which the bytes originate.

    • uri - The sanitized uri from which the bytes originate.

  • counter

    events_out_total

    The total number of events emitted by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    processed_events_total

    The total number of events processed by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • file - The file that produced the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

Examples

Given the following input:

{
"stream": "stdout",
"message": "150.75.72.205 - - [03/Oct/2020:16:11:29 +0000] "HEAD /initiatives HTTP/1.1" 504 117"
}

And the following configuration:

vector.toml
[sources.docker_logs]
type = "docker_logs"
include_images = ["mingrammer/flog"]

The following Vector log event will be output:

{
"container_created_at": "2020-10-03T16:11:29.443232Z",
"container_id": "fecc98177eca7fb75a2b2186c418bf9a0cd3a05a1169f2e2293bf8987a9d96ab",
"container_name": "flog",
"image": "mingrammer/flog",
"message": "150.75.72.205 - - [03/Oct/2020:16:11:29 +0000] \"HEAD /initiatives HTTP/1.1\" 504 117",
"stream": "stdout",
"host": "my-host.local"
}

How It Works

Context

By default, the docker_logs source will augment events with helpful context keys as shown in the "Output" section.

Merging Split Messages

Docker, by default, will split log messages that exceed 16kb. This can be a rather frustrating problem because it produces malformed log messages that are difficult to work with. Vector's solves this by default, automatically merging these messages into a single message. You can turn this off via the auto_partial_merge option. Furthermore, you can adjust the marker that we use to determine if an event is partial via the partial_event_marker_field option.

State

This component is stateless, meaning its behavior is consistent across each input.