Docker Source
The Vector docker_logs
source
collects logs from Docker.
Requirements
Warnings
Setup
This component is part of a larger setup strategy for the Docker platform.
Configuration
- Common
- Advanced
- vector.toml
- vector.yaml
- vector.json
[sources.my_source_id]type = "docker_logs" # requireddocker_host = "http://localhost:2375" # optional, no defaultinclude_containers = ["include_", "include_me_0", "ad08cc418cf9"] # optional, no defaultinclude_images = ["httpd", "redis"] # optional, no defaultinclude_labels = ["com.example.vendor=Timber Inc.", "com.example.name=Vector"] # optional, no default
- optionalbool
auto_partial_merge
Setting this to
false
will disable the automatic merging of partial events. See Merging Split Messages for more info.- Default:
true
- View examples
- Default:
- commonoptionalstring
docker_host
The Docker host to connect to. Use an HTTPS URL to enable TLS encryption. If absent, Vector will try to use
DOCKER_HOST
enviroment variable. IfDOCKER_HOST
is also absent, Vector will use default Docker local socket (/var/run/docker.sock
on Unix flatforms,//./pipe/docker_engine
on Windows).- Syntax:
literal
- View examples
- Syntax:
- optional[string]
exclude_containers
A list of container IDs or names to match against for containers you don't want to collect logs from. Prefix matches are supported, so you can supply just the first few characters of the ID or name of containers you want to exclude. This can be used in conjunction with
include_containers
.- View examples
- optionalstring
host_key
The key name added to each event representing the current host. This can also be globally set via the global [
host_key
](#host_key) option.- Syntax:
literal
- Default:
"host"
- Syntax:
- commonoptional[string]
include_containers
A list of container IDs or names to match against for containers you want to collect logs from. Prefix matches are supported, so you can supply just the first few characters of the ID or name of containers you want to include. This can be used in conjunction with
exclude_containers
.- View examples
- commonoptional[string]
include_images
A list of image names to match against. If not provided, all images will be included.
- View examples
- commonoptional[string]
include_labels
A list of container object labels to match against when filtering running containers. This should follow the described label's syntax in docker object labels docs.
- View examples
- optionaltable
multiline
Multiline parsing configuration. If not specified, multiline parsing is disabled.
- commonrequiredstring
condition_pattern
Condition regex pattern to look for. Exact behavior is configured via
mode
.This field accepts a valid [Rust regular expression]urls.rustregex_syntax. Wrapping
/
characters are _not required or permitted.- Syntax:
regex
- View examples
- Syntax:
- enumcommonrequiredstring
mode
Mode of operation, specifies how the
condition_pattern
is interpreted.- Syntax:
literal
- Enum, must be one of:
"continue_through"
"continue_past"
"halt_before"
"halt_with"
- View examples
- Syntax:
- commonrequiredstring
start_pattern
Start regex pattern to look for as a beginning of the message.
This field accepts a valid [Rust regular expression]urls.rustregex_syntax. Wrapping
/
characters are _not required or permitted.- Syntax:
regex
- View examples
- Syntax:
- commonrequireduint
timeout_ms
The maximum time to wait for the continuation. Once this timeout is reached, the buffered message is guaranteed to be flushed, even if incomplete.
- View examples
- optionaluint
retry_backoff_secs
The amount of time to wait before retrying after an error.
- Default:
1
(seconds)
- Default:
- optionaltable
tls
TLS options to connect to the Docker deamon. This has no effect unless
docker_host
is an HTTPS URL. If absent, Vector will try to use environment variableDOCKER_CERT_PATH
and thenDOCKER_CONFIG
. If both environment variables are absent, Vector will try to read certificates in~/.docker/
.- commonrequiredstring
ca_file
Path to CA certificate file.
- Syntax:
literal
- View examples
- Syntax:
- commonrequiredstring
crt_file
Path to TLS certificate file.
- Syntax:
literal
- View examples
- Syntax:
- commonrequiredstring
key_file
Path to TLS key file.
- Syntax:
literal
- View examples
- Syntax:
Env Vars
- commonoptionalstring
DOCKER_CERT_PATH
Path to look for TLS certificates when
tls
configuration is absent. Vector will use:$DOCKER_CERT_PATH/ca.pem
: CA certificate.$DOCKER_CERT_PATH/cert.pem
: TLS certificate.$DOCKER_CERT_PATH/key.pem
: TLS key.
- Syntax:
literal
- View examples
- commonoptionalstring
DOCKER_CONFIG
Path to look for TLS certificates when both
tls
configuration andDOCKER_CERT_PATH
are absent.- Syntax:
literal
- View examples
- Syntax:
- commonoptionalstring
DOCKER_HOST
The Docker host to connect to when
docker_host
configuration is absent.- Syntax:
literal
- View examples
- Syntax:
Output
This component outputs log events with the following fields:
{"*" : "Started GET / for 127.0.0.1 at 2012-03-10 14:28:14 +0100","container_created_at" : "2020-10-10T17:07:36+00:00","container_id" : "9b6247364a03","container_name" : "evil_ptolemy","host" : "my-host.local","image" : "ubuntu:latest","message" : "Started GET / for 127.0.0.1 at 2012-03-10 14:28:14 +0100","stream" : "stdout","timestamp" : "2020-10-10T17:07:36+00:00"}
- commonrequiredtimestamp
container_created_at
A UTC timestamp representing when the container was created.
- View examples
- commonrequiredstring
container_id
The Docker container ID that the log was collected from.
- Syntax:
literal
- View examples
- Syntax:
- commonrequiredstring
container_name
The Docker container name that the log was collected from.
- Syntax:
literal
- View examples
- Syntax:
- commonrequiredstring
host
The local hostname, equivalent to the
gethostname
command.- Syntax:
literal
- View examples
- Syntax:
- commonrequiredstring
image
The image name that the container is based on.
- Syntax:
literal
- View examples
- Syntax:
- commonrequiredstring
*
Each container label is inserted with it's exact key/value pair.
- Syntax:
literal
- View examples
- Syntax:
- commonrequiredstring
message
The raw log message.
- Syntax:
literal
- View examples
- Syntax:
- enumcommonrequiredstring
stream
The standard stream that the log was collected from.
- Syntax:
literal
- Enum, must be one of:
"stdout"
"stderr"
- View examples
- Syntax:
- commonrequiredtimestamp
timestamp
The UTC timestamp extracted from the Docker log event.
- View examples
Telemetry
This component provides the following metrics that can be retrieved through
the internal_metrics
source. See the
metrics section in the
monitoring page for more info.
- counter
events_in_total
The number of events accepted by this component either from tagged origin like file and uri, or cumulatively from other origins. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.container_name
- The name of the container from which the event originates.file
- The file from which the event originates.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.mode
- The connection mode used by the component.peer_addr
- The IP from which the event originates.peer_path
- The pathname from which the event originates.pod_name
- The name of the pod from which the event originates.uri
- The sanitized uri from which the event originates.
- counter
communication_errors_total
The total number of errors stemming from communication with the Docker daemon. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
container_metadata_fetch_errors_total
The total number of errors encountered when fetching container metadata. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
container_processed_events_total
The total number of container events processed. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
containers_unwatched_total
The total number of times Vector stopped watching for container logs. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
containers_watched_total
The total number of times Vector started watching for container logs. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
logging_driver_errors_total
The total number of logging driver errors encountered caused by not using either the
jsonfile
orjournald
driver. This metric includes the following tags:component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_bytes_total
The number of bytes processed by the component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.container_name
- The name of the container from which the bytes originate.file
- The file from which the bytes originate.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.mode
- The connection mode used by the component.peer_addr
- The IP from which the bytes originate.peer_path
- The pathname from which the bytes originate.pod_name
- The name of the pod from which the bytes originate.uri
- The sanitized uri from which the bytes originate.
- counter
events_out_total
The total number of events emitted by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_events_total
The total number of events processed by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.file
- The file that produced the errorinstance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
Examples
Given the following input:
{"stream": "stdout","message": "150.75.72.205 - - [03/Oct/2020:16:11:29 +0000] "HEAD /initiatives HTTP/1.1" 504 117"}
And the following configuration:
[sources.docker_logs]type = "docker_logs"include_images = ["mingrammer/flog"]
The following Vector log event will be output:
{"container_created_at": "2020-10-03T16:11:29.443232Z","container_id": "fecc98177eca7fb75a2b2186c418bf9a0cd3a05a1169f2e2293bf8987a9d96ab","container_name": "flog","image": "mingrammer/flog","message": "150.75.72.205 - - [03/Oct/2020:16:11:29 +0000] \"HEAD /initiatives HTTP/1.1\" 504 117","stream": "stdout","host": "my-host.local"}
How It Works
Context
By default, the docker_logs
source will augment events with helpful
context keys as shown in the "Output" section.
Merging Split Messages
Docker, by default, will split log messages that exceed 16kb. This can be a
rather frustrating problem because it produces malformed log messages that are
difficult to work with. Vector's solves this by default, automatically merging
these messages into a single message. You can turn this off via the
auto_partial_merge
option. Furthermore, you can adjust the marker
that we use to determine if an event is partial via the
partial_event_marker_field
option.
State
This component is stateless, meaning its behavior is consistent across each input.