Docker Source
The Vector docker_logs
source
collects logs from Docker.
Requirements
Setup
This component is part of a larger setup strategy for the Docker platform.
Configuration
- Common
- Advanced
- vector.toml
- vector.yaml
- vector.json
[sources.my_source_id]type = "docker_logs" # requireddocker_host = "http://localhost:2375" # optional, no defaultinclude_containers = ["include_", "include_me_0", "ad08cc418cf9"] # optional, no defaultinclude_images = ["httpd", "redis"] # optional, no defaultinclude_labels = ["com.example.vendor=Timber Inc.", "com.example.name=Vector"] # optional, no default
- optionalbool
auto_partial_merge
Setting this to
false
will disable the automatic merging of partial events. See Merging Split Messages for more info.- Default:
true
- View examples
- Default:
- optionalstring
docker_host
The Docker host to connect to. Use an HTTPS URL to enable TLS encryption. If absent, Vector will try to use
DOCKER_HOST
enviroment variable. IfDOCKER_HOST
is also absent, Vector will use default Docker local socket (/var/run/docker.sock
on Unix flatforms,\\.\pipe\docker_engine
on Windows).- View examples
- optional[string]
exclude_containers
A list of container IDs or names to match against for containers you don't want to collect logs from. Prefix matches are supported, so you can supply just the first few characters of the ID or name of containers you want to exclude. This can be used in conjunction with
include_containers
.- View examples
- optional[string]
include_containers
A list of container IDs or names to match against for containers you want to collect logs from. Prefix matches are supported, so you can supply just the first few characters of the ID or name of containers you want to include. This can be used in conjunction with
exclude_containers
.- View examples
- optional[string]
include_images
A list of image names to match against. If not provided, all images will be included.
- View examples
- optional[string]
include_labels
A list of container object labels to match against when filtering running containers. This should follow the described label's synatx in docker object labels docs.
- View examples
- optionaltable
multiline
Multiline parsing configuration. If not specified, multiline parsing is disabled.
- requiredstring
condition_pattern
Condition regex pattern to look for. Exact behavior is configured via
mode
.- View examples
- requiredstring
mode
Mode of operation, specifies how the
condition_pattern
is interpreted.- Enum, must be one of:
"continue_through"
"continue_past"
"halt_before"
"halt_with"
- View examples
- Enum, must be one of:
- requiredstring
start_pattern
Start regex pattern to look for as a beginning of the message.
- View examples
- requireduint
timeout_ms
The maximum time to wait for the continuation. Once this timeout is reached, the buffered message is guaranteed to be flushed, even if incomplete.
- View examples
- optionaluint
retry_backoff_secs
The amount of time to wait before retrying after an error.
- Default:
1
(seconds)
- Default:
- optionaltable
tls
TLS options to connect to the Docker deamon. This has no effect unless
docker_host
is an HTTPS URL. If absent, Vector will try to use environment variableDOCKER_CERT_PATH
and thenDOCKER_CONFIG
. If both environment variables are absent, Vector will try to read certificates in~/.docker/
.- requiredstring
ca_file
Path to CA certificate file.
- View examples
- requiredstring
crt_file
Path to TLS certificate file.
- View examples
- requiredstring
key_file
Path to TLS key file.
- View examples
Env Vars
- optionalstring
DOCKER_CERT_PATH
Path to look for TLS certificates when
tls
configuration is absent. Vector will use:$DOCKER_CERT_PATH/ca.pem
: CA certificate.$DOCKER_CERT_PATH/cert.pem
: TLS certificate.$DOCKER_CERT_PATH/key.pem
: TLS key.
- View examples
- optionalstring
DOCKER_CONFIG
Path to look for TLS certificates when both
tls
configuration andDOCKER_CERT_PATH
are absent.- View examples
- optionalstring
DOCKER_HOST
The Docker host to connect to when
docker_host
configuration is absent.- View examples
Output
This component outputs log events with the following fields:
{"*" : "Started GET / for 127.0.0.1 at 2012-03-10 14:28:14 +0100","container_created_at" : "2020-10-10T17:07:36+00:00","container_id" : "9b6247364a03","container_name" : "evil_ptolemy","image" : "ubuntu:latest","message" : "Started GET / for 127.0.0.1 at 2012-03-10 14:28:14 +0100","stream" : "stdout","timestamp" : "2020-10-10T17:07:36+00:00"}
- requiredstring
*
Each container label is inserted with it's exact key/value pair.
- View examples
- requiredtimestamp
container_created_at
A UTC timestamp representing when the container was created.
- View examples
- requiredstring
container_id
The Docker container ID that the log was collected from.
- View examples
- requiredstring
container_name
The Docker container name that the log was collected from.
- View examples
- requiredstring
image
The image name that the container is based on.
- View examples
- requiredstring
message
The raw log message.
- View examples
- requiredstring
stream
The standard stream that the log was collected from.
- Enum, must be one of:
"stdout"
"stderr"
- View examples
- Enum, must be one of:
- requiredtimestamp
timestamp
The UTC timestamp extracted from the Docker log event.
- View examples
Telemetry
This component provides the following metrics that can be retrieved through
the internal_metrics
source. See the
metrics section in the
monitoring page for more info.
- counter
communication_errors_total
The total number of errors stemming from communication with the Docker daemon. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
container_processed_events_total
The total number of container events processed. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
container_metadata_fetch_errors_total
The total number of errors encountered when fetching container metadata. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
containers_unwatched_total
The total number of times Vector stopped watching for container logs. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
containers_watched_total
The total number of times Vector started watching for container logs. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_events_total
The total number of events processed by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.file
- The file that produced the errorinstance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
logging_driver_errors_total
The total number of logging driver errors encountered caused by not using either the
jsonfile
orjournald
driver. This metric includes the following tags:component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_bytes_total
The total number of bytes processed by the component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
Examples
Given the following input:
{"stream": "stdout","message": "150.75.72.205 - - [03/Oct/2020:16:11:29 +0000] "HEAD /initiatives HTTP/1.1" 504 117"}
And the following configuration:
[sources.docker_logs]type = "docker_logs"include_images = ["mingrammer/flog"]
The following Vector log event will be output:
{"container_created_at": "2020-10-03T16:11:29.443232Z","container_id": "fecc98177eca7fb75a2b2186c418bf9a0cd3a05a1169f2e2293bf8987a9d96ab","container_name": "flog","image": "mingrammer/flog","message": "150.75.72.205 - - [03/Oct/2020:16:11:29 +0000] \"HEAD /initiatives HTTP/1.1\" 504 117","stream": "stdout"}
How It Works
Context
By default, the docker_logs
source will augment events with helpful
context keys as shown in the "Output" section.
Merging Split Messages
Docker, by default, will split log messages that exceed 16kb. This can be a
rather frustrating problem because it produces malformed log messages that are
difficult to work with. Vector's solves this by default, automatically merging
these messages into a single message. You can turn this off via the
auto_partial_merge
option. Furthermore, you can adjust the marker
that we use to determine if an event is partial via the
partial_event_marker_field
option.