Exec
Collect output from a process running on the host
Configuration
Example configurations
{
"sources": {
"my_source_id": {
"type": "exec",
"command": [
"echo"
],
"mode": "scheduled"
}
}
}
[sources.my_source_id]
type = "exec"
command = [ "echo" ]
mode = "scheduled"
sources:
my_source_id:
type: exec
command:
- echo
mode: scheduled
{
"sources": {
"my_source_id": {
"type": "exec",
"command": [
"echo"
],
"environment": {
"LANG": "es_ES.UTF-8",
"PATH": "/bin:/usr/bin:/usr/local/bin",
"TZ": "Etc/UTC"
},
"include_stderr": true,
"maximum_buffer_size_bytes": 1000000,
"mode": "scheduled"
}
}
}
[sources.my_source_id]
type = "exec"
command = [ "echo" ]
include_stderr = true
maximum_buffer_size_bytes = 1_000_000
mode = "scheduled"
[sources.my_source_id.environment]
LANG = "es_ES.UTF-8"
PATH = "/bin:/usr/bin:/usr/local/bin"
TZ = "Etc/UTC"
sources:
my_source_id:
type: exec
command:
- echo
environment:
LANG: es_ES.UTF-8
PATH: /bin:/usr/bin:/usr/local/bin
TZ: Etc/UTC
include_stderr: true
maximum_buffer_size_bytes: 1000000
mode: scheduled
clear_environment
optional boolfalse
command
required [string]decoding
optional objectdecoding.avro
required objectcodec = "avro"
decoding.avro.schema
required string literalThe Avro schema definition.
Please note that the following [apache_avro::types::Value
] variants are currently not supported:
Date
Decimal
Duration
Fixed
TimeMillis
decoding.avro.strip_schema_id_prefix
required booldecoding.codec
optional string literal enumOption | Description |
---|---|
avro | Decodes the raw bytes as as an Apache Avro message. |
bytes | Uses the raw bytes as-is. |
gelf | Decodes the raw bytes as a GELF message. This codec is experimental for the following reason: The GELF specification is more strict than the actual Graylog receiver.
Vector’s decoder currently adheres more strictly to the GELF spec, with
the exception that some characters such as Other GELF codecs such as Loki’s, use a Go SDK that is maintained by Graylog, and is much more relaxed than the GELF spec. Going forward, Vector will use that Go SDK as the reference implementation, which means the codec may continue to relax the enforcement of specification. |
influxdb | Decodes the raw bytes as an Influxdb Line Protocol message. |
json | Decodes the raw bytes as JSON. |
native | Decodes the raw bytes as native Protocol Buffers format. This codec is experimental. |
native_json | Decodes the raw bytes as native JSON format. This codec is experimental. |
protobuf | Decodes the raw bytes as protobuf. |
syslog | Decodes the raw bytes as a Syslog message. Decodes either as the RFC 3164-style format (“old” style) or the RFC 5424-style format (“new” style, includes structured data). |
vrl | Decodes the raw bytes as a string and passes them as input to a VRL program. |
bytes
decoding.gelf
optional objectcodec = "gelf"
decoding.gelf.lossy
optional boolDetermines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER
.
true
decoding.influxdb
optional objectcodec = "influxdb"
decoding.influxdb.lossy
optional boolDetermines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER
.
true
decoding.json
optional objectcodec = "json"
decoding.json.lossy
optional boolDetermines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER
.
true
decoding.native_json
optional objectcodec = "native_json"
decoding.native_json.lossy
optional boolDetermines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER
.
true
decoding.protobuf
optional objectcodec = "protobuf"
decoding.protobuf.desc_file
optional string literaldecoding.protobuf.message_type
optional string literaldecoding.syslog
optional objectcodec = "syslog"
decoding.syslog.lossy
optional boolDetermines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER
.
true
decoding.vrl
required objectcodec = "vrl"
decoding.vrl.source
required string literal.
target will be used as the decoding result.
Compilation error or use of ‘abort’ in a program will result in a decoding error.decoding.vrl.timezone
optional string literalThe name of the timezone to apply to timestamp conversions that do not contain an explicit
time zone. The time zone name may be any name in the TZ database, or local
to indicate system local time.
If not set, local
will be used.
environment
optional objectenvironment.*
required string literalframing
optional objectFraming configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.
framing.character_delimited
required objectmethod = "character_delimited"
framing.character_delimited.delimiter
required ascii_charframing.character_delimited.max_length
optional uintThe maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.
framing.chunked_gelf
optional objectmethod = "chunked_gelf"
framing.chunked_gelf.decompression
optional string literal enumOption | Description |
---|---|
Auto | Automatically detect the decompression method based on the magic bytes of the message. |
Gzip | Use Gzip decompression. |
None | Do not decompress the message. |
Zlib | Use Zlib decompression. |
Auto
framing.chunked_gelf.max_length
optional uintThe maximum length of a single GELF message, in bytes. Messages longer than this length will be dropped. If this option is not set, the decoder does not limit the length of messages and the per-message memory is unbounded.
Note that a message can be composed of multiple chunks and this limit is applied to the whole message, not to individual chunks.
This limit takes only into account the message’s payload and the GELF header bytes are excluded from the calculation. The message’s payload is the concatenation of all the chunks' payloads.
framing.chunked_gelf.pending_messages_limit
optional uintframing.chunked_gelf.timeout_secs
optional float5
framing.length_delimited
required objectmethod = "length_delimited"
framing.length_delimited.length_field_is_big_endian
optional booltrue
framing.length_delimited.length_field_length
optional uint4
framing.length_delimited.length_field_offset
optional uintframing.method
required string literal enumOption | Description |
---|---|
bytes | Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments). |
character_delimited | Byte frames which are delimited by a chosen character. |
chunked_gelf | Byte frames which are chunked GELF messages. |
length_delimited | Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length. |
newline_delimited | Byte frames which are delimited by a newline character. |
octet_counting | Byte frames according to the octet counting format. |
framing.newline_delimited
optional objectmethod = "newline_delimited"
framing.newline_delimited.max_length
optional uintThe maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.
framing.octet_counting
optional objectmethod = "octet_counting"
framing.octet_counting.max_length
optional uintinclude_stderr
optional booltrue
maximum_buffer_size_bytes
optional uint1e+06
mode
required string literal enumOption | Description |
---|---|
scheduled | The command is run on a schedule. |
streaming | The command is run until it exits, potentially being restarted. |
scheduled
optional objectscheduled.exec_interval_secs
optional uintThe interval, in seconds, between scheduled command runs.
If the command takes longer than exec_interval_secs
to run, it is killed.
60
streaming
optional objectstreaming.respawn_interval_secs
optional uint5
streaming.respawn_on_exit
optional booltrue
working_directory
optional string literalOutputs
<component_id>
Output Data
Logs
Warning
Line
stdout
stderr
gethostname
command.my-host.local
2019-02-13T19:48:34+00:00 [info] Started GET "/" for 127.0.0.1
60085
668
exec
2020-10-10T17:07:36.452332Z
Telemetry
Metrics
linkcommand_executed_total
countercommand_execution_duration_seconds
histogramcomponent_discarded_events_total
counterfilter
transform, or false if due to an error.component_errors_total
countercomponent_received_bytes_total
countercomponent_received_event_bytes_total
countercomponent_received_events_count
histogramA histogram of the number of events passed in each internal batch in Vector’s internal topology.
Note that this is separate than sink-level batching. It is mostly useful for low level debugging performance issues in Vector due to small internal batches.
component_received_events_total
countercomponent_sent_event_bytes_total
countercomponent_sent_events_total
countersource_lag_time_seconds
histogramExamples
Exec line
Given this event...64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.060 ms
sources:
my_source_id:
type: exec
[sources.my_source_id]
type = "exec"
{
"sources": {
"my_source_id": {
"type": "exec"
}
}
}
{
"data_stream": "stdout",
"host": "my-host.local",
"message": "64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.060 ms",
"pid": 5678,
"source_type": "exec",
"timestamp": "2020-03-13T20:45:38.119Z"
}
How it works
Line Delimiters
0xA
byte, is found or the end of the
maximum_buffer_size_bytes
is reached.Shutting Down
When Vector begins shutting down (typically due to a SIGTERM), this source will signal to the child process to terminate, if it is running, to shut down.
On *nix platforms, Vector will issue a SIGTERM to the child process, allowing it to
gracefully shutdown, and the source will continue reading until the process exits or
Vector’s shutdown grace period expires. The duration of the grace period can be
configured using --graceful-shutdown-limit-secs
.
On Windows, the subprocess will be issued a SIGKILL and terminate abruptly. In the future we hope to support graceful shutdown of Windows processes as well.