Socket
Collect logs using the socket client
Configuration
Example configurations
{
"sources": {
"my_source_id": {
"type": "socket",
"address": "0.0.0.0:9000",
"mode": "tcp",
"path": "/path/to/socket"
}
}
}
[sources.my_source_id]
type = "socket"
address = "0.0.0.0:9000"
mode = "tcp"
path = "/path/to/socket"
sources:
my_source_id:
type: socket
address: 0.0.0.0:9000
mode: tcp
path: /path/to/socket
{
"sources": {
"my_source_id": {
"type": "socket",
"address": "0.0.0.0:9000",
"max_length": 102400,
"mode": "tcp",
"path": "/path/to/socket",
"permit_origin": [
"192.168.0.0/16"
],
"port_key": "port",
"shutdown_timeout_secs": 30,
"socket_file_mode": 511
}
}
}
[sources.my_source_id]
type = "socket"
address = "0.0.0.0:9000"
max_length = 102_400
mode = "tcp"
path = "/path/to/socket"
permit_origin = [ "192.168.0.0/16" ]
port_key = "port"
shutdown_timeout_secs = 30
socket_file_mode = 511
sources:
my_source_id:
type: socket
address: 0.0.0.0:9000
max_length: 102400
mode: tcp
path: /path/to/socket
permit_origin:
- 192.168.0.0/16
port_key: port
shutdown_timeout_secs: 30
socket_file_mode: 511
address
required string literalThe socket address to listen for connections on, or systemd{#N}
to use the Nth socket passed by
systemd socket activation.
If a socket address is used, it must include a port.
mode = "tcp" or mode = "udp"
connection_limit
optional uintmode = "tcp"
decoding
optional objectdecoding.avro
required objectcodec = "avro"
decoding.avro.schema
required string literalThe Avro schema definition.
Please note that the following [apache_avro::types::Value
] variants are currently not supported:
Date
Decimal
Duration
Fixed
TimeMillis
decoding.avro.strip_schema_id_prefix
required booldecoding.codec
optional string literal enumOption | Description |
---|---|
avro | Decodes the raw bytes as as an Apache Avro message. |
bytes | Uses the raw bytes as-is. |
gelf | Decodes the raw bytes as a GELF message. This codec is experimental for the following reason: The GELF specification is more strict than the actual Graylog receiver.
Vector’s decoder currently adheres more strictly to the GELF spec, with
the exception that some characters such as Other GELF codecs such as Loki’s, use a Go SDK that is maintained by Graylog, and is much more relaxed than the GELF spec. Going forward, Vector will use that Go SDK as the reference implementation, which means the codec may continue to relax the enforcement of specification. |
influxdb | Decodes the raw bytes as an Influxdb Line Protocol message. |
json | Decodes the raw bytes as JSON. |
native | Decodes the raw bytes as native Protocol Buffers format. This codec is experimental. |
native_json | Decodes the raw bytes as native JSON format. This codec is experimental. |
protobuf | Decodes the raw bytes as protobuf. |
syslog | Decodes the raw bytes as a Syslog message. Decodes either as the RFC 3164-style format (“old” style) or the RFC 5424-style format (“new” style, includes structured data). |
vrl | Decodes the raw bytes as a string and passes them as input to a VRL program. |
bytes
decoding.gelf
optional objectcodec = "gelf"
decoding.gelf.lossy
optional boolDetermines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER
.
true
decoding.influxdb
optional objectcodec = "influxdb"
decoding.influxdb.lossy
optional boolDetermines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER
.
true
decoding.json
optional objectcodec = "json"
decoding.json.lossy
optional boolDetermines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER
.
true
decoding.native_json
optional objectcodec = "native_json"
decoding.native_json.lossy
optional boolDetermines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER
.
true
decoding.protobuf
optional objectcodec = "protobuf"
decoding.protobuf.desc_file
optional string literaldecoding.protobuf.message_type
optional string literaldecoding.syslog
optional objectcodec = "syslog"
decoding.syslog.lossy
optional boolDetermines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER
.
true
decoding.vrl
required objectcodec = "vrl"
decoding.vrl.source
required string literal.
target will be used as the decoding result.
Compilation error or use of ‘abort’ in a program will result in a decoding error.decoding.vrl.timezone
optional string literalThe name of the timezone to apply to timestamp conversions that do not contain an explicit
time zone. The time zone name may be any name in the TZ database, or local
to indicate system local time.
If not set, local
will be used.
framing
optional objectFraming configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.
framing.character_delimited
required objectmethod = "character_delimited"
framing.character_delimited.delimiter
required ascii_charframing.character_delimited.max_length
optional uintThe maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.
framing.length_delimited
required objectmethod = "length_delimited"
framing.length_delimited.length_field_is_big_endian
optional booltrue
framing.length_delimited.length_field_length
optional uint4
framing.length_delimited.length_field_offset
optional uintframing.method
required string literal enumOption | Description |
---|---|
bytes | Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments). |
character_delimited | Byte frames which are delimited by a chosen character. |
length_delimited | Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length. |
newline_delimited | Byte frames which are delimited by a newline character. |
octet_counting | Byte frames according to the octet counting format. |
framing.newline_delimited
optional objectmethod = "newline_delimited"
framing.newline_delimited.max_length
optional uintThe maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.
framing.octet_counting
optional objectmethod = "octet_counting"
framing.octet_counting.max_length
optional uinthost_key
optional string literalOverrides the name of the log field used to add the peer host to each event.
The value will be the peer host’s address, including the port i.e. 1.2.3.4:9000
.
By default, the global log_schema.host_key
option is used.
Set to ""
to suppress this key.
keepalive
optional objectkeepalive.time_secs
optional uintmode = "tcp"
max_connection_duration_secs
optional uintMaximum duration to keep each connection open. Connections open for longer than this duration are closed.
This is helpful for load balancing long-lived connections.
mode = "tcp"
max_length
optional uintThe maximum buffer size of incoming messages.
Messages larger than this are truncated.
102400
(bytes)mode = "udp"
mode
required string literal enumOption | Description |
---|---|
tcp | Listen on TCP. |
udp | Listen on UDP. |
unix_datagram | Listen on a Unix domain socket (UDS), in datagram mode. |
unix_stream | Listen on a Unix domain socket (UDS), in stream mode. |
path
required string literalThe Unix socket path.
This should be an absolute path.
mode = "unix_datagram" or mode = "unix_stream"
permit_origin
optional [string]mode = "tcp"
port_key
optional string literalOverrides the name of the log field used to add the peer host’s port to each event.
The value will be the peer host’s port i.e. 9000
.
By default, "port"
is used.
Set to ""
to suppress this key.
port
mode = "tcp" or mode = "udp"
receive_buffer_bytes
optional uintmode = "tcp" or mode = "udp"
shutdown_timeout_secs
optional uint30
(seconds)mode = "tcp"
socket_file_mode
optional uintUnix file mode bits to be applied to the unix socket file as its designated file permissions.
Note: The file mode value can be specified in any numeric format supported by your configuration language, but it is most intuitive to use an octal number.
mode = "unix_datagram" or mode = "unix_stream"
tls
optional objectsources
, adding metadata from the client certificate.tls.alpn_protocols
optional [string]Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
tls.ca_file
optional string literalAbsolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
tls.client_metadata_key
optional string literaltls.crt_file
optional string literalAbsolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
tls.enabled
optional boolWhether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file
for
more information.
tls.key_file
optional string literalAbsolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
tls.key_pass
optional string literalPassphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
tls.verify_certificate
optional boolEnables certificate verification. For components that create a server, this requires that the client connections have a valid client certificate. For components that initiate requests, this validates that the upstream has a valid certificate.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
tls.verify_hostname
optional boolEnables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
mode = "tcp"
Outputs
<component_id>
Output Data
Logs
Warning
Line
129.21.31.122
2019-02-13T19:48:34+00:00 [info] Started GET "/" for 127.0.0.1
2838
socket
2020-10-10T17:07:36.452332Z
Telemetry
Metrics
linkcomponent_discarded_events_total
counterfilter
transform, or false if due to an error.component_errors_total
countercomponent_received_bytes
histogramcomponent_received_bytes_total
countercomponent_received_event_bytes_total
countercomponent_received_events_count
histogramA histogram of the number of events passed in each internal batch in Vector’s internal topology.
Note that this is separate than sink-level batching. It is mostly useful for low level debugging performance issues in Vector due to small internal batches.
component_received_events_total
countercomponent_sent_event_bytes_total
countercomponent_sent_events_total
counterconnection_established_total
counterconnection_send_errors_total
counterconnection_shutdown_total
countersource_lag_time_seconds
histogramExamples
Socket line
Given this event...2019-02-13T19:48:34+00:00 [info] Started GET "/" for 127.0.0.1
sources:
my_source_id:
type: socket
[sources.my_source_id]
type = "socket"
{
"sources": {
"my_source_id": {
"type": "socket"
}
}
}
{
"host": "my-host.local",
"message": "2019-02-13T19:48:34+00:00 [info] Started GET \"/\" for 127.0.0.1",
"source_type": "socket",
"timestamp": "2020-10-10T17:07:36.452332Z"
}
How it works
Transport Layer Security (TLS)
tls.*
options and/or via an
OpenSSL configuration file. The file location defaults to
/usr/local/ssl/openssl.cnf
or can be specified with the OPENSSL_CONF
environment variable.