HTTP Server
Receive logs emitted by an HTTP server.
Alias
This component was previously called the http
source. Make sure to update your
Vector configuration to accommodate the name change:
[sources.my_http_server_source]
+type = "http_server"
-type = "http"
Configuration
Example configurations
{
"sources": {
"my_source_id": {
"type": "http_server",
"address": "0.0.0.0:80"
}
}
}
[sources.my_source_id]
type = "http_server"
address = "0.0.0.0:80"
---
sources:
my_source_id:
type: http_server
address: 0.0.0.0:80
{
"sources": {
"my_source_id": {
"type": "http_server",
"address": "0.0.0.0:80",
"encoding": "binary",
"headers": [
"User-Agent"
],
"method": "POST",
"path": "/",
"path_key": "path",
"query_parameters": [
"application"
],
"strict_path": true
}
}
}
[sources.my_source_id]
type = "http_server"
address = "0.0.0.0:80"
encoding = "binary"
headers = [ "User-Agent" ]
method = "POST"
path = "/"
path_key = "path"
query_parameters = [ "application" ]
strict_path = true
---
sources:
my_source_id:
type: http_server
address: 0.0.0.0:80
encoding: binary
headers:
- User-Agent
method: POST
path: /
path_key: path
query_parameters:
- application
strict_path: true
acknowledgements
optional objectControls how acknowledgements are handled by this source.
This setting is deprecated in favor of enabling acknowledgements
at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
acknowledgements.enabled
optional booladdress
required string literalThe socket address to listen for connections on.
It must include a port.
auth
optional objectauth.password
required string literalauth.username
required string literaldecoding
optional objectdecoding.codec
required string literal enumOption | Description |
---|---|
bytes | Uses the raw bytes as-is. |
gelf | Decodes the raw bytes as a GELF message. |
json | Decodes the raw bytes as JSON. |
native | Decodes the raw bytes as Vector’s native Protocol Buffers format. This codec is experimental. |
native_json | Decodes the raw bytes as Vector’s native JSON format. This codec is experimental. |
syslog | Decodes the raw bytes as a Syslog message. Will decode either as the RFC 3164-style format (“old” style) or the more modern RFC 5424-style format (“new” style, includes structured data). |
encoding
optional string literal enumThe expected encoding of received data.
Note that for json
and ndjson
encodings, the fields of the JSON objects are output as separate fields.
Option | Description |
---|---|
binary | Binary. |
json | JSON. |
ndjson | Newline-delimited JSON. |
text | Plaintext. |
framing
optional objectFraming configuration.
Framing deals with how events are separated when encoded in a raw byte form, where each event is a “frame” that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.
framing.character_delimited
required objectmethod = "character_delimited"
framing.character_delimited.delimiter
required uintframing.character_delimited.max_length
optional uintThe maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This will ensure that processing is not truly unbounded.
framing.method
required string literal enumOption | Description |
---|---|
bytes | Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments). |
character_delimited | Byte frames which are delimited by a chosen character. |
length_delimited | Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length. |
newline_delimited | Byte frames which are delimited by a newline character. |
octet_counting | Byte frames according to the octet counting format. |
framing.newline_delimited
optional objectmethod = "newline_delimited"
framing.newline_delimited.max_length
optional uintThe maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This will ensure that processing is not truly unbounded.
framing.octet_counting
optional objectmethod = "octet_counting"
framing.octet_counting.max_length
optional uintheaders
optional [string]A list of HTTP headers to include in the log event.
These will override any values included in the JSON payload with conflicting names.
method
optional string literal enumOption | Description |
---|---|
DELETE | HTTP DELETE method. |
GET | HTTP GET method. |
HEAD | HTTP HEAD method. |
PATCH | HTTP PATCH method. |
POST | HTTP POST method. |
PUT | HTTP Put method. |
POST
path_key
optional string literalpath
query_parameters
optional [string]A list of URL query parameters to include in the log event.
These will override any values included in the body with conflicting names.
strict_path
optional boolWhether or not to treat the configured path
as an absolute path.
If set to true
, only requests using the exact URL path specified in path
will be accepted. Otherwise,
requests sent to a URL path that starts with the value of path
will be accepted.
With strict_path
set to false
and path
set to ""
, the configured HTTP source will accept requests from
any URL path.
true
tls
optional objecttls.alpn_protocols
optional [string]Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.
tls.ca_file
optional string literalAbsolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
tls.crt_file
optional string literalAbsolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
tls.enabled
optional boolWhether or not to require TLS for incoming/outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file
for
more information.
tls.key_file
optional string literalAbsolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
tls.key_pass
optional string literalPassphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
tls.verify_certificate
optional boolEnables certificate verification.
If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
tls.verify_hostname
optional boolEnables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
Outputs
<component_id>
Output Data
Logs
Structured
application/json
requestpath_key
configuration setting/
/logs/event712
2020-10-10T17:07:36.452332Z
Text
text/plain
requestHello world
path_key
configuration setting/
/logs/event712
http_server
2020-10-10T17:07:36.452332Z
Telemetry
Metrics
linkcomponent_errors_total
countercomponent_id
instead. The value is the same as component_id
.component_received_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_received_event_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_received_events_total
countercomponent_id
instead. The value is the same as component_id
.component_sent_event_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_sent_events_total
countercomponent_id
instead. The value is the same as component_id
.events_in_total
countercomponent_received_events_total
instead.component_id
instead. The value is the same as component_id
.events_out_total
countercomponent_sent_events_total
instead.component_id
instead. The value is the same as component_id
.http_bad_requests_total
counter400 Bad Request
errors encountered.parse_errors_total
countersource_lag_time_seconds
histogramcomponent_id
instead. The value is the same as component_id
.Examples
text/plain
Given this event...POST / HTTP/1.1
Content-Type: text/plain
User-Agent: my-service/v2.1
X-Forwarded-For: my-host.local
Hello world
[sources.my_source_id]
type = "http_server"
address = "0.0.0.0:80"
encoding = "text"
headers = [ "User-Agent" ]
---
sources:
my_source_id:
type: http_server
address: 0.0.0.0:80
encoding: text
headers:
- User-Agent
{
"sources": {
"my_source_id": {
"type": "http_server",
"address": "0.0.0.0:80",
"encoding": "text",
"headers": [
"User-Agent"
]
}
}
}
[{"log":{"User-Agent":"my-service/v2.1","host":"my-host.local","message":"Hello world","path":"/","source_type":"http_server","timestamp":"2020-10-10T17:07:36.452332Z"}}]
application/json
Given this event...POST /events HTTP/1.1
Content-Type: application/json
User-Agent: my-service/v2.1
X-Forwarded-For: my-host.local
{"key": "val"}
[sources.my_source_id]
type = "http_server"
address = "0.0.0.0:80"
encoding = "json"
headers = [ "User-Agent" ]
path_key = "vector_http_path"
---
sources:
my_source_id:
type: http_server
address: 0.0.0.0:80
encoding: json
headers:
- User-Agent
path_key: vector_http_path
{
"sources": {
"my_source_id": {
"type": "http_server",
"address": "0.0.0.0:80",
"encoding": "json",
"headers": [
"User-Agent"
],
"path_key": "vector_http_path"
}
}
}
[{"log":{"User-Agent":"my-service/v2.1","host":"my-host.local","key":"val","source_type":"http_server","timestamp":"2020-10-10T17:07:36.452332Z"}}]
How it works
Decompression
Content-Encoding
header.
Supported algorithms are gzip
, deflate
, and snappy
.Transport Layer Security (TLS)
tls.*
options.