The Vector team is excited to announce version 0.47.0
!
Release highlights:
opentelemetry
source now supports metrics ingestion.window
transform has been introduced which enables log noise reduction by filtering out events when the system is in a healthy state.mqtt
source is now available, enabling ingestion from MQTT brokers.datadog_logs
sink now supports a new conforms_as_agent
option to format logs like the Datadog Agent, ensuring compatibility with reserved fields.opentelemetry
source now supports metrics ingestion.window
transform, a variant of ring buffer or backtrace logging implemented as a sliding window.
Allows for reduction of log volume by filtering out logs when the system is healthy, but preserving detailed
logs when they are most relevant.mqtt
source enabling Vector to receive logs from a MQTT broker.vector graph
websocket_server
sink, allowing this component to cache latest received messages per client.convert_to
of type ConversionUnit
. By default, timing values in milliseconds (ms
) are converted to seconds (s
). Users can set convert_to
to milliseconds
to preserve the original millisecond values.address
field is now available within VRL scripts when using the auth.strategy.custom
authentication method.Sec-WebSocket-Protocol
header in the websocket_server
sink to better accommodate clients that require it.timeout
config option to the healthcheck
sink configuration. Previously it was hardcoded to 10 seconds across all components, but now it can be configured per component.Add wildcard_matching
global config option to set wildcard matching mode for inputs. Relaxed mode allows configurations with wildcards that do not match any inputs to be accepted without causing an error.
Example config:
wildcard_matching: relaxed
sources:
stdin:
type: stdin
# note - no transforms
sinks:
stdout:
type: console
encoding:
codec: json
inputs:
- "runtime-added-transform-*"
framing.method = length_delimited
in sinks for significantly improved performance with large (more than 10MB) batches.from
and to
parameters. There are no changes to the function signatures.file
and files
fields of remap
transforms are now watched when --watch-config
is enabled. Changes to these files automatically trigger a configuration reload, so there’s no need to restart Vector.amqp
sink now supports setting the priority
for messages. The value can be templated to an integer 0-255 (inclusive).deferred.max_age_secs
and deferred.queue_url
options to the aws_s3
and aws_sqs
sinks, to automatically
route older event notifications to a separate queue, allowing prioritized processing of recent files.file
source bug where known small files were not deleted after the specified remove_after_secs
.region
was missing from the STS
authentication endpoint.aws_cloudwatch_logs
sink to ~1MB.kubernetes source
bug where use_apiserver_cache=true
but there is no resourceVersion=0
parameter in list request. Per this issue, when resourceVersion =0
and !page_size.is_none
inListParams
, the parameter resourceVersion=0
will be ignored by kube-rs
sdk. If no parameter resourceVersion
passed to the apiserver, the apiserver will list pods from ETCD instead of in memory cache.Add an option in the datadog_logs
sink to allow Vector to mutate the record to conform to the
protocol used by the Datadog Agent itself. To enable, use the conforms_as_agent
option or have the
appropriate agent header (DD-PROTOCOL: agent-json
) within the additional HTTP Headers list.
Any top-level fields that use Datadog-reserved keywords are moved into a new object named message
. If message
doesn’t exist, it is created first. For example:
{
"key1": "value1",
"key2": { "key2-1" : "value2" },
"message" : "Hello world",
... rest of reserved fields
}
will be modified to:
{
"message" : {
"message" : "Hello world",
"key1": "value1",
"key2": { "key2-1" : "value2" }
},
... rest of reserved fields
}
datadog_logs
sink where the content of the log message is dropped when logs namespacing is enabled.Add a new extra_headers
option to greptimedb_logs
sink configuration to set additional headers for outgoing requests.
Change greptimedb_logs
sink default content type to application/x-ndjson
to match the default content type of greptimedb
sink.
If you use the greptimedb version v0.12 or earlier, you need to set the content type to application/json
in the sink configuration.
Example:
sinks:
greptime_logs:
type: greptimedb_logs
inputs: ["my_source_id"]
endpoint: "http://localhost:4000"
table: "demo_logs"
dbname: "public"
extra_headers:
x-source: vector
[sinks.greptime_logs]
type = "greptimedb_logs"
inputs = ["my_source_id"]
endpoint = "http://localhost:4000"
table = "demo_logs"
dbname = "public"
[sinks.greptime_logs.extra_headers]
x-source = "vector"
VRL was updated to v0.24.0
. This includes the following changes:
The encode_gzip
, decode_gzip
, encode_zlib
, and decode_zlib
methods now use the zlib-rs backend.
which is much faster than the previous backend miniz_oxide
.
The decode_base64
, encode_base64
, and decode_mime_q
functions now use the SIMD backend.
which is faster than the previous backend.
Sign up to receive emails on the latest Vector content and new releases
Thank you for joining our Updates Newsletter