Vector was designed with a focus on providing clear guarantees. Below you'll find a support matrix so you know exactly what type of guarantee you can expect for your combination of sources and sinks. This helps you make the appropriate tradeoffs or your usecase.
at-least-once delivery guarantee ensures that an
event received by Vector will be delivered at least
once to the configured destination(s). While rare, it is possible for an event
to be delivered more than once. See the Does Vector support exactly once
delivery FAQ below).
In order to achieve at least once delivery between restarts your source must
be configured to use
disk based buffers:
[sinks.my_sink_id]# ...[sinks.my_sink_id.buffer]type = "disk"when_full = "block"max_size = 104900000 # 100MiB
Each sink will include documentation for its buffer options.
best-effort delivery guarantee means that Vector will make a best effort to
deliver each event, but cannot guarantee delivery. This is usually due to
limitations of the underlying protocol; outside the sccope of Vector. It's
important to note that while data is possible, it is usually rare and Vector
does everything within it's control to ensure data is not lost. For more info,
see the "Do I need at least once delivery?"
prod-ready status is a subjective status defined by the Vector team
intended to give you a general idea of a feature's reliability for production
environments. A feature is
prod-ready if it means the following general
- A meaningful amount of users (generally >50) have been using the feature in a production environment for sustained periods without issue.
- The feature has had sufficient time (generally >4 months) to be community tested.
- There are no major open bugs for the feature.
beta status means that a feature has not met the criteria outlined in
the Prod-Ready section and therefore should be used with caution
in production environments.
Do I need at least once delivery?
One of the unique advantages of the logging use case is that data is usually used for diagnostic purposes only. Therefore, losing the occasional event has little impact on your business. This affords you the opportunity to provision your pipeline towards performance, simplicity, and cost reduction. On the other hand, if you're using your data to perform business critical functions, then data loss is not acceptable and therefore requires "at least once" delivery.
To clarify, even though a source or sink is marked as "best effort" it does not mean Vector takes delivery lightly. In fact, once data is within the boundary of Vector it will not be lost if you've configured on-disk buffers. Data loss for "best effort" sources and sinks are almost always due to the limitations of the underlying protocol.
Does Vector support exactly once delivery?
No, Vector does not support exactly once delivery. There are future plans to partially support this for sources and sinks that support it (Kafka, for example), but it remains unclear if Vector will ever be able to achieve this. We recommend subscribing to our mailing list, which will keep you in the loop if this ever changes.
How can I find components that meet these guarantees?
Head over to the components section and use the guarantee filters.