Splunk HEC Improvements
Improved compatibility with Splunk HTTP Event Collector
We’re excited to share that we’ve made a few improvements to Vector’s support for Splunk HEC; Vector now supports Splunk HEC indexer acknowledgements and channel token passthrough routing.
With the new support for Splunk HEC indexer acknowledgement, Vector is now able
to provide higher guarantees that no data is lost when using the
source and sink.
As you may be aware, Splunk HEC does not guarantee that data is successfully written when it responds to an incoming request successfully. To confirm that data has been persisted, the indexer acknowledgements feature must be enabled. The status of sent events can then be queried via a separate endpoint.
splunk_hec source did not support indexer
acknowledgements and so would provide weaker delivery guarantees and
would not work with Splunk senders that required them.
Now, you can configure the
splunk_hec source to use the indexer acknowledgements protocol by configuring:
acknowledgements = true
When enabled, responses to incoming requests will include an ID that can be used to query for acknowledgement status at the newly exposed
/services/collector/ack endpoint (learn more here). The acknowledgement status is wired into Vector’s
forthcoming end-to-end acknowledgement feature. This feature will require that
events be sent by sinks or persisted into disk buffers before sources will
splunk_hec sink simply treated successful HEC requests as the
events being acknowledged by the Spunk receiver and so Vector would drop them
from any buffers. Now, it is possible to configure the sink to wait until the
Splunk receiver acknowledges the events via the index
acknowledgements part of the HEC protocol.
This has defaulted to on to provide higher guarantees, but can be disabled to restore the previous behavior by configuring:
acknowledgements.indexer_acknowledgements_enabled = false
Vector now also supports Splunk HEC token passthrough routing. When
store_hec_token is enabled in a
splunk_hec source, tokens included in
requests to the source will be stored and used by downstream
Any passed through token takes precedence over the
set in the sink.