framing options for sources
Often, when consuming data from a source, the first operation you have to do on
it is decode the data from its source representation. To make this easier, we’ve
decoding options to most sources.
For example, if you have a
kafka source that has JSON-encoded messages, now
you can simply add
decoding.codec = "json" to your source configuration like:
[sources.kafka] type = "kafka" bootstrap_servers = "localhost:9200" topics = ["my_topic"] decoding.codec = "json"
This will decode your messages from JSON, thus saving you from an additional
In addition, we’ve added a new
framing option to allow configuration for
sources that have non-standard framing (for example a custom-delimiter
For example, if you have an
http source where the messages are delimited by
commas instead of newlines, you can configure this like:
[sources.http] type = "http" address = "0.0.0.0:8080" framing.method = "character_delimited" framing.character_delimited.delimiter = ","
To have Vector parse each comma-delimited element as a new message. This can be
decoding option specified above. See the docs
for other framing options.