/ Mar 10th, 2020 /

New Dedupe Transform

Shed duplicate logs

For certain use cases log deduplication can be a useful tool. Not only does this promote your data integrity, but it can help protect against upstream mistakes that accidentally doplicate logs. This mistake can easily double (or more!) your log volume. To protect against this you can use our new dedupe transform.

Get Started

Simply add the transform to your pipeline:

# General
type = "dedupe" # required
inputs = ["my-source-id"] # required
# Fields
fields.match = ["timestamp", "host", "message"] # optional, default
explain this command

Get This Change

This change was made available in 0.9.0. You can get this change by downloading the latest stable release.

Like What You See?