Tag cardinality limit

This transform stores in memory a copy of the key for every tag on every metric event seen by this transform. In mode exact, a copy of every distinct value for each key is also kept in memory, until value_limit distinct values have been seen for a given key, at which point new values for that key will be rejected. So to estimate the memory usage of this transform in mode exact you can use the following formula:

(number of distinct field names in the tags for your metrics * average length of
the field names for the tags) + (number of distinct field names in the tags of
your metrics * `value_limit` * average length of the values of tags for your
metrics)

In mode probabilistic, rather than storing all values seen for each key, each distinct key has a bloom filter which can probabilistically determine whether a given value has been seen for that key. The formula for estimating memory usage in mode probabilistic is:

(number of distinct field names in the tags for your metrics * average length of
the field names for the tags) + (number of distinct field names in the tags of
-your metrics * `cache_size_per_key`)

The cache_size_per_key option controls the size of the bloom filter used for storing the set of acceptable values for any single key. The larger the bloom filter the lower the false positive rate, which in our case means the less likely we are to allow a new tag value that would otherwise violate a configured limit. If you want to know the exact false positive rate for a given cache_size_per_key and value_limit, there are many free on-line bloom filter calculators that can answer this. The formula is generally presented in terms of ’n’, ‘p’, ‘k’, and ’m’ where ’n’ is the number of items in the filter (value_limit in our case), ‘p’ is the probability of false positives (what we want to solve for), ‘k’ is the number of hash functions used internally, and ’m’ is the number of bits in the bloom filter. You should be able to provide values for just ’n’ and ’m’ and get back the value for ‘p’ with an optimal ‘k’ selected for you. Remember when converting from value_limit to the ’m’ value to plug into the calculator that value_limit is in bytes, and ’m’ is often presented in bits (1/8 of a byte).

per_tag_limits lets you override the cardinality settings for individual tag keys instead of changing the metric-level value_limit. It is supported at two scopes — the top level (applies to every metric that does not match a per_metric_limits entry) and inside a per_metric_limits.<name> block (applies only to that metric).

Each entry uses one of two mode values:

• mode: limit_override — track the tag with its own value_limit, independent of the surrounding metric’s value_limit.
• mode: excluded — bypass cardinality tracking for this tag entirely. Values pass through unchanged on every event, are not counted against any value_limit, and are never added to the cache.

type: tag_cardinality_limit
value_limit: 500
mode: exact

# Applies to every metric that does NOT match a per_metric_limits entry below.
per_tag_limits:
  kube_pod_name:
    # High cardinality is intentional for this tag — never track it.
    mode: excluded
  request_id:
    # Tighten the cap for this tag without lowering the metric-level limit.
    mode: limit_override
    value_limit: 50

per_metric_limits:
  http_requests_total:
    value_limit: 1000
    mode: exact
    # This metric has its own per-tag rules. The top-level per_tag_limits
    # above is IGNORED for http_requests_total — `kube_pod_name` on this
    # metric is therefore tracked against value_limit=1000.
    per_tag_limits:
      trace_id:
        mode: excluded

Precedence is “nearest wins”:

1. If the metric matches a per_metric_limits entry, only that entry’s per_tag_limits is consulted; the top-level per_tag_limits is ignored for that metric. (This mirrors how a per-metric value_limit shadows the global value_limit.)
2. Otherwise, the top-level per_tag_limits is consulted.
3. Tags not listed in the applicable per_tag_limits fall back to the surrounding metric’s value_limit (per-metric, or global).