Druid Retention Rule Syntax Reference

The single question that produces the most retention incidents is deceptively small: "what exactly do I put in the JSON array I POST to the rules endpoint, and in what order?" A retention rule ladder is evaluated like a firewall ACL — top to bottom, first match wins — so a rule that reads correctly in isolation can silently shadow every rule beneath it, leaving old data loaded or fresh data dropped. This page is the precise syntax reference for every load and drop rule Druid accepts, the tieredReplicants block that controls replica placement, and the ordering semantics that decide which rule actually fires. It sits under TTL mapping and data expiration, which covers the two-stage drop-then-kill lifecycle; here the focus is the rule grammar itself and the POST /druid/coordinator/v1/rules/{dataSource} call that installs it.

Failure Modes & Diagnostics #

Retention rules fail in ways the API never reports: a malformed rule is rejected wholesale, a shadowing rule masks a drop, or a period boundary math error unloads live data. Diagnose from the shell against the live Coordinator before touching the ladder.

# Read back the live ladder — POST replaces the whole array, so this is the source of truth
curl -s "http://druid-coordinator:8081/druid/coordinator/v1/rules/clickstream" | jq '.'

# Audit who changed the rules last and to what (the Coordinator keeps a change log)
curl -s "http://druid-coordinator:8081/druid/coordinator/v1/rules/clickstream/history?count=5" \
  | jq '.[] | {auditTime, comment, payload}'

The three recurring failures:

1. A broad load rule shadows an intended drop. Symptom: a dropByPeriod or dropForever you added never fires and old data stays loaded. Root cause: an earlier loadForever (or an over-wide loadByPeriod) matches the interval first, and first-match-wins means the drop below it is never reached. Confirm by reading the ladder top-to-bottom and checking whether any load* rule matches the interval you expect to drop; move absolute holds up and the terminal drop to the very bottom.

2. The whole POST was silently rejected. Symptom: the ladder you read back is not the one you sent. Root cause: one malformed rule (a bad period, an unknown type, a tier name that isn't in the Druid cluster) invalidates the entire array — the update is atomic. Validate every period as ISO-8601 and every tier name against the live tiers below before POSTing.

# Which tier names actually exist? A tieredReplicants key that isn't here is a config error.
curl -s "http://druid-coordinator:8081/druid/coordinator/v1/tiers" | jq '.'

3. A period rule matched the wrong intervals. Symptom: an interval you expected to keep was dropped, or vice versa. Root cause: rules match against the segment's data interval relative to the Coordinator's clock, and a period that is not a whole multiple of the segment granularity cuts through a time chunk. Read the actual per-interval load state to see what the ladder resolved to:

# Loaded intervals and their segment counts — compare against what the ladder should keep
curl -s "http://druid-coordinator:8081/druid/coordinator/v1/datasources/clickstream/intervals?full" \
  | jq 'to_entries | map({interval: .key, segments: .value.count}) | .[0:10]'

Target Spec & Validated JSON #

A retention policy is a JSON array POSTed to POST /druid/coordinator/v1/rules/{dataSource}. The array replaces the datasource's rules wholesale — there is no partial update — so always send the complete, ordered ladder. Every rule object has a type; load rules additionally carry a scope (period/interval) and a tieredReplicants map, and drop rules carry only a scope. The ladder below is copy-ready against current stable Druid and exercises every rule kind, annotated inline.

[
  {
    "type": "loadByPeriod",
    "period": "P7D",
    "includeFuture": true,
    "tieredReplicants": { "hot": 2 }
  },
  {
    "type": "loadByInterval",
    "interval": "2026-01-01T00:00:00.000Z/2026-01-08T00:00:00.000Z",
    "tieredReplicants": { "hot": 1, "cold": 1 }
  },
  {
    "type": "loadByPeriod",
    "period": "P90D",
    "tieredReplicants": { "cold": 1 }
  },
  {
    "type": "dropBeforeByPeriod",
    "period": "P2Y"
  },
  {
    "type": "dropByPeriod",
    "period": "P400D",
    "includeFuture": false
  },
  {
    "type": "dropForever"
  }
]

Rule-by-rule, this is the full vocabulary you will ever need:

• loadByPeriod — keep segments whose data interval falls within a trailing ISO-8601 period measured back from the Coordinator's clock, recomputed every cycle. period: "P7D" keeps the last seven days. includeFuture (default true) also matches intervals dated ahead of now, which matters for streaming datasources whose newest chunks sit slightly in the future — leave it true there so in-flight chunks are not dropped.
• loadByInterval — pin an absolute ISO-8601 interval rather than a rolling window. Use it for legal holds, audit windows, or a one-off backfill that must survive independently of the trailing rules. Placed above a period rule, it wins for its interval regardless of age.
• loadForever — load every unmatched interval, forever. A terminal catch-all for datasources that must never drop anything; mutually exclusive with dropForever as the tail.
• dropByPeriod — unload segments older than the trailing period; the mirror image of loadByPeriod. With includeFuture: false it will not touch future-dated intervals.
• dropBeforeByPeriod — unload everything before now − period as an explicit hard floor. Semantically similar to dropByPeriod but read as "drop before this age," which some teams find clearer as a stated boundary.
• dropForever — the terminal sweep: unload every interval not matched above. Every ladder must end in dropForever or loadForever, or unmatched intervals fall through to the Druid cluster-wide default rules — rarely what you intend.

The tieredReplicants map is the placement half of a load rule: keys are tier names (hot, cold, or _default_tier on an untiered cluster) and values are the replica count on that tier. { "hot": 2 } keeps two copies on the hot tier; { "hot": 1, "cold": 1 } splits one copy onto each. Demoting an interval from { "hot": 2 } to { "cold": 1 } as it ages is the core lever for cost control, developed as an operational procedure in the tiered storage migration runbook and as replica-placement theory under coordinator load rules and tier placement.

Two ordering rules govern the whole array. First, absolute holds go above rolling windows, which go above drops — a loadByInterval legal hold placed below a dropForever is dead code. Second, the resolution at which any period rule can act is the segment granularity: a P90D rule on DAY-granularity segments is exact, but on MONTH granularity it can only cut on month boundaries. That relationship is set at ingestion by the segment granularity settings, and the number of whole chunks a period keeps is:

$$ \text{chunksRetained} \approx \left\lceil \frac{\text{periodDays}}{\text{granularityDays}} \right\rceil $$

Python Automation Script #

Rules belong in version control and should be applied idempotently — read the live ladder, POST only on a diff, and re-read to confirm propagation, because the Coordinator applies rules on its own cycle and is only eventually consistent. The orchestrator below uses the standard library plus requests, with capped exponential backoff on the asynchronous, occasionally-503 rules API. It refuses to POST a ladder that does not terminate in a catch-all, so a malformed policy can never fall through to cluster defaults.

import time
import logging
import requests

logger = logging.getLogger("druid.rules")

COORDINATOR = "http://druid-coordinator:8081"
TERMINAL_TYPES = {"dropForever", "loadForever"}


def _request(method, url, *, retries=5, **kwargs):
    """HTTP with capped exponential backoff on 5xx and connection errors."""
    delay = 1.0
    for attempt in range(1, retries + 1):
        try:
            resp = requests.request(method, url, timeout=30, **kwargs)
            if resp.status_code < 500:
                resp.raise_for_status()
                return resp
            logger.warning("%s %s -> %s (attempt %d)", method, url, resp.status_code, attempt)
        except requests.RequestException as exc:
            logger.warning("%s %s failed: %s (attempt %d)", method, url, exc, attempt)
        if attempt == retries:
            raise RuntimeError(f"{method} {url} failed after {retries} attempts")
        time.sleep(delay)
        delay = min(delay * 2, 30.0)  # cap the backoff at 30s


def validate_ladder(rules):
    """Reject a ladder that would let intervals fall through to cluster defaults."""
    if not rules:
        raise ValueError("empty rule ladder")
    if rules[-1]["type"] not in TERMINAL_TYPES:
        raise ValueError("ladder must end in dropForever or loadForever")
    for i, rule in enumerate(rules[:-1]):
        if rule["type"] in TERMINAL_TYPES:
            raise ValueError(f"terminal rule at index {i} shadows every rule below it")


def apply_rules_if_changed(datasource, desired):
    """Idempotent replace: POST the full ordered array only when it differs from live."""
    validate_ladder(desired)
    current = _request("GET", f"{COORDINATOR}/druid/coordinator/v1/rules/{datasource}").json()
    if current == desired:
        logger.info("Rules for %s already match; no-op", datasource)
        return False
    _request(
        "POST",
        f"{COORDINATOR}/druid/coordinator/v1/rules/{datasource}",
        json=desired,
        headers={"Content-Type": "application/json"},
    )
    # Confirm the live ladder is what we sent before declaring success.
    readback = _request("GET", f"{COORDINATOR}/druid/coordinator/v1/rules/{datasource}").json()
    if readback != desired:
        raise RuntimeError("rules readback does not match desired ladder (partial rejection?)")
    logger.info("Applied %d retention rules to %s", len(desired), datasource)
    return True


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    ladder = [
        {"type": "loadByPeriod", "period": "P7D", "includeFuture": True,
         "tieredReplicants": {"hot": 2}},
        {"type": "loadByPeriod", "period": "P90D",
         "tieredReplicants": {"cold": 1}},
        {"type": "dropForever"},
    ]
    apply_rules_if_changed("clickstream", ladder)

Because the ladder is validated before the POST and read back after it, a shadowing terminal rule or a partial rejection is caught in the pipeline rather than discovered when a tier quietly empties days later.

Verification Steps #

Confirm the rules resolved to the load state you intended. First read the ladder back — it must be byte-for-byte the array you sent, since POST replaces the whole list:

curl -s "http://druid-coordinator:8081/druid/coordinator/v1/rules/clickstream" \
  | jq '[.[] | {type, period, interval, tieredReplicants}]'

Expected output for the two-rule ladder above:

[
  { "type": "loadByPeriod", "period": "P7D", "tieredReplicants": { "hot": 2 } },
  { "type": "loadByPeriod", "period": "P90D", "tieredReplicants": { "cold": 1 } },
  { "type": "dropForever" }
]

Then confirm the Coordinator has converged: recent intervals should be loaded on hot, the 8-to-90-day tail on cold, and everything older unloaded. Check per-tier replica counts against what the rules ask for:

# Under-replicated segments should be 0 once the Coordinator has caught up
curl -s "http://druid-coordinator:8081/druid/coordinator/v1/loadstatus?full" \
  | jq '{clickstream: .clickstream}'

An empty object or all-zero counts for the datasource means every rule is satisfied and no segment is waiting to load or drop. If counts stay non-zero for more than a few coordination cycles, the target tier lacks capacity for the replicas the ladder requests — reduce tieredReplicants on the aging window or add Historical capacity. Because a retention change can silently unload months of history, gate every ladder push behind the access controls described under configuring segment retention policies, and keep the ladder in Git with an audit comment on every change.

Related #

• TTL Mapping and Data Expiration — the two-stage drop-then-kill lifecycle these rules drive, and how dropped segments become reclaimable.
• Coordinator Load Rules and Tier Placement — how tieredReplicants places replicas across tiers for availability and balancing.
• Tiered Storage Migration Runbook — the operational procedure for aging segments from a hot tier to a cheaper cold tier using these load rules.
• Configuring Segment Retention Policies — who is authorised to change a rule ladder and how the change is audited.
• Understanding Druid Segment Granularity — the time-chunk boundaries that set the resolution at which any period rule can match.

Up one level: TTL Mapping and Data Expiration.

For authoritative rule semantics, see the official Apache Druid retention rules reference.