Writing Tier-Aware Load Rules for Hot and Cold Druid Tiers

You have a hot tier of fast NVMe Historicals and a cold tier of cheap high-density nodes, and you need recent data triple-served for latency while archival data survives on a single cheap replica — but the rule set you posted either loaded everything on one tier or left the newest week unavailable. The fix is a correctly ordered tieredReplicants rule set pushed through POST /druid/coordinator/v1/rules/{dataSource}, where each rule names real tiers and the specific windows precede the general ones. This page builds that exact JSON, applies it, and verifies per-tier placement. It is the concrete recipe behind Coordinator load rules and tier placement; read that for the matching semantics this page puts to work.

Failure Modes & Diagnostics #

Before writing rules, confirm the two facts every tier-aware rule set depends on: the tier names your Historicals actually advertise, and the current rules in force. A rule referencing a tier that does not exist is the number-one cause of "recent data won't load."

# 1. What tiers exist, and how many Historicals + how much capacity each?
curl -s http://coordinator:8081/druid/coordinator/v1/tiers?simple | jq

# 2. What rules are currently in force for the datasource (incl. inherited _default)?
curl -s "http://coordinator:8081/druid/coordinator/v1/rules/events?full" \
  | jq '.[] | {type, period, interval, tieredReplicants}'

# 3. Anything under-replicated or unavailable right now?
curl -s "http://coordinator:8081/druid/coordinator/v1/loadstatus?full" | jq '.events'

If step 1 lists only _default_tier, no hot/cold split exists yet — set druid.server.tier=hot (and =cold) in the respective Historicals' runtime.properties and restart them before any tiered rule can bind. If step 2 shows a loadForever above your intended specific rules, first-match will shadow them: the earlier rule wins and the later never runs. If step 3 shows a nonzero count for a tier, that tier is over-subscribed — the requested replicas exceed capacity, and adding more replication will only deepen the backlog.

Target Spec & Validated JSON #

The rule array below encodes the three-band policy shown above: two hot plus one cold for the trailing week, one cold from a week to three months, and drop beyond. It is posted verbatim to POST /druid/coordinator/v1/rules/events and is valid against current stable Druid. Order is load-bearing — the specific window sits above the general, and the catch-all is last.

[
  {
    "type": "loadByPeriod",
    "period": "P7D",
    "includeFuture": true,
    "tieredReplicants": { "hot": 2, "cold": 1 }
  },
  {
    "type": "loadByPeriod",
    "period": "P3M",
    "includeFuture": false,
    "tieredReplicants": { "cold": 1 }
  },
  {
    "type": "dropForever"
  }
]

Three details make this correct rather than merely plausible:

• includeFuture: true on the hot rule. Streaming ingestion can publish segments timestamped a little ahead of wall-clock; without this flag those future-dated segments match no load rule and never load. Set it on the newest band only.
• Both tiers in one map. {"hot": 2, "cold": 1} is a single rule producing three replicas — two hot, one cold — for a replication factor of three. The Broker prefers hot at query time because hot Historicals carry a higher druid.server.priority.
• dropForever last, not loadForever. Ending in a drop keeps aged data out of the serving tiers while leaving the bytes in deep storage; relaxing the P3M rule later reloads them with no re-ingest. A loadForever catch-all instead would silently keep every ancient segment on cold forever.

If you need a fixed window triple-replicated regardless of age — say a quarter under audit — prepend a loadByInterval above the P7D rule so first-match reaches it:

{
  "type": "loadByInterval",
  "interval": "2026-01-01T00:00:00Z/2026-04-01T00:00:00Z",
  "tieredReplicants": { "hot": 3 }
}

Python Automation Script #

The applier reads current rules, strips any prior copy of the managed rules to stay idempotent, posts the tier-aware set with audit headers, and polls to convergence with capped exponential backoff. Standard library plus requests only.

import time
import requests

COORDINATOR = "http://coordinator:8081"

TIER_RULES = [
    {"type": "loadByPeriod", "period": "P7D", "includeFuture": True,
     "tieredReplicants": {"hot": 2, "cold": 1}},
    {"type": "loadByPeriod", "period": "P3M", "includeFuture": False,
     "tieredReplicants": {"cold": 1}},
    {"type": "dropForever"},
]


def live_tiers() -> set:
    r = requests.get(f"{COORDINATOR}/druid/coordinator/v1/tiers", timeout=15)
    r.raise_for_status()
    return set(r.json())


def apply_rules(datasource: str, rules: list) -> None:
    referenced = {t for rule in rules
                  for t in rule.get("tieredReplicants", {})}
    missing = referenced - live_tiers()
    if missing:
        raise ValueError(f"rules reference non-existent tiers: {sorted(missing)}")
    r = requests.post(
        f"{COORDINATOR}/druid/coordinator/v1/rules/{datasource}",
        json=rules,
        headers={
            "Content-Type": "application/json",
            "X-Druid-Author": "tier-rule-bot",
            "X-Druid-Comment": "apply hot/cold tier-aware load rules",
        },
        timeout=30,
    )
    r.raise_for_status()


def wait_converged(datasource: str, max_wait: float = 1800.0) -> bool:
    delay, waited = 5.0, 0.0
    while waited < max_wait:
        r = requests.get(
            f"{COORDINATOR}/druid/coordinator/v1/loadstatus",
            params={"full": ""}, timeout=30,
        )
        r.raise_for_status()
        remaining = r.json().get(datasource, {})
        if not remaining or all(v == 0 for v in remaining.values()):
            return True
        time.sleep(delay)
        waited += delay
        delay = min(delay * 2, 60.0)  # capped exponential backoff
    return False


if __name__ == "__main__":
    apply_rules("events", TIER_RULES)
    if not wait_converged("events"):
        raise SystemExit("rules posted but replicas did not converge in budget")
    print("tier-aware rules applied and fully loaded")

The pre-flight missing = referenced - live_tiers() check is the guardrail that turns the most common outage — a rule naming a tier no node advertises — into a fast local error instead of silently unavailable segments. The convergence poll ensures the caller only proceeds once the new per-tier replicas are actually loaded.

Verification Steps #

After the applier reports success, confirm the rules landed and the segments distributed across tiers as intended. First, read back the effective rule set:

curl -s "http://coordinator:8081/druid/coordinator/v1/rules/events?full" \
  | jq '.[] | {type, period, interval, tieredReplicants}'

Expected — your three rules in order, specific window first, dropForever last:

{ "type": "loadByPeriod", "period": "P7D", "interval": null, "tieredReplicants": { "hot": 2, "cold": 1 } }
{ "type": "loadByPeriod", "period": "P3M", "interval": null, "tieredReplicants": { "cold": 1 } }
{ "type": "dropForever", "period": null, "interval": null, "tieredReplicants": null }

Next, confirm nothing remains under-replicated for the datasource — an empty object or all-zero counts means every desired replica is loaded:

curl -s "http://coordinator:8081/druid/coordinator/v1/loadstatus?full" | jq '.events // {}'

{}

Finally, verify the physical split: count how many segments each tier actually holds for the datasource by inspecting the Druid cluster server view. A hot node should hold only the trailing-week segments, a cold node the wider range:

curl -s "http://coordinator:8081/druid/coordinator/v1/servers?full" \
  | jq '[.[] | {tier, host: .host, segments: (.segments | length)}] | sort_by(.tier)'

[
  { "tier": "cold", "host": "hist-cold-1:8083", "segments": 640 },
  { "tier": "hot",  "host": "hist-hot-1:8083",  "segments": 168 },
  { "tier": "hot",  "host": "hist-hot-2:8083",  "segments": 168 }
]

The two hot nodes holding equal counts confirms the two hot replicas landed on distinct Historicals (never two copies on one node), and the larger cold count confirms the wider P3M window resolved there. If a hot node shows zero segments while loadstatus is clean, re-check druid.server.priority and that the node's druid.server.tier is exactly hot.

Related #

• Coordinator load rules and tier placement — parent guide: the first-match evaluation and tier mechanics this recipe applies.
• Segment replication and fault tolerance — what the replication factor these rules set buys you when a node dies.
• Configuring segment retention policies — the drop-and-retention rules that pair with these load rules for per-tenant expiry.
• Tiered storage migration runbook — moving already-loaded data from hot to cold by evolving these same rules.

Up one level: Coordinator load rules and tier placement.