Coordinator Load Rules and Tier Placement in Apache Druid

Load rules are the declarative contract that tells the Druid Coordinator how many replicas of each segment to hold, and on which Historical tier — the input side of the reconciliation loop described in Apache Druid segment replication, balancing & high availability. A rule set is an ordered list attached to a datasource: the Coordinator walks it top-down and applies the first rule that matches each segment, so placement is entirely a function of rule order and each rule's tieredReplicants map. Get the order or a tier name wrong and segments silently fail to load, over-replicate, or land on the wrong hardware. This guide covers the exact matching semantics, tier configuration, and the throttle that keeps a rule change from flooding the fleet.

Mechanics & Internals #

A load rule has two parts: a matcher that selects which segments it governs, and an action that says what to do with them. There are three load matchers and three drop matchers, and their names encode exactly how they select:

• loadByPeriod matches segments whose interval falls within a rolling window ending now, expressed as an ISO-8601 period like P7D or P3M. Because "now" advances, a loadByPeriod rule is a sliding window — yesterday's hot segment ages out of the P7D rule automatically seven days later. The optional includeFuture flag (default true) also matches future-dated intervals, which matters for streaming datasources that write slightly ahead of wall-clock.
• loadByInterval matches segments overlapping a fixed ISO interval like 2026-01-01/2026-02-01. It never slides; use it to pin a specific historical window to specific replication, for example triple-replicating a quarter under active audit.
• loadForever matches everything. It is the unconditional catch-all and must be the last load rule, because any rule after it is unreachable — every remaining segment already matched.

Each load matcher carries a tieredReplicants map from tier name to replica count, and the sum of those counts is the segment's replication factor. The drop matchers — dropByPeriod, dropByInterval, dropForever, plus dropBeforeByPeriod — carry no replicants; they remove segments from Historicals entirely (the bytes stay in deep storage, so the action is reversible by changing the rule back). A common shape is a load rule for recent data followed by a dropForever to keep aged intervals out of the serving tier without deleting them, which is precisely the retention pattern developed in configuring segment retention policies.

The first-match rule is the single most important mechanic. For every used segment, the Coordinator evaluates rules top-down and stops at the first whose matcher accepts the segment; that rule alone determines placement. Two consequences follow. First, order specific-before-general: a loadByInterval for an audited quarter must sit above the broad loadByPeriod that would otherwise swallow it. Second, a rule set must be exhaustive — if no rule matches a segment, the Coordinator logs it and does nothing, leaving the segment unloaded — which is why a terminal loadForever or dropForever is mandatory.

Rules resolve against tiers, and a tier is nothing more than a label. Each Historical declares druid.server.tier in its runtime.properties; every Historical sharing a value forms one tier. A tieredReplicants key must match a live tier's name exactly — a rule asking for {"hot": 2} when no Historical advertises druid.server.tier=hot produces segments that can never load, because the Coordinator has nowhere to send them. The default tier name is _default_tier, and the default rule set (attached to the special _default datasource) is loadForever with {"_default_tier": 2}, which is why an out-of-the-box cluster double-replicates everything onto one tier.

Two rules can also target different tiers for the same segment through one tieredReplicants map: {"hot": 2, "cold": 1} loads three total replicas, two on hot and one on cold, and the Broker prefers the higher-druid.server.priority tier at query time. This is how a datasource lives on fast and slow hardware simultaneously — the recipe worked through concretely in writing tier-aware load rules.

Finally, applying rules is throttled. When a rule change increases replication for many segments at once, the Coordinator does not enqueue them all in one run; replicationThrottleLimit caps how many new replicas per tier enter the load queue each run, and maxSegmentsInNodeLoadingQueue caps any single Historical's queue depth. This is what turns a sweeping rule edit into a gradual, bandwidth-safe ramp rather than a deep-storage read storm.

Validated Configuration Spec #

The rule document below is posted to POST /druid/coordinator/v1/rules/{dataSource} as a JSON array. It is copy-ready against current stable Druid and demonstrates all three matcher types in the correct specific-before-general order, ending in a mandatory catch-all.

[
  {
    "type": "loadByInterval",
    "interval": "2026-01-01T00:00:00Z/2026-04-01T00:00:00Z",
    "tieredReplicants": { "hot": 3 }
  },
  {
    "type": "loadByPeriod",
    "period": "P7D",
    "includeFuture": true,
    "tieredReplicants": { "hot": 2, "cold": 1 }
  },
  {
    "type": "loadByPeriod",
    "period": "P3M",
    "includeFuture": false,
    "tieredReplicants": { "cold": 1 }
  },
  {
    "type": "dropForever"
  }
]

Reading top to bottom, and remembering first-match wins:

• The loadByInterval rule pins the audited Q1 window to three hot replicas. It sits first so it wins even for segments also inside the P7D or P3M windows.
• The first loadByPeriod (P7D, includeFuture: true) gives the trailing week two hot and one cold replica; includeFuture ensures streaming segments written slightly ahead still match.
• The second loadByPeriod (P3M, includeFuture: false) demotes everything from a week to three months old to a single cold replica.
• dropForever removes anything older than three months from Historicals; the bytes remain in deep storage and reload instantly if the rule is relaxed.

The corresponding Historical runtime.properties must declare the tier names the rules reference. A hot node carries:

druid.server.tier=hot
druid.server.priority=10
druid.server.maxSize=2000000000000

and a cold node:

druid.server.tier=cold
druid.server.priority=1
druid.server.maxSize=8000000000000

The higher druid.server.priority on hot means the Broker routes to the hot replica when a segment exists on both tiers; the larger druid.server.maxSize on cold reflects denser, cheaper storage. The throttle that governs how fast a change to these rules rolls out lives in the Coordinator dynamic config at POST /druid/coordinator/v1/config; with smartSegmentLoading enabled it is auto-sized, and the field reference is maintained in the Apache Druid coordinator documentation.

Sizing Heuristics & Formulas #

A rule set's replication factor multiplies storage tier by tier. For a datasource, the capacity a tier must provide is the summed byte footprint of the segments a rule sends there, times that rule's replica count for the tier:

$$\text{tierBytes} \approx \sum_{\text{matched intervals}} \text{segmentBytes} \times \text{replicas}_{\text{tier}}$$

Take the spec above with a datasource writing 130 GB/day. The P7D hot rule holds seven days at two replicas: $7 \times 130 \times 2 \approx 1820$ GB on hot. The Q1 loadByInterval at three replicas over ~90 days adds $90 \times 130 \times 3 \approx 35100$ GB — a reminder that a fixed-interval triple-replication rule can dwarf the rolling window, so scope such rules tightly. Cold holds the P7D single replica plus everything back to three months: roughly $(90 - 7) \times 130 \times 1 \approx 10790$ GB.

Size each tier so steady-state utilization stays under about 85% of aggregate druid.server.maxSize, leaving room for balancing and node-loss re-replication:

$$N_{\text{tier}} \gtrsim \frac{\text{tierBytes}}{0.85 \times \text{maxSize}_{\text{node}}}$$

For 1820 GB on hot nodes advertising 2 TB each, that is $1820 / (0.85 \times 2000) \approx 1.07$, so a minimum of two hot Historicals just to hold the working set — and a third if the tier must survive losing one node, since the survivors alone must still fit the set.

The throttle sets convergence speed. Ramping $M$ new replicas at replicationThrottleLimit $= r$ per run over Coordinator period $p$ takes at least:

$$t_{\text{ramp}} \approx \left\lceil \frac{M}{r} \right\rceil \times p$$

so raising replication on 50,000 segments at $r = 500$ per run and $p = 60\text{s}$ takes on the order of $100 \times 60 \approx 6000\text{s}$, about 100 minutes — plan rule changes as ramps, not instant cutovers.

Python Orchestration Snippet #

The driver below fetches a datasource's current rules, prepends a fixed-interval override (idempotently), posts the new set with audit headers, and polls loadstatus to convergence with capped exponential backoff. It uses only the standard library plus requests.

import time
import requests

COORDINATOR = "http://coordinator:8081"


def get_rules(datasource: str) -> list:
    r = requests.get(
        f"{COORDINATOR}/druid/coordinator/v1/rules/{datasource}", timeout=15
    )
    r.raise_for_status()
    return r.json()


def put_rules(datasource: str, rules: list, author: str, comment: str) -> None:
    r = requests.post(
        f"{COORDINATOR}/druid/coordinator/v1/rules/{datasource}",
        json=rules,
        headers={
            "Content-Type": "application/json",
            "X-Druid-Author": author,
            "X-Druid-Comment": comment,
        },
        timeout=30,
    )
    r.raise_for_status()


def pin_interval_hot(datasource: str, interval: str, replicas: int = 3) -> None:
    """Idempotently prepend a loadByInterval hot-pin above existing rules."""
    override = {
        "type": "loadByInterval",
        "interval": interval,
        "tieredReplicants": {"hot": replicas},
    }
    rules = [r for r in get_rules(datasource)
             if not (r.get("type") == "loadByInterval"
                     and r.get("interval") == interval)]
    put_rules(datasource, [override] + rules,
              author="rule-bot", comment=f"pin {interval} to {replicas} hot")


def wait_converged(datasource: str, max_wait: float = 1800.0) -> bool:
    delay, waited = 5.0, 0.0
    while waited < max_wait:
        r = requests.get(
            f"{COORDINATOR}/druid/coordinator/v1/loadstatus",
            params={"full": ""}, timeout=30,
        )
        r.raise_for_status()
        remaining = r.json().get(datasource, {})
        if not remaining or all(v == 0 for v in remaining.values()):
            return True
        time.sleep(delay)
        waited += delay
        delay = min(delay * 2, 60.0)  # capped exponential backoff
    return False


if __name__ == "__main__":
    pin_interval_hot("events", "2026-01-01T00:00:00Z/2026-04-01T00:00:00Z", 3)
    if not wait_converged("events"):
        raise SystemExit("rules applied but replicas did not converge in budget")
    print("rules applied and fully loaded")

Two safeguards make this production-safe. The read-filter-write cycle removes any prior pin for the same interval before prepending, so re-running the script never stacks duplicate rules. The convergence poll blocks on loadstatus?full — which reports segments still queued per datasource and tier — so a downstream migration only proceeds once the new replicas are actually loaded, mirroring the submit-and-verify discipline in dynamic ingestion spec generation.

Failure Modes & Diagnostics #

1. Segments in a rule's window never load. The tieredReplicants names a tier no Historical advertises. Confirm the live tiers, then compare against the rules:

curl -s http://coordinator:8081/druid/coordinator/v1/tiers | jq
curl -s http://coordinator:8081/druid/coordinator/v1/rules/events | jq '.[].tieredReplicants'

Every key in the second output must appear in the first. A hot in the rules but only _default_tier in the Druid cluster means no node ever receives those replicas.

2. A new specific rule appears to do nothing. An earlier, broader rule matches first and shadows it. Fetch the effective rules and read top-down — the first matcher that accepts a segment wins:

curl -s "http://coordinator:8081/druid/coordinator/v1/rules/events?full" \
  | jq '.[] | {type, interval, period, tieredReplicants}'

If a loadForever or wide loadByPeriod sits above your loadByInterval, move the specific rule up.

3. Rule change floods the Druid cluster / query latency dips during rollout. Replication is ramping faster than deep storage can serve. Watch the per-tier remaining load:

curl -s "http://coordinator:8081/druid/coordinator/v1/loadstatus?full" | jq '.events'

A large, slowly shrinking count is a healthy throttled ramp; if it is not shrinking, capacity — not throttle — is the limit. Inspect per-server queues:

curl -s "http://coordinator:8081/druid/coordinator/v1/loadqueue?simple" | jq

4. Dropped data still served / not dropped. A dropByPeriod sits below a loadForever, so nothing reaches it. Reorder so drops precede the catch-all, and verify no live version remains:

curl -s "http://coordinator:8081/druid/coordinator/v1/datasources/events/segments?full=true" \
  | jq 'group_by(.interval)[] | {interval: .[0].interval, count: length}'

Automation Checklist #

Wire these into the pipeline that manages rule sets so placement stays correct and auditable.

• Terminal catch-all present — assert every rule set ends in loadForever or dropForever so no segment is left unmatched.
• Tier names validated — check every tieredReplicants key against GET /druid/coordinator/v1/tiers before posting a rule set.
• Specific-before-general order — lint that loadByInterval rules precede overlapping loadByPeriod rules, and that no rule follows the catch-all.
• Audited writes — always send X-Druid-Author and X-Druid-Comment; reject unattributed rule changes in CI.
• Idempotent edits — read-filter-write so re-applying a rule set never stacks duplicates.
• Convergence gate — after posting, poll loadstatus?full to zero-remaining before signalling success downstream.
• Capacity pre-check — compute projected tierBytes against aggregate maxSize and refuse a rule change that would push a tier past 85%.
• Throttle awareness — for large replication increases, estimate ramp time from replicationThrottleLimit and schedule the change in a low-traffic window.
• Drift audit — periodically diff live rules against the intended source-of-truth and alert on manual edits.
• Under-replication alert — page on druid_segment_underreplicated_count > 0 sustained, the direct signal a rule cannot be satisfied.

Related #

• Writing tier-aware load rules — concrete hot/cold replica-count JSON built and pushed through the rules API.
• Segment replication and fault tolerance — how the replica counts these rules declare translate into availability and node-loss survival.
• Historical tier balancing strategies — how the Coordinator chooses which Historical within a rule's tier receives each replica.
• Configuring segment retention policies — the retention-and-drop side of the same rule engine, driving per-tenant expiry.
• Query routing and segment discovery — how the Broker prefers the higher-priority tier when a segment is loaded on more than one.

Up one level: Apache Druid segment replication, balancing & high availability frames how load rules, replication factor, and balancing fit into one control loop.