Recovering From Historical Node Loss in Apache Druid

A Historical node has died — hardware fault, an OOM kill, or a spot-instance reclaim — and your segment/underReplicated/count is climbing while a slice of recent queries returns partial results. The immediate question is whether Druid is healing on its own and how to help it converge faster without making things worse. The short answer: the Coordinator detects the loss on its next run and re-replicates the orphaned segments from deep storage onto surviving Historicals in the same tier, bounded by druid.coordinator.load.timeout and the replication throttle. This page walks the recovery timeline, the settings that govern it, and how to force it along. It is the operational companion to segment replication and fault tolerance, which defines the under-replicated and unavailable states referenced throughout.

Failure Modes & Diagnostics #

First establish which failure you are in: transient under-replication that will self-heal, or true unavailability with no surviving replica. The distinction dictates urgency.

# 1. Confirm the node is actually gone from the Druid cluster view (not just flapping)
curl -s "http://coordinator:8081/druid/coordinator/v1/servers?simple" \
  | jq '[.[] | {host, tier, currSize, maxSize}]'

# 2. Anything with ZERO copies (unavailable) vs merely under target?
curl -s "http://coordinator:8081/druid/coordinator/v1/loadstatus?simple" | jq   # unavailable per ds
curl -s "http://coordinator:8081/druid/coordinator/v1/loadstatus?full"   | jq   # still-to-load per tier

# 3. Is re-replication actually enqueued and draining?
curl -s "http://coordinator:8081/druid/coordinator/v1/loadqueue?simple" | jq

If step 2's ?simple output is all zeros, every affected segment still has at least one replica — you are under-replicated but queryable, and the priority is simply to let recovery finish. If a datasource shows a nonzero unavailable count, that data lost its last loaded copy; it will reload from deep storage but is missing from queries until it does. If step 3's load queues are empty while under-replication persists, the Coordinator has nowhere to place copies — the surviving nodes are full. Check headroom:

curl -s "http://coordinator:8081/druid/coordinator/v1/servers?simple" \
  | jq '[.[] | select(.tier=="hot") | {host, usedPct: ((.currSize/.maxSize*100)|floor)}]'

Survivors near 100% confirm a capacity wall: recovery is blocked until you add a node to the tier or relax replication. This is the failure that turns a routine node loss into a sustained degradation, and it is why the sizing math in the parent guide reserves 15% headroom per tier.

Target Spec & Validated JSON #

Recovery speed and correctness are governed by three settings. The first, druid.coordinator.load.timeout, is a Coordinator runtime property (set in runtime.properties, not the dynamic config) that bounds how long the Coordinator waits for a Historical to confirm a queued load before it gives up on that assignment and re-tries elsewhere:

# coordinator/runtime.properties
druid.coordinator.load.timeout=PT15M
druid.coordinator.period=PT60S

The default PT15M is deliberately generous — a cold Historical pulling a large segment from remote deep storage can legitimately take minutes. Shortening it makes the Coordinator abandon slow loads sooner and try a different node, which helps when one survivor is pathologically slow but hurts if every node is merely busy, because abandoned-and-retried loads waste read bandwidth. Leave it at default unless you have measured a specific slow-node pattern.

The other two are in the Coordinator dynamic config at POST /druid/coordinator/v1/config, and control the recovery ramp:

{
  "smartSegmentLoading": true,
  "replicationThrottleLimit": 500,
  "maxSegmentsInNodeLoadingQueue": 1000,
  "replicantLifetime": 15
}

• replicationThrottleLimit — replica loads per tier per run. During a large recovery you may temporarily raise it (after setting smartSegmentLoading: false) to converge faster, provided deep-storage read bandwidth and survivor CPU can absorb it.
• maxSegmentsInNodeLoadingQueue — per-node queue depth; raising it lets each survivor accept more re-replication work at once.
• replicantLifetime — runs a replica may wait queued before the Coordinator alerts; a low value surfaces a stuck recovery quickly.

To force a recovery along when a survivor is not accepting work — or to drain a node you are decommissioning rather than one that crashed — mark it in decommissioningNodes, which tells the Coordinator to move its segments off proactively:

{
  "decommissioningNodes": ["hist-hot-3:8083"],
  "decommissioningMaxPercentOfMaxSegmentsToMove": 100
}

Setting the percent to 100 lets the drain use the entire move budget for a fast, deliberate evacuation. The authoritative field reference is the Apache Druid coordinator documentation.

Python Automation Script #

The recovery monitor below watches the Druid cluster heal after a node loss: it polls unavailable and under-replicated counts with capped exponential backoff, reports progress, and returns once full replication is restored — or raises if recovery stalls past a budget, the signal a human must intervene (usually to add capacity). Standard library plus requests only.

import time
import requests

COORDINATOR = "http://coordinator:8081"


def counts() -> tuple[int, int]:
    """Return (total_unavailable, total_under_replicated) across all datasources."""
    ua = requests.get(f"{COORDINATOR}/druid/coordinator/v1/loadstatus",
                      params={"simple": ""}, timeout=30)
    ua.raise_for_status()
    unavailable = sum(int(v) for v in ua.json().values())
    ur = requests.get(f"{COORDINATOR}/druid/coordinator/v1/loadstatus",
                      params={"full": ""}, timeout=30)
    ur.raise_for_status()
    under = sum(sum(t.values()) for t in ur.json().values())
    return unavailable, under


def monitor_recovery(max_wait: float = 3600.0) -> None:
    delay, waited = 10.0, 0.0
    last_under = None
    stalled_since = 0.0
    while waited < max_wait:
        unavailable, under = counts()
        print(f"t+{int(waited):>4}s  unavailable={unavailable}  under_replicated={under}")
        if unavailable == 0 and under == 0:
            print("recovery complete: full replication restored")
            return
        # Detect a stall: under-replication not shrinking over a sustained window.
        if last_under is not None and under >= last_under:
            stalled_since += delay
        else:
            stalled_since = 0.0
        if stalled_since >= 300.0:
            raise RuntimeError(
                f"recovery stalled at under_replicated={under} for 5 min — "
                "check tier capacity (survivors likely full) or add a Historical"
            )
        last_under = under
        time.sleep(delay)
        waited += delay
        delay = min(delay * 2, 60.0)  # capped exponential backoff
    raise RuntimeError("recovery did not complete within budget")


if __name__ == "__main__":
    monitor_recovery()

The stall detector is the operational heart of this script: automatic recovery either converges or it is blocked on capacity, and a flat under-replication count for five minutes is the unambiguous signal that survivors are full and no amount of waiting will help. That converts a silent, slow-burning degradation into an actionable alert.

Verification Steps #

Once the monitor reports completion, confirm the Druid cluster is genuinely whole. First, both availability counters at zero:

curl -s "http://coordinator:8081/druid/coordinator/v1/loadstatus?simple" | jq 'add // 0'
curl -s "http://coordinator:8081/druid/coordinator/v1/loadstatus?full" \
  | jq '[.[][]] | add // 0'

Expected — no unavailable, nothing left to load:

0
0

Next, confirm the surviving tier holds the full replica set on distinct hosts. Every segment should again have its desired number of copies spread across different Historicals:

curl -s "http://coordinator:8081/druid/coordinator/v1/servers?full" \
  | jq '[.[] | select(.tier=="hot") | {host, segments: (.segments | length)}]'

[
  { "host": "hist-hot-1:8083", "segments": 512 },
  { "host": "hist-hot-2:8083", "segments": 512 },
  { "host": "hist-hot-4:8083", "segments": 511 }
]

Roughly equal counts across the survivors confirm re-replication both restored redundancy and rebalanced — the replacement copies did not all pile onto one node. If the counts are lopsided, let the balancer settle or nudge it as described in Historical tier balancing strategies. Finally, when you replace the dead node, bring the new Historical up with the same druid.server.tier, and the Coordinator will automatically move a fair share of segments onto it — no manual assignment needed. If the loss was a planned decommission rather than a crash, clear the node from decommissioningNodes afterward so its slot is reusable.

Related #

• Segment replication and fault tolerance — parent guide: the under-replicated vs unavailable states and the replica-count math this recovery restores.
• Historical tier balancing strategies — how re-replicated copies spread evenly across the surviving tier.
• Coordinator load rules and tier placement — the rules that define how many replicas recovery must restore.
• Query routing and segment discovery — how the Broker drops a dead Historical from its timeline and picks it back up on recovery.

Up one level: Segment replication and fault tolerance.