Diagnosing Segment Balancing Storms in Apache Druid

Your Coordinator duty time is climbing past a minute, segment/moved/count is pinned near its ceiling every run, and Historicals are constantly pulling and dropping segments — yet utilization never actually evens out. This is a balancing storm: the Coordinator moves segments faster than the Druid cluster stabilizes, spending deep-storage read bandwidth and Historical CPU on churn that produces no net improvement. This page identifies the storm from its telemetry, traces it to one of three usual root causes, and throttles it back to calm. It is the failure-mode companion to Historical tier balancing strategies, which explains the cost balancer and move budget this page tunes.

Failure Modes & Diagnostics #

A storm has a clear signature across three signals: moves stuck at the ceiling, rising Coordinator duty time, and utilization that does not converge. Capture all three before changing anything.

# 1. Coordinator duty run time — is it climbing past the period (default 60s)?
curl -s "http://coordinator:8081/druid/coordinator/v1/config" \
  | jq '{smartSegmentLoading, maxSegmentsToMove, balancerComputeThreads}'

# 2. Per-node used bytes — is the spread actually shrinking between runs?
curl -s "http://coordinator:8081/druid/coordinator/v1/servers?simple" \
  | jq '[.[] | select(.tier=="hot") | {host, gb: ((.currSize/1073741824)|floor)}] | sort_by(.gb)'

# 3. Load/drop queues — churn shows as segments moving in AND out constantly
curl -s "http://coordinator:8081/druid/coordinator/v1/loadqueue?simple" | jq

Run the second command a few times a minute apart. In a healthy rebalance the spread narrows each run and moves taper off; in a storm the spread stays roughly constant while moves stay pinned — the Druid cluster is busy but not improving. Three root causes account for nearly every storm:

• Move budget too high for the Druid cluster. A large maxSegmentsToMove (usually a manual override with smartSegmentLoading off) lets the balancer relocate more than the tier can absorb per run, so queues never drain and the next run piles on more.
• A flapping Historical. A node that repeatedly leaves and rejoins the Druid cluster (GC pauses, network blips, an unhealthy disk) forces the balancer to re-evaluate placement every cycle — each disappearance orphans its segments, each return invites moving them back.
• Too many tiny segments. A datasource with millions of sub-target segments inflates the cost computation and the candidate set, so each run does enormous work for marginal gain. The fix is upstream: compact them via automated compaction task scheduling.

Distinguish them quickly: if smartSegmentLoading is off and maxSegmentsToMove is large, suspect the budget; if one host oscillates between present and absent in the server view, suspect flapping; if segment count per datasource is in the millions, suspect fragmentation.

Target Spec & Validated JSON #

The immediate remedy is to calm the move rate through the Coordinator dynamic config at POST /druid/coordinator/v1/config. The safest first move is simply to re-enable smart loading, which auto-sizes the budget to the Druid cluster:

{
  "smartSegmentLoading": true
}

If you must keep manual control — for instance while a specific migration is in flight — pin a conservative budget instead of the default-high value:

{
  "smartSegmentLoading": false,
  "maxSegmentsToMove": 20,
  "balancerComputeThreads": 2,
  "useRoundRobinSegmentAssignment": true
}

• maxSegmentsToMove: 20 — a deliberately low ceiling so each run moves only what the tier can comfortably absorb, letting queues drain and the spread converge instead of thrashing.
• useRoundRobinSegmentAssignment: true — keeps new segment loads off the expensive cost path, so the balancer's compute budget goes to convergence, not initial placement.
• balancerComputeThreads — leave modest; adding threads speeds a CPU-bound cost computation but does not fix a storm whose cause is churn or fragmentation.

If the cause is a flapping node, do not tune the budget at all — take the node out cleanly by adding it to decommissioningNodes so its segments drain in a controlled way rather than being repeatedly orphaned and re-placed:

{
  "decommissioningNodes": ["hist-hot-3:8083"],
  "decommissioningMaxPercentOfMaxSegmentsToMove": 100
}

The authoritative field reference for all of these is the Apache Druid coordinator documentation.

Python Automation Script #

The storm monitor below samples moves-per-run and per-node balance over several Coordinator cycles, decides whether balancing is converging or storming, and — if storming — throttles the move budget down and confirms the churn subsides. It uses capped exponential backoff and only the standard library plus requests.

import time
import statistics
import requests

COORDINATOR = "http://coordinator:8081"


def config() -> dict:
    r = requests.get(f"{COORDINATOR}/druid/coordinator/v1/config", timeout=30)
    r.raise_for_status()
    return r.json()


def set_config(patch: dict) -> None:
    cfg = config()
    cfg.update(patch)
    r = requests.post(
        f"{COORDINATOR}/druid/coordinator/v1/config", json=cfg,
        headers={"Content-Type": "application/json",
                 "X-Druid-Author": "storm-bot",
                 "X-Druid-Comment": "throttle balancing move budget"},
        timeout=30,
    )
    r.raise_for_status()


def tier_spread_gb(tier: str) -> float:
    r = requests.get(f"{COORDINATOR}/druid/coordinator/v1/servers?simple", timeout=30)
    r.raise_for_status()
    used = [s["currSize"] / 1073741824 for s in r.json() if s.get("tier") == tier]
    return statistics.pstdev(used) if len(used) > 1 else 0.0


def is_storming(tier: str, samples: int = 4, interval: float = 65.0) -> bool:
    """Storm = per-node spread not shrinking across several Coordinator runs."""
    spreads = []
    delay = interval
    for _ in range(samples):
        spreads.append(tier_spread_gb(tier))
        time.sleep(delay)
        delay = min(delay * 1.5, 180.0)  # capped backoff between samples
    # Not converging if the last spread is >= 90% of the first.
    return spreads[-1] >= 0.9 * spreads[0] and spreads[0] > 0


def tame_storm(tier: str = "hot") -> None:
    if not is_storming(tier):
        print(f"tier {tier}: balancing is converging, no action")
        return
    print(f"tier {tier}: storm detected — re-enabling smart loading")
    set_config({"smartSegmentLoading": True})
    # Confirm the churn subsides: spread should now start shrinking.
    time.sleep(130)
    if is_storming(tier, samples=3):
        raise RuntimeError(
            f"tier {tier}: still storming after throttle — "
            "check for a flapping Historical or millions of tiny segments"
        )
    print(f"tier {tier}: balancing settled")


if __name__ == "__main__":
    tame_storm("hot")

The monitor's logic is deliberately conservative: it declares a storm only when the per-node spread fails to shrink across several runs (busy-but-improving is not a storm and needs no intervention), and after throttling it re-checks rather than assuming the fix worked — if the churn persists, it escalates to the human causes a config change cannot fix, a flapping node or segment fragmentation.

Verification Steps #

After throttling, confirm the storm has actually subsided rather than merely paused. First, Coordinator duty time back under the period:

curl -s "http://coordinator:8081/druid/coordinator/v1/config" | jq '{smartSegmentLoading, maxSegmentsToMove}'

Expected — smart loading back on (or a low pinned budget):

{ "smartSegmentLoading": true, "maxSegmentsToMove": 0 }

(maxSegmentsToMove reads as the last-set value but is ignored while smartSegmentLoading is true.) Next, sample per-node used bytes twice a couple of minutes apart and confirm the spread is now shrinking:

curl -s "http://coordinator:8081/druid/coordinator/v1/servers?simple" \
  | jq '[.[] | select(.tier=="hot") | (.currSize/1073741824)] | {min: (min|floor), max: (max|floor), spread_gb: ((max|floor)-(min|floor))}'

{ "min": 1412, "max": 1449, "spread_gb": 37 }

A spread_gb that narrows between samples means the tier is converging, not churning. Finally, confirm the load and drop queues are draining rather than constantly refilling:

curl -s "http://coordinator:8081/druid/coordinator/v1/loadqueue?simple" \
  | jq 'to_entries | map({host: .key, queued: .value}) | sort_by(-.queued)'

Queues that shrink toward empty across successive calls confirm the storm is over. If they stay pinned, the root cause is not the move budget — hunt the flapping node in the server view or count segments per datasource and route the tiny-segment case to automated compaction task scheduling. A Druid cluster kept in the segment size band from optimizing segment size for Historical nodes rarely storms in the first place.

Related #

• Historical tier balancing strategies — parent guide: the cost balancer, move budget, and smartSegmentLoading this page throttles.
• Segment replication and fault tolerance — how a flapping node orphans replicas and triggers the re-placement churn behind many storms.
• Automated compaction task scheduling — merging tiny segments, the upstream fix for fragmentation-driven storms.
• Optimizing segment size for Historical nodes — keeping segments large enough that balancing stays cheap and calm.

Up one level: Historical tier balancing strategies.