Tiered Storage Migration Runbook

Query SLAs demand fast NVMe-backed Historical nodes, but keeping ninety days of history on that hardware is wasteful when only the last week is queried at interactive rates. The fix is a second, cheaper Historical tier — larger, slower disks, fewer replicas — and a retention ladder that migrates segments from hot to cold as they age. Done wrong, a tier migration under-replicates or unloads data mid-move and drops queries; done right, it is a boring, verifiable, zero-data-loss reshuffle driven entirely by load rules. This runbook is the operational procedure for that migration. It sits under segment size optimization strategies, because the segments you migrate must already be in the query-optimal size band for a cold-tier scan to stay economical; the tier-placement mechanics behind the rules are developed under coordinator load rules and tier placement.

Failure Modes & Diagnostics #

A tier migration is a change to load rules, so its failures are load-placement failures: segments stuck under-replicated, a cold tier with no nodes to receive them, or a move that unloaded the hot copy before the cold copy was serving. Diagnose from the shell before and during the migration.

# Do the tiers you're migrating between actually have Historical nodes and capacity?
curl -s "http://druid-coordinator:8081/druid/coordinator/v1/tiers?simple" | jq '.'

# Per-tier: how many segments are loaded vs the target the rules ask for?
curl -s "http://druid-coordinator:8081/druid/coordinator/v1/loadstatus?full" | jq '.clickstream'

# Segments the Coordinator wants loaded but nothing is serving — must be 0 mid-migration
curl -s "http://druid-coordinator:8081/druid/coordinator/v1/metrics" 2>/dev/null \
  || curl -s "http://druid-coordinator:8081/druid/coordinator/v1/datasources/clickstream?full" \
       | jq '{segments: .segments.count, replicated: .segments.replicatedSize}'

The three failures to watch:

1. Cold tier has no capacity or no nodes. Symptom: after applying the demotion rule, aging segments show as under-replicated and never land on cold. Root cause: no Historical is tagged into the cold tier, or its druid.segmentCache.locations is full. Remediation: confirm the tier membership (druid.server.tier=cold on the cold nodes) and that aggregate cold capacity exceeds the migrating byte volume before you push the rule.

2. Segments unavailable during the move. Symptom: queries over the aging window briefly return partial results while the migration runs. Root cause: a rule change that removed the hot placement in the same edit that added the cold placement, and the Coordinator dropped hot before cold finished loading. Remediation: the Coordinator's default behaviour is load-before-drop, but confirm druid.coordinator.loadqueuepeon throughput is high enough that the cold load completes promptly; stage the migration in age bands so only a bounded volume moves per cycle.

3. Balancing storm after the push. Symptom: CPU and network spike cluster-wide as the Coordinator reshuffles far more than the migrated segments. Root cause: too much moved at once, or an interaction with the cost balancer. Remediation: cap maxSegmentsToMove and migrate in age-banded steps rather than one wholesale rule change.

Target Spec & Validated JSON #

The migration is expressed entirely as a retention ladder POSTed to POST /druid/coordinator/v1/rules/{dataSource}. The array replaces the datasource's rules wholesale, so send the complete ordered ladder. This one keeps the last 7 days hot at 2 replicas, demotes days 8–90 to a single cold replica, and drops everything older — the classic hot-to-cold decay. Every field is annotated; the tieredReplicants grammar is detailed in the retention rule syntax reference.

[
  {
    "type": "loadByPeriod",
    "period": "P7D",
    "includeFuture": true,
    "tieredReplicants": { "hot": 2 }
  },
  {
    "type": "loadByPeriod",
    "period": "P90D",
    "tieredReplicants": { "cold": 1 }
  },
  {
    "type": "dropForever"
  }
]

• The first rule keeps the interactive window on fast disks at 2 replicas for both availability and query concurrency.
• The second rule matches everything the first did not (8–90 days old) and places a single replica on cold. Because the first rule already claimed the last 7 days, first-match-wins means this rule only ever sees the aging tail.
• dropForever sweeps everything past 90 days off both tiers. It is the only rule that unloads; the migration itself never drops data, it re-places it.

The loaded footprint on each tier follows directly from the window, the ingest rate, and the replica count:

$$ \text{tierBytes} \approx \text{ingestBytesPerDay} \times \text{tierWindowDays} \times \text{replicas} $$

For a datasource ingesting 120 GB/day of compressed segments, the hot tier holds $120 \times 7 \times 2 \approx 1{,}680$ GB and the cold tier holds $120 \times 83 \times 1 \approx 9{,}960$ GB. Moving the 83-day tail from 2 hot replicas to 1 cold replica is what turns roughly 20 TB of premium storage into 10 TB of cheap storage — the saving quantified under reducing Historical node storage costs.

Python Automation Script #

The runbook is: capture the current ladder, apply the migration ladder, wait for the Coordinator to converge with no unavailable segments, and only then declare the move complete. The script below does exactly that, keeping a rollback copy of the prior rules so a bad migration can be reverted in one call. It uses the standard library plus requests, with capped exponential backoff on the asynchronous rules and status APIs.

import time
import logging
import requests

logger = logging.getLogger("druid.tier_migrate")

COORDINATOR = "http://druid-coordinator:8081"


def _request(method, url, *, retries=5, **kwargs):
    """HTTP with capped exponential backoff on 5xx and connection errors."""
    delay = 1.0
    for attempt in range(1, retries + 1):
        try:
            resp = requests.request(method, url, timeout=30, **kwargs)
            if resp.status_code < 500:
                resp.raise_for_status()
                return resp
        except requests.RequestException as exc:
            logger.warning("%s %s failed: %s (attempt %d)", method, url, exc, attempt)
        if attempt == retries:
            raise RuntimeError(f"{method} {url} failed after {retries} attempts")
        time.sleep(delay)
        delay = min(delay * 2, 30.0)  # cap the backoff at 30s
    return None


def get_rules(datasource: str) -> list:
    return _request("GET", f"{COORDINATOR}/druid/coordinator/v1/rules/{datasource}").json()


def put_rules(datasource: str, rules: list) -> None:
    _request(
        "POST",
        f"{COORDINATOR}/druid/coordinator/v1/rules/{datasource}",
        json=rules,
        headers={"Content-Type": "application/json"},
    )
    logger.info("Applied %d rules to %s", len(rules), datasource)


def await_no_unavailable(datasource: str, deadline_s: int = 3600, poll_s: int = 30) -> bool:
    """Return True once nothing is unavailable or under-replicated for the datasource."""
    started = time.monotonic()
    while time.monotonic() - started < deadline_s:
        status = _request(
            "GET", f"{COORDINATOR}/druid/coordinator/v1/loadstatus?full"
        ).json().get(datasource, {})
        pending = sum(int(v) for v in status.values()) if status else 0
        logger.info("%s segments still pending placement: %d", datasource, pending)
        if pending == 0:
            return True
        time.sleep(poll_s)
    return False


def migrate(datasource: str, migration_rules: list) -> None:
    rollback = get_rules(datasource)
    logger.info("Captured rollback ladder (%d rules)", len(rollback))
    put_rules(datasource, migration_rules)
    if not await_no_unavailable(datasource):
        logger.error("Convergence deadline exceeded; rolling back")
        put_rules(datasource, rollback)
        raise RuntimeError("tier migration did not converge; rolled back")
    logger.info("Migration converged with zero pending placements")


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    ladder = [
        {"type": "loadByPeriod", "period": "P7D", "includeFuture": True,
         "tieredReplicants": {"hot": 2}},
        {"type": "loadByPeriod", "period": "P90D",
         "tieredReplicants": {"cold": 1}},
        {"type": "dropForever"},
    ]
    migrate("clickstream", ladder)

The rollback capture is what makes this safe to run unattended: if the cold tier cannot absorb the migrating segments within the deadline, the prior ladder is restored automatically and no interval is left stranded.

Verification Steps #

Confirm the migration placed replicas exactly where the rules ask. First, the load status for the datasource should reach zero pending placements — nothing waiting to load and nothing under-replicated:

curl -s "http://druid-coordinator:8081/druid/coordinator/v1/loadstatus?full" \
  | jq '{clickstream: (.clickstream // {})}'

Expected output once converged — an empty map means every rule is satisfied:

{ "clickstream": {} }

Next, confirm the byte split across tiers matches the intended hot/cold decay:

curl -s "http://druid-coordinator:8081/druid/coordinator/v1/tiers?simple" \
  | jq 'to_entries | map({tier: .key, segments: .value.segmentCount, bytes: .value.currSize})'

Expected: the hot tier carries only the last 7 days at 2 replicas while cold carries the 8–90-day tail at 1 replica, e.g.:

[
  { "tier": "hot",  "segments": 14, "bytes": 1803886264 },
  { "tier": "cold", "segments": 83, "bytes": 10695475712 }
]

Finally, run a representative query spanning both the hot window and the cold tail and confirm it returns complete results with no missing intervals — proof the migration never created an availability gap. Enforce these as gates in the runbook: migrate in age-banded steps so only bounded volume moves per cycle, keep the rollback ladder until verification passes, and alert on any non-zero under-replicated count that persists beyond a few coordination cycles. The segments you migrate should already sit in the target size band per segment size optimization strategies, so a cold-tier scan of the aged tail stays economical rather than fragmenting across thousands of tiny objects.

Related #

• Segment Size Optimization Strategies — the parent guide; keep segments in the query-optimal band before migrating so cold-tier scans stay cheap.
• Coordinator Load Rules and Tier Placement — the placement mechanics behind tieredReplicants and how the Coordinator moves replicas across tiers.
• Druid Retention Rule Syntax Reference — the full load/drop rule grammar the migration ladder is written in.
• Reducing Historical Node Storage Costs — the tier-and-replica cost accounting a hot-to-cold migration directly improves.
• TTL Mapping and Data Expiration — the expiration layer that eventually drops the cold tail off both tiers.

Up one level: Segment Size Optimization Strategies.

For authoritative tier and load-rule semantics, see the official Apache Druid retention rules reference.