Druid Segment Ops Segment & Ingestion Orchestration

Verifying Segment Handoff Completion in Druid

You have an interval you just ingested and one question: is it actually queryable yet? A task reporting SUCCESS does not answer that — the segments are published to the metadata store but not necessarily loaded onto a Historical, and a query will read empty or partial results until they are. This page is the concrete, curl-level procedure for confirming a specific interval has completed handoff and is being served, with the exact commands and the output you should expect from each. It is the hands-on companion to the parent guide on ingestion handoff validation, which explains why handoff and availability are distinct; here we simply prove availability from the shell and wrap it in a fail-closed gate.

A three-check verification ladder for confirming a Druid interval is queryable Three checks run in sequence, each a gate. Check one queries the Coordinator loadstatus endpoint and passes when the datasource reports one hundred percent loaded. Check two queries the per-datasource segments endpoint and passes when the target interval has a used segment with non-zero size. Check three runs a SQL count against the Broker and passes when it returns the expected row total. Only when all three pass is the interval declared queryable; any failure returns to waiting. 1 · loadstatus datasource == 100% Coordinator 2 · serverview interval used, size > 0 datasources/{ds}/segments 3 · SQL count rows == expected Broker Queryable gate passes pass pass pass Any check failing returns to waiting — the gate never reports queryable on a partial signal.

Failure Modes & Diagnostics

The verification exists because three things can each independently make an interval non-queryable after a SUCCESS, and each shows up in a different check. If check 1 (loadstatus) never reaches 100%, the Coordinator has not finished assigning and loading the datasource's segments — the load queue is draining or stuck on full Historicals. If check 1 passes but check 2 (per-interval serverview) finds no used segment for your window, the aggregate was satisfied by other intervals and yours specifically was excluded by a load rule or overshadowed. If checks 1 and 2 pass but check 3 (the SQL count) returns zero or a wrong total, the Broker's timeline has not caught up to the Historical's announcement, or a rollup collapsed rows differently than expected. Running all three in order localizes the fault immediately instead of leaving you guessing.

A fast first triage is the full loadstatus, which names exactly what is still pending:

# Anything still loading, per datasource and tier? Empty/absent == fully available.
curl -s "http://coordinator:8081/druid/coordinator/v1/loadstatus?full" \
  | jq '.events_web // "events_web fully loaded"'

If that shows a pending count that will not fall, look at the load queue to see whether any Historical is even accepting the segment:

# A queue that never drains means the target tier's Historicals are full.
curl -s "http://coordinator:8081/druid/coordinator/v1/loadqueue?simple" \
  | jq 'to_entries[] | select(.value.segmentsToLoad | length > 0)
        | {server: .key, toLoad: (.value.segmentsToLoad | length)}'

An empty result here with a non-queryable interval means the Coordinator never assigned the segment — a load-rule problem, not a capacity one — which is the boundary the parent ingestion handoff validation guide draws between "pending" and "never assigned".

Target Spec & Validated JSON

Handoff verification is a read-side procedure, so the "spec" here is the exact set of REST calls and the JSON each returns when handoff has completed. This is the contract the automation asserts against. First, the aggregate load status when the datasource is fully available — the datasource is simply absent from the response, or reports 100:

{}

An empty object from GET /druid/coordinator/v1/loadstatus means every used segment across all datasources is loaded. When a specific datasource is still loading, it appears with a percentage below 100:

{ "events_web": 66.7 }

Second, the per-interval serverview from GET /druid/coordinator/v1/datasources/events_web/segments?full=true, filtered to the target interval, returns one or more loaded segments with used: true and a real byte size:

[
  {
    "dataSource": "events_web",
    "interval": "2026-07-16T13:00:00.000Z/2026-07-16T14:00:00.000Z",
    "version": "2026-07-16T18:22:41.006Z",
    "size": 412998221,
    "used": true
  }
]

The three fields the verifier keys on are used: true (published and not overshadowed), size greater than zero (a real segment, not an empty tombstone), and a single version for the interval (no double-count). Third, the query-layer confirmation from the Broker's SQL endpoint returns the expected row total for the window, proving the Broker timeline actually serves it:

[ { "n": 4820551 } ]

Python Automation Script

The verifier below runs the three checks in order and only returns success when all pass, with capped exponential backoff between rounds. It uses the standard library plus requests, and returns a structured verdict so a CI/CD gate or an orchestrator sensor can branch on it.

import time
import requests

COORDINATOR = "http://coordinator:8081"
BROKER = "http://broker:8082"


def check_loadstatus(datasource: str) -> bool:
    """Check 1: the datasource is fully loaded cluster-wide."""
    r = requests.get(f"{COORDINATOR}/druid/coordinator/v1/loadstatus", timeout=15)
    r.raise_for_status()
    return float(r.json().get(datasource, 100.0)) >= 100.0


def check_interval_served(datasource: str, interval_start: str) -> dict | None:
    """Check 2: the target interval has a used segment with real bytes."""
    r = requests.get(
        f"{COORDINATOR}/druid/coordinator/v1/datasources/{datasource}/segments",
        params={"full": "true"}, timeout=15)
    r.raise_for_status()
    segs = [s for s in r.json()
            if s.get("interval", "").startswith(interval_start)
            and s.get("used", False) and s.get("size", 0) > 0]
    if not segs:
        return None
    return {"segments": len(segs),
            "versions": sorted({s["version"] for s in segs})}


def check_row_count(datasource: str, start: str, end: str) -> int:
    """Check 3: the Broker actually serves rows for the window."""
    sql = (f"SELECT COUNT(*) AS n FROM {datasource} "
           f"WHERE __time >= TIMESTAMP '{start}' AND __time < TIMESTAMP '{end}'")
    r = requests.post(f"{BROKER}/druid/v2/sql", json={"query": sql}, timeout=30)
    r.raise_for_status()
    return int(r.json()[0]["n"])


def verify_handoff(datasource: str, interval_start: str, sql_start: str,
                   sql_end: str, expected_rows: int | None = None,
                   timeout_s: int = 600) -> dict:
    """Fail-closed gate: all three checks must pass within the timeout."""
    deadline = time.monotonic() + timeout_s
    delay = 15.0
    while time.monotonic() < deadline:
        if check_loadstatus(datasource):
            served = check_interval_served(datasource, interval_start)
            if served and len(served["versions"]) == 1:
                rows = check_row_count(datasource, sql_start, sql_end)
                if expected_rows is None or rows == expected_rows:
                    return {"queryable": True, "rows": rows, **served}
        time.sleep(delay)
        delay = min(delay * 1.5, 30.0)
    return {"queryable": False,
            "loaded": check_loadstatus(datasource),
            "served": check_interval_served(datasource, interval_start),
            "hint": "load rule excludes interval or Historicals are full"}


if __name__ == "__main__":
    verdict = verify_handoff(
        "events_web", "2026-07-16T13",
        "2026-07-16 13:00:00", "2026-07-16 14:00:00",
        expected_rows=4_820_551)
    if not verdict["queryable"]:
        raise SystemExit(f"handoff NOT verified: {verdict}")
    print("handoff verified:", verdict)

The gate is fail-closed: it returns queryable: True only when loadstatus is 100%, the interval has exactly one live version with real bytes, and the Broker returns the expected row count. A single live version guards against double-count; the row-count assertion catches the case where segments are loaded but the Broker timeline has not caught up. This is exactly the assertion a deploy gate needs — plug it into the end-to-end ingestion CI/CD gate so a release only proceeds once new data is provably queryable.

Verification Steps

Run the three checks by hand to confirm, or to debug the automation. First, the aggregate load status:

curl -s "http://coordinator:8081/druid/coordinator/v1/loadstatus" | jq '.events_web // "fully loaded"'

Expected output when handoff is complete:

"fully loaded"

Second, the per-interval serverview, asserting a single used version with real bytes:

curl -s "http://coordinator:8081/druid/coordinator/v1/datasources/events_web/segments?full=true" \
  | jq '[.[] | select(.interval | startswith("2026-07-16T13"))]
        | {served: (all(.[]; .used and .size > 0)),
           versions: (map(.version) | unique), segments: length}'

Expected output — served, one version, a sane segment count:

{
  "served": true,
  "versions": ["2026-07-16T18:22:41.006Z"],
  "segments": 1
}

Third, the Broker row count for the window, which must match the known source total:

curl -s -H 'Content-Type: application/json' \
  -d '{"query":"SELECT COUNT(*) AS n FROM events_web WHERE __time >= TIMESTAMP '\''2026-07-16 13:00:00'\'' AND __time < TIMESTAMP '\''2026-07-16 14:00:00'\''"}' \
  http://broker:8082/druid/v2/sql | jq '.[0]'

Expected output — the expected row total, proving the Broker serves the interval:

{ "n": 4820551 }

When all three match their expected outputs, the interval has completed handoff and is queryable, and any downstream step — a dashboard refresh, a validation job, a deploy — can safely proceed. If check 3 returns roughly double the expected count, an old version is still used; resolve it with an atomic replace as covered in exactly-once Druid backfills.

Up one level: Ingestion handoff validation.

Back to Automated Ingestion Pipeline Orchestration