Druid Segment Ops Segment & Ingestion Orchestration

Exactly-Once Druid Backfills with dropExisting

A backfill re-ingests an interval that already holds data — to correct a bug, absorb late-arriving records, or apply a new rollup. The failure everyone hits at least once is the double-count: the re-run adds a second segment version on top of the first, both stay live, and every query over that interval now sums two copies of the same rows. This page shows how to make a backfill exactly-once in effect — the interval ends with precisely one live version regardless of how many times you run it — using dropExisting: true with an explicit, tightly bounded intervals, and how to verify afterward that only the newest version is being served. It is the concrete replacement mechanism behind the parent guide on idempotent ingestion & retry patterns, and it depends on the same segment-versioning contract that governs the Druid segment architecture and lifecycle.

A dropExisting backfill atomically replacing an interval so only the newest version is served The top row shows an interval before backfill holding version v1 across three partition segments, all served. A backfill task runs with dropExisting true and an explicit interval. The middle shows the transition: the task publishes version v2, and on handoff v1 is overshadowed. The bottom row shows the interval after backfill holding only version v2; v1 is marked unused and no longer served, so a query reads exactly one copy of the rows. A caption warns that a wider interval would also sweep adjacent data. BEFORE · v1 SERVED v1 · part 0 v1 · part 1 v1 · part 2 Backfill task dropExisting: true intervals: [exact window] publishes v2 · overshadows v1 AFTER · v2 SERVED, v1 UNUSED v2 · part 0 v2 · part 1 v2 · part 2 one live version → no double-count A wider interval than the data would also sweep adjacent segments — bound it exactly.

Failure Modes & Diagnostics

The whole point of an exactly-once backfill is to avoid two specific, silent corruptions, so start by learning to detect them. The first is the double-count: two live versions for one interval. It happens when a re-run uses appendToExisting: true, or uses dropExisting but with a mismatched interval, so the old version is never overshadowed. Detect it by grouping loaded segments by interval and listing distinct versions:

curl -s "http://coordinator:8081/druid/coordinator/v1/datasources/events_web/segments?full=true" \
  | jq 'group_by(.interval)[]
        | {interval: .[0].interval, versions: (map(.version) | unique), n: length}'

Any interval whose versions array has more than one entry — where both are used — is double-counting. A correct backfill leaves exactly one version per interval.

The second corruption is the over-sweep: dropExisting with an interval wider than the data being replaced silently drops adjacent, healthy segments, because dropExisting removes everything in the stated window, not just what the task rewrites. Detect it by confirming the replaced interval's boundaries match the intended window exactly and that neighboring intervals still have their data:

# Are the hours on either side of the backfilled window still present?
curl -s "http://coordinator:8081/druid/coordinator/v1/datasources/events_web/segments?full=true" \
  | jq -r '.[].interval' | sort -u | grep -E '2026-07-15|2026-07-16|2026-07-17'

A gap where an adjacent day should be means the interval was too wide. The guard is mechanical: the spec's granularitySpec.intervals must bound exactly the window being corrected — no rounding out to a full day when only an hour changed. A third, rarer symptom is a partial handoff where v2 publishes but never loads, so the Broker keeps serving v1; that is a handoff problem, diagnosed in verifying segment handoff completion, and until it resolves the backfill has not actually taken effect.

Target Spec & Validated JSON

An exactly-once backfill spec is an index_parallel task with three non-negotiable settings: dropExisting: true, appendToExisting: false, and a granularitySpec.intervals that bounds the corrected window precisely. The block below is complete and copy-ready — all required keys present. Here only the single hour 13:00–14:00 is being corrected, so the interval is that hour and nothing wider.

{
  "type": "index_parallel",
  "id": "backfill_events_web_20260716T13_v4",
  "spec": {
    "dataSchema": {
      "dataSource": "events_web",
      "timestampSpec": { "column": "event_ts", "format": "iso" },
      "dimensionsSpec": {
        "dimensions": ["country", "device", { "name": "page_id", "type": "long" }]
      },
      "metricsSpec": [
        { "type": "count", "name": "rows" },
        { "type": "longSum", "name": "bytes", "fieldName": "resp_bytes" }
      ],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "HOUR",
        "queryGranularity": "MINUTE",
        "rollup": true,
        "intervals": ["2026-07-16T13:00:00Z/2026-07-16T14:00:00Z"]
      }
    },
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "s3",
        "prefixes": ["s3://lake/events/web/2026-07-16/hour=13/"]
      },
      "inputFormat": { "type": "json" },
      "appendToExisting": false,
      "dropExisting": true
    },
    "tuningConfig": {
      "type": "index_parallel",
      "maxRowsInMemory": 1000000,
      "maxNumConcurrentSubTasks": 4,
      "forceGuaranteedRollup": true,
      "partitionsSpec": {
        "type": "range",
        "partitionDimensions": ["country"],
        "targetRowsPerSegment": 5000000
      }
    }
  }
}

dropExisting: true is what makes the replacement atomic: on handoff, Druid marks every prior used segment inside 2026-07-16T13:00:00Z/2026-07-16T14:00:00Z as unused in a single metadata transaction, so there is never a window where both v1 and v2 are served. Because segmentGranularity is HOUR, the interval aligns to a segment boundary — an interval that splits a segment would force Druid to partially rewrite it, which dropExisting does not support cleanly, so always align the backfill window to the datasource's segment granularity. The pinned id ties this into the idempotency contract: re-running the same backfill collides and is rejected rather than producing a v3.

Python Automation Script

The script below runs an exactly-once backfill and refuses to declare success until it has verified that exactly one version is live for the interval. It uses only the standard library plus requests, with capped exponential backoff on submission.

import time
import requests

OVERLORD = "http://overlord:8090"
COORDINATOR = "http://coordinator:8081"


def submit_backfill(spec: dict, retries: int = 5) -> str:
    """POST the dropExisting spec with a pinned id; backoff on transient errors."""
    task_id = spec["id"]
    delay = 2.0
    for attempt in range(retries):
        try:
            r = requests.post(f"{OVERLORD}/druid/indexer/v1/task",
                              json=spec, timeout=30)
            if r.status_code == 400 and task_id in r.text:
                return task_id                # already ran: idempotent
            r.raise_for_status()
            return r.json()["task"]
        except requests.RequestException:
            if attempt == retries - 1:
                raise
            time.sleep(delay)
            delay = min(delay * 2, 30.0)
    raise RuntimeError("unreachable")


def poll_terminal(task_id: str, timeout_s: int = 3600) -> str:
    deadline = time.monotonic() + timeout_s
    delay = 5.0
    while time.monotonic() < deadline:
        r = requests.get(f"{OVERLORD}/druid/indexer/v1/task/{task_id}/status",
                         timeout=15)
        r.raise_for_status()
        state = r.json()["status"]["status"]
        if state in {"SUCCESS", "FAILED"}:
            return state
        time.sleep(delay)
        delay = min(delay * 2, 30.0)
    raise TimeoutError(f"{task_id} not terminal within {timeout_s}s")


def assert_single_version(datasource: str, interval_start: str,
                          tries: int = 30) -> str:
    """After handoff, exactly one used version must cover the interval."""
    delay = 5.0
    for _ in range(tries):
        r = requests.get(
            f"{COORDINATOR}/druid/coordinator/v1/datasources/{datasource}/segments",
            params={"full": "true"}, timeout=15)
        r.raise_for_status()
        used = [s for s in r.json()
                if s.get("interval", "").startswith(interval_start)]
        versions = sorted({s["version"] for s in used})
        if len(versions) == 1 and used:
            return versions[0]
        time.sleep(delay)
        delay = min(delay * 2, 30.0)
    raise RuntimeError(
        f"{datasource}@{interval_start}: expected 1 live version, got {versions}")


def run_exactly_once_backfill(spec: dict) -> str:
    task_id = submit_backfill(spec)
    if poll_terminal(task_id) != "SUCCESS":
        raise RuntimeError(f"backfill {task_id} did not succeed")
    ds = spec["spec"]["dataSchema"]["dataSource"]
    start = spec["spec"]["dataSchema"]["granularitySpec"]["intervals"][0].split("/")[0]
    version = assert_single_version(ds, start)
    return version

The load-bearing step is assert_single_version: the pipeline does not trust that dropExisting worked, it verifies that the Coordinator reports exactly one live version for the interval after handoff. If a race or a mismatched interval left two versions live, this raises rather than reporting a clean backfill — the same fail-closed discipline the idempotent ingestion & retry patterns guide applies to submission.

Verification Steps

After the backfill, three checks confirm exactly-once took effect. First, confirm exactly one live version covers the interval and the old version is gone:

curl -s "http://coordinator:8081/druid/coordinator/v1/datasources/events_web/segments?full=true" \
  | jq '[.[] | select(.interval | startswith("2026-07-16T13"))]
        | {versions: (map(.version) | unique), segments: length}'

Expected output — one version only:

{
  "versions": ["2026-07-16T18:22:41.006Z"],
  "segments": 1
}

Second, confirm the superseded v1 segments are now unused, not merely hidden, by asking for the full history including unused segments:

curl -s "http://coordinator:8081/druid/coordinator/v1/datasources/events_web/segments?full=true&includeUnused=true" \
  | jq '[.[] | select(.interval | startswith("2026-07-16T13"))]
        | group_by(.version)[] | {version: .[0].version, used: .[0].used}'

Expected output — the old version present but retired, the new version live:

{ "version": "2026-07-16T09:14:02.511Z", "used": false }
{ "version": "2026-07-16T18:22:41.006Z", "used": true }

Third, prove the query layer agrees by counting rows for the interval and comparing against the known source-row count — a double-count would show as roughly twice the expected total:

curl -s -H 'Content-Type: application/json' \
  -d '{"query":"SELECT COUNT(*) AS n FROM events_web WHERE __time >= TIMESTAMP '\''2026-07-16 13:00:00'\'' AND __time < TIMESTAMP '\''2026-07-16 14:00:00'\''"}' \
  http://broker:8082/druid/v2/sql | jq '.[0].n'

A single value matching the expected source count confirms exactly-once. If it is near double, an old version is still used — re-check the interval boundaries and that dropExisting was actually true. The full loadstatus and serverview checks that precede a queryable state are detailed in verifying segment handoff completion.

Up one level: Idempotent ingestion & retry patterns.

Back to Automated Ingestion Pipeline Orchestration