Automating Segment Drop Workflows

The recurring surprise for teams new to Druid is that dropping data does not reclaim any storage: a drop rule unloads a segment from Historical nodes, but its file still sits in deep storage and its row still occupies the metadata store, so the object-storage bill never moves. Actually reclaiming bytes is a distinct, irreversible step — a kill task — and the only segments it may delete are those explicitly marked unused. Automating drop safely therefore means sequencing a reversible markUnused before an irreversible kill, with a deliberate reconsideration window between them. This page sits under automated compaction task scheduling and details the drop-to-removal state machine, the Coordinator kill duty that can run it for you, and the guardrails that keep an automated workflow from deleting data a human still needs.

Failure Modes & Diagnostics #

Every automated drop workflow fails in one of two directions: it deletes nothing (drop rules fired but no kill ran, so storage never shrinks) or it deletes too much (a kill ran against an interval that was not safely marked). Diagnose both from the shell.

# How many segments does the metadata store still consider used for this datasource?
# A dropForever rule that "worked" but left this count unchanged means nothing was reclaimed.
curl -s "http://druid-coordinator:8081/druid/coordinator/v1/datasources/clickstream/segments" \
  | jq 'length'

# Is the automatic kill duty even enabled, and what is its safety floor?
curl -s "http://druid-coordinator:8081/status/properties" \
  | jq '{killOn: ."druid.coordinator.kill.on",
         retain: ."druid.coordinator.kill.durationToRetain",
         period: ."druid.coordinator.kill.period"}'

The failure signatures:

1. Dropped data still costs money. Symptom: a dropForever rule is live, the interval no longer appears in queries, but deep-storage bytes and the metadata segment count are flat. Root cause: drop only unloads; nothing was marked unused or killed. Remediation: enable the kill duty (druid.coordinator.kill.on=true) or run an explicit markUnused → kill step for the expired interval.

2. Kill task fails or hangs. Symptom: the kill task sits in RUNNING or flips to FAILED. Root cause: the Overlord/MiddleManager IAM role lacks s3:DeleteObject (or the equivalent on your deep store), or a lock conflict with a compaction or ingestion task on the same interval. Inspect the task log and active locks:

# Tail the kill task log for the real deep-storage delete error
curl -s "http://druid-overlord:8090/druid/indexer/v1/task/<KILL_TASK_ID>/log" | tail -50

# Any active locks on the interval you're trying to kill?
curl -s "http://druid-overlord:8090/druid/indexer/v1/lockedIntervals" \
  -H 'Content-Type: application/json' -d '{"clickstream": 0}' | jq '.'

3. A kill deleted still-wanted data. Symptom: an interval that should exist returns empty and cannot be re-loaded. Root cause: an automated kill ran with too small a durationToRetain, or a script marked-unused a live interval before killing. This is the failure the reversible-before-irreversible pattern exists to prevent: kill is metadata-destroying, so recovery requires re-ingesting from source. Remediation: widen durationToRetain, and never let a workflow kill an interval it marked unused in the same run without a reconsideration gap.

Target Spec & Validated JSON #

There are two ways to automate the workflow: hand it entirely to the Coordinator's kill duty, or drive the markUnused → kill sequence yourself from an orchestrator. Both are shown; most teams run the duty for steady-state expiry and reserve explicit kills for targeted cleanup.

Coordinator kill duty — set in coordinator/runtime.properties. This makes the Coordinator periodically mark segments outside the retention rules unused and issue kill tasks itself:

# Master switch: without this, drop rules unload data but NOTHING is ever deleted.
druid.coordinator.kill.on=true
# How often the kill duty runs (ISO-8601). Keep it well above coordinator.period.
druid.coordinator.kill.period=PT1H
# Hard safety floor: never delete data younger than this, regardless of drop rules.
druid.coordinator.kill.durationToRetain=P14D
# Cap segments removed per kill task, bounding metadata-store and deep-storage load.
druid.coordinator.kill.maxSegments=1000

• druid.coordinator.kill.on — the master switch. A conservative cluster leaves it false, which is exactly why so many clusters accumulate unused segments forever. Turn it on only once durationToRetain is set to a value you trust.
• druid.coordinator.kill.durationToRetain — the floor that overrides the rules. Even if a rule drops an interval, the duty will not delete anything newer than now − durationToRetain. This is the reconsideration window; size it to at least the on-call response time so a bad rule push is recoverable before physical delete.
• druid.coordinator.kill.maxSegments — bounds Overlord load when a large backlog of unused segments accumulates.

Explicit kill task — the minimal payload POSTed to POST /druid/indexer/v1/task. Only segments already marked unused in the interval are deleted; used segments in the same interval are untouched, so an over-broad interval is safe with respect to live data:

{
  "type": "kill",
  "dataSource": "clickstream",
  "interval": "2025-01-01T00:00:00.000Z/2025-04-01T00:00:00.000Z"
}

The markUnused call that makes segments eligible is a separate POST to the Coordinator:

{
  "interval": "2025-01-01T00:00:00.000Z/2025-04-01T00:00:00.000Z"
}

posted to POST /druid/coordinator/v1/datasources/clickstream/markUnused. Its inverse, markUsed, restores the segments as long as no kill has run — the reversibility the safe pattern depends on. The retention rules that decide which intervals should be dropped in the first place are covered in the retention rule syntax reference, and the byte accounting that tells you how much a reclaim actually saves is under reducing Historical node storage costs.

Python Automation Script #

A safe drop workflow separates the reversible and irreversible steps in time. The script below marks an interval unused, then enforces a reconsideration gap before killing — and refuses to kill anything younger than the same durationToRetain floor the Coordinator honours, so a mistaken interval argument cannot delete fresh data. It uses only the standard library plus requests, with capped exponential backoff on the asynchronous task APIs.

import time
import logging
from datetime import datetime, timedelta, timezone
import requests

logger = logging.getLogger("druid.drop")

COORDINATOR = "http://druid-coordinator:8081"
OVERLORD = "http://druid-overlord:8090"
KILL_FLOOR = timedelta(days=14)  # mirror druid.coordinator.kill.durationToRetain


def _request(method, url, *, retries=5, **kwargs):
    """HTTP with capped exponential backoff on 5xx and connection errors."""
    delay = 1.0
    for attempt in range(1, retries + 1):
        try:
            resp = requests.request(method, url, timeout=30, **kwargs)
            if resp.status_code < 500:
                resp.raise_for_status()
                return resp
        except requests.RequestException as exc:
            logger.warning("%s %s failed: %s (attempt %d)", method, url, exc, attempt)
        if attempt == retries:
            raise RuntimeError(f"{method} {url} failed after {retries} attempts")
        time.sleep(delay)
        delay = min(delay * 2, 30.0)  # cap the backoff at 30s
    return None


def _interval_end(interval: str) -> datetime:
    end = interval.split("/")[1]
    return datetime.fromisoformat(end.replace("Z", "+00:00"))


def mark_unused(datasource: str, interval: str) -> None:
    """Reversible: flag segments used=0 so they become kill-eligible."""
    _request(
        "POST",
        f"{COORDINATOR}/druid/coordinator/v1/datasources/{datasource}/markUnused",
        json={"interval": interval},
        headers={"Content-Type": "application/json"},
    )
    logger.info("Marked %s %s unused (reversible via markUsed)", datasource, interval)


def kill_interval(datasource: str, interval: str, poll_s: int = 10) -> str:
    """Irreversible: refuse young intervals, then submit and await the kill task."""
    if _interval_end(interval) > datetime.now(timezone.utc) - KILL_FLOOR:
        raise ValueError(f"interval {interval} is inside the {KILL_FLOOR} safety floor; refusing to kill")
    resp = _request(
        "POST",
        f"{OVERLORD}/druid/indexer/v1/task",
        json={"type": "kill", "dataSource": datasource, "interval": interval},
        headers={"Content-Type": "application/json"},
    )
    task_id = resp.json()["task"]
    logger.info("Kill task %s submitted for %s %s", task_id, datasource, interval)
    while True:
        status = _request(
            "GET", f"{OVERLORD}/druid/indexer/v1/task/{task_id}/status"
        ).json()["status"]["statusCode"]
        if status == "SUCCESS":
            logger.info("Kill task %s complete", task_id)
            return task_id
        if status == "FAILED":
            raise RuntimeError(f"Kill task {task_id} FAILED")
        time.sleep(poll_s)


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    expired = "2025-01-01T00:00:00.000Z/2025-04-01T00:00:00.000Z"
    mark_unused("clickstream", expired)
    # In production, the kill runs in a later scheduled pass, not the same run —
    # leaving a reconsideration window in which markUsed can still restore the data.
    kill_interval("clickstream", expired)

The two calls are deliberately decoupled in production: mark_unused runs in one scheduled pass and kill_interval in a later one, so the reversible state persists long enough for a human to catch a mistake. Killing in the same run that marks-unused defeats the entire safety model.

Verification Steps #

Confirm the reclaim actually freed storage rather than merely unloading. First, the metadata segment count for the interval should drop to zero after the kill:

curl -s "http://druid-coordinator:8081/druid/coordinator/v1/datasources/clickstream/intervals?full" \
  | jq 'to_entries | map(select(.key | startswith("2025-01"))) | length'

Expected output once the kill has completed — the interval no longer has any metadata rows:

0

Next, confirm the kill task itself reached a terminal success rather than silently failing on a permissions error:

curl -s "http://druid-overlord:8090/druid/indexer/v1/task/<KILL_TASK_ID>/status" \
  | jq '.status | {statusCode, duration}'

Expected:

{ "statusCode": "SUCCESS", "duration": 8421 }

Finally, verify deep storage actually shrank — list the datasource prefix on your object store and confirm the killed interval's segment directories are gone. Enforce these as workflow gates: never enable druid.coordinator.kill.on without a durationToRetain that covers your response window, alert on FAILED kill tasks and on DeleteObject permission errors in Overlord logs, and keep the markUnused and kill stages in separate scheduled runs so the reversible window is real. Coordinate the cadence with automated compaction task scheduling so intervals bound for deletion are never recompacted first — burning compute to rewrite data a kill will immediately purge.

Related #

• Automated Compaction Task Scheduling — the parent guide; align kill cadence with the compaction duty so doomed intervals are not rewritten before deletion.
• TTL Mapping and Data Expiration — the retention layer that decides which intervals should be dropped before this workflow reclaims them.
• Druid Retention Rule Syntax Reference — the load/drop rule grammar whose dropForever fires the first stage of this lifecycle.
• Reducing Historical Node Storage Costs — the byte accounting that quantifies what a reclaim actually saves.
• Configuring Segment Retention Policies — the access controls that must gate destructive kill and rule-change operations.

Up one level: Automated Compaction Task Scheduling.

For authoritative kill-task and data-deletion semantics, see the official Apache Druid data management reference.