Druid Segment Ops Segment & Ingestion Orchestration

Reducing Historical Node Storage Costs: Segment Management & Orchestration Patterns

Historical node storage costs in Apache Druid scale non-linearly when segment proliferation, misaligned rollup thresholds, and desynchronized retention policies compound over time. Effective cost reduction requires deterministic compaction, strict TTL enforcement, and automated pipeline orchestration. This reference details operational patterns for segment lifecycle management, focusing on failure modes, Python-driven orchestration, CI/CD validation, and idempotent recovery.

Failure Modes & Diagnostics

Storage bloat typically manifests through three deterministic failure modes that directly impact deep storage I/O and coordinator memory footprints.

  1. Segment Sprawl & Metadata Overhead: Ingestion pipelines writing to deep storage without partition alignment generate sub-50MB segments. This inflates coordinator metadata caches and degrades query planner efficiency. Diagnose via GET /druid/coordinator/v1/metadata/datasources/{ds}/segments?full=true and filter for size < 52428800. Sprawl directly correlates with increased S3/HDFS LIST operations and historical node memory pressure. Cross-reference with a sys.segments aggregation (SELECT datasource, COUNT(*) FROM sys.segments GROUP BY datasource HAVING COUNT(*) > 10000) to identify datasources exceeding 10,000 segments per day.
  2. Compaction Lock Contention: Overlapping compaction intervals or concurrent compact tasks targeting identical partition boundaries trigger SegmentLock timeouts. Monitor GET /druid/indexer/v1/tasks for RUNNING compaction tasks sharing the same ioConfig.interval. Unresolved contention leaves intermediate segments stranded in pending state, consuming deep storage without serving queries. Implement interval hashing or coordinator-level task queuing to serialize writes.
  3. Retention Desync: Coordinator TTL rules applied post-ingestion cause temporary storage spikes until the next duty cycle. Misalignment between ingestion queryGranularity and coordinator dropByPeriod boundaries results in orphaned segments that bypass expiration. Validate with GET /druid/coordinator/v1/rules/{ds} and cross-reference against metadata.maxTime distributions. For granular control over partition sizing and lifecycle mapping, consult Segment Size Optimization Strategies.

Compaction Threshold Tuning & Validated Specs

Compaction must be idempotent, bounded, and aligned with underlying storage block sizes. Target segment sizes should match cloud object storage optimal read throughput (typically 256MB–512MB). Over-compaction increases CPU burn on historical nodes, while under-compaction leaves storage costs unchecked.

Deploy the following validated compact task specification to enforce deterministic rollup and size boundaries:

{
  "type": "compact",
  "dataSource": "events_stream",
  "ioConfig": {
    "type": "compact",
    "inputSpec": {
      "type": "interval",
      "interval": "2024-01-01T00:00:00.000Z/2024-01-02T00:00:00.000Z"
    }
  },
  "granularitySpec": { "segmentGranularity": "DAY" },
  "tuningConfig": {
    "type": "index_parallel",
    "targetCompactionSizeBytes": 524288000,
    "partitionsSpec": {
      "type": "dynamic",
      "maxRowsPerSegment": 10000000,
      "maxTotalRows": 20000000
    }
  }
}

Key tuning parameters:

  • targetCompactionSizeBytes: Aligns with S3 multipart upload thresholds. Values below 256MB trigger excessive HTTP overhead; values above 1GB risk historical node heap fragmentation during segment load.
  • maxRowsPerSegment: Prevents memory spikes during index merging. Keep this ≤ 15M for typical OLAP workloads.
  • segmentGranularity: Must match ingestion granularitySpec to avoid cross-day boundary splits that bypass compaction windows.

Python-Driven Orchestration & Idempotent Execution

Automated compaction requires stateful orchestration to prevent duplicate task submission and handle transient coordinator failures. The following Python implementation uses the Druid Overlord API to submit, poll, and verify compaction tasks with exponential backoff and idempotency guards.

import requests
import time
import hashlib
from typing import Optional

class DruidCompactionOrchestrator:
    def __init__(self, overlord_url: str, timeout: int = 30):
        self.overlord_url = overlord_url.rstrip("/")
        self.timeout = timeout
        self.session = requests.Session()
        self.session.headers.update({"Content-Type": "application/json"})

    def _generate_task_id(self, datasource: str, interval: str) -> str:
        """Deterministic ID generation for idempotent retries."""
        payload = f"{datasource}_{interval}"
        return f"compact_{hashlib.md5(payload.encode()).hexdigest()[:12]}"

    def submit_compaction(self, spec: dict) -> str:
        task_id = self._generate_task_id(
            spec["dataSource"],
            spec["ioConfig"]["inputSpec"]["interval"]
        )
        spec["id"] = task_id

        resp = self.session.post(
            f"{self.overlord_url}/druid/indexer/v1/task",
            json=spec,
            timeout=self.timeout
        )
        resp.raise_for_status()
        return task_id

    def wait_for_completion(self, task_id: str, poll_interval: int = 15) -> str:
        """Blocks until task reaches terminal state. Returns final status."""
        url = f"{self.overlord_url}/druid/indexer/v1/task/{task_id}/status"
        while True:
            resp = self.session.get(url, timeout=self.timeout)
            resp.raise_for_status()
            status = resp.json()["status"]["status"]
            if status in ("SUCCESS", "FAILED"):
                return status
            time.sleep(poll_interval)

This pattern ensures pipeline resilience by deriving task IDs from deterministic hashes of the datasource and interval. If a network partition occurs mid-submission, re-execution will either attach to the existing task or return a 409 Conflict, both of which are safely handled upstream. For comprehensive guidance on integrating this into production workflows, review Segment Compaction, Retention & Storage Optimization.

CI/CD Validation & Recovery Patterns

Storage cost controls must be validated before deployment. Integrate schema and compaction spec validation into your CI pipeline using JSON Schema validation against Druid's task specification contract. Enforce strict linting rules:

  • Reject targetCompactionSizeBytes outside [134217728, 1073741824]
  • Require segmentGranularity to match queryGranularity
  • Validate interval boundaries against ISO-8601 standards

Idempotent Recovery from Stranded Segments

When compaction fails mid-flight, intermediate segments may remain in deep_storage but unregistered in the metadata store. Execute the following recovery sequence:

  1. Identify Orphaned Intervals: Query sys.segments for is_published = false and is_realtime = false. Cross-reference with deep storage prefixes using cloud provider CLI tools (aws s3 ls s3://druid-deep-storage/...).
  2. Re-enable Dropped Segments: If segments were previously marked unused, POST /druid/coordinator/v1/datasources/{ds}/markUsed flips them back to used so the Coordinator reloads them. This only updates metadata state — it does not recover never-published intermediate segments.
  3. Cleanup Unreferenced Data: DELETE /druid/coordinator/v1/datasources/{ds}/segments/{segmentId} only marks a segment unused in the metadata store; it does not remove data from deep storage. To physically reclaim deep storage, submit a kill task to the Overlord for the unused interval. Always verify against sys.segments before issuing either call to prevent query failures.

Implement automated TTL mapping by aligning coordinator dropByPeriod rules with business data retention SLAs. Misaligned retention windows cause coordinator duty cycles to repeatedly scan expired intervals, increasing CPU overhead without reclaiming storage. Reference the official Python Requests Documentation for advanced session management and retry strategies when integrating with Druid's REST APIs.

Storage optimization in Druid is not a one-time configuration but a continuous feedback loop between ingestion throughput, compaction cadence, and retention enforcement. By enforcing deterministic task IDs, validating compaction thresholds at the CI layer, and automating orphaned segment recovery, platform teams can maintain predictable historical node costs while preserving query performance.

Back to Apache Druid Segment Lifecycle