Configuring Druid Native Compaction Rules

Apache Druid native compaction operates as a deterministic, background reconciliation process that merges fragmented segments, enforces strict row limits, and aligns physical storage footprints with downstream query patterns. When misconfigured, compaction rules silently degrade query latency, inflate deep storage egress costs, and trigger coordinator heap exhaustion. This reference outlines production-grade specification patterns, Python-based orchestration workflows, and diagnostic signatures for OLAP data engineers and platform developers managing high-throughput ingestion pipelines.

Deterministic Task Specification #

Compaction tasks execute as compact type jobs routed through the Druid Overlord. The specification must explicitly define granularity boundaries, size thresholds, and temporal offsets to prevent ingestion overlap and lock contention. The following JSON enforces strict rollup, caps segment size, and isolates recent data from active ingestion streams:

{
  "type": "compact",
  "dataSource": "analytics_events",
  "granularitySpec": {
    "type": "uniform",
    "segmentGranularity": "DAY",
    "queryGranularity": "HOUR",
    "rollup": true
  },
  "tuningConfig": {
    "type": "index_parallel",
    "maxRowsPerSegment": 5000000,
    "targetCompactionSizeBytes": 536870912,
    "maxNumConcurrentSubTasks": 4,
    "chatHandlerTimeout": "PT10M"
  },
  "ioConfig": {
    "type": "compact",
    "inputSpec": {
      "type": "interval",
      "interval": "2024-01-01/2024-02-01"
    }
  },
  "context": {
    "forceTimeChunkLock": true
  }
}

Key configuration constraints for production deployment:

• skipOffsetFromLatest (an auto-compaction config field, set on the Coordinator compaction config rather than in a manual task's context) must exceed your ingestion pipeline’s maximum watermark lag. Setting it below actual ingestion latency triggers TaskLock conflicts and stalls downstream queries.
• targetCompactionSizeBytes should align with your storage tier’s optimal read block size. Values between 512MB and 1GB typically balance I/O throughput and memory pressure on historical nodes.
• forceTimeChunkLock: true forces time-chunk locking, so the compaction task holds an exclusive lock over the entire time chunk rather than individual segments — preventing concurrent tasks from interleaving writes on the same interval.

For architectural alignment across distributed clusters, map compaction boundaries directly to your storage topology using established Segment Compaction, Retention & Storage Optimization frameworks.

Python Pipeline Integration #

Automating compaction rule submission requires idempotent HTTP interactions with the Druid Overlord API. Production Python orchestrators should implement exponential backoff, session reuse, and explicit timeout boundaries to prevent task queue saturation.

import requests
import time
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

DRUID_OVERLORD = "https://druid-overlord.internal:8090"
COMPACTION_ENDPOINT = f"{DRUID_OVERLORD}/druid/indexer/v1/task"

def submit_compaction_task(spec: dict, retries: int = 3) -> str:
    session = requests.Session()
    retry_strategy = Retry(
        total=retries,
        backoff_factor=1.5,
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=["POST"]
    )
    adapter = HTTPAdapter(max_retries=retry_strategy)
    session.mount("http://", adapter)
    session.mount("https://", adapter)
    
    response = session.post(
        COMPACTION_ENDPOINT,
        json=spec,
        headers={"Content-Type": "application/json"},
        timeout=(5, 30)
    )
    response.raise_for_status()
    return response.json().get("task")

# Example execution with validated spec
# task_id = submit_compaction_task(compaction_spec)

Leverage persistent connection pooling and structured retry logic as documented in Python requests Session Management to maintain stable throughput during peak ingestion windows.

Threshold Tuning & Failure Diagnostics #

Compaction failures rarely surface as explicit API errors. Instead, they manifest as silent segment bloat, historical node OOM conditions, or coordinator scheduling deadlocks. Isolate root causes using these diagnostic signatures:

When implementing recurring execution windows, decouple task submission from real-time ingestion pipelines. Reference Automated Compaction Task Scheduling for watermark-driven DAG patterns that prevent coordinator starvation.

TTL Mapping & Storage Alignment #

Compaction rules must operate in concert with data lifecycle policies. Misaligned TTL configurations cause compaction to repeatedly process segments scheduled for imminent deletion, wasting compute and deep storage I/O. Align segmentGranularity with your retention policy cadence: daily segments pair cleanly with 30–90 day TTLs, while hourly segments require aggressive compaction windows before expiration triggers.

Enforce strict interval boundaries in ioConfig to prevent historical nodes from scanning expired partitions. When integrating with cloud object storage, verify that compaction tasks respect lifecycle transition policies (e.g., S3 Intelligent-Tiering or GCS Coldline) to avoid premature storage class downgrades. For comprehensive retention mapping, consult TTL Mapping and Data Expiration to synchronize expiration hooks with compaction execution cycles.

Production compaction orchestration requires continuous validation of segment metadata, coordinator queue depth, and historical node memory profiles. Implement automated alerting on druid/compaction/task/duration and druid/coordinator/queue/size metrics to preemptively adjust targetCompactionSizeBytes and concurrency limits before query latency degrades.