Druid Segment Ops Segment & Ingestion Orchestration

LZ4 vs ZSTD Codec Selection by Storage Tier

The question surfaces the first time a cold-tier storage bill outruns the value of the data sitting on it, or the first time a hot-tier Historical burns query CPU decompressing archival segments that should never have been LZ4. Apache Druid compresses every column block after encoding, and the codec is not a global constant — it is an indexSpec field you set per datasource and, more importantly, per storage tier the segment will live on. LZ4 decompresses almost for free and suits the hot tier; ZSTD packs markedly denser at a real CPU cost and suits cold and archival tiers. This page applies the encoding theory from its parent, columnar storage formats in Druid, to the concrete decision of which codec belongs on which tier and how to switch a segment from one to the other without re-ingesting.

LZ4 versus ZSTD placed on a scan-CPU against storage-cost trade-off plane by tier A trade-off plane with decompression CPU on the horizontal axis rising left to right and compressed storage footprint on the vertical axis rising bottom to top. LZ4 sits lower-left region flagged for the hot and warm tier: near-free decompression during scans but a larger on-disk footprint. ZSTD sits upper-right region flagged for the cold and archival tier: a much smaller footprint but higher decompression CPU on the occasional scan. An arrow labelled re-compaction connects LZ4 to ZSTD, showing that a segment moves between codecs by re-encoding, not in place. A guidance band across the middle reads: match the codec to how often the tier is scanned. DECOMPRESSION CPU PER SCAN —> STORAGE FOOTPRINT —> LZ4 near-free decompression larger on-disk footprint hot / warm tier ZSTD denser: ~20–40% smaller higher scan CPU cold / archival tier re-compaction switches codec (re-encode, not in place)

Failure Modes & Diagnostics

Codec mismatch never announces itself as "codec" — it reads as an unexpected storage bill or as hot-tier query CPU that will not come down. Two directions dominate: LZ4 left on segments that have aged onto a cold tier (paying storage for speed nobody uses), and ZSTD applied to hot-tier segments (paying scan CPU for density nobody needs). Isolate which one you have before re-compacting anything:

# What codec is actually recorded on published segments for an interval?
curl -s -X POST "http://<coordinator-host>:8081/druid/coordinator/v1/metadata/datasources/events_columnar/segments?full" \
  -H 'Content-Type: application/json' \
  -d '["2026-06-01T00:00:00.000Z/2026-07-01T00:00:00.000Z"]' \
  | jq '.[] | {interval: .interval, dimComp: .indexSpec.dimensionCompression, metComp: .indexSpec.metricCompression, longEnc: .indexSpec.longEncoding, size_mb: (.size/1048576|floor)}'

# Which tier are those segments loaded on right now?
curl -s "http://<coordinator-host>:8081/druid/coordinator/v1/tiers" | jq
curl -s "http://<coordinator-host>:8081/druid/coordinator/v1/servers?full" \
  | jq '.[] | {host: .host, tier: .tier, currSize_gb: (.currSize/1073741824|floor)}'

A cold-tier interval whose dimComp/metComp come back as lz4 is the classic overspend: ZSTD would reclaim 20–40% of that footprint. Conversely, if a hot-tier Historical shows sustained CPU during scans, confirm the codec is the culprit rather than the encoding itself:

# Historical CPU while a scan-heavy query runs — ZSTD decompression shows here
top -b -n 3 -d 1 -p <historical_pid> | grep <historical_pid>

# Segment/scan timing from the Broker's query metrics log (query/segment/time)
curl -s "http://<broker-host>:8082/druid/v2/sql" -H 'Content-Type: application/json' \
  -d '{"query":"SELECT COUNT(*) FROM events_columnar WHERE country = '\''DE'\''","context":{"enableMetrics":true}}'

If decompression CPU tracks the presence of ZSTD segments on the hot tier, the codec is misplaced for how often that tier is scanned. The rule the diagnostics keep confirming: match the codec to scan frequency, not to a global default.

Target Spec & Validated JSON

The codec lives in indexSpec inside tuningConfig, set independently for dimensions and metrics via dimensionCompression and metricCompression, and interacts with longEncoding for numeric columns. For a hot-tier datasource, keep LZ4 so scans stay cheap. The batch (index_parallel) spec below is copy-ready against a recent stable Druid release:

{
  "type": "index_parallel",
  "spec": {
    "dataSchema": {
      "dataSource": "events_columnar",
      "timestampSpec": { "column": "ts", "format": "iso" },
      "dimensionsSpec": {
        "dimensions": [
          { "type": "string", "name": "country", "createBitmapIndex": true },
          { "type": "long", "name": "status_code" }
        ]
      },
      "metricsSpec": [
        { "type": "count", "name": "events" },
        { "type": "longSum", "name": "bytes_sent", "fieldName": "bytes" }
      ],
      "granularitySpec": { "type": "uniform", "segmentGranularity": "DAY", "rollup": true }
    },
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": { "type": "s3", "prefixes": ["s3://analytics-bucket/raw/"] },
      "inputFormat": { "type": "json" }
    },
    "tuningConfig": {
      "type": "index_parallel",
      "maxRowsInMemory": 1000000,
      "partitionsSpec": { "type": "hashed", "targetRowsPerSegment": 5000000 },
      "indexSpec": {
        "bitmap": { "type": "roaring" },
        "dimensionCompression": "lz4",
        "metricCompression": "lz4",
        "longEncoding": "auto"
      }
    }
  }
}

The indexSpec fields that decide the trade-off:

  • dimensionCompression — compresses dictionary-encoded string column blocks. lz4 (default) for hot/warm where filters and GROUP BY scan constantly; zstd for cold/archival where a smaller footprint outweighs occasional scan cost. uncompressed exists but is almost never correct.
  • metricCompression — compresses numeric metric column blocks. Same tier logic. none is available for tiny metric sets but rarely pays off.
  • longEncoding — auto picks table/delta packing per long column before the codec runs, so ZSTD compounds on top of already-compact numerics; use longs (fixed 8-byte) only when measured to win.

To move an aged interval from LZ4 to ZSTD you do not re-ingest — you re-compact with a codec override. A compaction task carries its own tuningConfig.indexSpec, so the same rows are re-encoded under the denser codec:

{
  "type": "compact",
  "dataSource": "events_columnar",
  "ioConfig": {
    "type": "compact",
    "inputSpec": { "type": "interval", "interval": "2026-06-01/2026-07-01" }
  },
  "tuningConfig": {
    "type": "index_parallel",
    "partitionsSpec": { "type": "hashed", "targetRowsPerSegment": 5000000 },
    "indexSpec": {
      "bitmap": { "type": "roaring" },
      "dimensionCompression": "zstd",
      "metricCompression": "zstd",
      "longEncoding": "auto"
    }
  }
}

Aligning codec with tier is the same lifecycle event as moving the data itself; the retention rules that shift segments between tiers are covered under configuring segment retention policies, and the re-encoding is scheduled through automated compaction task scheduling.

Python Automation Script

Codec placement only stays correct if a pipeline re-compacts intervals to ZSTD as they age onto the cold tier. The orchestrator below submits a ZSTD re-compaction for a given interval, applies capped exponential backoff for transient Overlord errors, polls to a terminal state, and reports the storage reclaimed. It uses only the standard library plus requests.

import time
import logging
import requests

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("druid_codec")


class DruidCodecMigrator:
    def __init__(self, overlord_url, coordinator_url, datasource, auth=None):
        self.overlord = overlord_url.rstrip("/")
        self.coordinator = coordinator_url.rstrip("/")
        self.datasource = datasource
        self.session = requests.Session()
        self.session.auth = auth

    def _interval_bytes(self, interval):
        resp = self.session.post(
            f"{self.coordinator}/druid/coordinator/v1/metadata/datasources/"
            f"{self.datasource}/segments?full",
            json=[interval], timeout=30,
        )
        resp.raise_for_status()
        return sum(s["size"] for s in resp.json())

    def recompact_to_zstd(self, interval, target_rows=5_000_000):
        payload = {
            "type": "compact",
            "dataSource": self.datasource,
            "ioConfig": {
                "type": "compact",
                "inputSpec": {"type": "interval", "interval": interval},
            },
            "tuningConfig": {
                "type": "index_parallel",
                "partitionsSpec": {"type": "hashed", "targetRowsPerSegment": target_rows},
                "indexSpec": {
                    "bitmap": {"type": "roaring"},
                    "dimensionCompression": "zstd",
                    "metricCompression": "zstd",
                    "longEncoding": "auto",
                },
            },
        }
        resp = self.session.post(
            f"{self.overlord}/druid/indexer/v1/task", json=payload, timeout=30
        )
        resp.raise_for_status()
        task_id = resp.json()["task"]
        logger.info("Submitted ZSTD re-compaction %s for %s", task_id, interval)
        return task_id

    def poll_until_terminal(self, task_id, base_delay=5, max_delay=60, max_wait=3600):
        start = time.time()
        delay = base_delay
        while time.time() - start < max_wait:
            resp = self.session.get(
                f"{self.overlord}/druid/indexer/v1/task/{task_id}/status", timeout=10
            )
            resp.raise_for_status()
            status = resp.json().get("status", {}).get("status")
            if status in ("SUCCESS", "FAILED", "INTERRUPTED"):
                logger.info("Task %s terminal state: %s", task_id, status)
                return status
            logger.debug("Task %s pending (%s); sleeping %ss", task_id, status, delay)
            time.sleep(delay)
            delay = min(delay * 2, max_delay)  # exponential backoff, capped
        raise TimeoutError(f"Task {task_id} exceeded {max_wait}s")

    def migrate(self, interval):
        before = self._interval_bytes(interval)
        if self.poll_until_terminal(self.recompact_to_zstd(interval)) != "SUCCESS":
            raise SystemExit("ZSTD re-compaction failed; codec unchanged")
        after = self._interval_bytes(interval)
        saved_pct = 100 * (before - after) / before if before else 0
        logger.info("Interval %s: %.1f MB -> %.1f MB (%.1f%% reclaimed)",
                    interval, before / 1048576, after / 1048576, saved_pct)
        return saved_pct


# Usage
# m = DruidCodecMigrator("http://overlord:8090", "http://coordinator:8081", "events_columnar")
# m.migrate("2026-06-01/2026-07-01")

Because a non-SUCCESS state raises loudly, a codec migration that fails never leaves the interval half-converted. For templating this across many datasources and environments, see dynamic ingestion spec generation.

Verification Steps

Confirm the switch after re-compaction completes. First re-read the recorded indexSpec for the interval — every published segment should now report zstd:

curl -s -X POST "http://<coordinator-host>:8081/druid/coordinator/v1/metadata/datasources/events_columnar/segments?full" \
  -H 'Content-Type: application/json' \
  -d '["2026-06-01T00:00:00.000Z/2026-07-01T00:00:00.000Z"]' \
  | jq '[.[] | {dimComp: .indexSpec.dimensionCompression, metComp: .indexSpec.metricCompression}] | unique'

Expected output shows a single, uniform ZSTD entry — no stragglers left on LZ4:

[
  {
    "dimComp": "zstd",
    "metComp": "zstd"
  }
]

Next confirm the footprint actually shrank and that segments stayed inside the target band after re-encoding:

curl -s "http://<coordinator-host>:8081/druid/coordinator/v1/datasources/events_columnar/segments?full=true" \
  | jq '[.[] | (.size / 1048576)] | {count: length, avg_mb: (add/length|floor), max_mb: (max|floor)}'

Finally, spot-check that a representative query against the re-compacted interval still returns identical row counts and that hot-tier queries were untouched — ZSTD belongs only on the cold interval you migrated. If a cold-tier scan now costs more CPU than the storage saving justifies, the interval is scanned too often to be ZSTD; move it back with an LZ4 re-compaction using the same override. Reference the Druid ingestion spec reference for the full indexSpec field set and the compaction documentation for override semantics.

Back to Apache Druid Segment Architecture & Lifecycle Fundamentals