Blocking Deploys on Segment Size Regression

A partitioning change lands, every ingestion task reports SUCCESS, and the deploy is marked green — yet the datasource has quietly started emitting 90 MB segments where it used to emit 550 MB ones, and three weeks later the Coordinator run time has doubled and Historical heaps are thrashing. Task success is the wrong signal for segment shape: a spec can succeed and still produce a distribution that regresses the Druid cluster. This guide builds the check that catches it at deploy time — read the post-ingest size distribution back from the Coordinator, compare it against the target band, and exit non-zero when too many segments fall outside it. It is the third stage of the end-to-end ingestion CI/CD gate, isolated here as a standalone check you can run on its own against any datasource.

Failure Modes & Diagnostics #

A size regression presents in two opposite directions, and both are invisible to task-level success checks. Undersizing — a swarm of sub-300 MB segments — inflates the metadata store, multiplies Coordinator timeline entries, and fragments the columnar storage layout; oversizing — segments past 750 MB drifting toward gigabytes — pushes large column vectors onto the Historical JVM heap and triggers stop-the-world GC on load. The full memory model behind the band is documented under optimizing segment size for Historical nodes. Start by reading the actual distribution the deploy produced, straight from the Coordinator:

# Full size distribution in MB per segment for a datasource
curl -s "http://<coordinator-host>:8081/druid/coordinator/v1/datasources/<datasource>/segments?full=true" \
  | jq '[.[] | (.size / 1048576 | floor)] | sort'

# Bucketed counts: undersized / in-band / oversized
curl -s "http://<coordinator-host>:8081/druid/coordinator/v1/datasources/<datasource>/segments?full=true" \
  | jq '[.[] | (.size/1048576)] as $m
        | {under: [$m[] | select(. < 300)] | length,
           inband: [$m[] | select(. >= 300 and . <= 750)] | length,
           over:  [$m[] | select(. > 750)] | length}'

# Count segments per interval — a sudden spike is the earliest fragmentation signal
curl -s "http://<coordinator-host>:8081/druid/coordinator/v1/datasources/<datasource>/segments?full=true" \
  | jq 'group_by(.interval)[] | {interval: .[0].interval, segments: length}'

An under count that dominates confirms targetRowsPerSegment is set too low or the rollup ratio used to size it was wrong, so each segment holds too few rows. A nonzero over count with a skewed distribution means the partitionDimensions choice concentrates rows into a handful of hot partitions. A segment-per-interval count that jumped versus the prior deploy is fragmentation from an over-fine segmentGranularity or a dynamic partitionsSpec producing many small segments; the temporal driver is the segment granularity settings. Capture the pre-change distribution as a baseline so the gate compares against a known-good shape, not just a static band.

Target Spec & Validated JSON #

The gate needs two things: a spec whose partitioning is sized to hit the band, and an explicit threshold policy the check enforces. The index_parallel spec below uses range partitioning with a targetRowsPerSegment derived from the sizing formula, and includes every required top-level key so it is copy-ready against the latest stable Druid release:

{
  "type": "index_parallel",
  "spec": {
    "dataSchema": {
      "dataSource": "events_web",
      "timestampSpec": { "column": "event_ts", "format": "iso" },
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "HOUR",
        "queryGranularity": "MINUTE",
        "rollup": true,
        "intervals": ["2026-03-03T00:00:00Z/2026-03-04T00:00:00Z"]
      }
    },
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": { "type": "s3", "prefixes": ["s3://lake/events/web/2026-03-03/"] },
      "inputFormat": { "type": "json" },
      "dropExisting": true
    },
    "tuningConfig": {
      "type": "index_parallel",
      "maxRowsInMemory": 1000000,
      "forceGuaranteedRollup": true,
      "partitionsSpec": {
        "type": "range",
        "partitionDimensions": ["country"],
        "targetRowsPerSegment": 5000000
      }
    }
  }
}

The row target follows from the standard sizing relation, where avgRowBytes is measured from real segment telemetry rather than guessed:

$$ \text{targetRows} \approx \frac{\text{targetMB} \times 1048576}{\text{avgRowBytes}} $$

For a 600 MB target over rows averaging $\approx 120$ bytes, $\text{targetRows} \approx 5.24 \times 10^{6}$. The gate's pass condition is that the out-of-band fraction stays within tolerance $\tau$:

$$ \frac{\lvert { s : \text{size}(s) < 300,\text{MB} \ \lor\ \text{size}(s) > 750,\text{MB} } \rvert}{N} \le \tau $$

with $\tau \approx 0.10$. Encode the band and tolerance as an explicit policy the check reads, so tightening the gate is a config change, not a code edit.

Python Automation Script #

The gate below reads the produced distribution from the Coordinator with capped exponential backoff, buckets it against the band, and exits non-zero when the out-of-band fraction exceeds tolerance. It uses only the standard library plus requests and is written to drop directly into a CI step: its process exit code is the deploy decision.

import sys
import time
import requests

COORDINATOR = "http://coordinator:8081"
SIZE_MIN_MB, SIZE_MAX_MB = 300.0, 750.0
OUT_OF_BAND_TOLERANCE = 0.10


def fetch_segment_sizes(datasource: str, retries: int = 5) -> list[float]:
    """Return per-segment sizes in MB, retrying transient failures with backoff."""
    url = (f"{COORDINATOR}/druid/coordinator/v1/datasources/"
           f"{datasource}/segments?full=true")
    delay = 2.0
    for attempt in range(retries):
        try:
            r = requests.get(url, timeout=30)
            r.raise_for_status()
            return [s.get("size", 0) / 1048576 for s in r.json()]
        except requests.RequestException:
            if attempt == retries - 1:
                raise
            time.sleep(delay)
            delay = min(delay * 2, 30.0)  # capped backoff
    raise RuntimeError("unreachable")


def evaluate(sizes: list[float]) -> dict:
    """Bucket sizes against the band and compute the out-of-band fraction."""
    if not sizes:
        raise ValueError("no segments returned; cannot assess size regression")
    under = [m for m in sizes if m < SIZE_MIN_MB]
    over = [m for m in sizes if m > SIZE_MAX_MB]
    out_fraction = (len(under) + len(over)) / len(sizes)
    return {
        "count": len(sizes),
        "under": len(under),
        "over": len(over),
        "out_fraction": round(out_fraction, 4),
        "min_mb": round(min(sizes)),
        "max_mb": round(max(sizes)),
    }


def main() -> int:
    datasource = sys.argv[1]
    report = evaluate(fetch_segment_sizes(datasource))
    print(f"segment size report for {datasource}: {report}")
    if report["out_fraction"] > OUT_OF_BAND_TOLERANCE:
        print(
            f"DEPLOY BLOCKED: {report['out_fraction']:.1%} of segments outside "
            f"{int(SIZE_MIN_MB)}-{int(SIZE_MAX_MB)} MB "
            f"(tolerance {OUT_OF_BAND_TOLERANCE:.0%})",
            file=sys.stderr,
        )
        return 1
    print("DEPLOY OK: segment size distribution within tolerance")
    return 0


if __name__ == "__main__":
    sys.exit(main())

The check is intentionally read-only: it never mutates the Druid cluster, so it is safe to run repeatedly and safe to run in a dry-run stage before the change is promoted. When it blocks, the fix is a partitioning change — adjust targetRowsPerSegment, correct the rollup-ratio estimate, or switch partitionDimensions — followed by a re-ingest, or a compaction pass via automated compaction task scheduling to re-write the already-published interval into the band without re-reading the source.

Verification Steps #

After adjusting partitioning and re-ingesting (or compacting), confirm the distribution now clears the gate. First re-read the spread:

curl -s "http://<coordinator-host>:8081/druid/coordinator/v1/datasources/events_web/segments?full=true" \
  | jq '[.[] | (.size / 1048576 | floor)] | {min: min, max: max, count: length}'

Expected output shows the spread contained within the band:

{
  "min": 486,
  "max": 712,
  "count": 24
}

Then run the gate itself and confirm a zero exit code:

python3 size_gate.py events_web
echo "exit=$?"
# segment size report for events_web: {'count': 24, 'under': 0, 'over': 0, ...}
# DEPLOY OK: segment size distribution within tolerance
# exit=0

If older oversized segments for the interval are still being served alongside the new ones, mark them unused so the Coordinator stops loading them and re-evaluates its load queue:

curl -X POST "http://<coordinator-host>:8081/druid/coordinator/v1/datasources/events_web/markUnused" \
  -H "Content-Type: application/json" \
  -d '{"interval": "2026-03-03T00:00:00Z/2026-03-04T00:00:00Z"}'

Wire the gate as a required CI step so no deploy that regresses segment shape can be promoted, and export the report dict as a build artifact so drift is trended over time rather than judged one deploy at a time. Reference the Druid compaction documentation for the targetRowsPerSegment overrides that bring a drifted interval back into band.

Related #

• End-to-end ingestion CI/CD gate — parent guide: the full four-stage deploy gate this size check is the third stage of.
• Optimizing segment size for Historical nodes — the heap and GC model that makes the 300–750 MB band the right boundary to enforce.
• Automated compaction task scheduling — how to re-write an already-published interval back into the band when a re-ingest is not warranted.

Up one level: End-to-end ingestion CI/CD gate chains this size check with schema validation, handoff confirmation, and monitoring into one deploy decision.