JSON Schema Validation for Druid Ingestion Specs

An assembled ingestion spec that is missing tuningConfig, carries a segmentGranularity typo, or pairs forceGuaranteedRollup with a dynamic partitionsSpec will be rejected by the Overlord — but only after it has consumed an API round-trip, occupied a task slot, and produced a failure report that a human then has to read. A JSON Schema check moves that rejection left, into CI, where it costs a millisecond of local validation and returns a precise path to the offending field. This guide builds that gate: a jsonschema contract that an assembled spec must satisfy before any call to POST /druid/indexer/v1/task. It is the structural half of schema validation for Druid specs — the machine-checkable shape of the spec, ahead of the Druid-specific semantic rules that layer on top.

Failure Modes & Diagnostics #

The failures a JSON Schema catches are structural — the ones that make a spec malformed regardless of cluster state. The three that dominate are a missing required key, a wrong-typed value, and an unknown key introduced by a typo. Before writing the schema, inspect the assembled spec's shape so the contract you write matches reality:

# Confirm the required top-level and spec-level keys are present
jq '{top: keys, spec_keys: (.spec | keys),
     schema_keys: (.spec.dataSchema | keys)}' spec.json

# Surface a segmentGranularity typo (must be one of the Druid period names)
jq '.spec.dataSchema.granularitySpec.segmentGranularity' spec.json

# Check the rollup/partitioning pairing the schema will also enforce
jq '{forceGR: .spec.tuningConfig.forceGuaranteedRollup,
     partType: .spec.tuningConfig.partitionsSpec.type}' spec.json

A missing spec.tuningConfig, a segmentGranularity of "HOURLY" instead of "HOUR", or a metricsSpec that is an object rather than an array are all structural defects a schema rejects instantly. What a JSON Schema deliberately does not catch are semantic contradictions that are individually well-formed — a dropExisting: true without bounded intervals, or an intervals window that overlaps a live supervisor. Those belong to the semantic layer in the parent guide and, for the overlap case, to preventing batch/streaming segment overlap. Keep the two layers distinct: the schema owns shape, the semantic checks own intent. Handling a genuinely new source column so it does not silently become the wrong dimension is the domain of handling schema evolution in Druid ingestion.

Target Spec & Validated JSON #

The contract below is a JSON Schema (Draft 2020-12) that pins the structural shape of an index_parallel spec: required keys at every level, enumerated segmentGranularity values, metricsSpec as an array, and a partitionsSpec discriminated by type. It is intentionally strict — additionalProperties: false on the objects it fully owns so a typo'd key fails loudly rather than being silently ignored by Druid.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Druid index_parallel ingestion spec",
  "type": "object",
  "required": ["type", "spec"],
  "properties": {
    "type": { "const": "index_parallel" },
    "spec": {
      "type": "object",
      "required": ["dataSchema", "ioConfig", "tuningConfig"],
      "properties": {
        "dataSchema": {
          "type": "object",
          "required": ["dataSource", "timestampSpec", "granularitySpec"],
          "properties": {
            "dataSource": { "type": "string", "minLength": 1 },
            "timestampSpec": {
              "type": "object",
              "required": ["column"],
              "properties": {
                "column": { "type": "string", "minLength": 1 },
                "format": { "type": "string" }
              }
            },
            "dimensionsSpec": { "type": "object" },
            "metricsSpec": { "type": "array" },
            "granularitySpec": {
              "type": "object",
              "required": ["segmentGranularity"],
              "properties": {
                "type": { "type": "string" },
                "segmentGranularity": {
                  "enum": ["MINUTE", "FIVE_MINUTE", "HOUR", "DAY",
                           "WEEK", "MONTH", "YEAR"]
                },
                "queryGranularity": { "type": "string" },
                "rollup": { "type": "boolean" },
                "intervals": {
                  "type": "array",
                  "items": { "type": "string" }
                }
              }
            }
          }
        },
        "ioConfig": {
          "type": "object",
          "required": ["type", "inputSource", "inputFormat"],
          "properties": {
            "type": { "const": "index_parallel" },
            "dropExisting": { "type": "boolean" },
            "appendToExisting": { "type": "boolean" }
          }
        },
        "tuningConfig": {
          "type": "object",
          "required": ["type", "partitionsSpec"],
          "properties": {
            "type": { "const": "index_parallel" },
            "forceGuaranteedRollup": { "type": "boolean" },
            "maxRowsInMemory": { "type": "integer", "minimum": 1 },
            "partitionsSpec": {
              "type": "object",
              "required": ["type"],
              "properties": {
                "type": { "enum": ["dynamic", "hashed", "range"] },
                "targetRowsPerSegment": { "type": "integer", "minimum": 1 },
                "maxRowsPerSegment": { "type": "integer", "minimum": 1 }
              }
            }
          }
        }
      }
    }
  }
}

The schema encodes exactly the structural guarantees the Overlord depends on: the presence of the three spec children, a non-empty dataSource, a segmentGranularity drawn from Druid's period vocabulary, and a partitionsSpec.type from the three legal partitioning strategies. Field semantics beyond shape are maintained in the official Apache Druid ingestion spec reference; the schema's job is only to guarantee the JSON is well-formed against that reference before the Druid cluster ever sees it.

Python Automation Script #

The gate below validates an assembled spec against the schema using the jsonschema library, and only on success submits it to the Overlord with capped exponential backoff. jsonschema is a widely-used open-source validator; it is the one external dependency here beyond requests. Validation failures raise ValidationError carrying a JSON path (json_path) that points straight at the offending field, which the gate surfaces and then exits non-zero without ever calling the Druid cluster.

import sys
import time
import json
import requests
from jsonschema import Draft202012Validator

OVERLORD = "http://overlord:8090"


def load_validator(schema_path: str) -> Draft202012Validator:
    with open(schema_path) as fh:
        schema = json.load(fh)
    Draft202012Validator.check_schema(schema)  # fail fast on a broken schema
    return Draft202012Validator(schema)


def validate_spec(validator: Draft202012Validator, spec: dict) -> list[str]:
    """Return a sorted list of human-readable errors; empty means valid."""
    errors = sorted(validator.iter_errors(spec), key=lambda e: e.json_path)
    return [f"{e.json_path}: {e.message}" for e in errors]


def submit_with_backoff(spec: dict, retries: int = 5) -> str:
    """POST the validated spec to the Overlord, retrying transient failures."""
    delay = 2.0
    for attempt in range(retries):
        try:
            r = requests.post(f"{OVERLORD}/druid/indexer/v1/task",
                              json=spec, timeout=30)
            r.raise_for_status()
            return r.json()["task"]
        except requests.RequestException:
            if attempt == retries - 1:
                raise
            time.sleep(delay)
            delay = min(delay * 2, 30.0)  # capped backoff
    raise RuntimeError("unreachable")


def main() -> int:
    schema_path, spec_path = sys.argv[1], sys.argv[2]
    validator = load_validator(schema_path)
    with open(spec_path) as fh:
        spec = json.load(fh)

    errors = validate_spec(validator, spec)
    if errors:
        print("SPEC REJECTED (not submitted):", file=sys.stderr)
        for err in errors:
            print(f"  {err}", file=sys.stderr)
        return 1

    task_id = submit_with_backoff(spec)
    print(f"SPEC VALID: submitted {task_id}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Two choices make this gate reliable. check_schema runs first so a mistake in the contract itself fails immediately rather than silently passing every spec. And iter_errors collects all violations in one pass — sorted by json_path — rather than stopping at the first, so an author fixes an entire spec in one iteration instead of one field per CI run. Submission is reached only when the error list is empty, guaranteeing no malformed spec consumes an Overlord slot.

Verification Steps #

Confirm the gate rejects a malformed spec and accepts a valid one. First, feed it a spec with tuningConfig removed and check for a non-zero exit with a precise path:

jq 'del(.spec.tuningConfig)' spec.json > broken.json
python3 spec_gate.py druid-index-schema.json broken.json
echo "exit=$?"
# SPEC REJECTED (not submitted):
#   $.spec: 'tuningConfig' is a required property
# exit=1

Then confirm a well-formed spec validates and is submitted:

python3 spec_gate.py druid-index-schema.json spec.json
echo "exit=$?"
# SPEC VALID: submitted index_parallel_events_web_2025-09-22T...
# exit=0

As a cross-check, confirm the schema itself is valid before trusting any result it produces:

python3 -c "import json; from jsonschema import Draft202012Validator; \
Draft202012Validator.check_schema(json.load(open('druid-index-schema.json'))); \
print('schema OK')"
# schema OK

Wire the gate as a required CI step ahead of every ingestion submit, keep the schema in version control beside the specs it guards, and treat a schema change as a reviewed change like any other. Pair it with the semantic checks in the parent guide so structure and intent are both enforced before the Overlord is called.

Related #

• Schema validation for Druid specs — parent guide: the full pre-flight validation contract, combining this structural schema with Druid-specific semantic rules.
• Handling schema evolution in Druid ingestion — how new source columns are absorbed safely rather than silently misclassified by the schema.
• Dynamic ingestion spec generation — the runtime builder whose output this schema gates before submission.

Up one level: Schema validation for Druid specs frames how structural JSON Schema checks and semantic rules compose into one pre-flight gate.