Druid Segment Ops Segment & Ingestion Orchestration

Troubleshooting Stale Broker Timelines

It shows up as a query that returns yesterday's answer: a freshly ingested interval is missing, a just-dropped interval still returns rows, or results go partial for a few minutes after a segment moves between tiers. The cause is almost always the same — the Broker answers from an in-memory segment timeline that lags the authoritative state in the metadata store, because segment discovery has not yet propagated. Apache Druid's Broker builds its timeline from announcements it learns through the discovery layer, so any gap between what the Coordinator has published and what the Broker has heard produces stale or partial results. This page sits under query routing and segment discovery, which explains how the timeline is built; here we focus on diagnosing and clearing the drift when it happens.

How a Broker timeline drifts behind the authoritative metadata state after an ingest, move, or drop A vertical propagation chain from the source of truth down to the Broker. At the top the metadata store holds the authoritative used-segment set, updated the instant an ingest, move, or drop commits. The Coordinator polls the metadata store and issues load and drop instructions to Historicals. Historicals load or drop segments and announce the change through the discovery layer. The Broker consumes those announcements to rebuild its in-memory timeline. A dashed lag arrow beside the chain marks the propagation window during which the Broker timeline is stale, so a query routed in that window misses new data or still sees dropped data and returns partial results. Metadata store · authoritative used-segment set updated the instant ingest / move / drop commits Coordinator polls metadata, issues load / drop druid.coordinator.period Historicals load / drop, announce via discovery ZooKeeper or HTTP segment discovery Broker · in-memory timeline (what queries hit) rebuilt from announcements — lags the top propagation window Broker timeline stale here → partial / stale results PROPAGATION CHAIN

Failure Modes & Diagnostics

Stale-timeline symptoms all reduce to a mismatch between what the metadata store says is used and what the Broker is serving. Establish the authoritative state first, then compare it against the Broker's view. Start at the source of truth:

# Authoritative used-segment intervals from the Coordinator (backed by metadata store)
curl -s "http://<coordinator-host>:8081/druid/coordinator/v1/datasources/events_routed/intervals" | jq

# Full used-segment set with load status the Coordinator believes is current
curl -s "http://<coordinator-host>:8081/druid/coordinator/v1/datasources/events_routed?full" \
  | jq '{segments: (.segments | length), intervals: .segments | map(.interval) | unique | length}'

Now ask the Broker what it thinks it can serve, and diff the two:

# Segments the Broker currently has in its timeline (server view)
curl -s "http://<broker-host>:8082/druid/broker/v1/loadstatus" | jq

# The Broker's own datasource/segment view — compare interval set to the Coordinator's
curl -s "http://<broker-host>:8082/druid/v2/datasources/events_routed?full" \
  | jq '{dimensions: (.dimensions|length), intervals: .segments? }'

New interval missing after ingest. Symptom: a SUCCESS ingestion task, rows visible via the Coordinator's interval list, but queries return nothing for the new interval. Root cause is the Broker timeline not yet including the handed-off segments. Confirm the segment is used and loaded but simply not yet announced to this Broker, then force a refresh (below). Handoff completion itself is verified under verifying segment handoff completion.

Dropped interval still returns rows. Symptom: after a markUnused or retention drop, queries still count rows in the retired interval. Root cause is the Broker holding a stale "used" entry it has not yet been told to remove:

# Is the interval still announced anywhere from the Broker's perspective?
curl -s "http://<broker-host>:8082/druid/broker/v1/loadstatus?full" \
  | jq '.events_routed // "not in broker timeline"'

Partial results after a segment move. Symptom: for a minute or two after a tier migration or rebalance, row counts dip then recover. Root cause is the drop-from-source announcement reaching the Broker before the load-on-target announcement — a transient during balancing. Check the Coordinator load queue to see moves still in flight:

curl -s "http://<coordinator-host>:8081/druid/coordinator/v1/loadqueue?full" \
  | jq 'to_entries[] | {server: .key, toLoad: (.value.segmentsToLoad|length), toDrop: (.value.segmentsToDrop|length)}'

A non-empty toLoad/toDrop confirms propagation is mid-flight; the timeline will settle once the queue drains. Persistent drift instead of transient drift points at discovery itself — a ZooKeeper partition or an HTTP-discovery gap between Broker and Historicals, which ties back to the trust and connectivity boundaries in security boundaries for segment access.

Target Spec & Validated JSON

There is no ingestion spec to fix here — the levers are Broker and discovery runtime properties that bound how long the propagation window can last. The Broker-side properties below (set in broker/runtime.properties) shrink the drift and make the timeline sync predictable:

{
  "druid.broker.segment.watchedTiers": ["_default_tier", "cold"],
  "druid.serverview.type": "http",
  "druid.broker.http.numConnections": 20,
  "druid.sql.planner.metadataRefreshPeriod": "PT1M",
  "druid.coordinator.period": "PT30S",
  "druid.coordinator.startDelay": "PT10S",
  "druid.manager.segments.pollDuration": "PT1M"
}

The properties that govern timeline freshness:

  • druid.serverview.type — http (recommended on current Druid) makes Brokers pull segment announcements over HTTP rather than watching ZooKeeper, which removes a class of ZK-lag staleness. Ensure every Broker and Coordinator agrees on this setting.
  • druid.broker.segment.watchedTiers — if set, the Broker only builds a timeline for these tiers; a segment that moves to an unwatched tier vanishes from that Broker's view. A frequent silent cause of "missing after move" is a watchedTiers list that omits the destination tier.
  • druid.coordinator.period — how often the Coordinator reconciles the metadata store and issues load/drop. Shorter periods narrow the propagation window at the cost of Coordinator CPU.
  • druid.manager.segments.pollDuration — how often the Coordinator re-reads the used-segment set from the metadata store; this bounds how quickly a new or dropped segment even becomes visible to the Coordinator.
  • druid.sql.planner.metadataRefreshPeriod — how often the Broker refreshes SQL schema/segment metadata; a long value can make the SQL layer report stale availability even after the native timeline updates.

These settings bound the window; they do not eliminate it. The metadata store internals that the Coordinator polls are detailed in Druid segment metadata storage deep dive.

Python Automation Script

The reliable way to avoid querying into a stale timeline is to poll until the Broker's view converges with the Coordinator's authoritative interval set before signalling a pipeline healthy. The monitor below diffs the two, applies capped exponential backoff, and can trigger a Broker Coordinator refresh if convergence stalls. It uses only the standard library plus requests.

import time
import logging
import requests

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("druid_timeline")


class BrokerTimelineMonitor:
    def __init__(self, broker_url, coordinator_url, datasource, auth=None):
        self.broker = broker_url.rstrip("/")
        self.coordinator = coordinator_url.rstrip("/")
        self.datasource = datasource
        self.session = requests.Session()
        self.session.auth = auth

    def _coordinator_intervals(self):
        resp = self.session.get(
            f"{self.coordinator}/druid/coordinator/v1/datasources/"
            f"{self.datasource}/intervals", timeout=15,
        )
        resp.raise_for_status()
        return set(resp.json())

    def _broker_intervals(self):
        resp = self.session.get(
            f"{self.broker}/druid/v2/datasources/{self.datasource}?full", timeout=15,
        )
        resp.raise_for_status()
        # The Broker reports the interval span it can currently serve.
        body = resp.json()
        return set(body.get("segments", {}).get("intervals", [])) or {
            body.get("segments", {}).get("minTime", "") + "/" +
            body.get("segments", {}).get("maxTime", "")
        }

    def wait_for_convergence(self, base_delay=2, max_delay=30, max_wait=600):
        start = time.time()
        delay = base_delay
        want = self._coordinator_intervals()
        while time.time() - start < max_wait:
            have = self._broker_intervals()
            missing = want - have
            if not missing:
                logger.info("Broker timeline converged with Coordinator (%d intervals)",
                            len(want))
                return True
            logger.info("Broker still missing %d interval(s); sleeping %ss",
                        len(missing), delay)
            time.sleep(delay)
            delay = min(delay * 2, max_delay)  # exponential backoff, capped
            want = self._coordinator_intervals()  # re-read in case more committed
        raise TimeoutError(
            f"{self.datasource} timeline did not converge within {max_wait}s"
        )


# Usage
# mon = BrokerTimelineMonitor("http://broker:8082", "http://coordinator:8081",
#                             "events_routed")
# mon.wait_for_convergence()   # blocks until queries will see the current state

Because a stalled convergence raises TimeoutError, a pipeline never marks an ingest available while the serving Broker is still stale. For running this without blocking a whole batch of tasks, see non-blocking Druid task status polling.

Verification Steps

Confirm the timeline is fresh after the propagation window. First check that the Broker load status reports no outstanding segments to load for the datasource:

curl -s "http://<broker-host>:8082/druid/broker/v1/loadstatus" | jq

Expected output reports a fully-loaded inventory (100% — nothing pending):

{
  "inventoryInitialized": true,
  "0%": 0
}

Then confirm a query now returns the freshly ingested interval — the count for the new day should be non-zero and the dropped interval should be empty:

curl -s "http://<broker-host>:8082/druid/v2/sql" -H 'Content-Type: application/json' \
  -d '{"query":"SELECT FLOOR(__time TO DAY) AS d, COUNT(*) AS rows FROM events_routed GROUP BY 1 ORDER BY 1 DESC LIMIT 3"}'

Expected output shows the newest day present with rows, confirming the Broker timeline caught up:

[
  { "d": "2026-02-10T00:00:00.000Z", "rows": 1843200 },
  { "d": "2026-02-09T00:00:00.000Z", "rows": 1799040 }
]

If drift persists past the configured druid.coordinator.period plus pollDuration, it is no longer a propagation transient — inspect discovery health directly (ZooKeeper connectivity or the HTTP server view) and confirm every Broker shares the same druid.serverview.type and watchedTiers. Reference the Druid Broker documentation and the segment discovery configuration reference for the authoritative property list.

Back to Apache Druid Segment Architecture & Lifecycle Fundamentals