Security Boundaries for Segment Access in Apache Druid

Segment access control is where multi-tenant Druid deployments either hold their compliance line or quietly leak dimension distributions across tenant boundaries. Unlike row-level ACLs in relational engines, Druid enforces access at the datasource level through a resource/action authorization model, then reinforces that boundary with time-based partitioning and deep storage IAM policies that govern the raw segment files themselves. For platform engineers running shared analytics clusters, these boundaries must be codified as automation rather than configured by hand — which is why they sit directly under Apache Druid Segment Architecture & Lifecycle Fundamentals, where ingestion velocity, query concurrency, and regulatory retention all converge on the segment as the unit of enforcement.

Mechanics & Internals #

Druid processes every request through a two-stage pipeline before a single segment is scanned. The Authenticator chain (druid.auth.authenticatorChain) establishes identity — mapping an inbound credential to a principal — and the Authorizer (druid.auth.authorizers) evaluates whether that principal may perform a specific action on a specific resource. Authorization is not row- or value-aware: the atomic decision is a triple of resource name, resource type, and action.

For segment access the relevant resource type is DATASOURCE, and the two actions are READ (query paths through the Broker) and WRITE (ingestion, compaction, kill, and retention-rule mutation routed through the Overlord and Coordinator). A role is a named collection of permissions, each permission binding one resource pattern to one action. Resource names are matched as regular expressions, so a permission on clickstream_.* grants access to every datasource sharing that prefix — a property that makes tenant isolation a naming-convention problem as much as a policy problem.

Identity resolution happens at whichever process first receives the request. External queries hit the Broker (or Router); ingestion submissions hit the Overlord. Each process independently runs the authenticator chain and authorizer, so a misconfigured runtime.properties on a single Historical or Broker silently widens the boundary. Internal service-to-service traffic (Coordinator instructing a Historical to load a segment, Broker scattering a sub-query) authenticates through the escalator (druid.escalator), a dedicated internal principal that must itself be granted the permissions needed to move segments.

The second, deeper boundary is the segment file. Because Druid's columnar storage formats persist metadata dictionaries, bitmap indexes, and value columns as separate structures inside each segment, direct read access to raw files in deep storage can leak dimension cardinality and statistical distributions without ever touching the query API. Datasource ACLs are therefore necessary but not sufficient: the segments/ prefix in object storage must be locked down to the Druid service principals with an IAM policy, and ingestion workers must operate under short-lived credentials that expire when the task completes. The way data is partitioned by segment granularity determines how cleanly those file-level boundaries map to tenant and residency constraints — a DAY-granular datasource yields one enforceable file group per day per tenant, whereas coarse ALL granularity collapses everything into a single blast radius.

The primary control-plane endpoints an orchestrator touches are:

• GET /druid/coordinator/v1/security/roles — list defined roles.
• POST /druid/coordinator/v1/security/roles/{roleName} — create a role.
• POST /druid/coordinator/v1/security/roles/{roleName}/permissions — bind permissions (the resource/action list) to a role.
• POST /druid/coordinator/v1/security/users/{userName}/roles/{roleName} — assign a role to a user.
• GET /druid/coordinator/v1/metadata/datasources — enumerate datasources to reconcile against the permission set.

Validated Configuration Spec #

The following blocks form a complete, copy-ready datasource-isolation baseline using the druid-basic-security extension. Start with the common runtime.properties shared by every process:

# --- Load the extension on every process ---
druid.extensions.loadList=["druid-basic-security", "druid-s3-extensions"]

# --- Authenticator chain: identify the principal ---
druid.auth.authenticatorChain=["basic"]
druid.auth.authenticator.basic.type=basic
druid.auth.authenticator.basic.initialAdminPassword=
druid.auth.authenticator.basic.initialInternalClientPassword=
druid.auth.authenticator.basic.authorizerName=basic
druid.auth.authenticator.basic.credentialsValidator.type=metadata

# --- Authorizer: evaluate resource + action against roles ---
druid.auth.authorizers=["basic"]
druid.auth.authorizer.basic.type=basic

# --- Escalator: the internal principal for service-to-service calls ---
druid.escalator.type=basic
druid.escalator.internalClientUsername=druid_system
druid.escalator.internalClientPassword=
druid.escalator.authorizerName=basic

Field notes: authenticatorChain is evaluated in order, so a fallback (e.g. append "trustedProxy") must come after "basic". credentialsValidator.type=metadata stores password hashes in the metadata database; switch to ldap to delegate identity to a directory without changing the authorizer. The initialAdminPassword and initialInternalClientPassword seed the metadata store only on first boot — rotate them immediately afterward through the users API. The escalator username must be granted READ/WRITE on all datasources it moves, or internal segment loads fail with authorization errors that look like cluster-wide outages.

Next, the permission payload that scopes a tenant's analyst role to exactly one datasource family, read-only:

[
  {
    "resource": { "name": "clickstream_.*", "type": "DATASOURCE" },
    "action": "READ"
  },
  {
    "resource": { "name": "clickstream_.*", "type": "DATASOURCE" },
    "action": "WRITE"
  }
]

Field notes: name is a regex anchored implicitly; keep tenant prefixes disjoint (clickstream_, billing_) so no pattern accidentally overlaps another tenant's namespace. Grant WRITE only to the ingestion service principal — analyst roles should carry READ alone. To also restrict the Druid console and system tables, add CONFIG and STATE resource types in a separate operator role.

The deep storage boundary is enforced outside Druid with an object-storage policy scoping the segments/ prefix to the Druid service principals only:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DruidSegmentReadWrite",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::analytics-deep-storage/segments/*"
    },
    {
      "Sid": "DruidSegmentList",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::analytics-deep-storage",
      "Condition": { "StringLike": { "s3:prefix": "segments/*" } }
    }
  ]
}

Field notes: attach this policy to the role assumed by Historical and Coordinator processes (via IRSA on Kubernetes or an instance profile), never to ingestion workers, which should assume a PutObject-only variant scoped to the interval prefix they write. s3:DeleteObject is required for the kill/retention path; withhold it from any principal that must not purge history. The matching Druid-side storage config points at the same prefix:

druid.storage.type=s3
druid.storage.bucket=analytics-deep-storage
druid.storage.baseKey=segments
druid.s3.enablePathStyleAccess=false

Finally, align the ingestion spec's dataSource and granularitySpec with the permission and IAM prefixes so a tenant's data lands only where its policy reaches:

{
  "type": "index_parallel",
  "spec": {
    "dataSchema": {
      "dataSource": "clickstream_tenant_a",
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "DAY",
        "queryGranularity": "HOUR",
        "rollup": true
      },
      "timestampSpec": { "column": "ts", "format": "iso" },
      "dimensionsSpec": { "dimensions": ["path", "referrer", "country"] }
    },
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": { "type": "s3", "prefixes": ["s3://tenant-a-landing/events/"] },
      "inputFormat": { "type": "json" }
    },
    "tuningConfig": {
      "type": "index_parallel",
      "maxRowsPerSegment": 5000000,
      "partitionsSpec": { "type": "hashed", "numShards": 4 }
    }
  }
}

The dataSource value (clickstream_tenant_a) matches the clickstream_.* permission regex, and segmentGranularity: "DAY" produces one enforceable file group per day, keeping residency and retention boundaries physical rather than logical.

Sizing Heuristics & Formulas #

Security boundaries carry quantifiable budgets. The permission-set size an orchestrator must reconcile is the product of datasources and the actions each role touches:

$$ P \approx D \times |A| \times R $$

where $D$ is the number of datasources, $|A|$ the actions per role (typically $2$: READ, WRITE), and $R$ the number of distinct roles. A 40-datasource cluster with 6 tenant roles yields $P \approx 40 \times 2 \times 6 = 480$ permission bindings — well within API limits, but past a few thousand you should collapse per-datasource grants into prefix regexes to keep the reconciliation loop fast.

The most common security incident in automated pipelines is a credential that expires mid-task. The ingestion credential TTL must dominate the task's wall-clock plus the polling tail plus clock skew:

$$ \text{tokenTTL} \geq t_{\text{wall}} + t_{\text{poll}} + \Delta_{\text{clock}} $$

For a batch task with a 45-minute wall-clock, a 5-minute poll tail, and 1 minute of allowed skew, provision a token TTL of at least $51$ minutes and round up to a standard $1\text{h}$ lease. Under-provisioning here manifests as PutObject failures deep into a task, after compute has already been spent.

Blast radius scales with how coarsely you partition. If a tenant owns $n_{\text{days}}$ days of retained data at DAY granularity with replication factor $r$, the number of independently permissioned segment file groups is:

$$ F \approx n_{\text{days}} \times r $$

Coarsening to MONTH granularity divides $F$ by roughly $30$, shrinking the audit surface but also coarsening the smallest unit you can drop for a residency request — a direct trade between enforcement granularity and operational overhead, decided at ingestion time and only reversible by re-compaction.

Python Orchestration Snippet #

This orchestrator provisions no static keys: it submits an ingestion task under a caller-supplied short-lived bearer token, polls the Overlord for terminal status, and retries transient failures with exponential backoff. It uses only the standard library plus requests.

import time
import requests

OVERLORD = "https://druid-overlord.internal:8090"


def submit_ingestion(spec: dict, token: str) -> str:
    """Submit a task under a scoped, short-lived token; return the task id."""
    resp = requests.post(
        f"{OVERLORD}/druid/indexer/v1/task",
        json=spec,
        headers={"Authorization": f"Bearer {token}"},
        timeout=30,
    )
    if resp.status_code == 403:
        raise PermissionError(
            f"WRITE denied for dataSource "
            f"{spec['spec']['dataSchema']['dataSource']!r} — check role bindings"
        )
    resp.raise_for_status()
    return resp.json()["task"]


def poll_until_complete(task_id: str, token: str, deadline_s: float = 3600) -> str:
    """Poll task status with exponential backoff until terminal or deadline."""
    headers = {"Authorization": f"Bearer {token}"}
    url = f"{OVERLORD}/druid/indexer/v1/task/{task_id}/status"
    delay, waited = 2.0, 0.0
    while waited < deadline_s:
        try:
            r = requests.get(url, headers=headers, timeout=15)
            if r.status_code == 401:
                raise PermissionError("Token expired mid-task — widen tokenTTL")
            r.raise_for_status()
            state = r.json()["status"]["status"]  # RUNNING | SUCCESS | FAILED
            if state in ("SUCCESS", "FAILED"):
                return state
        except requests.RequestException:
            pass  # transient — fall through to backoff
        time.sleep(delay)
        waited += delay
        delay = min(delay * 2, 60.0)  # cap backoff at 60s
    raise TimeoutError(f"{task_id} did not finish within {deadline_s}s")


def run(spec: dict, token: str) -> None:
    task_id = submit_ingestion(spec, token)
    outcome = poll_until_complete(task_id, token)
    if outcome != "SUCCESS":
        raise RuntimeError(f"{task_id} terminated as {outcome}")

The 403 path distinguishes an authorization gap (bad role binding) from the 401 path (an expired lease) — collapsing them into one handler is the mistake that turns a routine token-TTL fix into a fruitless permissions audit. This submission pattern pairs directly with schema-first pipelines described under dynamic ingestion spec generation, where the spec itself is templated per tenant.

Failure Modes & Diagnostics #

1. Blanket 403 Forbidden on every query for a datasource. Symptom: analysts see access denied where they had access yesterday. Confirm the role still carries the permission:

curl -s -u admin:"$ADMIN_PW" \
  "$COORDINATOR/druid/coordinator/v1/security/roles/tenant_a_read/permissions" \
  | jq '.[] | {name: .resource.name, type: .resource.type, action: .action}'

If the clickstream_.* READ binding is absent, a role-sync job overwrote it. Re-POST the permission list from source control.

2. New datasource is invisible / unqueryable. A regex gap means the datasource exists but matches no permission. Diff live datasources against the permitted patterns:

curl -s -u admin:"$ADMIN_PW" \
  "$COORDINATOR/druid/coordinator/v1/metadata/datasources" \
  | jq -r '.[]' | grep -vE 'clickstream_|billing_'

Any line printed is an orphaned, unpermissioned datasource — either extend a prefix pattern or rename the datasource to fit an existing one.

3. Internal segment loads fail; Historicals log authorization errors. The escalator principal lacks permission. Verify druid_system holds cluster-wide access:

curl -s -u admin:"$ADMIN_PW" \
  "$COORDINATOR/druid/coordinator/v1/security/users/druid_system/roles" | jq .

If it returns [], bind an admin-equivalent role to druid_system — without it the Coordinator cannot instruct Historicals to load segments and the Druid cluster appears to stall.

4. Ingestion writes fail at PutObject late in the task. The credential expired before the task finished. Check the task's failure detail and the lease TTL:

curl -s "$OVERLORD/druid/indexer/v1/task/$TASK_ID/reports" \
  | jq '.ingestionStatsAndErrors.payload.errorMsg'

An Access Denied or ExpiredToken message confirms a TTL shortfall — re-provision per the $\text{tokenTTL}$ formula above.

5. Raw segment files readable outside Druid. Audit that no non-Druid principal can list the segments/ prefix. From a machine holding a suspect role:

aws s3 ls s3://analytics-deep-storage/segments/ 2>&1 | head

A successful listing from anything other than the Historical/Coordinator role is a boundary breach — tighten the IAM policy before proceeding.

Automation Checklist #

• Extension druid-basic-security present in druid.extensions.loadList on every process, not just the Coordinator.
• Escalator principal (druid_system) holds READ/WRITE across all datasources it must move.
• Tenant datasource prefixes are disjoint; every live datasource matches exactly one permission regex (pre-ingestion validation gate).
• Ingestion dataSource values are validated against the permission regex before task submission, alongside schema validation for Druid specs.
• segments/ object-storage prefix is scoped to Historical/Coordinator roles only; ingestion workers hold a PutObject-only, interval-scoped variant.
• Ingestion credentials are short-lived and satisfy $\text{tokenTTL} \geq t_{\text{wall}} + t_{\text{poll}} + \Delta_{\text{clock}}$; no static keys in specs or env.
• Role/permission definitions are version-controlled and applied through CI/CD, never edited live in the console.
• Post-ingestion audit reconciles live datasources against permitted patterns and alerts on orphans.
• Retention and kill paths hold s3:DeleteObject and are wired to segment retention policies so expired data is purged deterministically.
• Authorization decisions and denied requests are logged and surfaced to the same monitoring stack as query and compaction metrics.

Authorizer behavior and the resource/action model are specified in the Apache Druid basic security documentation.

Related #

• Configuring Segment Retention Policies — codify the deletion side of the boundary so segments never outlive their compliance window.
• Understanding Druid Segment Granularity — the partition sizing that decides how cleanly file-level boundaries map to tenants and residency rules.
• Columnar Storage Formats in Druid — why raw segment files leak cardinality and distributions when the file-level boundary is loose.
• Query Routing and Segment Discovery — how the Broker resolves and dispatches reads that datasource authorization gates.
• TTL Mapping and Data Expiration — the storage-side lifecycle rules that keep unauthorized historical exposure bounded.

Up one level: Apache Druid Segment Architecture & Lifecycle Fundamentals.