Druid Segment Ops Segment & Ingestion Orchestration

Datasource Access Control with Basic Auth

The requirement lands the day a shared cluster gains a second tenant: analyst A must query sales_events but never hr_events, and the ingestion service must write both without a human ever holding those credentials. Apache Druid answers this with the bundled druid-basic-security extension, which pairs an HTTP Basic authenticator (who are you) with an authorizer (what may you touch) and enforces access at the datasource level using resource:action permissions — DATASOURCE resources gated by READ and WRITE actions. This page sits under security boundaries for segment access, which frames the trust planes around a segment; here we implement the query-path boundary concretely with users, roles, and permissions.

A Basic-auth request flowing through the authenticator and authorizer to a datasource resource check A left-to-right request flow. A client sends an HTTP Basic credential to the Router or Broker. The authenticator stage validates the username and password against the basic-security credentials store and establishes an identity, or returns 401 if the credential is wrong. The authenticated identity passes to the authorizer stage, which looks up the user's roles and their permissions and evaluates the requested resource and action, for example DATASOURCE sales_events with action READ, against the permission matrix. A match permits the query to reach the datasource and scatter to Historicals; no matching permission returns 403 Forbidden. A side note shows WRITE actions gate ingestion and compaction on the same datasource resource. client HTTP Basic cred Authenticator verify user + password establish identity 401 bad credential Authorizer roles → permissions DATASOURCE sales_events action = READ ? 403 no permission match datasource query scatters to Historicals WRITE gates ingestion & compaction on the same DATASOURCE resource a service account needs DATASOURCE + action WRITE, not admin

Failure Modes & Diagnostics

Access-control problems present as a 403 Forbidden where a query should succeed, a 401 Unauthorized where the credential should be valid, or — most dangerous — a query succeeding where it should have been denied. Diagnose against the basic-security REST API, which the Coordinator hosts. First confirm the extension is actually loaded and the authenticator/authorizer are wired:

# Are the basic authenticator and authorizer active? (empty/404 means not loaded)
curl -s -u admin:<admin_pw> \
  "http://<coordinator-host>:8081/druid/coordinator/v1/config/global/authentication/basic" >/dev/null && echo "authenticator reachable"

# List users known to the basic authenticator
curl -s -u admin:<admin_pw> \
  "http://<coordinator-host>:8081/druid/coordinator/v1/security/authentication/db/basic/users" | jq

# List roles known to the basic authorizer
curl -s -u admin:<admin_pw> \
  "http://<coordinator-host>:8081/druid/coordinator/v1/security/authorization/db/basic/roles" | jq

Unexpected 403 for a legitimate user. Symptom: the user authenticates but every query is denied. Inspect exactly which permissions their roles grant and whether the DATASOURCE regex matches the datasource name:

# Full permission set attached to a role
curl -s -u admin:<admin_pw> \
  "http://<coordinator-host>:8081/druid/coordinator/v1/security/authorization/db/basic/roles/analyst_read?full" | jq

# Which roles is the user actually mapped to?
curl -s -u admin:<admin_pw> \
  "http://<coordinator-host>:8081/druid/coordinator/v1/security/authorization/db/basic/users/analyst_a?full" | jq

Root cause is usually a DATASOURCE permission whose name regex is anchored too tightly (for example sales_events instead of sales_.*) or a user never bound to the role. Remediation: attach the correct role or widen the regex, then re-test.

401 despite correct password. Symptom: authentication itself fails. Root cause is often that the two metadata-store credential caches (authenticator and authorizer) have not polled the latest change, or the request hit an escalated internal endpoint. Confirm the user exists and force a re-check by re-reading the users endpoint above. The authenticator and authorizer poll the metadata store on an interval; a just-created user is not instantly live.

A query that should be denied succeeds. Symptom: a tenant reads another tenant's datasource. This is the boundary failure that matters most. Confirm no wildcard DATASOURCE .* permission leaked into a shared role:

# Hunt for over-broad datasource grants across all roles
for r in $(curl -s -u admin:<admin_pw> "http://<coordinator-host>:8081/druid/coordinator/v1/security/authorization/db/basic/roles" | jq -r '.[]'); do
  echo "== $r =="
  curl -s -u admin:<admin_pw> "http://<coordinator-host>:8081/druid/coordinator/v1/security/authorization/db/basic/roles/$r?full" \
    | jq '.permissions[]? | select(.resource.type=="DATASOURCE") | {name: .resource.name, action: .action}'
done

Root cause is a broad .* grant on a role more users share than intended. Remediation: split the role and scope each DATASOURCE regex to the tenant prefix. These datasource grants are the query-path half of the two boundaries in security boundaries for segment access; the file-layer half (deep-storage IAM) is enforced separately, and retention-driven placement is governed by configuring segment retention policies.

Target Spec & Validated JSON

Two layers configure this: static runtime properties that turn the extension on, and dynamic REST-managed users/roles/permissions. The runtime properties (set on the common config and every service's runtime.properties) declare the authenticator and authorizer:

{
  "druid.auth.authenticatorChain": ["basic"],
  "druid.auth.authenticator.basic.type": "basic",
  "druid.auth.authenticator.basic.credentialsValidator.type": "metadata",
  "druid.auth.authenticator.basic.initialAdminPassword": "<admin_pw>",
  "druid.auth.authenticator.basic.initialInternalClientPassword": "<internal_pw>",
  "druid.auth.authenticator.basic.authorizerName": "basic",
  "druid.auth.authorizers": ["basic"],
  "druid.auth.authorizer.basic.type": "basic",
  "druid.escalator.type": "basic",
  "druid.escalator.internalClientUsername": "druid_system",
  "druid.escalator.internalClientPassword": "<internal_pw>",
  "druid.escalator.authorizerName": "basic"
}
  • druid.auth.authenticatorChain — ordered list of authenticators; basic here does HTTP Basic against the metadata credential store (credentialsValidator.type: metadata).
  • druid.auth.authorizers — the authorizer that evaluates resource:action. It must match the authenticator's authorizerName so identities route to the right permission model.
  • druid.escalator — how Druid services authenticate to each other; the internal druid_system account must exist and hold broad permissions, or inter-service calls (Broker to Historical, Coordinator to Overlord) fail with 401.

With the extension live, model a read-only analyst and a write-only ingestion service through the REST API. A DATASOURCE permission is a resource (type + a name regex) plus an action:

{
  "permissions": [
    { "resource": { "type": "DATASOURCE", "name": "sales_.*" }, "action": "READ" },
    { "resource": { "type": "STATE", "name": ".*" }, "action": "READ" }
  ]
}

That JSON is the payload assigned to a role's permissions. The STATE READ grant lets the user see system tables (sys.segments) needed by the SQL console; omit it for a pure data-plane account. A write-scoped ingestion role instead carries {"resource": {"type": "DATASOURCE", "name": "sales_.*"}, "action": "WRITE"}, which authorizes submitting ingestion and compaction tasks against those datasources — the same WRITE action gates the compaction covered in automated compaction task scheduling. Grant WRITE without READ so a pipeline account cannot exfiltrate the data it loads.

Python Automation Script

Users, roles, and permissions must be provisioned as code so a new tenant is reproducible and auditable. The provisioner below creates a role, assigns scoped DATASOURCE permissions, creates a user, sets a password, and binds the user to the role — each step idempotent, with capped exponential backoff over the basic-security REST API. It uses only the standard library plus requests.

import time
import logging
import requests

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("druid_basic_auth")

_BASE = "/druid/coordinator/v1/security"


class DruidBasicAuthProvisioner:
    def __init__(self, coordinator_url, admin_user, admin_password):
        self.coordinator = coordinator_url.rstrip("/")
        self.session = requests.Session()
        self.session.auth = (admin_user, admin_password)

    def _call(self, method, path, json_body=None,
              base_delay=2, max_delay=20, max_attempts=5):
        url = f"{self.coordinator}{path}"
        delay = base_delay
        for attempt in range(1, max_attempts + 1):
            resp = self.session.request(method, url, json=json_body, timeout=15)
            if resp.status_code < 400 or resp.status_code == 404:
                return resp
            if resp.status_code in (429, 500, 502, 503, 504) and attempt < max_attempts:
                logger.warning("%s %s -> %s; retry %d in %ss",
                               method, path, resp.status_code, attempt, delay)
                time.sleep(delay)
                delay = min(delay * 2, max_delay)  # exponential backoff, capped
                continue
            resp.raise_for_status()
        raise RuntimeError(f"{method} {path} failed after {max_attempts} attempts")

    def ensure_role(self, role, permissions):
        self._call("POST", f"{_BASE}/authorization/db/basic/roles/{role}")
        self._call("POST",
                   f"{_BASE}/authorization/db/basic/roles/{role}/permissions",
                   json_body=permissions)
        logger.info("Role %s ensured with %d permission(s)", role, len(permissions))

    def ensure_user(self, user, password, role):
        self._call("POST", f"{_BASE}/authentication/db/basic/users/{user}")
        self._call("POST",
                   f"{_BASE}/authentication/db/basic/users/{user}/credentials",
                   json_body={"password": password})
        self._call("POST", f"{_BASE}/authorization/db/basic/users/{user}")
        self._call("POST",
                   f"{_BASE}/authorization/db/basic/users/{user}/roles/{role}")
        logger.info("User %s ensured and bound to role %s", user, role)


# Usage
# p = DruidBasicAuthProvisioner("http://coordinator:8081", "admin", "<admin_pw>")
# p.ensure_role("analyst_read", [
#     {"resource": {"type": "DATASOURCE", "name": "sales_.*"}, "action": "READ"},
#     {"resource": {"type": "STATE", "name": ".*"}, "action": "READ"},
# ])
# p.ensure_user("analyst_a", "<user_pw>", "analyst_read")

Because each step is idempotent and retried under backoff, re-running the provisioner converges the Druid cluster to the declared access model rather than duplicating grants. For generating these role definitions from a tenant manifest, the same templating approach as dynamic ingestion spec generation applies.

Verification Steps

Confirm the boundary holds after provisioning. First prove the allowed path succeeds — the analyst can query the datasource they were granted:

curl -s -u analyst_a:<user_pw> "http://<broker-host>:8082/druid/v2/sql" \
  -H 'Content-Type: application/json' \
  -d '{"query":"SELECT COUNT(*) AS rows FROM sales_events"}'

Expected output is a normal result set — authentication and authorization both passed:

[
  { "rows": 4821030 }
]

Then prove the denied path is actually denied — the same user must be refused a datasource outside their grant, returning HTTP 403:

curl -s -o /dev/null -w "%{http_code}\n" -u analyst_a:<user_pw> \
  "http://<broker-host>:8082/druid/v2/sql" -H 'Content-Type: application/json' \
  -d '{"query":"SELECT COUNT(*) FROM hr_events"}'

Expected output is exactly 403, confirming the DATASOURCE regex did not leak across tenants:

403

Finally confirm the write boundary: the read-only user cannot submit an ingestion task (403), while the ingestion service account can. If any of these three checks returns the wrong status, re-inspect the role's DATASOURCE permission regex and action with the diagnostic commands above before granting access. Reference the Druid basic-security extension documentation for the full authenticator, authorizer, and permission-model reference.

Back to Apache Druid Segment Architecture & Lifecycle Fundamentals