Authentication & tokens

Closed by default

Every read and write needs a token. Only /health and /ready answer without one. What a node does at boot depends on what it was given:

What a token is

A token is a JWT signed with HMAC-SHA256 over TSDB_AUTH_SECRET, sent as Authorization: Bearer <jwt>. The header's alg must be exactly HS256; it is checked before any signature is computed, so a token naming another algorithm — including none — is refused as Unsupported token algorithm, never as a signature mismatch. The claims:

A request the data plane refuses answers 401 with the reason in the body: Missing Authorization header, Invalid token format, Invalid token signature, Unsupported token algorithm, Token expired, Token not yet valid, Invalid token claims, or Refresh token is not an API credential. A valid token without the permission the route needs answers 403 Insufficient permissions.

Mint a token for a client

POST /auth/token mints tokens for other clients. Two callers may use it, and nothing else: an admin access token, which may mint for any subject; and, behind the secure overlay, an mTLS identity the sidecar stamped on the request, mapped to roles by TSDB_AUTH_SAN_ROLES, which mints for that identity only. A non-admin access token may not mint — if it could, every access token would be a refresh token and short lifetimes would mean nothing.

The request names the subject, its roles and, optionally, ttl_s (clamped to the configured access lifetime — a caller may ask for less, never more) and "refresh":true to receive a refresh token as well:

POST /auth/token
Authorization: Bearer <admin access token>

{"sub":"svc-a","roles":["writer"],"ttl_s":600,"refresh":true}

{"access_token":"eyJ…","token_use":"access","expires_in":600,
 "refresh_token":"eyJ…","refresh_expires_in":604800}

Delegation, never escalation. A minted token carries only roles whose every permission the minter already holds. Containment is by permission, not by role name: an admin holds everything reader grants, so an admin may mint a read-only token.

The first admin credential. A bootstrapped node prints its admin refresh token at boot. A cluster with an operator secret has no such moment: the first admin token is signed with the secret itself, which is what the quickstart's mint step does and what $TOKEN means below. From there, mint a pair for a service and prove the access half writes — without the write-forwarding overlay only the node that leads the metric's group answers 200, the others 503:

$ NODE=http://localhost:8080
$ PAIR=$(curl -s -X POST "$NODE/auth/token" -H "Authorization: Bearer $TOKEN" \
    -d '{"sub":"svc-a","roles":["writer"],"refresh":true}')
$ ACCESS=$(printf '%s' "$PAIR" | sed -E 's/.*"access_token":"([^"]+)".*/\1/')
$ REFRESH=$(printf '%s' "$PAIR" | sed -E 's/.*"refresh_token":"([^"]+)".*/\1/')
$ for n in 8080 8081 8082; do
    curl -s -o /dev/null -w "$n %{http_code}\n" -X POST "http://localhost:$n/ingest" \
      -H "Authorization: Bearer $ACCESS" -d "svc_metric,host=svc-a value=1 $(date +%s)000000000"
  done

Hand svc-a its refresh token, not the access token: the refresh token is what it exchanges for new access tokens on its own, and the one you can kill.

Refresh

POST /auth/refresh exchanges a refresh token for an access token. The refresh token is the credential — no admin token, no certificate — and the body is {"refresh_token":"…"}. What it buys is deliberately narrow: an access token with the same subject and roles (an exchange can never gain authority), carrying no jti, so it cannot itself be exchanged again. Refresh tokens are reusable until they expire or are revoked; they are not rotated per exchange, which keeps concurrent clients simple and makes revocation the one kill switch.

A refresh token is not an API credential. Presented as the bearer on any data route it is refused 401 Refresh token is not an API credential, however valid its signature: the data plane never consults the revocation list, so honouring it there would make a leaked refresh token as good as the data even after revocation.

$ ACCESS2=$(curl -s -X POST http://localhost:8081/auth/refresh \
    -d "{\"refresh_token\":\"$REFRESH\"}" | sed -E 's/.*"access_token":"([^"]+)".*/\1/')
$ Q="http://localhost:8082/query?zql=SELECT%20avg(value)%20FROM%20svc_metric"
$ curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $ACCESS2" "$Q"
$ curl -s -H "Authorization: Bearer $REFRESH" "$Q"

200
{"error":"Refresh token is not an API credential"}

The CLI's credential

zigtsdb-cli sends a credential on every command that talks to a node. It reads two variables, and the difference matters:

TSDB_REFRESH_TOKEN wins when both are set: an operator who has configured both is mid-migration, and preferring the static one would silently keep the un-revocable credential in play. A rejected exchange is an error, not a fallback: if the refresh token has been revoked or has expired, the CLI says so and exits 1 rather than quietly retrying with the static token and defeating the revocation:

auth: the refresh token was rejected (RefreshRejected) — it may have been
revoked or expired. Not falling back to TSDB_AUTH_TOKEN: that credential
cannot be revoked.

The image ships the CLI at /app/zigtsdb-cli, so a service's pair can be tried from inside any node. With the refresh token minted above:

$ docker compose -f docker-compose.ha-3node.yml exec -T \
    -e TSDB_REFRESH_TOKEN="$REFRESH" node0 /app/zigtsdb-cli license; echo "exit $?"

Exit codes: 0 the command did what it said; 1 it failed — a non-2xx answer (Error: HTTP <n> plus the body), a transport error, a rejected exchange; 2 invalid arguments. Scripts written against a 1.5 or older CLI, which printed the error and still exited 0, must check the status they were already meant to check. Later CLI releases add auth commands for the minting and revocation steps shown with curl on this page.

Revoke

POST /auth/revoke kills a refresh token. It is admin-gated and takes either the token itself ({"refresh_token":"…"}) or its identity from an audit line ({"jti":"<32 hex>"}). It is idempotent, and answers 200 {"revoked":"<jti>"}.

The revocation is a Raft write: it replicates to every node, because a denylist that reached only one of them is defeated by retrying the refresh against another. Send it to the leader — a follower answers 503 with a leader_url to follow, the same hint a write gets. It is refused with 409 while any peer runs a release before 1.6.0, since a node that cannot read the entry would silently drop it and go on honouring the token; finish the rollout first. Revocations survive a restart and a snapshot (every node keeps token_revocations.json in its data directory); a node downgraded below 1.6.0 forgets them. Access tokens cannot be revoked — they expire within the access lifetime, which is why it is short.

Decode the refresh token's jti — the payload is plain base64url, no key needed — and revoke around the ring until a node other than a follower answers:

$ claim() { p=$(printf '%s' "$1" | cut -d. -f2 | tr '_-' '/+'); pad=$(( (4 - ${#p} % 4) % 4 ));
    printf '%s%s' "$p" "$(printf '%*s' "$pad" '' | tr ' ' '=')" | openssl base64 -d -A \
      | grep -oE "\"$2\":\"[^\"]*\"" | cut -d'"' -f4; }
$ JTI=$(claim "$REFRESH" jti)
$ for n in 8080 8081 8082; do
    code=$(curl -s -o revoke.json -w '%{http_code}' -X POST "http://localhost:$n/auth/revoke" \
      -H "Authorization: Bearer $TOKEN" -d "{\"jti\":\"$JTI\"}")
    [ "$code" = 503 ] || break
  done
$ cat revoke.json

{"revoked":"3f9c…"}

From that moment the refresh token is dead on every node, and the CLI holding it says so. The access tokens it already bought keep working until they expire:

$ curl -s -X POST http://localhost:8082/auth/refresh -d "{\"refresh_token\":\"$REFRESH\"}"
$ docker compose -f docker-compose.ha-3node.yml exec -T \
    -e TSDB_REFRESH_TOKEN="$REFRESH" node0 /app/zigtsdb-cli license; echo "exit $?"
$ curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $ACCESS2" "$Q"

{"error":"revoked"}
auth: the refresh token was rejected (RefreshRejected) — it may have been revoked or expired. …
exit 1
200

Lifetimes and clock skew

The lower bound is one second above the 300 s clock-skew grace, because a lifetime at or below the grace is meaningless; the upper bound is 30 days, because a year-long "short-lived" token is the problem short lifetimes exist to solve. A ttl_s in a mint request above the configured lifetime is clamped, not refused.

Time claims are checked with 300 s of tolerated clock skew: a token is rejected as expired only once now > exp + 300, and as not yet valid when nbf (or a future-dated iat) is more than 300 s ahead. The grace exists so an NTP correction cannot cause a cluster-wide authentication outage; the cost is that a token stays usable for up to 300 s past its stated expiry, which is why short lifetimes matter.

Through the dispatcher

The dispatcher has a credential of its own, TSDB_DISPATCHER_AUTH_TOKEN, which authenticates a caller to the dispatcher; writes need only that, because the dispatcher forwards them to the nodes on its internal scope. A query is different: it is answered by a node's own /query, which checks the caller's JWT, so the dispatcher forwards the Authorization header it received verbatim — send the cluster token on queries through the dispatcher. A node's 401 or 403 on a query is the caller's answer: the dispatcher returns it as its own status, never a 200 or a retryable 503, and does not fail over to another replica. The /auth/* routes are node routes; the dispatcher does not serve them.

Rotate the signing secret with no downtime

A node signs with TSDB_AUTH_SECRET and accepts tokens signed with either it or TSDB_AUTH_SECRET_NEXT, the outgoing secret. Three phases, each a rolling restart of every node:

1. Introduce. TSDB_AUTH_SECRET=<new> TSDB_AUTH_SECRET_NEXT=<old>. New tokens are signed with the new secret; everything clients still hold keeps working. Restart node by node.
2. Wait at least TSDB_AUTH_REFRESH_TTL_S + 300 s (the clock-skew grace): by then every refresh token minted under the old secret has expired or been exchanged for an access token signed with the new one. Clients that hold a static TSDB_AUTH_TOKEN must be re-issued in this window — a static token has no refresh path.
3. Retire. Drop TSDB_AUTH_SECRET_NEXT, restart node by node. Anything still signed with the old secret is refused as Invalid token signature.

TSDB_AUTH_SECRET_NEXT without TSDB_AUTH_SECRET refuses to start: an overlap needs an operator holding both. A revocation is keyed by the token's jti, not by the secret, so a refresh token revoked during the overlap stays revoked after it. A bootstrapped single node has no overlap of its own; moving it to an operator secret is a repo runbook (below).

For the quickstart cluster, phase 1 is this overlay, saved as docker-compose.ha-3node-rotate.override.yml. It reads the new value from TSDB_AUTH_SECRET_NEW in .env and demotes the current one to _NEXT:

# Secret rotation, phase 1 (introduce): every node signs with the new secret
# and still accepts tokens signed with the old one. Use it with the base file,
# one node at a time:
#
#   docker compose -f docker-compose.ha-3node.yml \
#                  -f docker-compose.ha-3node-rotate.override.yml up -d --no-deps node0

x-rotate: &rotate
  TSDB_AUTH_SECRET: ${TSDB_AUTH_SECRET_NEW:?set TSDB_AUTH_SECRET_NEW in .env}
  TSDB_AUTH_SECRET_NEXT: ${TSDB_AUTH_SECRET:?set TSDB_AUTH_SECRET in .env}

services:
  node0:
    environment: *rotate
  node1:
    environment: *rotate
  node2:
    environment: *rotate

Generate the new secret and roll the overlay through the nodes, waiting for each to answer before the next. Tokens signed with the old secret — $TOKEN among them — keep working throughout:

$ printf 'TSDB_AUTH_SECRET_NEW=%s\n' "$(openssl rand -hex 32)" >> .env
$ for n in node0 node1 node2; do
    docker compose -f docker-compose.ha-3node.yml \
      -f docker-compose.ha-3node-rotate.override.yml up -d --no-deps "$n"
    until curl -fsS -m 3 "http://localhost:$((8080 + ${n#node}))/health" >/dev/null 2>&1; do sleep 2; done
  done

After the wait, retire the old secret: make the new value TSDB_AUTH_SECRET in .env and roll the nodes once more, this time without the overlay, so nothing is set as _NEXT:

$ NEW=$(grep '^TSDB_AUTH_SECRET_NEW=' .env | cut -d= -f2-)
$ { grep -vE '^TSDB_AUTH_SECRET(_NEW)?=' .env; printf 'TSDB_AUTH_SECRET=%s\n' "$NEW"; } > .env.retire
$ mv .env.retire .env && chmod 600 .env
$ for n in node0 node1 node2; do
    docker compose -f docker-compose.ha-3node.yml up -d --no-deps "$n"
    until curl -fsS -m 3 "http://localhost:$((8080 + ${n#node}))/health" >/dev/null 2>&1; do sleep 2; done
  done

A request with the old $TOKEN now answers 401 Invalid token signature; mint a new admin token from the new secret the same way as the first. Tokens minted during the overlap are signed with the new secret and are unaffected.

Rotate the scoped internal tokens

The secure overlay gives the nodes four scoped bearer tokens — raft_peer, cluster_admin, dispatcher, block_transfer — each gating one internal surface. A node accepts two values per scope during a rotation: the file named by TSDB_<SCOPE>_TOKEN_FILE and the one named by TSDB_<SCOPE>_TOKEN_NEXT_FILE (TSDB_RAFT_PEER_TOKEN_NEXT_FILE, TSDB_CLUSTER_ADMIN_TOKEN_NEXT_FILE, TSDB_DISPATCHER_TOKEN_NEXT_FILE, TSDB_BLOCK_TRANSFER_TOKEN_NEXT_FILE). The order matters — receivers first, senders second, retire last:

1. Write the next value the way the generator wrote the first (openssl rand -hex 32 > secure/tokens/<scope>.next.token).
2. Set TSDB_<SCOPE>_TOKEN_NEXT_FILE on every node and restart them; the nodes now accept both values.
3. Switch every sender to the new value — the other nodes' outbound use of the scope, and for dispatcher the dispatcher's own forward token — and restart them.
4. Move the new value into TSDB_<SCOPE>_TOKEN_FILE, unset the NEXT file, restart.

In the secure overlay, step 2 is one line per scope under x-secure-env, for example:

x-secure-env: &secure-env
  TSDB_DISPATCHER_TOKEN_FILE: /run/tsdb-secrets/dispatcher.token
  TSDB_DISPATCHER_TOKEN_NEXT_FILE: /run/tsdb-secrets/dispatcher.next.token

Read more

The ZigTSDB repository's docs/SECURITY.md holds the full credential inventory, the move from a bootstrapped secret to an operator secret, certificate rotation and the compromise-response order; docs/HTTP_API.md is the complete endpoint reference. Questions go to contact@anzaran.com.