Documents ZigTSDB v1.5.0 · applies to the 1.5.x line

Get started in 5 minutes

Run Anzaran in Docker, write your first points, and query them with ZQL — no config files, no schema setup.

1. Run a single node

$ docker run -d --name anzaran \
    -p 8080:8080 \
    -e TSDB_WAL_DIR=/data \
    -v anzaran-data:/data \
    ghcr.io/samisoftb/zigtsdb:latest

The volume mount is mandatory: every acked write is fsynced to the write-ahead log on this volume — a container restart loses nothing. (TSDB_WAL_DIR points the WAL at the volume; without it the node starts with a loud warning that acknowledged writes are not durable.) The restart contract — ingest, restart, every acknowledged point still there and countable — is pinned by an automated chaos test in every release.

Production notes. The image runs as a non-root user (uid 10001) and ships x86_64. The server listens on all interfaces with authentication OFF by default — set TSDB_AUTH_SECRET before exposing it beyond localhost. For immutable deploys, pin the release tag (:1.5.0) or, better, the image digest — :latest moves with releases and monthly base-image security refreshes:

ghcr.io/samisoftb/zigtsdb:1.5.0
ghcr.io/samisoftb/zigtsdb@sha256:3b2617b893e55ba0534d85a5bc7a38bf4068c10169445430ba59049a707c438c

Whatever you pull, docker inspect --format='{{index .RepoDigests 0}}' anzaran prints the digest the running container was started from.

Check the node is up:

$ curl http://localhost:8080/health
{"is_healthy":true,"node_id":0,"node_count":1,...,"storage_state":"read_write","disk_state":"ok"}

2. Write your first points

Anzaran ingests InfluxDB-style line protocol, one point per line — <metric>[,<label>=<value>...] value=<float> <timestamp_ns>, with up to 16 labels per line and timestamps in Unix nanoseconds:

$ curl -X POST http://localhost:8080/ingest \
    -d "cpu_usage,host=web-1 value=75.5 1704067200000000000"
{"status":"ok","points":1,"buffered":1,"s3_flushed":0}

3. Run your first query

Queries are ZQL, submitted URL-encoded to GET /query. Scalar results print four decimals by default (&fmt=full for shortest round-trip digits):

# SELECT avg(value) FROM cpu_usage
$ curl "http://localhost:8080/query?zql=SELECT%20avg(value)%20FROM%20cpu_usage"
{"status":"ok","result":75.5000}

The same query through the bundled CLI (shipped in the container image at /app/zigtsdb-cli):

$ docker exec anzaran /app/zigtsdb-cli query "SELECT avg(value) FROM cpu_usage"

4. Optional: enable the S3 cold tier

Tiered storage moves older data into any S3-compatible object store — included on the Free tier, no license required. Recent data stays in memory; your history lives durably in your own bucket at object-storage prices.

Create a bucket

Anzaran does not create the bucket for you — create it first (keep the account default "Block Public Access" ON; Anzaran needs no public access):

$ aws s3 mb s3://my-anzaran-data --region us-east-1

Grant the minimum permissions

Create a dedicated IAM user (or role) for Anzaran with exactly the four actions the server performs — write, read, delete, and list blocks:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AnzaranBlocks",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::my-anzaran-data/*"
    },
    {
      "Sid": "AnzaranList",
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::my-anzaran-data"
    }
  ]
}

No s3:CreateBucket, no s3:*, no public access — least privilege by default.

Point Anzaran at it

$ docker run -d --name anzaran -p 8080:8080 \
    -v anzaran-data:/data -e TSDB_WAL_DIR=/data \
    -e S3_ENDPOINT=https://s3.us-east-1.amazonaws.com \
    -e S3_BUCKET=my-anzaran-data \
    -e AWS_ACCESS_KEY_ID=... -e AWS_SECRET_ACCESS_KEY=... \
    ghcr.io/samisoftb/zigtsdb:latest

Verify by flushing and checking the response counts blocks written to the tier:

$ curl -X POST http://localhost:8080/flush
Local alternative. No AWS account? Garage is a lightweight, actively maintained S3-compatible store that creates the bucket and key for you in single-node mode: docker run -d -p 3900:3900 -e GARAGE_DEFAULT_ACCESS_KEY=... -e GARAGE_DEFAULT_SECRET_KEY=... -e GARAGE_DEFAULT_BUCKET=anzaran garage server --single-node --default-bucket, then point Anzaran at S3_ENDPOINT=http://host.docker.internal:3900. Use an HTTPS endpoint and real credentials in production.

5. Optional: run a 3-node HA cluster

Three nodes replicate every write through Raft, so the cluster keeps serving when one of them is lost. The whole topology is the compose file below — copy it, generate its secrets, and start it. There is nothing to build and nothing else to download; the nodes run the same published image as a single node. It defines four services:

  • garage — a lightweight S3-compatible store for the cold tier (creates its bucket and key on startup, no extra steps)
  • node0 / node1 / node2 — three replicated Anzaran nodes on host ports 8080–8082, each with its own WAL volume, joined into a Raft cluster via TSDB_PEERS

Save it as docker-compose.ha-3node.yml:

name: anzaran-ha

# Three Anzaran nodes in one Raft cluster, plus a small S3-compatible store
# for the cold tier. Every secret comes from .env; the stack does not start
# without them.

x-node: &node
  image: ghcr.io/samisoftb/zigtsdb:1.5.0@sha256:3b2617b893e55ba0534d85a5bc7a38bf4068c10169445430ba59049a707c438c
  restart: unless-stopped
  depends_on:
    garage:
      condition: service_healthy
  networks: [cluster]
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
    interval: 10s
    timeout: 5s
    retries: 3

x-node-env: &node-env
  NODE_COUNT: 3
  TSDB_PEERS: http://node0:8080,http://node1:8080,http://node2:8080
  TSDB_WAL_DIR: /var/lib/zigtsdb/wal
  TSDB_RAFT_ADMIN_TOKEN: ${TSDB_RAFT_ADMIN_TOKEN:?set TSDB_RAFT_ADMIN_TOKEN in .env}
  TSDB_RAFT_GROUPS: "16"
  TSDB_RAFT_REPLICATION_FACTOR: "3"
  S3_ENDPOINT: http://garage:3900
  S3_BUCKET: anzaran
  AWS_ACCESS_KEY_ID: ${GARAGE_ACCESS_KEY:?set GARAGE_ACCESS_KEY in .env}
  AWS_SECRET_ACCESS_KEY: ${GARAGE_SECRET_KEY:?set GARAGE_SECRET_KEY in .env}

services:
  garage:
    image: dxflrs/garage:v2.3.0
    command: ["/garage", "server", "--single-node", "--default-bucket"]
    restart: unless-stopped
    environment:
      GARAGE_DEFAULT_ACCESS_KEY: ${GARAGE_ACCESS_KEY:?set GARAGE_ACCESS_KEY in .env}
      GARAGE_DEFAULT_SECRET_KEY: ${GARAGE_SECRET_KEY:?set GARAGE_SECRET_KEY in .env}
      GARAGE_DEFAULT_BUCKET: anzaran
    configs:
      - source: garage_toml
        target: /etc/garage.toml
    volumes:
      - garage_meta:/var/lib/garage/meta
      - garage_data:/var/lib/garage/data
    networks: [cluster]
    healthcheck:
      test: ["CMD", "/garage", "status"]
      interval: 5s
      timeout: 5s
      retries: 20

  node0:
    <<: *node
    environment:
      <<: *node-env
      NODE_ID: 0
    ports: ["8080:8080"]
    volumes: [node0_wal:/var/lib/zigtsdb/wal]

  node1:
    <<: *node
    environment:
      <<: *node-env
      NODE_ID: 1
    ports: ["8081:8080"]
    volumes: [node1_wal:/var/lib/zigtsdb/wal]

  node2:
    <<: *node
    environment:
      <<: *node-env
      NODE_ID: 2
    ports: ["8082:8080"]
    volumes: [node2_wal:/var/lib/zigtsdb/wal]

configs:
  garage_toml:
    content: |
      metadata_dir = "/var/lib/garage/meta"
      data_dir = "/var/lib/garage/data"
      db_engine = "sqlite"
      replication_factor = 1
      rpc_bind_addr = "[::]:3901"
      rpc_secret = "${GARAGE_RPC_SECRET:?set GARAGE_RPC_SECRET in .env}"
      [s3_api]
      api_bind_addr = "[::]:3900"
      s3_region = "us-east-1"
      [admin]
      api_bind_addr = "[::]:3903"

networks:
  cluster:
    driver: bridge

volumes:
  garage_meta:
  garage_data:
  node0_wal:
  node1_wal:
  node2_wal:

It reads four secrets from a .env file beside it. Generate them once — they are yours, and nothing ships with a default:

$ cat > .env <<'EOF'
TSDB_RAFT_ADMIN_TOKEN=$(openssl rand -hex 32)
GARAGE_ACCESS_KEY=GK$(openssl rand -hex 12)
GARAGE_SECRET_KEY=$(openssl rand -hex 32)
GARAGE_RPC_SECRET=$(openssl rand -hex 32)
EOF
$ chmod 600 .env

$ docker compose -f docker-compose.ha-3node.yml up -d

# Every node answers, and any node reports on the whole cluster
$ curl http://localhost:8080/health
$ curl http://localhost:8081/health
$ curl http://localhost:8082/health
$ curl http://localhost:8080/cluster/status
No secret, no start. Every credential in the file is required, not defaulted, so the stack refuses to come up rather than running on a value that was published in a document. TSDB_RAFT_ADMIN_TOKEN is the one to guard: membership changes and migrations need it, and internal endpoints fail closed without it.

Writes go to the leader of the metric's shard group. A node that does not lead answers 503 {"error":"not_leader","leader_url":"…"} — retry against the hint, which is what a client library does for you. To let clients target any node instead, add this second file as docker-compose.ha-3node-forward.override.yml: each node then relays a write it does not lead to the node that does.

# Write forwarding: a node that does not lead the metric's group relays the
# write to the leader instead of answering 503, so a client can target any
# node. Use it together with the base file:
#
#   docker compose -f docker-compose.ha-3node.yml \
#                  -f docker-compose.ha-3node-forward.override.yml up -d

services:
  node0:
    environment:
      TSDB_NODE_FORWARD_WRITES: "1"
  node1:
    environment:
      TSDB_NODE_FORWARD_WRITES: "1"
  node2:
    environment:
      TSDB_NODE_FORWARD_WRITES: "1"
$ docker compose -f docker-compose.ha-3node.yml \
    -f docker-compose.ha-3node-forward.override.yml up -d

Secure overlay for production networks

On an untrusted network, put an Envoy sidecar in front of every node. The database then binds plaintext to loopback inside the sidecar's network namespace and has no reachable address of its own: clients speak mTLS to the sidecar on port 9443 (host ports 18443–18445), and peer traffic leaves through the sidecar too, so no hop between nodes is ever in the clear. The node trusts the identity its own sidecar stamps on a request, and nothing a caller sends.

Three files. First the material — a private CA for the cluster, one certificate per node, one for your own tools, and the four scoped bearer tokens the nodes read at boot. Save it as make-secure-material.sh and run it once:

make-secure-material.sh 
#!/usr/bin/env bash
# Generate the material the secure overlay mounts: a private CA, one server
# certificate per node, one client certificate for your own tools, and the
# four scoped bearer tokens the nodes read at boot.
#
# Everything lands in ./secure, which is created mode 0700. Re-running
# replaces the material and invalidates certificates already issued.
set -euo pipefail

DIR="${ANZARAN_SECURE_DIR:-./secure}"
DAYS="${ANZARAN_CERT_DAYS:-365}"

mkdir -p "${DIR}"/certs "${DIR}"/tokens "${DIR}"/node0 "${DIR}"/node1 "${DIR}"/node2
chmod 0700 "${DIR}"

# Scoped bearer tokens. Each gates one internal surface; the node fails to
# boot if a file is missing, so none of them can be silently skipped.
for scope in raft_peer cluster_admin dispatcher block_transfer; do
  openssl rand -hex 32 > "${DIR}/tokens/${scope}.token"
done

# A private CA for this cluster. keyUsage is explicit: strict X.509
# verifiers reject a CA without it.
openssl req -x509 -newkey rsa:2048 -days "${DAYS}" -nodes \
  -keyout "${DIR}/certs/ca.key" -out "${DIR}/certs/ca.crt" \
  -subj "/CN=anzaran-cluster-ca" \
  -addext "basicConstraints=critical,CA:TRUE" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" 2>/dev/null

# One leaf per node, plus a client leaf for curl and your client libraries.
# The URI SAN is the identity the node records in its audit line.
for name in node0 node1 node2 client; do
  cat > "${DIR}/certs/${name}.cnf" <<CNF
[req]
distinguished_name=req
req_extensions=v3_req
prompt=no
[v3_req]
basicConstraints=CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=DNS:${name},DNS:localhost,DNS:node0,DNS:node1,DNS:node2,URI:spiffe://anzaran/${name}
CNF
  openssl req -newkey rsa:2048 -nodes \
    -keyout "${DIR}/certs/${name}.key" -out "${DIR}/certs/${name}.csr" \
    -subj "/CN=${name}" -config "${DIR}/certs/${name}.cnf" 2>/dev/null
  openssl x509 -req -days "${DAYS}" -in "${DIR}/certs/${name}.csr" \
    -CA "${DIR}/certs/ca.crt" -CAkey "${DIR}/certs/ca.key" -CAcreateserial \
    -out "${DIR}/certs/${name}.crt" -extensions v3_req \
    -extfile "${DIR}/certs/${name}.cnf" 2>/dev/null
done

# Each sidecar mounts only its own leaf, at a fixed path, so all three share
# one envoy.yaml.
for n in node0 node1 node2; do
  cp "${DIR}/certs/${n}.crt" "${DIR}/${n}/tls.crt"
  cp "${DIR}/certs/${n}.key" "${DIR}/${n}/tls.key"
done

rm -f "${DIR}"/certs/*.csr "${DIR}"/certs/*.cnf "${DIR}"/certs/*.srl

# The CA key and your client key stay private to you - nothing in a
# container reads them. The material the containers DO mount has to be
# readable by the unprivileged users they run as, so it is world-readable
# inside a directory that is not. Use your own PKI and a secrets manager
# for anything you cannot afford to regenerate.
chmod 0600 "${DIR}"/certs/*.key
chmod 0644 "${DIR}"/certs/*.crt "${DIR}"/tokens/* "${DIR}"/node?/*
chmod 0755 "${DIR}"/certs "${DIR}"/tokens "${DIR}"/node0 "${DIR}"/node1 "${DIR}"/node2

echo "Wrote ${DIR}: a CA, 3 node certificates, a client certificate, 4 tokens."

Then the sidecar configuration, saved as envoy.yaml. One file serves all three nodes: each sidecar mounts only its own key pair, so the configuration never names a node.

envoy.yaml 
# One sidecar configuration, shared by all three nodes. Each sidecar mounts
# only its own key pair, at /etc/envoy/self, so the file never names a node.
static_resources:
  listeners:
  # Client-facing mTLS. A caller must present a certificate this cluster's
  # CA signed; the sidecar then tells the node who it was.
  - name: ingress_mtls
    address:
      socket_address: { address: 0.0.0.0, port_value: 9443 }
    filter_chains:
    - transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
          require_client_certificate: true
          common_tls_context:
            tls_certificates:
            - certificate_chain: { filename: /etc/envoy/self/tls.crt }
              private_key: { filename: /etc/envoy/self/tls.key }
            validation_context:
              trusted_ca: { filename: /etc/envoy/certs/ca.crt }
      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress
          access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
              path: /dev/stdout
              log_format:
                text_format_source:
                  inline_string: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(:PATH)%\" %RESPONSE_CODE% san=\"%DOWNSTREAM_PEER_URI_SAN%\"\n"
          route_config:
            name: local_app
            virtual_hosts:
            - name: app
              domains: ["*"]
              # The identity headers are the node's trust boundary, so a
              # caller is never allowed to supply its own: strip whatever
              # arrived, then stamp what the handshake actually proved.
              request_headers_to_remove: ["x-tsdb-mtls-verified", "x-tsdb-mtls-san"]
              request_headers_to_add:
              - header: { key: "X-TSDB-MTLS-Verified", value: "true" }
              - header: { key: "X-TSDB-MTLS-SAN", value: "%DOWNSTREAM_PEER_URI_SAN%" }
              routes:
              - match: { prefix: "/" }
                route:
                  cluster: local_app
                  # A pooled connection the node is closing at that moment
                  # fails before the request is seen. It never reached the
                  # database, so it is safe to retry on a fresh one.
                  retry_policy:
                    retry_on: "reset,connect-failure"
                    num_retries: 2
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  # The node's own outbound calls to its peers leave through here and go
  # back out as mTLS, so no hop between nodes is ever in the clear.
  - name: egress_plaintext
    address:
      socket_address: { address: 127.0.0.1, port_value: 10080 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: egress
          route_config:
            name: peer_routes
            virtual_hosts:
            - name: peers
              domains: ["node0:9443", "node1:9443", "node2:9443", "node0", "node1", "node2"]
              routes:
              - match: { prefix: "/" }
                route: { cluster_header: ":authority" }
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: local_app
    connect_timeout: 0.25s
    type: STATIC
    # Drop pooled upstream connections before the node's own idle timeout
    # closes them underneath us.
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        common_http_protocol_options:
          idle_timeout: 2s
        explicit_http_config:
          http_protocol_options: {}
    load_assignment:
      cluster_name: local_app
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: 127.0.0.1, port_value: 8080 }
  - name: node0
    connect_timeout: 0.25s
    type: STRICT_DNS
    transport_socket: &peer_tls
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        common_tls_context:
          tls_certificates:
          - certificate_chain: { filename: /etc/envoy/self/tls.crt }
            private_key: { filename: /etc/envoy/self/tls.key }
          validation_context:
            trusted_ca: { filename: /etc/envoy/certs/ca.crt }
    load_assignment:
      cluster_name: node0
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: node0, port_value: 9443 }
  - name: node1
    connect_timeout: 0.25s
    type: STRICT_DNS
    transport_socket: *peer_tls
    load_assignment:
      cluster_name: node1
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: node1, port_value: 9443 }
  - name: node2
    connect_timeout: 0.25s
    type: STRICT_DNS
    transport_socket: *peer_tls
    load_assignment:
      cluster_name: node2
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: node2, port_value: 9443 }

And the overlay itself, saved as docker-compose.ha-3node-secure.override.yml:

# Secure overlay: an Envoy sidecar in front of every node. Clients speak
# mTLS to the sidecar on 9443; the database itself binds plaintext to
# loopback only, inside the sidecar's network namespace, so it has no
# reachable address of its own. Peer traffic leaves through the sidecar
# too, so every hop between nodes is encrypted and authenticated.
#
# Run ./make-secure-material.sh first, then:
#
#   docker compose -f docker-compose.ha-3node.yml \
#                  -f docker-compose.ha-3node-secure.override.yml up -d

x-secure-env: &secure-env
  TSDB_BIND_ADDR: 127.0.0.1
  TSDB_SECURE_MODE: required
  TSDB_INTERNAL_EGRESS_PROXY: http://127.0.0.1:10080
  TSDB_PEERS: https://node0:9443,https://node1:9443,https://node2:9443
  TSDB_RAFT_PEER_TOKEN_FILE: /run/tsdb-secrets/raft_peer.token
  TSDB_CLUSTER_ADMIN_TOKEN_FILE: /run/tsdb-secrets/cluster_admin.token
  TSDB_DISPATCHER_TOKEN_FILE: /run/tsdb-secrets/dispatcher.token
  TSDB_BLOCK_TRANSFER_TOKEN_FILE: /run/tsdb-secrets/block_transfer.token
  TSDB_ADMIN_RATE_LIMIT_RPS: "1"
  TSDB_INTERNAL_RATE_LIMIT_RPS: "100"

x-sidecar: &sidecar
  image: envoyproxy/envoy:v1.37.0@sha256:d9b4a70739d92b3e28cd407f106b0e90d55df453d7d87773efd22b4429777fe8
  restart: unless-stopped
  command: ["envoy", "-c", "/etc/envoy/envoy.yaml", "--log-level", "warning"]

services:
  node0:
    ports: !override ["18443:9443"]
    environment: *secure-env
    volumes:
      - node0_wal:/var/lib/zigtsdb/wal
      - ./secure/tokens:/run/tsdb-secrets:ro

  node1:
    ports: !override ["18444:9443"]
    environment: *secure-env
    volumes:
      - node1_wal:/var/lib/zigtsdb/wal
      - ./secure/tokens:/run/tsdb-secrets:ro

  node2:
    ports: !override ["18445:9443"]
    environment: *secure-env
    volumes:
      - node2_wal:/var/lib/zigtsdb/wal
      - ./secure/tokens:/run/tsdb-secrets:ro

  node0-envoy:
    <<: *sidecar
    network_mode: service:node0
    depends_on: [node0]
    volumes:
      - ./envoy.yaml:/etc/envoy/envoy.yaml:ro
      - ./secure/node0:/etc/envoy/self:ro
      - ./secure/certs:/etc/envoy/certs:ro

  node1-envoy:
    <<: *sidecar
    network_mode: service:node1
    depends_on: [node1]
    volumes:
      - ./envoy.yaml:/etc/envoy/envoy.yaml:ro
      - ./secure/node1:/etc/envoy/self:ro
      - ./secure/certs:/etc/envoy/certs:ro

  node2-envoy:
    <<: *sidecar
    network_mode: service:node2
    depends_on: [node2]
    volumes:
      - ./envoy.yaml:/etc/envoy/envoy.yaml:ro
      - ./secure/node2:/etc/envoy/self:ro
      - ./secure/certs:/etc/envoy/certs:ro
$ ./make-secure-material.sh
$ docker compose -f docker-compose.ha-3node.yml \
    -f docker-compose.ha-3node-secure.override.yml up -d

Clients then write through any node's sidecar. Presenting a certificate this cluster's CA signed is required for every request; POST /dispatch/ingest/simd additionally requires the dispatcher-scoped bearer token, and answers 401 without it:

POST https://localhost:18443/dispatch/ingest/simd
Authorization: Bearer <contents of secure/tokens/dispatcher.token>
(client certificate presented at the TLS handshake)

cpu_usage,host=web-1 value=75.5 1704067200000000000

A node that does not lead the metric's group answers 503 {"error":"not_leader","leader_url":"https://nodeN:9443"} — follow the hint — or, with the write-forwarding overlay above, relays the write over the egress listener to the leader's sidecar, so every hop stays mTLS and the leader's audit line records the relaying node's identity.

About the generated material. The tokens and the private key each sidecar loads have to be readable by the unprivileged users the containers run as, so make-secure-material.sh leaves them world-readable inside a directory that is not. The CA key and your client key stay private to you. For a deployment you cannot afford to regenerate, issue the certificates from your own PKI and mount the tokens from your secrets manager instead.
One limit. zigtsdb-cli cannot present a client certificate — use curl or a client library against the sidecar.

6. Optional: put a dispatcher in front of the cluster

The cluster above is complete without a dispatcher: clients write to a node, follow its leader_url hint or use the forwarding overlay, and query any node. A dispatcher is a stateless router you can put in front of it so that clients have one address, writes are split by metric and sent to the right group leader, and queries are routed to a replica that holds the data. It is the same image as the nodes, started with a different command. The full reference is on the Dispatcher page.

Save this as docker-compose.ha-3node-dispatcher.override.yml next to the cluster file:

services:
  dispatcher:
    image: ghcr.io/samisoftb/zigtsdb:1.5.0@sha256:3b2617b893e55ba0534d85a5bc7a38bf4068c10169445430ba59049a707c438c
    command: ["/app/zigtsdb-dispatcher", "--port", "8090"]
    ports:
      - "8090:8090"
    environment:
      STORAGE_NODES: http://node0:8080,http://node1:8080,http://node2:8080
      # The nodes accept forwarded writes only with their internal credential,
      # so the dispatcher presents the same token the nodes were given.
      TSDB_DISPATCHER_FORWARD_TOKEN: ${TSDB_RAFT_ADMIN_TOKEN:?set TSDB_RAFT_ADMIN_TOKEN in .env}
    networks: [cluster]
    depends_on:
      node0: {condition: service_healthy}
      node1: {condition: service_healthy}
      node2: {condition: service_healthy}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8090/health"]
      interval: 5s
      timeout: 3s
      retries: 5
$ docker compose -f docker-compose.ha-3node.yml \
    -f docker-compose.ha-3node-dispatcher.override.yml up -d

Clients then write to http://localhost:8090/ingest and query http://localhost:8090/query; the dispatcher follows leader changes on its own. Every write is all-or-nothing on its input — one unusable line refuses the whole body and nothing is stored:

$ curl -s -X POST http://localhost:8090/ingest \
    -d 'cpu_usage,host=web-1 value=75.5 1704067200000000000'
{"status":"ok","total":1}

$ curl -s -X POST http://localhost:8090/ingest \
    -d $'cpu_usage,host=web-1 value=75.5 1704067200000000000\nnot line protocol'
{"error":"malformed_line_protocol","rejected":1,"stored":0}
Guard the dispatcher like a node. As published here it listens in the clear on port 8090 with no client authentication, holding a credential that can write to the whole cluster — fine on a private compose network, not on anything reachable. Set TSDB_DISPATCHER_AUTH_TOKEN to require a bearer token from clients, and TSDB_DISPATCHER_SECURE_MODE=required to make it refuse to start unless a client token, a forward token and a non-wildcard bind address are all configured. Running it behind the mTLS sidecars of the secure overlay is not covered on this page.

7. Next steps