Dispatcher

What it is

A cluster is complete without a dispatcher. Consensus runs node to node; a client can write to any node and follow its leader_url hint, or turn on write forwarding so any node accepts the write. The dispatcher exists for the cases where that is not enough:

• One address. Clients see a single endpoint instead of a node list and leader hints.
• Fan-in off the storage nodes. Heavy client concurrency lands on the dispatcher instead of competing with storage for the same CPU.
• Subsetted clusters. When each node holds only some of the groups, no single node can answer every query; the dispatcher routes each query to a replica that holds the metric's group.

It keeps no data and no state beyond its view of the cluster's topology, which it refreshes from the nodes at most once per second and adopts only when the epoch moves forward. Losing a dispatcher loses nothing; start another.

Running it

The dispatcher is the same image as the nodes, started with a different command:

$ docker run --rm -p 8090:8090 \
    -e STORAGE_NODES=http://node0:8080,http://node1:8080,http://node2:8080 \
    -e TSDB_DISPATCHER_FORWARD_TOKEN="$TSDB_RAFT_ADMIN_TOKEN" \
    ghcr.io/samisoftb/zigtsdb:1.5.0 \
    /app/zigtsdb-dispatcher --port 8090

The binary is /app/zigtsdb-dispatcher. Its default port is 8081; --port (or -p) changes it, and --version (or -V) prints the version and exits. The compose files in this documentation put it on 8090 so it does not collide with the three nodes' host ports. The quickstart gives the compose override that adds it to the 3-node cluster.

Configuration

Everything is an environment variable. A _FILE variant, where listed, reads the value from a file (trimmed of surrounding whitespace, 1–512 bytes) and wins over the plain variable.

Routes

Paths are matched as prefixes in this order. token means the route requires TSDB_DISPATCHER_AUTH_TOKEN when one is set; open routes never require it.

A path that matches no route answers 404 {"error":"not_found"}; a method the route does not accept answers 405 with an Allow header. The unauthenticated scrape surface discloses nothing about the cluster: the node label in the metrics is the index into STORAGE_NODES, never a URL.

Writes

Send line protocol to /ingest (or /dispatch/ingest), one point per line, at most 2 MiB and 50 000 points per request. The dispatcher parses every line, groups the points by the group their metric maps to, and forwards one batch per group to that group's leader — following a leader hint once if the node it asked has stepped down. The response tells you exactly what happened to the input:

The two failure classes call for different remedies. 400 and 413 mean the client must change the body before retrying — a partially usable body is never partially ingested, so nothing was stored. 503 means the same body is worth retrying: the input was valid and a node could not take it. A point is therefore either stored or named in the response, never silently absent.

The dispatcher re-serialises each point as metric value=<float> <timestamp>. Tags are part of the metric token and survive; any field other than value= is discarded.

A node that stops answering trips a circuit breaker after three consecutive failures and is skipped for ten seconds before it is tried again; while the breaker is open, points routed to that node are the dropped count in a 503.

Queries

GET /query?zql=… takes the same ZQL as a node. The dispatcher reads the first metric in the statement, resolves its group, and sends the query — body unchanged — to one replica of that group, walking the replica list in placement order from the current leader and failing over to the next replica if one does not answer. A statement that JOINs several metrics goes whole to the node owning the first metric.

There is no fan-out and no merge, by design: writes are replicated by consensus, so every replica of a group holds the same data, and summing across replicas would multiply every result by the replication factor. If every replica of the group is down, the answer is 503 {"error":"partial_result","failed_nodes":F,"responded":0}.

Observability

/health and /stats are the two counters most people need: received is points accepted from clients, forwarded is points a node acknowledged. /metrics/prometheus exposes the full set:

dispatcher_requests_total
dispatcher_forwarded_total
dispatcher_route_epoch
dispatcher_topology_epoch
dispatcher_topology_groups
dispatcher_group_routed_points_total{group="…"}
dispatcher_group_leader{group="…",node="…"}
dispatcher_route_refresh_failures_total
dispatcher_leader_hint_retries_total
dispatcher_circuit_breaker_state{node="…"}
dispatcher_circuit_breaker_failures_total{node="…"}

Two of these are worth an alert. dispatcher_route_refresh_failures_total rising means the dispatcher cannot read the cluster's topology from any node and is routing on a stale view; dispatcher_circuit_breaker_state at 1 for a node means that node is being skipped. The breaker transitions are also logged, one line per change.