Developers
Extend the cluster protocol
A cluster is nodes of different ages, some of them pods that come and go, talking over one authenticated stream. Anything you add to that conversation has to survive an older node, a missing node, a pod, and a node that lies. This page is how.
Background for users is in Build a cluster. Upgrading a cluster one node at a time is in Upgrades, backups, and moving in.
How a read works
- The node that was asked is the entry node. For a federated read it asks each reachable peer over the cluster stream and merges the answers with its own.
- The request is one JSON value of the
Readenum incrates/wrendns/src/federated.rs, sent with kindapi.read. The peer decodes it, answers from its own data inanswer(), and replies with JSON. - A peer that doesn't answer within two seconds, or errors, is left out and listed in the result's
missingNodes: a dead node costs a read two seconds, never more (CLU-002). - Reads run on blocking threads, never on the DNS path (DNS never depends on the cluster).
The rules
| Rule | Why |
|---|---|
Old nodes must not answer wrongly. If adding a field changes what a read means, make it a new Read variant, with a doc comment saying "older peers don't know it". | A peer decodes what it is sent leniently: it ignores a field it does not know. An old node would answer as if you had not asked, and the entry node would merge a wrong answer into a right one. An unknown variant is an error, so the old node is listed as not reached, which is true. |
An answer fits about a mebibyte. One frame is at most 1 MiB (MAX_FRAME in crates/wrendns-cluster/src/wire.rs), and an answer a little under it. A larger answer fails with "answer too large", and the node is listed as missing. | The stream is shared with heartbeats and replication. Cap what you return (a page limit), paginate, or compress; do not assume it fits. |
| An ephemeral member can be asked, but may not change or reveal much. A resolver pod is a member with less trust. Requests that change a node's state, forward configuration writes, or return anything sensitive are refused when the asker is ephemeral (the handler already does this for pauses, cache flushes, maintenance, rollouts, and node removal). | Pods are created by anyone who can schedule one, and they come and go. They get replicated configuration and their own queries, not the users and tokens. |
| Nothing a peer says is trusted. Check sizes, counts, and paths in what it sends. When you report that a node failed, use your own words, not its text. | Another node is another machine. A bug or a compromise there must not become a bug or an injection here. |
A secret in a request is held in memory and never printable. Give the type a Debug that hides it. Don't log it, store it, or put it in an audit entry. | The Read enum derives Debug; whatever prints a read (a log, a panic message) would print the secret. |
| The query path never waits for the cluster. A read has a deadline and a fallback; a write is forwarded or refused, never blocked on. | CLU-004: DNS never depends on anything else. |
Don't change the frames casually. The frames and their versions are the cluster protocol (CLU-010), and versions one apart must work together. Extending Read is not a protocol change; changing a frame is. | A cluster upgraded one node at a time spends days with two versions in it. |
The recipe
- Decide whether it should federate at all. Configuration is replicated from the primary and is the same everywhere; a read that asks each node about its own data (its query log, its cache, its health) is what federates.
- Add the
Readvariant with a doc comment (the requirement, and "older peers don't know it"). - Add a method to the
Backendtrait incrates/wrendns-api/src/lib.rswith a default that returns "unavailable", so a node that is not in a cluster still compiles. Implement it for the node's own data inapi_backend.rs. - Answer it. An arm in
answer()that calls the node's own implementation. - Ask for it. In the
Federatedimplementation, build the reads for the peers in scope, call them (gatherruns them in parallel with the two-second deadline), merge what came back with the local answer, and keep the failures in the missing list. - Gate it. If it changes state or returns anything sensitive, refuse an ephemeral asker in the handler next to the others.
- Bound it. Cap the rows, the bytes, and the time.
- Test it (below), document it, and write the ADR if the choice was not obvious.
The checks
- Unit tests in
crates/wrendns-clusterrun real nodes inside one process over real TLS (net_tests.rs), and the federated merge functions are plain functions you can test directly. - The end-to-end scripts run real processes:
deploy/cluster/e2e.shstarts a primary and a replica and walks the features (federated reads, write forwarding, shipped logs, maintenance), and CI'snativejob also runs the failover, GitOps failover, and node-removal scripts. Add a step for your read there: it is the only test that crosses a real stream with real sizes. - Say what you did not run. A mixed-version cluster is hard to test; if you did not, the task's Done note says so.
- Wait for exactly what you assert. A cluster test waits for the condition it then checks. One test asserted a second condition straight after waiting for the first and failed once under load; the suspected cause is that the second is derived from state that arrives separately.
Worked example: a negated search
The query log's "not" filters (Add an API parameter) are sent to peers inside the existing Queries read. An older peer would have ignored the new flags and answered with ordinary rows. So a search that uses a flag goes out as QueriesNegated, with the same fields, and the live view as TailOpenNegated. A search without a flag still uses the old variants, so a mixed cluster keeps working for everything that existed before. The peer that doesn't know the new variant errors, and shows as not reached.