mirror of
https://github.com/block/buzz.git
synced 2026-11-07 09:52:03 +00:00
27ab6a085a
## Problem
Each relay node runs one consumer for events that other nodes publish to
Redis. It delivered them one at a time and awaited the access gate
(visibility and membership lookups) for each event. Under load it fell
more than 4,096 events behind the pub/sub broadcast. Each `Lagged(n)`
then skipped `n` events for good: nothing replayed them, and plain
subscribers were never told.
In a 15-node load test, about 50% of cross-node events were skipped at
10x load and about 75% at 13x. Throughput at 13x was also *lower* than
at 10x. Loadgen didn't show this, because it only times events that
arrive.
## Change
- **Concurrent delivery.** A dispatcher reads the broadcast and only
routes. It drops local echoes and queues each event on one of 16 workers
by `(community, topic)`. Each worker delivers its queue in order, so a
channel's events keep their order while different channels deliver
concurrently. `handlers/multinode_fanout.rs`
- **Known loss closes the affected connections.** An event is lost when
its worker's bounded queue (1,024 events) is full, or when it isn't
delivered within 2 s of this node reading it from Redis, whether it
waited in the broadcast, behind a backlog, or in a stalled delivery.
`ChannelEvent` now carries `received_at`, stamped by the pub/sub
subscriber. For a lost event, the relay sends `CLOSED "resync: live
delivery lagged"` to every sync the event would have reached (NIP-AR
Sync rule 5). It also closes every connection the event would have
reached, so those clients reconnect and catch up from their cursor.
Matching runs before the access gate, so a recipient that would have
been filtered out may be closed too. That costs a reconnect, never a
leak.
- **A lost ephemeral event is only counted.** Presence, typing and
observer frames are never stored, so no reconnect can replay them.
Losing one fences and closes nothing; it is still never delivered past
its deadline. Without this, one stale global presence event would close
nearly every mobile connection in the community, since every mobile
client subscribes to presence.
- **Unknown loss closes every subscribed connection.** If the dispatcher
itself lags the broadcast, it can't know which events it skipped. It
resyncs every sync (unchanged) and closes every connection that has a
subscription.
- **Immediate fence.** Each affected connection is fenced as soon as the
loss is detected. The live fan-out send path (`send_to_text_bytes`,
local and cross-node) delivers it nothing from then on. The fence is
written under the connection map's write guard, and the send path checks
it and queues its frame under one read guard, so no frame slips in
between the check and the send.
- **Why 2 s.** The mobile client resumes from the newest `created_at` it
received, minus 5 s of slack. The desktop app (block/buzz-app) resumes
960 s before its connection last heard the relay, which already covers
the full 900 s `created_at` drift. While an event waits, local fan-out
and other workers keep moving that cursor. The fence stops the cursor,
and the deadline means no withheld event is more than 2 s older than the
fence, so a reconnect replays it. Without the deadline, an event stuck
behind a long backlog could be withheld past the slack and never
replayed.
- **Paced closes.** Only the close is delayed. One closer task closes at
most 50 connections per 10 ms (5,000/s per node). A drop in a hot
channel therefore can't become a reconnect storm (Review-Proven Rule 9).
- `fan_out_pubsub_event` is now split into `take_local_echo` and
`deliver_pubsub_event`, so the dispatcher can check echoes before
queueing. Its behavior is unchanged.
- **Limits (mobile):** replay works while the 2 s deadline, plus the
writer's clock skew and its publish delay, stays under the client's 5 s
slack. The relay accepts `created_at` up to 15 min off, so a writer
clock more than about 3 s slow can still put an event behind the cursor.
An ordinary reconnect has the same limit today; closing it needs a
client rewind instruction, which is out of scope here. The desktop app's
960 s resume window has no such gap. A dispatcher that falls behind
without overflowing the broadcast still never delivers a late event, but
it detects the loss, and fences, late. Events skipped by dispatcher lag
are never seen. Neither case is bounded by the deadline, which is a
cutoff for attempting delivery, not a guaranteed fence time.
- **What a load-test rerun should watch:** under sustained overload the
deadline turns backlog into steady closes (up to 5,000/s per node), and
reconnecting clients replay from the same database pool. Watch
`buzz_multinode_fanout_dropped_total{reason="stale"}`,
`buzz_multinode_fanout_lag_closed_total` and the reconnect REQ rate next
to pool wait. On dispatcher lag, the dispatcher scans the subscription
registry and fences each subscribed connection synchronously, so also
watch how long it stays behind after a `Lagged`.
Metrics:
- `buzz_multinode_fanout_lag_total` (dispatcher lag) and
`buzz_multinode_fanout_total` (delivered) keep their meaning.
- `buzz_multinode_fanout_dropped_total{reason}` counts lost events:
`queue_full`, or `stale` for one not delivered by its deadline. A lost
ephemeral event counts as `ephemeral` instead and closes nothing.
- `buzz_multinode_fanout_lag_closed_total` counts close attempts for a
loss, one batch at a time.
## Rollout
- Staging first. The load-test rerun at 1x, 10x and 13x happens after
merge, with the staging load tests planned once the current relay
changes land. At 1x, expect zero `dropped_total` for stored kinds and
zero `lag_closed_total`. Report delivered %, `dropped_total{reason}`,
`lag_closed_total`, reconnect REQ rate and pool wait.
- Then one production cluster before the rest.
- **Rollback trigger:** `lag_closed_total` climbing steadily, or
reconnect REQs pushing pool wait up, at normal load. Rollback is a plain
revert and redeploy: there are no schema, config or wire changes.
## Ordering
Order is kept within a `(community, topic)`. Events on different topics,
for example a channel event and a global event, can now reach one
subscriber in either order. Before this change they arrived in Redis
arrival order. Nothing in the relay depends on order across topics: the
access gate re-reads membership and visibility per event. Clients order
by `created_at`.
## What got simpler / known debt
- The fan-out loop in `main.rs` shrinks to one `spawn` call, and the lag
handling lives next to the delivery code.
- No new config knobs. The worker count and queue size are constants
until a load test shows they need tuning.
- **Known debt, not in this PR:** a Redis pub/sub reconnect can still
lose events silently for plain subscribers. Syncs are already resynced
in that case (`artifact_sync::watch_link`), but ordinary subscriptions
are not.
## Tests
`handlers::multinode_fanout::tests`, each checked with a mutant on Blox:
- events reach their subscriber through the workers, in publish order
- a full worker queue closes exactly the dropped event's recipients
- a lost ephemeral event (full queue or stale) is counted as `ephemeral`
and neither fences nor closes, while a stored event lost on the same
queue still closes
- a local echo on a full queue closes nothing
- dispatcher lag closes every subscribed connection
- after a known loss or dispatcher lag, a recipient gets no newer live
event while it waits to be closed
- the fence also withholds relay-local fan-out
- an event that waited out its deadline before dispatch is lost, never
delivered
- an event behind a backlog is delivered inside its deadline, and past
it is lost (its recipient fenced and queued for close), never delivered
late (paused clock)
- a delivery that stalls on a database that never answers is cut off at
the deadline and handled as lost
- the closer cancels in paced batches (paused clock), and counts each
batch as it closes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>