mirror of
https://github.com/block/buzz.git
synced 2026-11-12 20:31:12 +00:00
1b190d21fc
**Context:** #9302 (merged) stopped the relay from closing connections when cross-node fan-out falls behind. It also removed the 30 s delivery deadline. This PR is the fix from the review of #9302. **Problem:** Since #9302, a fan-out worker waits for the access-gate database read with no time limit. If a database connection stops answering in the middle of that read, the worker never continues. Its queue fills, and one sixteenth of the node's channels get no live events from other nodes until the relay restarts. Before #9302 the deadline cut that delivery off after 30 s. **Fix:** Each delivery attempt has a 30 s limit (`MAX_DELIVERY_ATTEMPT`), measured from when the worker starts it. When the limit passes, the relay counts the event in `buzz_multinode_fanout_dropped_total{reason="stale"}`, resyncs the artifact syncs the event matched, and continues with the next event. It closes no connection. An event that only waits in the queue is still delivered late, as #9302 intends. The queue-full drop and the new stale drop share one helper. ### Validation - Added test: `hung_delivery_is_dropped_at_the_bound_and_the_worker_moves_on` drives the production worker with a database that accepts the connection and never answers. It checks that the delivery stops at the limit, is counted, resyncs the matched sync, closes nothing, and that the next event on the same worker is delivered. With the limit removed it fails (the worker waits out the pool's one-hour acquire timeout instead). - The other `handlers::multinode_fanout` tests pass. The full local `buzz-relay` suite was not run; CI on this head covers it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Tom Brow <tomb@squareup.com> Co-authored-by: Tom Brow <tomb@squareup.com>