mirror of
https://github.com/block/buzz.git
synced 2026-10-30 14:01:37 +00:00
ae689ab54d
## Summary A relay rolling deploy closes every socket with 1012. On reconnect, desktop replayed each live subscription's **original filter** (startup `since`, full `limit`) unless that subscription had received an event. Quiet subscriptions therefore re-read history back to app start every time. Paged channels did this twice: they resent the original filter *and* ran the HTTP repair pages over the same window. A 1012 close also reset the backoff to the 1s minimum, so a client bounced from pod to pod during a deploy came back at roughly the same moment as every other client. **This PR makes three changes:** 1. **Resume cursor.** The subscription records `liveUntil`, when the connection closes, if it reached EOSE on that connection. That time is the last frame received from the relay, not the close itself. The session tracks that time per peer frame, independent of the stall watchdog, so a close delivered late (after a sleep, or while replay is still running) doesn't overstate how long the subscription was live. Close and error notices don't count as liveness. A frame processed more than 70s after the previous one (the stall watchdog's idle timeout plus its check interval) may have been queued through a sleep, so from then on it doesn't extend liveness for the rest of that connection. Liveness also starts fresh on every connection. If the time is unknown, the previous `liveUntil` is kept rather than guessed from the clock. The replay then uses `since = max(original since, lastSeen − 5s, min(liveUntil − 960s, now − 3600s))`. - **Why the one-hour window:** the relay can drop a fan-out frame for a client whose send buffer is full without closing the socket (`ConnectionState::send`), so continued pings don't prove a subscription missed nothing. The liveness term therefore never resumes later than an hour ago, which is the same window as the agent harness's silent-drop repair (#8653). The saving applies to history older than an hour, so it matters most for long-running clients, which is the deploy-bounce case. Older drops still reach the timeline, because auto-heal refetches the open channel's newest window on reconnect. What they miss are live-only side effects such as unread and mention badges. - **Why 960s:** the relay accepts `created_at` up to ±900s from its clock (`MAX_TIMESTAMP_DRIFT_SECS` in `crates/buzz-relay/src/handlers/ingest.rs`). An event stored after the close can therefore be backdated by that much, and the extra 60s covers client clock skew. - **Gap floors win:** a pinned failed-backfill floor still takes precedence. - **No cursor:** a subscription without one replays as before. - **Paged channels** keep their 1865s event-time lookback. 2. **Live-only REQ for paged channels.** When the HTTP repair covers the gap, the live REQ goes out as the original filter with `limit: 0`. The relay serves no stored events for it (`req.rs` passes the limit through to a SQL `LIMIT 0`). - **Backfill/live overlap:** the repair waits for that REQ's EOSE before it starts. EOSE arrives only after the subscription is registered, so live and repair always overlap. - **If EOSE never arrives**, a single 25s deadline, measured from when the REQ was sent, leads to the bounded history fallback below. The stall watchdog runs during replay, so a half-open socket that never sends EOSE or a close is reset by the watchdog rather than holding recovery. - **A reset, disposal or supersession** wakes that EOSE wait immediately, and the wait never starts for a retired connection, so a dead attempt never holds `connect()` (and the next reconnect) behind it. A `connect()` whose connection was replaced during replay settles as superseded instead of arming the watchdog. - **If EOSE never arrives, or all three repair attempts fail**, the live REQ is re-issued under the same ID with the window's bounded history (`since` = floor, `limit` ≤ 500): the pre-change behavior. The missed window still fills on this connection, and the floor stays pinned for anything the cap left out. Duplicate suppression stays on until that request's own EOSE, so events live already delivered aren't dispatched twice. If the original EOSE was never seen, a late one can't be told apart from the fallback's, so only the deadline or teardown ends suppression. Teardown also ends it, and so does a 25s deadline in case the relay drops the EOSE. A subscription disposed while the fallback is in flight still ends with a CLOSE. 3. **Reconnect spreading.** After a 1012 or 1001 close, the first attempt waits a uniform random delay in [0, 10s] instead of resetting to 1s. Ordinary backoff jitter is a symmetric ±25%, capped at 30s. The backoff resets only after the existing 60s stable-connection check, never just because a socket opened. **Simpler?** The backoff state, including its 60s stable-connection reset timer, moved out of `relayClientSession.ts` into a small `RelayReconnectBackoff` class in `relayReconnectPolicy.ts`. That makes the jitter math testable on its own. It also keeps the session file at 1199 lines, the same as `main` (its cap is 1200), even with the new liveness and replay logic; the liveness gap rule lives in `relayResumeCursor.ts`. The stall watchdog's API is unchanged from `main`. **Identified debt (not changed here):** - `relayClosedRecovery.ts` retries a retryable live CLOSED with the original filter (1s → 30s backoff), with no terminal state. - `useRelayAutoHeal.ts` sends an invalidation wave after reconnect. - The `lastSeen − 5s` term keeps its existing gap for events backdated after a close. - `liveUntil` means the subscription was registered, not that every event was delivered. The one-hour window bounds this: a dropped fan-out frame older than an hour on a quiet subscription isn't re-read over the websocket. The fix is a relay-side drop signal, the same follow-up as #8653's. - An HTTP repair page that is already in flight when the socket resets still holds `connect()` until it returns, as on `main`. The native HTTP client sets no request timeout. - A `limit: 0` REQ still makes the relay acquire a reader for an empty query. ### Related issue N/A. This is part of a set of reconnect-load reductions. Companion PRs: relay read-pool warm (#8651), agent harness (#8653) and mobile (#8659). ### Testing - **Full desktop gate** on a Linux workstation at `045030c4`: - `pnpm typecheck`, `pnpm check` and the file-size check pass. - `pnpm test`: 6901/6901, plus 165/165 jsdom. - `pnpm build:e2e` passes. - The `relay-reconnect` Playwright spec passes, including the restart spread draw, which passed 10/10 repeats. - **New or extended tests** (`node:test`): - `relayResumeCursor.test.mjs`: a table test over the cursor inputs, including the one-hour window, plus the liveness gap rule. - `relayClientResumeCursor.test.mjs`: drives the real `RelayClient`. It covers a dropped fan-out followed by pings, a frame queued through a sleep, fresh credit per connection, a half-open socket reset by the watchdog during replay, and a reset or disposal while parked on EOSE. - Extended: `relayReconnectReplay` (deadline from send, a single timeout → fallback, a late original EOSE, fallback dedupe and CLOSE convergence), `relayReconnectPolicy` and `relayClosedRecovery`. - **Mutations:** each guard was reverted and failed at least one test. That covers every cursor term, the 900 + 60 skew, the one-hour window, `limit: 0`, the EOSE wait, its deadline and its timeout, floor precedence, EOSE generation stamping, the last-inbound cap and gap rule, the per-connection reset, the watchdog starting before replay, fallback dedupe, the late-EOSE guard, the 1001 match, the backoff reset, the restart spread and jitter symmetry. - **Exercised against a real relay** (PG16 and Redis) in a relay-backed desktop build, with Chromium on a Linux workstation, base `b310fea0` vs head `045030c4`. Each channel was seeded with 200 reactions and 3 messages, all two hours old: | | Base | Head | |---|---:|---:| | Quiet channel, stored events replayed over the websocket | 204 | 1 | | Paged channel, stored events replayed over the websocket | 50 | 0 (`limit:0`) | | Paged HTTP repair, events returned | 204 | 1 | | Event-query calls / rows returned | 68 / 895 | 73 / 441 (−50.7% rows) | - **21-minute process freeze:** the watchdog fired first. The quiet cursor matched `max(lastSeen − 5, min(preFreeze − 960, now − 3600))` exactly, and 3/3 messages posted during the freeze were recovered once each. - **Drop before the paged EOSE:** `connect()` released in 13ms. The next dial followed on the normal backoff, and the message arrived once. - **Half-open socket** (all relay frames withheld, no close): the watchdog reset it at 60.0s, the next dial came 1.2s later, and both messages were recovered once each. - **Earlier run at `180dc9ae`, before the one-hour window, relay restart (1012):** the quiet channel resumed from last inbound − 960s. At this head that cursor is also capped at now − 3600s. The paged channel sent `limit:0`, with repair starting about 45ms after EOSE. Both messages posted during the bounce were delivered, including one backdated 600s. The first reconnect came about 8s after the close. buzz-review-completed 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>