Files
buzz/desktop/src/shared/api/relayClientShared.ts
Luke Tornquist ae689ab54d fix(desktop): resume reconnect replay from the last live point (#8652)
## Summary

A relay rolling deploy closes every socket with 1012. On reconnect,
desktop replayed each live subscription's **original filter** (startup
`since`, full `limit`) unless that subscription had received an event.
Quiet subscriptions therefore re-read history back to app start every
time. Paged channels did this twice: they resent the original filter
*and* ran the HTTP repair pages over the same window. A 1012 close also
reset the backoff to the 1s minimum, so a client bounced from pod to pod
during a deploy came back at roughly the same moment as every other
client.

**This PR makes three changes:**

1. **Resume cursor.** The subscription records `liveUntil`, when the
connection closes, if it reached EOSE on that connection. That time is
the last frame received from the relay, not the close itself. The
session tracks that time per peer frame, independent of the stall
watchdog, so a close delivered late (after a sleep, or while replay is
still running) doesn't overstate how long the subscription was live.
Close and error notices don't count as liveness. A frame processed more
than 70s after the previous one (the stall watchdog's idle timeout plus
its check interval) may have been queued through a sleep, so from then
on it doesn't extend liveness for the rest of that connection. Liveness
also starts fresh on every connection. If the time is unknown, the
previous `liveUntil` is kept rather than guessed from the clock. The
replay then uses `since = max(original since, lastSeen − 5s,
min(liveUntil − 960s, now − 3600s))`.
- **Why the one-hour window:** the relay can drop a fan-out frame for a
client whose send buffer is full without closing the socket
(`ConnectionState::send`), so continued pings don't prove a subscription
missed nothing. The liveness term therefore never resumes later than an
hour ago, which is the same window as the agent harness's silent-drop
repair (#8653). The saving applies to history older than an hour, so it
matters most for long-running clients, which is the deploy-bounce case.
Older drops still reach the timeline, because auto-heal refetches the
open channel's newest window on reconnect. What they miss are live-only
side effects such as unread and mention badges.
- **Why 960s:** the relay accepts `created_at` up to ±900s from its
clock (`MAX_TIMESTAMP_DRIFT_SECS` in
`crates/buzz-relay/src/handlers/ingest.rs`). An event stored after the
close can therefore be backdated by that much, and the extra 60s covers
client clock skew.
- **Gap floors win:** a pinned failed-backfill floor still takes
precedence.
   - **No cursor:** a subscription without one replays as before.
   - **Paged channels** keep their 1865s event-time lookback.
2. **Live-only REQ for paged channels.** When the HTTP repair covers the
gap, the live REQ goes out as the original filter with `limit: 0`. The
relay serves no stored events for it (`req.rs` passes the limit through
to a SQL `LIMIT 0`).
- **Backfill/live overlap:** the repair waits for that REQ's EOSE before
it starts. EOSE arrives only after the subscription is registered, so
live and repair always overlap.
- **If EOSE never arrives**, a single 25s deadline, measured from when
the REQ was sent, leads to the bounded history fallback below. The stall
watchdog runs during replay, so a half-open socket that never sends EOSE
or a close is reset by the watchdog rather than holding recovery.
- **A reset, disposal or supersession** wakes that EOSE wait
immediately, and the wait never starts for a retired connection, so a
dead attempt never holds `connect()` (and the next reconnect) behind it.
A `connect()` whose connection was replaced during replay settles as
superseded instead of arming the watchdog.
- **If EOSE never arrives, or all three repair attempts fail**, the live
REQ is re-issued under the same ID with the window's bounded history
(`since` = floor, `limit` ≤ 500): the pre-change behavior. The missed
window still fills on this connection, and the floor stays pinned for
anything the cap left out. Duplicate suppression stays on until that
request's own EOSE, so events live already delivered aren't dispatched
twice. If the original EOSE was never seen, a late one can't be told
apart from the fallback's, so only the deadline or teardown ends
suppression. Teardown also ends it, and so does a 25s deadline in case
the relay drops the EOSE. A subscription disposed while the fallback is
in flight still ends with a CLOSE.
3. **Reconnect spreading.** After a 1012 or 1001 close, the first
attempt waits a uniform random delay in [0, 10s] instead of resetting to
1s. Ordinary backoff jitter is a symmetric ±25%, capped at 30s. The
backoff resets only after the existing 60s stable-connection check,
never just because a socket opened.

**Simpler?** The backoff state, including its 60s stable-connection
reset timer, moved out of `relayClientSession.ts` into a small
`RelayReconnectBackoff` class in `relayReconnectPolicy.ts`. That makes
the jitter math testable on its own. It also keeps the session file at
1199 lines, the same as `main` (its cap is 1200), even with the new
liveness and replay logic; the liveness gap rule lives in
`relayResumeCursor.ts`. The stall watchdog's API is unchanged from
`main`.

**Identified debt (not changed here):**
- `relayClosedRecovery.ts` retries a retryable live CLOSED with the
original filter (1s → 30s backoff), with no terminal state.
- `useRelayAutoHeal.ts` sends an invalidation wave after reconnect.
- The `lastSeen − 5s` term keeps its existing gap for events backdated
after a close.
- `liveUntil` means the subscription was registered, not that every
event was delivered. The one-hour window bounds this: a dropped fan-out
frame older than an hour on a quiet subscription isn't re-read over the
websocket. The fix is a relay-side drop signal, the same follow-up as
#8653's.
- An HTTP repair page that is already in flight when the socket resets
still holds `connect()` until it returns, as on `main`. The native HTTP
client sets no request timeout.
- A `limit: 0` REQ still makes the relay acquire a reader for an empty
query.

### Related issue
N/A. This is part of a set of reconnect-load reductions. Companion PRs:
relay read-pool warm (#8651), agent harness (#8653) and mobile (#8659).

### Testing
- **Full desktop gate** on a Linux workstation at `045030c4`:
  - `pnpm typecheck`, `pnpm check` and the file-size check pass.
  - `pnpm test`: 6901/6901, plus 165/165 jsdom.
  - `pnpm build:e2e` passes.
- The `relay-reconnect` Playwright spec passes, including the restart
spread draw, which passed 10/10 repeats.
- **New or extended tests** (`node:test`):
- `relayResumeCursor.test.mjs`: a table test over the cursor inputs,
including the one-hour window, plus the liveness gap rule.
- `relayClientResumeCursor.test.mjs`: drives the real `RelayClient`. It
covers a dropped fan-out followed by pings, a frame queued through a
sleep, fresh credit per connection, a half-open socket reset by the
watchdog during replay, and a reset or disposal while parked on EOSE.
- Extended: `relayReconnectReplay` (deadline from send, a single timeout
→ fallback, a late original EOSE, fallback dedupe and CLOSE
convergence), `relayReconnectPolicy` and `relayClosedRecovery`.
- **Mutations:** each guard was reverted and failed at least one test.
That covers every cursor term, the 900 + 60 skew, the one-hour window,
`limit: 0`, the EOSE wait, its deadline and its timeout, floor
precedence, EOSE generation stamping, the last-inbound cap and gap rule,
the per-connection reset, the watchdog starting before replay, fallback
dedupe, the late-EOSE guard, the 1001 match, the backoff reset, the
restart spread and jitter symmetry.
- **Exercised against a real relay** (PG16 and Redis) in a relay-backed
desktop build, with Chromium on a Linux workstation, base `b310fea0` vs
head `045030c4`. Each channel was seeded with 200 reactions and 3
messages, all two hours old:

| | Base | Head |
|---|---:|---:|
| Quiet channel, stored events replayed over the websocket | 204 | 1 |
| Paged channel, stored events replayed over the websocket | 50 | 0
(`limit:0`) |
| Paged HTTP repair, events returned | 204 | 1 |
| Event-query calls / rows returned | 68 / 895 | 73 / 441 (−50.7% rows)
|

- **21-minute process freeze:** the watchdog fired first. The quiet
cursor matched `max(lastSeen − 5, min(preFreeze − 960, now − 3600))`
exactly, and 3/3 messages posted during the freeze were recovered once
each.
- **Drop before the paged EOSE:** `connect()` released in 13ms. The next
dial followed on the normal backoff, and the message arrived once.
- **Half-open socket** (all relay frames withheld, no close): the
watchdog reset it at 60.0s, the next dial came 1.2s later, and both
messages were recovered once each.
- **Earlier run at `180dc9ae`, before the one-hour window, relay restart
(1012):** the quiet channel resumed from last inbound − 960s. At this
head that cursor is also capped at now − 3600s. The paged channel sent
`limit:0`, with repair starting about 45ms after EOSE. Both messages
posted during the bounce were delivered, including one backdated 600s.
The first reconnect came about 8s after the close.

buzz-review-completed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 13:50:33 -04:00

197 lines
6.4 KiB
TypeScript

import type { RelayEvent } from "@/shared/api/types";
/**
* Observable connection state for the relay singleton.
*
* - `idle` — never tried to connect yet (post-init, pre-community).
* - `connecting` — initial socket + AUTH handshake in flight.
* - `connected` — socket open and AUTH'd.
* - `reconnecting` — socket dropped, waiting for the backoff timer.
* - `stalled` — socket is *open* per the WS layer but no inbound frames
* for a long time (half-open socket / VPN split-brain). We
* surface this so the UI can warn even though tungstenite
* hasn't reported anything wrong yet.
* - `disconnected` — final/terminal disconnect (auth rejected, community
* switch, etc.) — no auto-reconnect scheduled.
*/
export type ConnectionState =
| "idle"
| "connecting"
| "connected"
| "reconnecting"
| "stalled"
| "disconnected";
/** True when the UI should surface a "connection lost" indicator. */
export function isRelayConnectionDegraded(state: ConnectionState): boolean {
return (
state === "reconnecting" || state === "stalled" || state === "disconnected"
);
}
/**
* Aggregate cap on explicit `#h` channel values the relay accepts across one
* REQ's filters before rejecting it as `restricted: too many explicit
* channels`. Mirrors `MAX_EXPLICIT_CHANNEL_VALUES` in
* `crates/buzz-relay/src/handlers/req.rs`. A subscription over more hidden DMs
* than this must be split into multiple REQs, each within the cap.
*/
export const MAX_EXPLICIT_CHANNEL_VALUES = 128;
export type RelaySubscriptionFilter = {
ids?: string[];
kinds: number[];
limit: number;
authors?: string[];
since?: number;
until?: number;
/** Relay extension: composite pagination tiebreak paired with `until`. */
before_id?: string;
} & Partial<Record<`#${string}`, string[]>>;
type HistorySubscription = {
mode: "history";
filter: RelaySubscriptionFilter;
events: RelayEvent[];
resolve: (events: RelayEvent[]) => void;
reject: (error: Error) => void;
timeout: number;
timeoutMs: number;
closedRetryAttempt?: number;
};
type FirstEventSubscription = {
mode: "first";
onEvent: (event: RelayEvent) => void;
resolve: (event: RelayEvent | null) => void;
reject: (error: Error) => void;
timeout: number;
};
export type LiveSubscriptionReadiness = "eose" | "closed" | "timeout";
type LiveSubscription = {
mode: "live";
filter: RelaySubscriptionFilter;
/** Client-side admission only; interactive consumers still obey cooldown/pacing. */
priority?: "interactive";
onEvent: (event: RelayEvent) => void;
resolveReady?: (readiness: LiveSubscriptionReadiness) => void;
/** Release readiness/cancellation listeners when this entry is retired. */
onRemoved?: () => void;
lastSeenCreatedAt?: number;
/** Connection generation on which this subscription last reached EOSE. */
eoseGeneration?: number;
/**
* Unix seconds up to which this subscription was registered on a connection
* that delivered its EOSE: that connection's last inbound frame, stamped on
* close. Not proof of delivery; see `relayResumeCursor.ts`.
*/
liveUntil?: number;
/**
* Lower bound of a reconnect backfill window that has not yet completed.
*
* Events on the restored live REQ advance `lastSeenCreatedAt` regardless of
* backfill success, so after an exhausted backfill the cursor alone would
* make the next reconnect skip the unresolved older window — silent message
* loss. This floor is pinned when paging starts and cleared only when a
* backfill pass completes; the next replay starts from
* `min(pendingReplaySince, cursor window)`.
*/
pendingReplaySince?: number;
/** Dispatch-level duplicate suppression shared by restored-live and repair. */
reconnectReplay?: {
generation: number;
seenEventIds: Set<string>;
liveEose: boolean;
repairDone: boolean;
/** When the restored live REQ went out; its EOSE deadline runs from here. */
liveReqSentAtMs?: number;
/**
* Set when the history fallback follows an unacknowledged live REQ: a late
* original EOSE is then indistinguishable from the fallback's, so no EOSE
* ends dedupe; only the fallback's deadline or teardown does.
*/
ignoreEose?: boolean;
/**
* Wakes the repair worker waiting for the restored live REQ's EOSE: on
* EOSE, or when a newer replay supersedes this one.
*/
liveEoseWaiter?: () => void;
};
closedRetryAttempt?: number;
closedRetryTimeout?: number;
};
export type PendingEvent = {
event: RelayEvent;
resolve: (event: RelayEvent) => void;
reject: (error: Error) => void;
timeout: number;
};
export type RelaySubscription =
| HistorySubscription
| FirstEventSubscription
| LiveSubscription;
export type SubscriptionEventBufferItem = {
subId: string;
event: RelayEvent;
generation: number;
};
export function sortEvents(events: RelayEvent[]) {
return [...events].sort((left, right) => {
if (left.created_at !== right.created_at) {
return left.created_at - right.created_at;
}
// Same (created_at, id) tiebreak as the cache sort (sortMessages) so a
// history REQ resolves same-second events in a stable, relay-matching
// order. Currently every consumer re-sorts downstream, but keeping the
// two sorts on one invariant avoids a latent ordering drift.
return left.id < right.id ? -1 : left.id > right.id ? 1 : 0;
});
}
/**
* Splits a native websocket delivery into individual frames.
*
* The native plugin coalesces inbound frames into one IPC delivery, so a
* delivery is an array of frames. Older single-frame deliveries stay valid:
* `getTextPayload` rejects arrays, so every consumer must unwrap here rather
* than per-client — an un-unwrapped consumer drops batched frames silently
* instead of failing.
*/
export function toRelayFrames(message: unknown): unknown[] {
return Array.isArray(message) ? message : [message];
}
export function getTextPayload(message: unknown) {
if (typeof message === "string") {
return message;
}
if (
typeof message === "object" &&
message !== null &&
"type" in message &&
message.type === "Text" &&
"data" in message &&
typeof message.data === "string"
) {
return message.data;
}
if (
typeof message === "object" &&
message !== null &&
"Text" in message &&
typeof message.Text === "string"
) {
return message.Text;
}
return null;
}