mirror of
https://github.com/block/buzz.git
synced 2026-10-31 14:31:02 +00:00
## Summary A relay rolling deploy closes every socket with 1012. On reconnect, desktop replayed each live subscription's **original filter** (startup `since`, full `limit`) unless that subscription had received an event. Quiet subscriptions therefore re-read history back to app start every time. Paged channels did this twice: they resent the original filter *and* ran the HTTP repair pages over the same window. A 1012 close also reset the backoff to the 1s minimum, so a client bounced from pod to pod during a deploy came back at roughly the same moment as every other client. **This PR makes three changes:** 1. **Resume cursor.** The subscription records `liveUntil`, when the connection closes, if it reached EOSE on that connection. That time is the last frame received from the relay, not the close itself. The session tracks that time per peer frame, independent of the stall watchdog, so a close delivered late (after a sleep, or while replay is still running) doesn't overstate how long the subscription was live. Close and error notices don't count as liveness. A frame processed more than 70s after the previous one (the stall watchdog's idle timeout plus its check interval) may have been queued through a sleep, so from then on it doesn't extend liveness for the rest of that connection. Liveness also starts fresh on every connection. If the time is unknown, the previous `liveUntil` is kept rather than guessed from the clock. The replay then uses `since = max(original since, lastSeen − 5s, min(liveUntil − 960s, now − 3600s))`. - **Why the one-hour window:** the relay can drop a fan-out frame for a client whose send buffer is full without closing the socket (`ConnectionState::send`), so continued pings don't prove a subscription missed nothing. The liveness term therefore never resumes later than an hour ago, which is the same window as the agent harness's silent-drop repair (#8653). The saving applies to history older than an hour, so it matters most for long-running clients, which is the deploy-bounce case. Older drops still reach the timeline, because auto-heal refetches the open channel's newest window on reconnect. What they miss are live-only side effects such as unread and mention badges. - **Why 960s:** the relay accepts `created_at` up to ±900s from its clock (`MAX_TIMESTAMP_DRIFT_SECS` in `crates/buzz-relay/src/handlers/ingest.rs`). An event stored after the close can therefore be backdated by that much, and the extra 60s covers client clock skew. - **Gap floors win:** a pinned failed-backfill floor still takes precedence. - **No cursor:** a subscription without one replays as before. - **Paged channels** keep their 1865s event-time lookback. 2. **Live-only REQ for paged channels.** When the HTTP repair covers the gap, the live REQ goes out as the original filter with `limit: 0`. The relay serves no stored events for it (`req.rs` passes the limit through to a SQL `LIMIT 0`). - **Backfill/live overlap:** the repair waits for that REQ's EOSE before it starts. EOSE arrives only after the subscription is registered, so live and repair always overlap. - **If EOSE never arrives**, a single 25s deadline, measured from when the REQ was sent, leads to the bounded history fallback below. The stall watchdog runs during replay, so a half-open socket that never sends EOSE or a close is reset by the watchdog rather than holding recovery. - **A reset, disposal or supersession** wakes that EOSE wait immediately, and the wait never starts for a retired connection, so a dead attempt never holds `connect()` (and the next reconnect) behind it. A `connect()` whose connection was replaced during replay settles as superseded instead of arming the watchdog. - **If EOSE never arrives, or all three repair attempts fail**, the live REQ is re-issued under the same ID with the window's bounded history (`since` = floor, `limit` ≤ 500): the pre-change behavior. The missed window still fills on this connection, and the floor stays pinned for anything the cap left out. Duplicate suppression stays on until that request's own EOSE, so events live already delivered aren't dispatched twice. If the original EOSE was never seen, a late one can't be told apart from the fallback's, so only the deadline or teardown ends suppression. Teardown also ends it, and so does a 25s deadline in case the relay drops the EOSE. A subscription disposed while the fallback is in flight still ends with a CLOSE. 3. **Reconnect spreading.** After a 1012 or 1001 close, the first attempt waits a uniform random delay in [0, 10s] instead of resetting to 1s. Ordinary backoff jitter is a symmetric ±25%, capped at 30s. The backoff resets only after the existing 60s stable-connection check, never just because a socket opened. **Simpler?** The backoff state, including its 60s stable-connection reset timer, moved out of `relayClientSession.ts` into a small `RelayReconnectBackoff` class in `relayReconnectPolicy.ts`. That makes the jitter math testable on its own. It also keeps the session file at 1199 lines, the same as `main` (its cap is 1200), even with the new liveness and replay logic; the liveness gap rule lives in `relayResumeCursor.ts`. The stall watchdog's API is unchanged from `main`. **Identified debt (not changed here):** - `relayClosedRecovery.ts` retries a retryable live CLOSED with the original filter (1s → 30s backoff), with no terminal state. - `useRelayAutoHeal.ts` sends an invalidation wave after reconnect. - The `lastSeen − 5s` term keeps its existing gap for events backdated after a close. - `liveUntil` means the subscription was registered, not that every event was delivered. The one-hour window bounds this: a dropped fan-out frame older than an hour on a quiet subscription isn't re-read over the websocket. The fix is a relay-side drop signal, the same follow-up as #8653's. - An HTTP repair page that is already in flight when the socket resets still holds `connect()` until it returns, as on `main`. The native HTTP client sets no request timeout. - A `limit: 0` REQ still makes the relay acquire a reader for an empty query. ### Related issue N/A. This is part of a set of reconnect-load reductions. Companion PRs: relay read-pool warm (#8651), agent harness (#8653) and mobile (#8659). ### Testing - **Full desktop gate** on a Linux workstation at `045030c4`: - `pnpm typecheck`, `pnpm check` and the file-size check pass. - `pnpm test`: 6901/6901, plus 165/165 jsdom. - `pnpm build:e2e` passes. - The `relay-reconnect` Playwright spec passes, including the restart spread draw, which passed 10/10 repeats. - **New or extended tests** (`node:test`): - `relayResumeCursor.test.mjs`: a table test over the cursor inputs, including the one-hour window, plus the liveness gap rule. - `relayClientResumeCursor.test.mjs`: drives the real `RelayClient`. It covers a dropped fan-out followed by pings, a frame queued through a sleep, fresh credit per connection, a half-open socket reset by the watchdog during replay, and a reset or disposal while parked on EOSE. - Extended: `relayReconnectReplay` (deadline from send, a single timeout → fallback, a late original EOSE, fallback dedupe and CLOSE convergence), `relayReconnectPolicy` and `relayClosedRecovery`. - **Mutations:** each guard was reverted and failed at least one test. That covers every cursor term, the 900 + 60 skew, the one-hour window, `limit: 0`, the EOSE wait, its deadline and its timeout, floor precedence, EOSE generation stamping, the last-inbound cap and gap rule, the per-connection reset, the watchdog starting before replay, fallback dedupe, the late-EOSE guard, the 1001 match, the backoff reset, the restart spread and jitter symmetry. - **Exercised against a real relay** (PG16 and Redis) in a relay-backed desktop build, with Chromium on a Linux workstation, base `b310fea0` vs head `045030c4`. Each channel was seeded with 200 reactions and 3 messages, all two hours old: | | Base | Head | |---|---:|---:| | Quiet channel, stored events replayed over the websocket | 204 | 1 | | Paged channel, stored events replayed over the websocket | 50 | 0 (`limit:0`) | | Paged HTTP repair, events returned | 204 | 1 | | Event-query calls / rows returned | 68 / 895 | 73 / 441 (−50.7% rows) | - **21-minute process freeze:** the watchdog fired first. The quiet cursor matched `max(lastSeen − 5, min(preFreeze − 960, now − 3600))` exactly, and 3/3 messages posted during the freeze were recovered once each. - **Drop before the paged EOSE:** `connect()` released in 13ms. The next dial followed on the normal backoff, and the message arrived once. - **Half-open socket** (all relay frames withheld, no close): the watchdog reset it at 60.0s, the next dial came 1.2s later, and both messages were recovered once each. - **Earlier run at `180dc9ae`, before the one-hour window, relay restart (1012):** the quiet channel resumed from last inbound − 960s. At this head that cursor is also capped at now − 3600s. The paged channel sent `limit:0`, with repair starting about 45ms after EOSE. Both messages posted during the bounce were delivered, including one backdated 600s. The first reconnect came about 8s after the close. buzz-review-completed 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
197 lines
6.4 KiB
TypeScript
197 lines
6.4 KiB
TypeScript
import type { RelayEvent } from "@/shared/api/types";
|
|
|
|
/**
|
|
* Observable connection state for the relay singleton.
|
|
*
|
|
* - `idle` — never tried to connect yet (post-init, pre-community).
|
|
* - `connecting` — initial socket + AUTH handshake in flight.
|
|
* - `connected` — socket open and AUTH'd.
|
|
* - `reconnecting` — socket dropped, waiting for the backoff timer.
|
|
* - `stalled` — socket is *open* per the WS layer but no inbound frames
|
|
* for a long time (half-open socket / VPN split-brain). We
|
|
* surface this so the UI can warn even though tungstenite
|
|
* hasn't reported anything wrong yet.
|
|
* - `disconnected` — final/terminal disconnect (auth rejected, community
|
|
* switch, etc.) — no auto-reconnect scheduled.
|
|
*/
|
|
export type ConnectionState =
|
|
| "idle"
|
|
| "connecting"
|
|
| "connected"
|
|
| "reconnecting"
|
|
| "stalled"
|
|
| "disconnected";
|
|
|
|
/** True when the UI should surface a "connection lost" indicator. */
|
|
export function isRelayConnectionDegraded(state: ConnectionState): boolean {
|
|
return (
|
|
state === "reconnecting" || state === "stalled" || state === "disconnected"
|
|
);
|
|
}
|
|
|
|
/**
|
|
* Aggregate cap on explicit `#h` channel values the relay accepts across one
|
|
* REQ's filters before rejecting it as `restricted: too many explicit
|
|
* channels`. Mirrors `MAX_EXPLICIT_CHANNEL_VALUES` in
|
|
* `crates/buzz-relay/src/handlers/req.rs`. A subscription over more hidden DMs
|
|
* than this must be split into multiple REQs, each within the cap.
|
|
*/
|
|
export const MAX_EXPLICIT_CHANNEL_VALUES = 128;
|
|
|
|
export type RelaySubscriptionFilter = {
|
|
ids?: string[];
|
|
kinds: number[];
|
|
limit: number;
|
|
authors?: string[];
|
|
since?: number;
|
|
until?: number;
|
|
/** Relay extension: composite pagination tiebreak paired with `until`. */
|
|
before_id?: string;
|
|
} & Partial<Record<`#${string}`, string[]>>;
|
|
|
|
type HistorySubscription = {
|
|
mode: "history";
|
|
filter: RelaySubscriptionFilter;
|
|
events: RelayEvent[];
|
|
resolve: (events: RelayEvent[]) => void;
|
|
reject: (error: Error) => void;
|
|
timeout: number;
|
|
timeoutMs: number;
|
|
closedRetryAttempt?: number;
|
|
};
|
|
|
|
type FirstEventSubscription = {
|
|
mode: "first";
|
|
onEvent: (event: RelayEvent) => void;
|
|
resolve: (event: RelayEvent | null) => void;
|
|
reject: (error: Error) => void;
|
|
timeout: number;
|
|
};
|
|
|
|
export type LiveSubscriptionReadiness = "eose" | "closed" | "timeout";
|
|
|
|
type LiveSubscription = {
|
|
mode: "live";
|
|
filter: RelaySubscriptionFilter;
|
|
/** Client-side admission only; interactive consumers still obey cooldown/pacing. */
|
|
priority?: "interactive";
|
|
onEvent: (event: RelayEvent) => void;
|
|
resolveReady?: (readiness: LiveSubscriptionReadiness) => void;
|
|
/** Release readiness/cancellation listeners when this entry is retired. */
|
|
onRemoved?: () => void;
|
|
lastSeenCreatedAt?: number;
|
|
/** Connection generation on which this subscription last reached EOSE. */
|
|
eoseGeneration?: number;
|
|
/**
|
|
* Unix seconds up to which this subscription was registered on a connection
|
|
* that delivered its EOSE: that connection's last inbound frame, stamped on
|
|
* close. Not proof of delivery; see `relayResumeCursor.ts`.
|
|
*/
|
|
liveUntil?: number;
|
|
/**
|
|
* Lower bound of a reconnect backfill window that has not yet completed.
|
|
*
|
|
* Events on the restored live REQ advance `lastSeenCreatedAt` regardless of
|
|
* backfill success, so after an exhausted backfill the cursor alone would
|
|
* make the next reconnect skip the unresolved older window — silent message
|
|
* loss. This floor is pinned when paging starts and cleared only when a
|
|
* backfill pass completes; the next replay starts from
|
|
* `min(pendingReplaySince, cursor window)`.
|
|
*/
|
|
pendingReplaySince?: number;
|
|
/** Dispatch-level duplicate suppression shared by restored-live and repair. */
|
|
reconnectReplay?: {
|
|
generation: number;
|
|
seenEventIds: Set<string>;
|
|
liveEose: boolean;
|
|
repairDone: boolean;
|
|
/** When the restored live REQ went out; its EOSE deadline runs from here. */
|
|
liveReqSentAtMs?: number;
|
|
/**
|
|
* Set when the history fallback follows an unacknowledged live REQ: a late
|
|
* original EOSE is then indistinguishable from the fallback's, so no EOSE
|
|
* ends dedupe; only the fallback's deadline or teardown does.
|
|
*/
|
|
ignoreEose?: boolean;
|
|
/**
|
|
* Wakes the repair worker waiting for the restored live REQ's EOSE: on
|
|
* EOSE, or when a newer replay supersedes this one.
|
|
*/
|
|
liveEoseWaiter?: () => void;
|
|
};
|
|
closedRetryAttempt?: number;
|
|
closedRetryTimeout?: number;
|
|
};
|
|
|
|
export type PendingEvent = {
|
|
event: RelayEvent;
|
|
resolve: (event: RelayEvent) => void;
|
|
reject: (error: Error) => void;
|
|
timeout: number;
|
|
};
|
|
|
|
export type RelaySubscription =
|
|
| HistorySubscription
|
|
| FirstEventSubscription
|
|
| LiveSubscription;
|
|
|
|
export type SubscriptionEventBufferItem = {
|
|
subId: string;
|
|
event: RelayEvent;
|
|
generation: number;
|
|
};
|
|
|
|
export function sortEvents(events: RelayEvent[]) {
|
|
return [...events].sort((left, right) => {
|
|
if (left.created_at !== right.created_at) {
|
|
return left.created_at - right.created_at;
|
|
}
|
|
// Same (created_at, id) tiebreak as the cache sort (sortMessages) so a
|
|
// history REQ resolves same-second events in a stable, relay-matching
|
|
// order. Currently every consumer re-sorts downstream, but keeping the
|
|
// two sorts on one invariant avoids a latent ordering drift.
|
|
return left.id < right.id ? -1 : left.id > right.id ? 1 : 0;
|
|
});
|
|
}
|
|
|
|
/**
|
|
* Splits a native websocket delivery into individual frames.
|
|
*
|
|
* The native plugin coalesces inbound frames into one IPC delivery, so a
|
|
* delivery is an array of frames. Older single-frame deliveries stay valid:
|
|
* `getTextPayload` rejects arrays, so every consumer must unwrap here rather
|
|
* than per-client — an un-unwrapped consumer drops batched frames silently
|
|
* instead of failing.
|
|
*/
|
|
export function toRelayFrames(message: unknown): unknown[] {
|
|
return Array.isArray(message) ? message : [message];
|
|
}
|
|
|
|
export function getTextPayload(message: unknown) {
|
|
if (typeof message === "string") {
|
|
return message;
|
|
}
|
|
|
|
if (
|
|
typeof message === "object" &&
|
|
message !== null &&
|
|
"type" in message &&
|
|
message.type === "Text" &&
|
|
"data" in message &&
|
|
typeof message.data === "string"
|
|
) {
|
|
return message.data;
|
|
}
|
|
|
|
if (
|
|
typeof message === "object" &&
|
|
message !== null &&
|
|
"Text" in message &&
|
|
typeof message.Text === "string"
|
|
) {
|
|
return message.Text;
|
|
}
|
|
|
|
return null;
|
|
}
|