mirror of
https://github.com/block/buzz.git
synced 2026-10-29 05:21:31 +00:00
fix(relay): reject a frame on its own acknowledgement channel (#6961)
Pinky, an AI agent, updated this description on Wes's behalf after
taking over the startup investigation.
**Category:** fix
**User Impact:** An EVENT refused by WebSocket admission or handler
saturation receives a correlated `OK(event_id, false, reason)` instead
of an uncorrelated NOTICE, so the client can settle that refusal without
waiting for its publish timeout. Rate-limited refusals also arm client
backoff. This fixes a protocol failure mechanism; it does not establish
that every startup send will succeed or that the reported Desktop
startup incident is fully resolved.
**Problem:** Startup opens several live subscriptions and publishes at
once, and the relay's WebSocket admission gate is a fixed 5-second
window (`ws_admission_budget` = `human_ws_events_per_sec * 5`). If that
shared per-principal quota is exhausted, `enforce_ws_admission`
previously rejected an EVENT with a bare `["NOTICE", reason]`. Quota
pressure is a possible trigger, not proof of the original incident's
complete cause.
A NOTICE carries no event id. Both clients settle a pending publish
*only* from an `OK` keyed by event id (desktop `pendingEvents`, mobile
`_pendingEvents`), so nothing settled — and `handle_text_message`
returns early, so no `OK` ever followed either. The send **could not
fail**; it could only time out at `PUBLISH_TIMEOUT_MS` = 25s. That
explains how this rejection mechanism can produce a roughly 25-second
timeout; attributing the original report to it still requires the actual
startup/send workflow.
The handler-semaphore saturation path had the identical defect, and that
one needs no quota burst to fire.
**Solution:** NIP-01 gives each request type its own acknowledgement
channel, and a rejection is only actionable on the same one. Reject a
REQ with `CLOSED`, an EVENT with `OK(id, false, reason)`, and fall back
to `NOTICE` only where no per-request correlation exists. COUNT refusals
now also use `CLOSED(query_id, reason)` per NIP-45, covering both quota
admission and handler saturation (added in
`cd12c93804b87a24b61075dfd171dc471a0a527f`).
Reason strings are unchanged, so the `rate-limited:` prefix and `retry
in {N}s` hint that existing client gates parse keep working (desktop
`parseRateLimitHint`, mobile `RelayRateLimitGate`, buzz-acp
`set_rate_limit_gate`). Only the frame *type* changes, so
`docs/multi-tenant-relay.md` L7 stays satisfied.
Two notes on how this landed, both worth a reviewer's attention:
1. **A survived mutation became a design change.**
`send_admission_result` originally took a `RejectionTarget` parameter,
and reverting the *second* call site (the per-minute message quota)
survived the whole suite — with Redis unreachable the first quota check
short-circuits, so that line is unreachable in test. Rather than test
around it, the parameter is gone: the target is derived from the frame,
so no call site can name the wrong channel.
2. **The relay fix would have caused a client regression on its own.**
Gate arming lived only in the NOTICE branch. Once rejections arrive as
`OK:false`, `handleOk` failed the send without ever backing off — the
client would retry straight into the same quota. Desktop and Mobile now
arm on a `rate-limited:` OK rejection. ACP was subsequently fixed in
`3b06dd32493596ec650f20abf8805791c50fdc24`: it arms the gate and
re-parks only the refused observer frame, preserving other in-flight
frames. Desktop gets `activateRateLimitIfSignalled` as the single owner
of that prefix test, called from both `handleOk` and the NOTICE branch.
<details>
<summary>File changes</summary>
**crates/buzz-relay/src/rejection.rs** (new)
Owns the admission-rejection concern: `RejectionTarget`,
`rejection_target_for`, `request_rejection_message`,
`send_admission_result`, and `enforce_ws_admission`, moved out of
`connection.rs`. Six tests, two of which drive the real
`enforce_ws_admission` against a real `AppState`.
**crates/buzz-relay/src/connection.rs**
Fix the EVENT handler-semaphore rejection to correlate to the event id;
delegate admission to the new module. Add two tests that drive the real
`handle_text_message` with every handler permit held. Down from 1319 to
1116 lines.
**crates/buzz-relay/src/state.rs**
Widen the existing `test_state` helper to `pub(crate)` so the rejection
tests reuse it rather than adding a ninth copy of `AppState`
construction.
**desktop/src/shared/api/relayRateLimitGate.ts**
Add `activateRateLimitIfSignalled` — one owner for the `rate-limited:`
prefix test, since three inbound frame types now carry it.
**desktop/src/shared/api/relayClientSession.ts**
Arm the gate on a rate-limited OK rejection; route the NOTICE branch
through the same helper. Net zero lines, which keeps this
already-oversized file within the differential ratchet.
**desktop/src/shared/api/relayClientPublishRejection.test.mjs** (new)
Four tests against the real `RelayClient`: a rate-limited OK settles the
pending publish and arms the gate; an ordinary rejection does not arm
it; an accepted OK still resolves.
**mobile/lib/shared/relay/relay_session.dart**
Arm the gate in `_handleOk` for a rate-limited rejection.
**mobile/test/shared/relay/relay_session_test.dart**
Two tests driving the real `publish` + `debugHandleMessage` path.
</details>
<details>
<summary>Validation</summary>
**Mutation-tested — 5 mutations, all now killed.** Each production call
site was reverted to the defective behaviour to confirm a test fails.
This caught two false-negative tests:
| # | Mutation | Result |
|---|----------|--------|
| 1 | `rejection_target_for`: EVENT → `Connection` | 4 tests fail |
| 2 | EVENT handler-semaphore call site → bare NOTICE | **survived at
first** |
| 3 | per-minute quota call site → `Connection` | **survived**; fixed by
removing the parameter |
| 4 | desktop `handleOk` gate arming removed | 1 test fails |
| 5 | mobile `_handleOk` gate arming removed | 1 test fails |
Mutation 2 is the lesson: my first saturation test called
`request_rejection_message` directly, so reverting the real call site
inside the `match` arm left it green. It now drives
`handle_text_message` itself and dies on that mutation.
- `cargo test -p buzz-relay` — 928 passed, 1 failed:
`api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo`,
**pre-existing**, reproduced with all changes stashed at `4dd4d73de`.
- `cd desktop && npm test` — 5721 passed, 0 failed (full suite).
- `cd mobile && flutter test` — 1876 passed, 0 failed (full suite).
- `just fmt-check`, `just clippy`, `just desktop-check`, `just
mobile-check`, `just file-size-check` — clean. Desktop's 5 biome
warnings are pre-existing (reproduced with changes stashed).
- All 9 pre-push lanes green, including `rust-tests` and
`desktop-tauri-checks`.
**Not verified:** not reproduced end-to-end against a live relay under a
forced quota burst. The causal chain is source-proven and
mutation-proven at the frame level; the ~25s attribution follows from
`PUBLISH_TIMEOUT_MS` but is not directly measured. A packaged-build
click-through would close that gap.
</details>
Related work: #6957 bounds Desktop HTTP event submission, but safe
retained-operation recovery after exhausted/ambiguous outcomes remains
unfinished. #6998 is the separately reviewable Desktop
readiness/duplicate-subscription slice. Neither is claimed to complete
native before/after startup-send validation.
Diagnosis note: `RESEARCH/DESKTOP_STARTUP_SEND_STALL_2026_08_27.md`
(Brain's workspace).
## Current review disposition (2026-08-28)
The [review on
`cd12c938`](https://github.com/block/buzz/pull/6961#pullrequestreview-5052902510)
identified ACP's missing rate-limited-OK handling. Commit
`3b06dd32493596ec650f20abf8805791c50fdc24` fixes gate arming, re-parking
the specifically refused observer frame, and the stale NOTICE comment.
Two regressions drive the real frame dispatcher. See [the implementation
and validation
response](https://github.com/block/buzz/pull/6961#issuecomment-5455032054).
The Mobile generation-check inline thread is resolved: its `async
publish` returns a failed Future when superseded; it does not throw
synchronously at invocation. No further production change was indicated
by that comment.
The validation counts above describe the original slice, not a new
rerun. At `3b06dd324`, the current GitHub check rollup has successful
completed test/build checks (non-applicable jobs skipped). The
security-review comment still requires review for the current base/head
range; do not read a green authorization job as a completed security
review. Approval and merge remain human decisions.
---------
Signed-off-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz>
Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Co-authored-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz>
Co-authored-by: Carl <9d00794d3df50972eb8b615511783cab12a77a8fd5dd5edd58073ec73b54bd8b@buzz.block.builderlab.xyz>
This commit is contained in:
@@ -9,6 +9,10 @@ import {
|
||||
type RelaySubscriptionFilter,
|
||||
} from "@/shared/api/relayClientShared";
|
||||
import { closeWebSocket } from "@/shared/api/relayWebSocketClose";
|
||||
import {
|
||||
activateRateLimitIfSignalled,
|
||||
waitForRateLimit,
|
||||
} from "@/shared/api/relayRateLimitGate";
|
||||
import {
|
||||
AUTH_TIMEOUT_MS,
|
||||
HISTORY_TIMEOUT_MS,
|
||||
@@ -107,7 +111,10 @@ export class ReadOnlyRelayClient {
|
||||
|
||||
async publishEvent(event: RelayEvent): Promise<void> {
|
||||
await this.connect();
|
||||
if (this.wsId === null) {
|
||||
const generation = this.generation;
|
||||
await waitForRateLimit();
|
||||
|
||||
if (generation !== this.generation || this.wsId === null) {
|
||||
throw new Error("Read-only relay socket is not connected.");
|
||||
}
|
||||
|
||||
@@ -281,6 +288,7 @@ export class ReadOnlyRelayClient {
|
||||
if (success) {
|
||||
publish.resolve();
|
||||
} else {
|
||||
activateRateLimitIfSignalled(message);
|
||||
publish.reject(
|
||||
new Error(message || "Observer relay rejected the event."),
|
||||
);
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
// ReadOnlyRelayClient publishes to inactive communities, but it shares the
|
||||
// process-wide relay rate-limit gate with the primary session. Addressed EVENT
|
||||
// refusals therefore need to settle this client's pending publish, arm the
|
||||
// shared gate, and defer later sends until the advertised window expires.
|
||||
import assert from "node:assert/strict";
|
||||
import test from "node:test";
|
||||
|
||||
let fakeNow = 0;
|
||||
const pendingTimers = new Map();
|
||||
let nextTimerId = 1;
|
||||
const sends = [];
|
||||
|
||||
globalThis.window = {
|
||||
setTimeout: (fn, ms) => {
|
||||
const id = nextTimerId++;
|
||||
pendingTimers.set(id, { fn, fireAt: fakeNow + ms });
|
||||
return id;
|
||||
},
|
||||
clearTimeout: (id) => pendingTimers.delete(id),
|
||||
__TAURI_INTERNALS__: {
|
||||
invoke: async (command, args) => {
|
||||
if (command === "plugin:websocket|send") sends.push(args);
|
||||
},
|
||||
},
|
||||
};
|
||||
Date.now = () => fakeNow;
|
||||
|
||||
const { ReadOnlyRelayClient } = await import("./readOnlyRelayClient.ts");
|
||||
const { activateRateLimit, isRateLimited, resetRateLimitGate } = await import(
|
||||
"./relayRateLimitGate.ts"
|
||||
);
|
||||
|
||||
function tickTo(ms) {
|
||||
fakeNow = ms;
|
||||
for (const [id, { fn, fireAt }] of Array.from(pendingTimers.entries())) {
|
||||
if (fireAt <= fakeNow) {
|
||||
pendingTimers.delete(id);
|
||||
fn();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function reset() {
|
||||
resetRateLimitGate();
|
||||
fakeNow = 0;
|
||||
pendingTimers.clear();
|
||||
nextTimerId = 1;
|
||||
sends.length = 0;
|
||||
}
|
||||
|
||||
function connectedClient() {
|
||||
const client = new ReadOnlyRelayClient("wss://inactive.example");
|
||||
client.wsId = 7;
|
||||
client.connect = async () => {};
|
||||
return client;
|
||||
}
|
||||
|
||||
function armPendingPublish(client, eventId) {
|
||||
const settled = new Promise((resolve, reject) => {
|
||||
client.publishes.set(eventId, {
|
||||
resolve,
|
||||
reject,
|
||||
timeout: window.setTimeout(() => {}, 25_000),
|
||||
});
|
||||
});
|
||||
return settled.then(
|
||||
() => ({ status: "resolved" }),
|
||||
(error) => ({ status: "rejected", error }),
|
||||
);
|
||||
}
|
||||
|
||||
function deliver(client, frame) {
|
||||
return client.handleWsMessage(
|
||||
{ type: "Text", data: JSON.stringify(frame) },
|
||||
client.generation,
|
||||
);
|
||||
}
|
||||
|
||||
test("a rate-limited OK rejects the named publish and arms the shared gate", async () => {
|
||||
reset();
|
||||
const client = connectedClient();
|
||||
const eventId = "a".repeat(64);
|
||||
const settled = armPendingPublish(client, eventId);
|
||||
|
||||
await deliver(client, [
|
||||
"OK",
|
||||
eventId,
|
||||
false,
|
||||
"rate-limited: quota exceeded; retry in 4s",
|
||||
]);
|
||||
|
||||
const outcome = await settled;
|
||||
assert.equal(outcome.status, "rejected");
|
||||
assert.match(outcome.error.message, /rate-limited/);
|
||||
assert.equal(client.publishes.has(eventId), false);
|
||||
assert.equal(isRateLimited(), true);
|
||||
});
|
||||
|
||||
test("an ordinary OK rejection does not arm the shared gate", async () => {
|
||||
reset();
|
||||
const client = connectedClient();
|
||||
const eventId = "b".repeat(64);
|
||||
const settled = armPendingPublish(client, eventId);
|
||||
|
||||
await deliver(client, ["OK", eventId, false, "invalid: bad signature"]);
|
||||
|
||||
assert.equal((await settled).status, "rejected");
|
||||
assert.equal(isRateLimited(), false);
|
||||
});
|
||||
|
||||
test("publish waits outside its timeout and pending state, then sends and settles", async () => {
|
||||
reset();
|
||||
activateRateLimit(4);
|
||||
const client = connectedClient();
|
||||
const event = { id: "c".repeat(64), kind: 5 };
|
||||
|
||||
const published = client.publishEvent(event);
|
||||
await Promise.resolve();
|
||||
await Promise.resolve();
|
||||
|
||||
assert.equal(sends.length, 0, "EVENT must remain unsent while gated");
|
||||
assert.equal(
|
||||
client.publishes.has(event.id),
|
||||
false,
|
||||
"publish timeout and pending ownership start only after the gate expires",
|
||||
);
|
||||
assert.equal(pendingTimers.size, 1, "only the gate timer should be armed");
|
||||
|
||||
tickTo(4_000);
|
||||
await Promise.resolve();
|
||||
await Promise.resolve();
|
||||
|
||||
assert.equal(sends.length, 1);
|
||||
assert.deepEqual(JSON.parse(sends[0].message.data), ["EVENT", event]);
|
||||
assert.equal(client.publishes.has(event.id), true);
|
||||
|
||||
await deliver(client, ["OK", event.id, true, ""]);
|
||||
await published;
|
||||
assert.equal(client.publishes.has(event.id), false);
|
||||
});
|
||||
|
||||
test("a disconnected client does not send after the shared gate expires", async () => {
|
||||
reset();
|
||||
activateRateLimit(4);
|
||||
const client = connectedClient();
|
||||
const event = { id: "d".repeat(64), kind: 5 };
|
||||
|
||||
const published = client.publishEvent(event);
|
||||
await Promise.resolve();
|
||||
client.disconnect();
|
||||
tickTo(4_000);
|
||||
|
||||
await assert.rejects(published, /not connected/);
|
||||
assert.equal(sends.length, 0);
|
||||
assert.equal(client.publishes.has(event.id), false);
|
||||
});
|
||||
295
desktop/src/shared/api/relayClientPublishRejection.test.mjs
Normal file
295
desktop/src/shared/api/relayClientPublishRejection.test.mjs
Normal file
@@ -0,0 +1,295 @@
|
||||
// A relay rejection addressed to one event must settle that event's pending
|
||||
// publish *and* arm the rate-limit gate.
|
||||
//
|
||||
// History: the relay rejected an over-quota EVENT with a bare
|
||||
// `["NOTICE", "rate-limited: ..."]`. A NOTICE carries no event id, and
|
||||
// `pendingEvents` is keyed by event id, so nothing settled — the publish sat
|
||||
// until PUBLISH_TIMEOUT_MS (25s) and surfaced as a message stuck on
|
||||
// "Sending…". Startup quota exhaustion made that routine in the first seconds
|
||||
// after launch. The relay now rejects on the OK channel instead, so the gate
|
||||
// arming that used to live in the NOTICE branch has to happen here too.
|
||||
import assert from "node:assert/strict";
|
||||
import test from "node:test";
|
||||
|
||||
const fakeNow = 0;
|
||||
const pendingTimers = new Map();
|
||||
let nextTimerId = 1;
|
||||
const sendAttempts = [];
|
||||
const deliveredFrames = [];
|
||||
let sendTransport = async (args) => {
|
||||
deliveredFrames.push(args);
|
||||
};
|
||||
|
||||
globalThis.window = {
|
||||
setTimeout: (fn, ms) => {
|
||||
const id = nextTimerId++;
|
||||
pendingTimers.set(id, { fn, fireAt: fakeNow + ms });
|
||||
return id;
|
||||
},
|
||||
clearTimeout: (id) => pendingTimers.delete(id),
|
||||
__TAURI_INTERNALS__: {
|
||||
invoke: async (command, args) => {
|
||||
if (command === "plugin:websocket|send") {
|
||||
sendAttempts.push(args);
|
||||
return sendTransport(args);
|
||||
}
|
||||
},
|
||||
},
|
||||
};
|
||||
Date.now = () => fakeNow;
|
||||
|
||||
const { RelayClient } = await import("./relayClientSession.ts");
|
||||
const { activateRateLimit, isRateLimited, resetRateLimitGate } = await import(
|
||||
"./relayRateLimitGate.ts"
|
||||
);
|
||||
|
||||
function reset() {
|
||||
resetRateLimitGate();
|
||||
pendingTimers.clear();
|
||||
nextTimerId = 1;
|
||||
sendAttempts.length = 0;
|
||||
deliveredFrames.length = 0;
|
||||
sendTransport = async (args) => {
|
||||
deliveredFrames.push(args);
|
||||
};
|
||||
}
|
||||
|
||||
function connectedClient() {
|
||||
const client = new RelayClient();
|
||||
client.wsId = 7;
|
||||
return client;
|
||||
}
|
||||
|
||||
function eventFrames() {
|
||||
return deliveredFrames.filter(
|
||||
({ message }) => JSON.parse(message.data)[0] === "EVENT",
|
||||
);
|
||||
}
|
||||
|
||||
async function flushUntil(predicate, attempts = 20) {
|
||||
for (let attempt = 0; attempt < attempts; attempt++) {
|
||||
if (predicate()) return;
|
||||
await Promise.resolve();
|
||||
}
|
||||
assert.fail("condition did not become true before the microtask limit");
|
||||
}
|
||||
|
||||
function deferred() {
|
||||
let resolve;
|
||||
let reject;
|
||||
const promise = new Promise((promiseResolve, promiseReject) => {
|
||||
resolve = promiseResolve;
|
||||
reject = promiseReject;
|
||||
});
|
||||
return { promise, resolve, reject };
|
||||
}
|
||||
|
||||
/**
|
||||
* Registers a pending publish the way `publishEvent` does, without needing a
|
||||
* socket: the OK dispatch under test only reads `pendingEvents`.
|
||||
*/
|
||||
function armPendingPublish(client, eventId) {
|
||||
const event = { id: eventId };
|
||||
const settled = new Promise((resolve, reject) => {
|
||||
client.pendingEvents.set(eventId, {
|
||||
event,
|
||||
resolve,
|
||||
reject,
|
||||
timeout: window.setTimeout(() => {}, 25_000),
|
||||
});
|
||||
});
|
||||
// Keep the rejection from surfacing as an unhandled rejection.
|
||||
return settled.then(
|
||||
(value) => ({ status: "resolved", value }),
|
||||
(error) => ({ status: "rejected", error }),
|
||||
);
|
||||
}
|
||||
|
||||
/** Feeds a raw relay frame through the real inbound dispatch path. */
|
||||
function deliver(client, frame) {
|
||||
return client.handleWsMessage(
|
||||
{ type: "Text", data: JSON.stringify(frame) },
|
||||
client.connectionGeneration,
|
||||
);
|
||||
}
|
||||
|
||||
test("a rate-limited OK rejection settles the pending publish", async () => {
|
||||
resetRateLimitGate();
|
||||
pendingTimers.clear();
|
||||
const client = new RelayClient();
|
||||
const eventId = "a".repeat(64);
|
||||
const settled = armPendingPublish(client, eventId);
|
||||
|
||||
await deliver(client, [
|
||||
"OK",
|
||||
eventId,
|
||||
false,
|
||||
"rate-limited: quota exceeded; retry in 4s",
|
||||
]);
|
||||
|
||||
const outcome = await settled;
|
||||
assert.equal(
|
||||
outcome.status,
|
||||
"rejected",
|
||||
"an over-quota publish must fail fast, not hang until the 25s publish timeout",
|
||||
);
|
||||
assert.match(outcome.error.message, /rate-limited/);
|
||||
assert.equal(
|
||||
client.pendingEvents.has(eventId),
|
||||
false,
|
||||
"the pending entry must be cleared",
|
||||
);
|
||||
});
|
||||
|
||||
test("a rate-limited OK rejection arms the rate-limit gate", async () => {
|
||||
resetRateLimitGate();
|
||||
pendingTimers.clear();
|
||||
const client = new RelayClient();
|
||||
const eventId = "b".repeat(64);
|
||||
const settled = armPendingPublish(client, eventId);
|
||||
|
||||
assert.equal(isRateLimited(), false, "gate starts closed");
|
||||
|
||||
await deliver(client, [
|
||||
"OK",
|
||||
eventId,
|
||||
false,
|
||||
"rate-limited: quota exceeded; retry in 4s",
|
||||
]);
|
||||
await settled;
|
||||
|
||||
assert.equal(
|
||||
isRateLimited(),
|
||||
true,
|
||||
"back-pressure now arrives on the OK channel — without arming here the " +
|
||||
"client fails the send and immediately retries into the same quota",
|
||||
);
|
||||
});
|
||||
|
||||
test("an ordinary OK rejection does not arm the gate", async () => {
|
||||
resetRateLimitGate();
|
||||
pendingTimers.clear();
|
||||
const client = new RelayClient();
|
||||
const eventId = "c".repeat(64);
|
||||
const settled = armPendingPublish(client, eventId);
|
||||
|
||||
await deliver(client, ["OK", eventId, false, "invalid: bad signature"]);
|
||||
const outcome = await settled;
|
||||
|
||||
assert.equal(outcome.status, "rejected");
|
||||
assert.equal(
|
||||
isRateLimited(),
|
||||
false,
|
||||
"only `rate-limited:` rejections signal back-pressure",
|
||||
);
|
||||
});
|
||||
|
||||
test("an accepted OK still resolves the pending publish", async () => {
|
||||
reset();
|
||||
const client = new RelayClient();
|
||||
const eventId = "d".repeat(64);
|
||||
const settled = armPendingPublish(client, eventId);
|
||||
|
||||
await deliver(client, ["OK", eventId, true, ""]);
|
||||
const outcome = await settled;
|
||||
|
||||
assert.equal(outcome.status, "resolved");
|
||||
assert.equal(outcome.value.id, eventId);
|
||||
});
|
||||
|
||||
test("a publish started during an ordinary outage reconnects once and settles", async () => {
|
||||
reset();
|
||||
const client = new RelayClient();
|
||||
const event = { id: "0".repeat(64), kind: 1 };
|
||||
let reconnects = 0;
|
||||
client.ensureConnected = async () => {
|
||||
reconnects++;
|
||||
client.connectionGeneration++;
|
||||
client.wsId = 8;
|
||||
return client.connectionGeneration;
|
||||
};
|
||||
|
||||
const published = client.publishEvent(event, "timed out", "send failed");
|
||||
await flushUntil(() => eventFrames().length === 1);
|
||||
|
||||
assert.equal(reconnects, 1);
|
||||
assert.equal(sendAttempts.length, 1);
|
||||
assert.equal(client.pendingEvents.has(event.id), true);
|
||||
|
||||
await deliver(client, ["OK", event.id, true, ""]);
|
||||
assert.equal(await published, event);
|
||||
assert.equal(client.pendingEvents.size, 0);
|
||||
});
|
||||
|
||||
test("a community switch while gated cannot publish through its replacement socket", async () => {
|
||||
reset();
|
||||
activateRateLimit(4);
|
||||
const client = connectedClient();
|
||||
const event = { id: "e".repeat(64), kind: 1 };
|
||||
|
||||
const published = client.publishEvent(event, "timed out", "send failed");
|
||||
await Promise.resolve();
|
||||
assert.equal(client.pendingEvents.size, 0);
|
||||
|
||||
client.disconnect();
|
||||
resetRateLimitGate();
|
||||
client.wsId = 8;
|
||||
|
||||
await assert.rejects(published, /community switch/);
|
||||
assert.equal(client.pendingEvents.size, 0);
|
||||
assert.equal(eventFrames().length, 0);
|
||||
});
|
||||
|
||||
test("a community switch after send failure cannot retry through its replacement socket", async () => {
|
||||
reset();
|
||||
const client = connectedClient();
|
||||
const event = { id: "f".repeat(64), kind: 1 };
|
||||
const reconnect = deferred();
|
||||
client.ensureConnected = async () => {
|
||||
await reconnect.promise;
|
||||
return client.connectionGeneration;
|
||||
};
|
||||
sendTransport = async () => {
|
||||
throw new Error("old socket failed");
|
||||
};
|
||||
|
||||
const published = client.publishEvent(event, "timed out", "send failed");
|
||||
const outcome = published.then(
|
||||
() => ({ status: "resolved" }),
|
||||
(error) => ({ status: "rejected", error }),
|
||||
);
|
||||
await flushUntil(() => client.connectionGeneration === 1);
|
||||
assert.equal(sendAttempts.length, 1);
|
||||
assert.equal(eventFrames().length, 0);
|
||||
assert.equal(
|
||||
client.connectionGeneration,
|
||||
1,
|
||||
"the failed send reset its socket",
|
||||
);
|
||||
assert.equal(
|
||||
client.pendingEvents.has(event.id),
|
||||
true,
|
||||
"the original publish remains owned while reconnect is pending",
|
||||
);
|
||||
|
||||
client.disconnect();
|
||||
client.wsId = 8;
|
||||
const settledBeforeReconnect = await outcome;
|
||||
assert.equal(
|
||||
settledBeforeReconnect.status,
|
||||
"rejected",
|
||||
"community switch must settle the publish without waiting for reconnect",
|
||||
);
|
||||
assert.match(settledBeforeReconnect.error.message, /community switch/);
|
||||
|
||||
reconnect.resolve();
|
||||
await published.catch(() => {});
|
||||
await Promise.resolve();
|
||||
assert.equal(client.pendingEvents.size, 0);
|
||||
assert.equal(
|
||||
sendAttempts.length,
|
||||
1,
|
||||
"the replacement socket must not be used",
|
||||
);
|
||||
assert.equal(eventFrames().length, 0);
|
||||
});
|
||||
@@ -38,11 +38,8 @@ import {
|
||||
} from "@/shared/api/relayClosedRecovery";
|
||||
import { getChannelReconnectRepairEvents } from "@/shared/api/channelReconnectRepair";
|
||||
import { replayLiveSubscriptions } from "@/shared/api/relayReconnectReplay";
|
||||
import {
|
||||
activateRateLimit,
|
||||
parseRateLimitHint,
|
||||
waitForRateLimit,
|
||||
} from "@/shared/api/relayRateLimitGate";
|
||||
import { publishSessionEvent } from "@/shared/api/relayEventPublisher";
|
||||
import { activateRateLimitIfSignalled } from "@/shared/api/relayRateLimitGate";
|
||||
import {
|
||||
fetchChunkedHistory,
|
||||
requestFirstEventGated,
|
||||
@@ -64,7 +61,6 @@ import {
|
||||
BACKOFF_RESET_STABLE_MS,
|
||||
EVENT_BATCH_MS,
|
||||
HISTORY_TIMEOUT_MS,
|
||||
PUBLISH_TIMEOUT_MS,
|
||||
RECONNECT_BASE_DELAY_MS,
|
||||
RECONNECT_MAX_DELAY_MS,
|
||||
STALL_CHECK_INTERVAL_MS,
|
||||
@@ -82,7 +78,7 @@ import { buildThreadReferenceTags } from "@/features/messages/lib/threading";
|
||||
export class RelayClient {
|
||||
private wsId: number | null = null;
|
||||
private relayUrl: string | null = null;
|
||||
private connectPromise: Promise<void> | null = null;
|
||||
private connectPromise: Promise<number> | null = null;
|
||||
private reconnectTimeout: number | null = null;
|
||||
private reconnectWaiters = new RelayReconnectWaiters();
|
||||
private reconnectDelayMs = RECONNECT_BASE_DELAY_MS;
|
||||
@@ -97,6 +93,7 @@ export class RelayClient {
|
||||
private notifyReconnectListeners = false;
|
||||
private onMessageChannel: Channel<unknown> | null = null;
|
||||
private connectionGeneration = 0;
|
||||
private sessionEpoch = 0;
|
||||
private stabilityTimer: number | null = null;
|
||||
private visibleChannelId: string | null = null;
|
||||
private authOkTracker = new AuthOkTracker();
|
||||
@@ -126,6 +123,7 @@ export class RelayClient {
|
||||
this.stabilityTimer = null;
|
||||
}
|
||||
this.stallWatchdog.stop();
|
||||
this.sessionEpoch++;
|
||||
this.connectionGeneration++;
|
||||
this.keepAliveRequested = false;
|
||||
this.relayUrl = null;
|
||||
@@ -496,7 +494,7 @@ export class RelayClient {
|
||||
}
|
||||
|
||||
if (this.wsId !== null) {
|
||||
return;
|
||||
return this.connectionGeneration;
|
||||
}
|
||||
|
||||
if (
|
||||
@@ -507,14 +505,14 @@ export class RelayClient {
|
||||
// The reconnect coordinator owns outage pacing. Query, publish, and
|
||||
// subscription callers must wait for its scheduled attempt instead of
|
||||
// clearing the timer and creating an immediate reconnect storm.
|
||||
return this.reconnectWaiters.wait();
|
||||
return this.reconnectWaiters.wait().then(() => this.connectionGeneration);
|
||||
}
|
||||
|
||||
const connectPromise = this.connect();
|
||||
this.connectPromise = connectPromise;
|
||||
|
||||
try {
|
||||
await connectPromise;
|
||||
return await connectPromise;
|
||||
} finally {
|
||||
if (this.connectPromise === connectPromise) {
|
||||
this.connectPromise = null;
|
||||
@@ -584,6 +582,7 @@ export class RelayClient {
|
||||
await this.replayLiveSubscriptions();
|
||||
this.stallWatchdog.start();
|
||||
this.emitReconnectIfNeeded();
|
||||
return generation;
|
||||
} catch (error) {
|
||||
const connectionError = this.normalizeRelayError(
|
||||
error,
|
||||
@@ -663,6 +662,17 @@ export class RelayClient {
|
||||
});
|
||||
}
|
||||
|
||||
private async sendRawForGeneration(payload: unknown[], generation: number) {
|
||||
if (generation !== this.connectionGeneration || this.wsId === null) {
|
||||
throw new Error("Relay publish was superseded by a session change.");
|
||||
}
|
||||
const wsId = this.wsId;
|
||||
await invoke("plugin:websocket|send", {
|
||||
id: wsId,
|
||||
message: { type: "Text", data: JSON.stringify(payload) },
|
||||
});
|
||||
}
|
||||
|
||||
private normalizeRelayError(error: unknown, fallbackMessage: string) {
|
||||
return error instanceof Error ? error : new Error(fallbackMessage);
|
||||
}
|
||||
@@ -712,47 +722,23 @@ export class RelayClient {
|
||||
timeoutMessage: string,
|
||||
sendErrorMessage: string,
|
||||
) {
|
||||
// Await the gate before sending EVENT; op timeout starts after the wait.
|
||||
await waitForRateLimit();
|
||||
|
||||
return new Promise<RelayEvent>((resolve, reject) => {
|
||||
const timeout = window.setTimeout(() => {
|
||||
this.pendingEvents.delete(event.id);
|
||||
reject(new Error(timeoutMessage));
|
||||
}, PUBLISH_TIMEOUT_MS);
|
||||
|
||||
this.pendingEvents.set(event.id, {
|
||||
event,
|
||||
resolve,
|
||||
reject,
|
||||
timeout,
|
||||
});
|
||||
|
||||
void this.sendRaw(["EVENT", event]).catch(async (error) => {
|
||||
const pendingEvent = this.pendingEvents.get(event.id);
|
||||
this.pendingEvents.delete(event.id);
|
||||
const normalizedError = this.recoverFromSocketFailure(
|
||||
error,
|
||||
sendErrorMessage,
|
||||
);
|
||||
|
||||
try {
|
||||
await this.ensureConnected();
|
||||
if (!pendingEvent) {
|
||||
throw normalizedError;
|
||||
}
|
||||
|
||||
this.pendingEvents.set(event.id, pendingEvent);
|
||||
await this.sendRaw(["EVENT", event]);
|
||||
} catch (retryError) {
|
||||
window.clearTimeout(timeout);
|
||||
this.pendingEvents.delete(event.id);
|
||||
reject(
|
||||
this.recoverFromSocketFailure(retryError, normalizedError.message),
|
||||
);
|
||||
}
|
||||
});
|
||||
});
|
||||
return publishSessionEvent(
|
||||
{
|
||||
generation: () => this.connectionGeneration,
|
||||
ownership: () => this.sessionEpoch,
|
||||
pendingEvents: this.pendingEvents,
|
||||
send: (payload, generation) =>
|
||||
this.sendRawForGeneration(payload, generation),
|
||||
reconnect: () => this.ensureConnected(),
|
||||
normalizeError: (error, fallback) =>
|
||||
this.normalizeRelayError(error, fallback),
|
||||
recoverSocketFailure: (error, fallback) =>
|
||||
this.recoverFromSocketFailure(error, fallback),
|
||||
},
|
||||
event,
|
||||
timeoutMessage,
|
||||
sendErrorMessage,
|
||||
);
|
||||
}
|
||||
|
||||
private async handleWsMessage(message: unknown, generation: number) {
|
||||
@@ -829,11 +815,8 @@ export class RelayClient {
|
||||
}
|
||||
|
||||
if (type === "NOTICE" && typeof rest[0] === "string") {
|
||||
const notice: string = rest[0];
|
||||
// Relay back-pressure — arm the gate until the window expires.
|
||||
if (notice.startsWith("rate-limited:")) {
|
||||
activateRateLimit(parseRateLimitHint(notice));
|
||||
}
|
||||
// Connection-scoped back-pressure — arm the gate until it expires.
|
||||
activateRateLimitIfSignalled(rest[0]);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -922,6 +905,10 @@ export class RelayClient {
|
||||
if (success) {
|
||||
pendingEvent.resolve(pendingEvent.event);
|
||||
} else {
|
||||
// Back-pressure now arrives here rather than as a NOTICE: the relay
|
||||
// rejects an over-quota EVENT on the OK channel so this pending publish
|
||||
// can be settled at all. Unarmed, the send retries into the same quota.
|
||||
activateRateLimitIfSignalled(message);
|
||||
pendingEvent.reject(new Error(message || "Relay rejected the event."));
|
||||
}
|
||||
}
|
||||
|
||||
84
desktop/src/shared/api/relayEventPublisher.ts
Normal file
84
desktop/src/shared/api/relayEventPublisher.ts
Normal file
@@ -0,0 +1,84 @@
|
||||
import type { RelayEvent } from "@/shared/api/types";
|
||||
import type { PendingEvent } from "@/shared/api/relayClientShared";
|
||||
import { waitForRateLimit } from "@/shared/api/relayRateLimitGate";
|
||||
import { PUBLISH_TIMEOUT_MS } from "@/shared/api/relayClientTimings";
|
||||
|
||||
type PublishSession = {
|
||||
generation: () => number;
|
||||
ownership: () => number;
|
||||
pendingEvents: Map<string, PendingEvent>;
|
||||
send: (payload: unknown[], generation: number) => Promise<void>;
|
||||
reconnect: () => Promise<number>;
|
||||
normalizeError: (error: unknown, fallback: string) => Error;
|
||||
recoverSocketFailure: (error: unknown, fallback: string) => Error;
|
||||
};
|
||||
|
||||
/** Publish once, with one reconnect retry, without crossing session ownership. */
|
||||
export async function publishSessionEvent(
|
||||
session: PublishSession,
|
||||
event: RelayEvent,
|
||||
timeoutMessage: string,
|
||||
sendErrorMessage: string,
|
||||
): Promise<RelayEvent> {
|
||||
const publishOwnership = session.ownership();
|
||||
await waitForRateLimit();
|
||||
if (publishOwnership !== session.ownership()) {
|
||||
throw new Error("Relay disconnected for community switch.");
|
||||
}
|
||||
const publishGeneration = session.generation();
|
||||
|
||||
return new Promise<RelayEvent>((resolve, reject) => {
|
||||
const timeout = window.setTimeout(() => {
|
||||
session.pendingEvents.delete(event.id);
|
||||
reject(new Error(timeoutMessage));
|
||||
}, PUBLISH_TIMEOUT_MS);
|
||||
const pendingEvent = { event, resolve, reject, timeout };
|
||||
session.pendingEvents.set(event.id, pendingEvent);
|
||||
|
||||
void session
|
||||
.send(["EVENT", event], publishGeneration)
|
||||
.catch(async (error) => {
|
||||
// A disconnect may already have rejected this operation while the send
|
||||
// was in flight. Its late failure must not reset the replacement session.
|
||||
if (
|
||||
publishOwnership !== session.ownership() ||
|
||||
publishGeneration !== session.generation() ||
|
||||
session.pendingEvents.get(event.id) !== pendingEvent
|
||||
) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Expected socket recovery must not reject the operation being retried.
|
||||
session.pendingEvents.delete(event.id);
|
||||
const sendError = session.recoverSocketFailure(error, sendErrorMessage);
|
||||
session.pendingEvents.set(event.id, pendingEvent);
|
||||
let retryGeneration: number | null = null;
|
||||
|
||||
try {
|
||||
retryGeneration = await session.reconnect();
|
||||
if (
|
||||
publishOwnership !== session.ownership() ||
|
||||
session.generation() !== retryGeneration ||
|
||||
session.pendingEvents.get(event.id) !== pendingEvent
|
||||
) {
|
||||
throw new Error(
|
||||
"Relay publish was superseded by a session change.",
|
||||
);
|
||||
}
|
||||
await session.send(["EVENT", event], retryGeneration);
|
||||
} catch (retryError) {
|
||||
if (session.pendingEvents.get(event.id) !== pendingEvent) return;
|
||||
|
||||
window.clearTimeout(timeout);
|
||||
session.pendingEvents.delete(event.id);
|
||||
reject(
|
||||
publishOwnership === session.ownership() &&
|
||||
retryGeneration !== null &&
|
||||
session.generation() === retryGeneration
|
||||
? session.recoverSocketFailure(retryError, sendError.message)
|
||||
: session.normalizeError(retryError, sendError.message),
|
||||
);
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
@@ -87,6 +87,24 @@ export function activateRateLimit(retryInSeconds: number | null): void {
|
||||
}, durationMs);
|
||||
}
|
||||
|
||||
/**
|
||||
* Arms the gate if `message` is a relay back-pressure signal, and reports
|
||||
* whether it was.
|
||||
*
|
||||
* The relay marks back-pressure with a `rate-limited:` prefix on whichever
|
||||
* frame carries the rejection — `NOTICE` for connection-scoped limits, `OK`
|
||||
* for one addressed to a single event, `CLOSED` for a subscription. Every
|
||||
* inbound path needs the same test, so it lives here with the gate rather than
|
||||
* being re-derived per call site.
|
||||
*/
|
||||
export function activateRateLimitIfSignalled(message: string): boolean {
|
||||
if (!message.startsWith("rate-limited:")) {
|
||||
return false;
|
||||
}
|
||||
activateRateLimit(parseRateLimitHint(message));
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Returns `true` when the relay has signalled back-pressure and the gate is active. */
|
||||
export function isRateLimited(): boolean {
|
||||
return expiresAt !== null && Date.now() < expiresAt;
|
||||
|
||||
Reference in New Issue
Block a user