mirror of
https://github.com/block/buzz.git
synced 2026-10-27 04:30:52 +00:00
9b1c4f98e2
Long threads (over ~100 replies) time out in desktop. The aux hop that
loads reactions and other e-tag references builds its filter as one
`tags @> …` term per id, joined with `OR`. Past about 100 ids the
planner switches to walking each partition's primary key backward,
expecting to stop early. When few rows actually match, it scans the
whole community. With a dirty GIN pending list, a 188-reply thread never
finishes.
### What changes
- **Set-based e-tag reads (`buzz-db`).** The aux e-tag query now puts
the id set behind a `MATERIALIZED` CTE and matches with `ANY`. Each
e-tag read also runs under a transaction-local `statement_timeout` of 20
s.
- **Terminal timeout contract (`buzz-relay`).** When a read hits the
deadline, the client gets a terminal timeout (`CLOSED` for REQ, the
timeout error for COUNT and search), not a retryable failure.
- **Subscription lifecycle (`buzz-relay`).** The per-connection
subscription map lock is now the lifecycle lock. Claims, replacement,
client `CLOSE`, owner-checked timeout and search cleanup, revoke, and
connection teardown all change the map, the registry, topic refcounts
and terminal frames while holding it. As a result, a timed-out
subscription's `CLOSED` goes out before any replacement with the same id
can claim it, and claims made after teardown are refused.
- **Dropped terminal frame disconnects (`buzz-relay`).** If the `CLOSED`
for a retired subscription (owner-checked timeout or channel eviction)
can't be queued, the relay cancels the connection instead of leaving the
client holding a subscription that will never receive fan-out. A
client-initiated `CLOSE` ack is exempt.
- **COUNT under the same deadline (`buzz-db`).** The fast COUNT path
runs e-tag counts under the same transaction-local 20 s cap as SELECT,
so WS COUNT and HTTP `/count` produce the terminal timeout too.
- **Replica reads surface the deadline (`buzz-db`).** When a
replica-routed read is cancelled by `statement_timeout` (SQLSTATE
`57014`), the error is returned instead of re-running the query on the
writer, which would run the same slow statement. Other replica errors
still fall back to the writer.
`PLANS/REPLICA_FULL_READ_ROUTING_DESIGN.md` records the exception.
- **Deadline handling (desktop).** Only the relay's `query timed out`
counts as a query deadline. A client-side `request timed out` stays
retryable. For one-shot reads a deadline is terminal: the global query
retry (budget 1), thread-replies retry and reaction hydration don't
re-send the same slow query. Reaction ids held back by a deadline are
released when the channel is opened again, so reopening retries them
while same-channel re-renders don't. Live subscriptions, in both the
renderer and the native archive-sync socket, keep treating a deadline
`CLOSED` as transient: they back off and resubscribe with the same
filter instead of being dropped.
- **Deadline handling (mobile).** Mobile recognizes the deadline on both
transports: HTTP `503 {"error":"query timed out"}` and the WS `CLOSED`
reason.
- The root `ProviderScope` retry policy returns no retry for deadline
errors and keeps Riverpod's default backoff for everything else.
- Background thread-summary recounts don't retry a deadline.
- Forum posts polling pauses while the last load settled on a deadline;
reopening the forum retries.
- A channel's newest-window load surfaces a deadline instead of falling
back to legacy WS history, and the channel list's batched queries return
"unavailable" on a deadline instead of fanning out to per-filter
fallbacks. An unavailable batch is not treated as empty. Known
last-message timestamps and unread state are kept, and unread catch-up
moves `lastMessageAt` forward (never backward) the same way live events
do, so a cold start with no known timestamp still gets one.
- A live subscription that gets a deadline `CLOSED` backs off and
resubscribes with the same filter. History requests still fail on it.
- Settled load failures show a Retry control, matching desktop's
thread-replies error card: a shared `LoadErrorView` on the channel
timeline, forum thread and forum posts list, and an icon button beside
the thread replies summary. While the thread query has no value, the
thread page shows the route snapshot plus live, cached and optimistic
replies, with the loading or error status still visible.
- Other re-sends (reconnect, foreground resume, live events, the unread
backstop, scroll) are not suppressed; each is bounded by the relay's 20
s statement deadline.
- **Long-thread seeder.** `just seed-long-thread [replies]` posts one
thread (default 187 replies) with some reactions to the local dev relay,
for reproducing long threads in the apps. It uses a throwaway key,
refuses non-loopback relays and ignores ambient `BUZZ_*` credentials.
Writes are paced to fit the stock relay rate limit.
**Lifecycle concurrency.** The lock serializes one connection's
subscription changes. Other connections can still be delayed by pressure
on the shared pubsub command queue, and by serial eviction loops waiting
behind a blocked connection. This PR does not claim strict isolation
between connections.
**Recovery vs. load.** Clients stop automatic retries of the same
one-shot read, but they don't remember deadlines across other triggers:
reconnect, resume, live events and the unread backstop can each send the
read again later, bounded by the 20 s statement deadline. A live
subscription that hits the deadline resubscribes with its original
filter under the existing 1–30 s backoff, and against a filter that
keeps timing out it retries at the 30 s cap indefinitely, so the worst
case is one bounded query per subscription every 30 s. `since` is
deliberately not advanced to now on resubscribe, because that would skip
events missed while the subscription was closed. Unchanged from `main`:
when mobile's channel history sync fails with cached messages available,
the cached messages are shown without the error or Retry control.
### Benchmark
Local reproduction on an isolated bench stack (not committed):
- PG16, migrations applied, one 188-reply thread plus 5M background
events in the same community.
- Each request replays the desktop thread-replies filter through the
relay, with `include_aux` on. Times are total ms for 3 runs, with a 60 s
client cap.
GIN pending-list state is controlled with `VACUUM` (clean) or by
refilling the pending list (dirty), and recorded per partition with
`pgstatginindex`. Dirty runs held 402 pending pages on every partition's
tags index at the start and end of each matrix. Clean runs held 0–1
pages (relay startup writes).
| build | GIN pending state | limit 10 | 50 | 100 | 200 (188 ids) |
|---|---|---|---|---|---|
| `main` | clean | 26 / 16 / 13 | 56 / 51 / 52 | 92 / 93 / 95 | 662 /
639 / 641 |
| `main` | dirty | 315 / 298 / 327 | 22141 / 15888 / 16769 | 28992 /
27819 / 28080 | timeout ×3 (>60 s) |
| this PR | clean | 35 / 21 / 14 | 57 / 49 / 48 | 82 / 80 / 75 | 153 /
150 / 135 |
| this PR | dirty | 290 / 254 / 251 | 1257 / 1204 / 1206 | 2432 / 2378 /
2382 | 4333 / 4706 / 5104 |
Baseline `origin/main` `b37e47721`; this PR at `5f2650f5f`, whose e-tag
read builders are identical to the current head. At head `1ad81d868`
against `origin/main` `15d44dcce`, limit 200 measured 145–179 ms clean
(`main` 683–715) and 4.41–4.51 s dirty (`main` timeout ×3).
Every completed cell returns the same signed event set: 209 events, 187
replies at limit 200. With `include_aux` off, every cell is ≤66 ms. The
thread-window loader (`select_aux`) is unchanged from `main`, and its
timings and captured plans match `main`'s.
Actual aux-hop statements captured from each relay binary with
`auto_explain` (analyze + buffers), execution ms:
| build | clean, 100 ids | clean, 188 ids | dirty, 100 ids | dirty, 188
ids |
|---|---|---|---|---|
| `main` (N-way `OR`) | 40 | 298 | 28785 | 62892 |
| this PR (`MATERIALIZED` fence + `ANY`) | 23 | 45 | 1992 | 3539 |
**What the fence changes**
- On `main`, the planner combines `ORDER BY created_at … LIMIT` with an
overestimated match count and walks each partition's primary key
backward, expecting to stop early. With few real matches it scans the
whole community: the dirty 188-id statement touches ~4.9M shared buffers
on `main` vs ~0.6M with the fence.
- The fence prevents that early-stop plan. Across the measured cases
(1–188 ids, both pending states), the planner used the GIN tags index.
The fence doesn't force GIN in general; a broad enough filter could
still get a different plan.
**Tradeoff:** the fence materializes every match before sorting. In this
bench small limits stayed within run-to-run noise of `main`. A single
hot id (5,000 reactions, limit 100) measured 30–34 ms unfenced vs 35 ms
fenced.
**Deadline:** each e-tag read (SELECT and COUNT) runs under a
transaction-local `statement_timeout` of 20 s. It only tightens: a
shorter operator timeout (`BUZZ_DB_STATEMENT_TIMEOUT_MS`) is kept. It's
a per-statement limit, not a per-request one, so a request that runs
several pages or hops can take longer than 20 s in total. The benchmark
does not force the deadline.
---------
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz>
77 lines
4.3 KiB
Markdown
77 lines
4.3 KiB
Markdown
---
|
|
title: "Replica Full-Read Routing — Caller Classification"
|
|
tags: [relay, replica-routing, consistency]
|
|
status: active
|
|
created: 2026-08-28
|
|
---
|
|
|
|
# Replica Full-Read Routing — Caller Classification
|
|
|
|
`Db::query_events` always reads the **writer** pool. `Db::query_events_routed`
|
|
(and its `_bounded` / `count` / feed siblings) opt a read into **replica
|
|
routing**: the read may be served from a read replica when
|
|
`BUZZ_REPLICA_READ_MAX_AGE_MS` is set, subject to the soundness predicate
|
|
`RoutePredicate::for_query` derives from the query shape. The seam fails closed
|
|
to the writer on replica errors, with one deliberate exception: a statement
|
|
cancelled by `statement_timeout` (SQLSTATE `57014`) is surfaced rather than
|
|
re-run, because the writer would run the same slow statement. It is a genuine
|
|
no-op until the budget is configured (`crates/buzz-db/src/lib.rs`).
|
|
|
|
## Routing rule
|
|
|
|
**If a read's result influences a write or a permission decision, it reads from
|
|
the writer.** A replica can lag behind the caller's own just-committed write; a
|
|
read that gates the next write against that lag would make a wrong decision
|
|
(e.g. validate a save against a stale head, or miss the caller's own accepted
|
|
event during post-write verification and report a false conflict). Every such
|
|
read stays on `query_events` / writer.
|
|
|
|
Display reads — a page the user scrolls, a count on a badge, a history list —
|
|
tolerate bounded staleness and take the routed path.
|
|
|
|
Adding, removing, or reclassifying a caller **requires updating the table
|
|
below**; the `query_events_routed` doc-comment points here.
|
|
|
|
### Client-carried intent: the `consistency` extension field
|
|
|
|
Some write-influencing reads are issued by clients (Desktop, CLI) through the
|
|
HTTP `/query` bridge, and are **indistinguishable by query shape** from display
|
|
reads — a kind-40100 `limit:1` read serves both `get_canvas` (display) and a
|
|
canvas save's head precondition. The client therefore signals intent with the
|
|
`consistency` extension field on the raw filter:
|
|
|
|
- `"consistency": "strong"` → the bridge serves that filter from the writer
|
|
(`query_events`), never a replica.
|
|
- absent → the default routed path.
|
|
- any other value → `400 Bad Request` (fails loud, never silently degrades).
|
|
|
|
The field only ever forces the **writer**, which is always the sound direction;
|
|
there is deliberately **no** inverse "force replica" value, so it cannot be
|
|
sprayed to bypass the replica-staleness guard on reads that should not. Parsed
|
|
in `crates/buzz-relay/src/api/bridge.rs` (`extract_consistency`).
|
|
|
|
## Caller classification table
|
|
|
|
Rows below are every `query_events_routed` / `query_events_routed_bounded`
|
|
call site at head, plus the writer-pinned canvas row this change adds. (The
|
|
`count` and feed routed families — `count_events_routed`,
|
|
`get_events_by_ids_routed`, `query_feed_*_routed`, and the `get_channel_window`
|
|
cursor/head reads — are all display or bounded-count surfaces on the routed
|
|
path; they carry their own soundness notes at their definitions in
|
|
`crates/buzz-db/src/lib.rs` and are out of scope for this table.)
|
|
|
|
| Caller / path label | Pool | Justification |
|
|
|---|---|---|
|
|
| `bridge_query` (default `/query` filter) | routed | Display reads over the HTTP bridge; bounded staleness acceptable. |
|
|
| `bridge_query` + `consistency: strong` | **writer** | Client-declared write-influencing read (canvas save precondition, restore precondition, post-write ancestry verification). |
|
|
| `req_historical` (WS REQ historical page) | routed | Display backfill of a subscription; per-row re-filter absorbs a briefly-stale row. |
|
|
| `bridge_thread_aux` (`AuxReader::Routed`, thread aux page) | routed | Thread reply hydration; display, post-verified against the fence wall. |
|
|
| `bridge_count_fallback` (`query_events_routed_bounded`) | routed (bounded arm) | COUNT fallback that materializes rows; bounded arm only, never covered. |
|
|
| `count_req_fallback` (`query_events_routed_bounded`) | routed (bounded arm) | WS COUNT fallback that materializes rows; bounded arm only. |
|
|
|
|
The **writer** row is the only write-influencing entry; every other caller is a
|
|
display or count surface that tolerates bounded staleness. Client canvas reads
|
|
that gate a write set `consistency: strong` (Desktop
|
|
`desktop/src-tauri/src/commands/canvas.rs`, CLI
|
|
`crates/buzz-cli/src/commands/channels.rs`) so they land on the writer row.
|