6 Commits

Author	SHA1	Message	Date
Jemiah Westerman	cec47c20a3	perf(db): hydrate search hits with exact event timestamps (#9126) ... ## Summary Keep each search hit's existing signed `created_at` alongside its event ID during canonical hydration. WebSocket search, HTTP search, and profile-name search now use a bounded exact-pair database lookup so PostgreSQL can prune unrelated event months. Also constrain channel-create replay detection by the timestamp of the already-available signed event. ## Scope / compatibility - Combined-tree verification also passed: repository-wide `just ci`, 220 unit/source-policy tests, and the full 722-test buzz-db/search PostgreSQL suite. All three temporary timestamp-constraint mutations were killed by functional/plan assertions and restored. - Independent of companion post-insert push and deletion PRs; based directly on `main`. - Deploy/code-only change. No schema/migrations/backfills, persistent or serialized formats, indexes, triggers, functions, procedures, rules, or foreign keys added. - Existing ID-only event APIs remain available for callers without timestamps. - Maximum 500 input pairs, validated before checkout; duplicate inputs do not multiply results; missing/deleted/wrong-community/wrong-timestamp hits are omitted. - Caller ordering, authorization, observability, bounded-only replica routing, proved connection use, non-cancel fallback, and cancellation propagation remain intact. A timestamp is not treated as a completeness proof. - Existing/new relays coexist against unchanged schema; no rollout/prebuild dependency. ## Verification - Compiled `buzz-db`, `buzz-search`, and `buzz-relay` library/tests. - 220 infrastructure-free unit/source-policy tests passed. - 35 focused PostgreSQL tests passed, including cross-month exact pairs, tenant/deleted/missing scope, same-second IDs, duplicate/empty/invalid/oversized/full batches, routing/fallback/cancellation, channel replay, and existing search regressions. - `EXPLAIN ANALYZE BUFFERS` of production hydration SQL verifies only requested event months execute under forced custom **and generic** plans, including the full **500-distinct-pair** shape. A wrong-timestamp decoy detects cross-product matching. - Targeted clippy with `-D warnings`, formatting, diff checks, PostgreSQL discovery, and differential file-size checks passed. - Live isolated relay smoke: exact channel-create replay, HTTP NIP-50 search hydration, profile-name search hydration, and authenticated WebSocket NIP-50 results/EOSE. All three hydration route labels observed in local metrics. - Independent review completed; optional maximum-batch plan coverage added and independently re-reviewed. Compile fixes for SQLx 0.9 API compatibility verified. No production latency improvement or workload-percentage claim is made. --------- Signed-off-by: Jemiah Westerman <jemiah@squareup.com> Co-authored-by: Son of Anton <8e0ef2ab02d69a7ac089ef5eca90d1a3fbbdc546f6322dc34d0c4f3deecea7f4@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 15:43:28 -08:00
Luke Tornquist	47be2e2d3f	fix(db): keep busy-reader history reads off the writer and cap them per pod (#9132) ... ## Problem A routed read that misses the reader's 150ms acquire budget (`reader_acquire_timeout`) falls back to the writer, whatever the read is. Under a burst of subscription-history reads (REQ history, channel and thread windows, feeds, search hydration, COUNT), the reader pool fills, and every miss moves a heavy history query onto the writer. There it competes with the authentication, tenant-resolution and moderation lookups that only the writer serves, and their latency climbs with it. Nothing bounds how many history reads a pod sends to the reader at once, so the reader queue grows without limit as well. Separately, the compatibility families `buzz_db_pool_acquire_wait_seconds` and `buzz_db_pool_acquisitions_total` have no `operation` label, so a breakdown by operation reports every sample as unlabelled. ## Changes 1. **History reads no longer spill onto the writer when the reader is busy** (`Db::route_read`). `route_read` now returns `Result<RouteDecision>`. For `ReaderOperation::SubscriptionHistory`, a `reader_acquire_timeout` on a saturated reader pool rejects the read with `sqlx::Error::PoolTimedOut`. Saturated means every connection the pool may hold exists and none is idle (`Db::reader_saturated`). The check is needed because sqlx also reports a reader that refuses dials as `PoolTimedOut`, and that reader is down, not busy. The relay already answers that with its generic retryable `CLOSED` `error: database error`, so there is no new client contract. A reader that is wrong rather than busy (unreachable, validation failure, proof timeout, stale token) and prechecks that never reach the reader still fall back to the writer. Authorization reads (`is_relay_member`) keep the writer fallback unchanged. 2. **Per-pod cap on reader-served history reads** (`runtime/history_reads.rs`). A FIFO `tokio::sync::Semaphore` admits at most the reader pool size less one fifth (at least one) history reads at once. For example, the default pool of 20 admits 16. The remaining connections are normally free for authorization reads. A read waits at most 1s for a permit and is otherwise rejected the same way. The permit is owned by the reader transaction (`ReaderTx`) and released when the transaction is dropped; routed reads drop it before re-running a failed replica read on the writer. That holds whether the transaction is a per-call routed read or a `ReadSession` the relay keeps for follow-up queries. The cap bounds history transactions, not connections: sqlx returns a dropped transaction's connection asynchronously through the reader release check. Released connections can therefore briefly occupy the free slots, for one round trip normally and at most the 5s release timeout when sockets stall. In that window, authorization reads fall back to the writer as before, and history reads on the saturated pool are rejected, not spilled. 3. **Rejections are observable**: `buzz_db_route_decision{decision="rejected",reason="history_read_cap"|"reader_acquire_timeout"}`. 4. **Legacy pool families gain `operation`.** The population is unchanged, and summing over `operation` gives the original series. 5. **HTTP reads answer 503 `database unavailable` when the database can't be reached** (`api::db_read_error`). This covers a pool timeout, closed pool or socket failure on `POST /query`, `/count` and every bridge `/query` branch that serves a history read (channel window, the three feeds, thread replies), including a shed history read. These used to return a generic 500. The thread-window path already answered 503 for pool timeouts; this applies the same mapping to the shared helper. Pool sizes are unchanged on purpose. A larger pool adds lock contention on the writer rather than relieving it. Docs: `deploy/charts/buzz/README.md` (operator contract) and `PLANS/REPLICA_FULL_READ_ROUTING_DESIGN.md` (fail-closed exceptions). ## What got simpler The concept count is roughly flat. Instead of a caller-side permit, the permit lives in the one type every replica transaction already flows through (`RouteDecision::Replica` / `ReadSessionInner::Replica`). The 11 routed call sites change only by a `?`, and the compiler enforces that change: a call site cannot silently fall through to the writer on a rejection. No new error variant, CLOSED reason or env var. ## Tests - `postgres_tests::saturated_reader_spends_one_acquire_budget_and_rejects_history_reads` (rewrites the former `routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold`). A size-1 reader is held saturated with a cold Aurora cache. A history read (`count_events_routed`) is rejected with `PoolTimedOut` within one acquire budget, records `rejected/reader_acquire_timeout`, and releases its permit. An authorization read on the same reader still falls back as `writer/reader_acquire_timeout`. - `postgres_tests::match_nothing_reads_take_no_connection` (unchanged): its control still requires a history read against an unreachable reader to fall back to the writer. It caught the first cut of this change, which rejected that read. - `postgres_tests::history_permit_follows_the_read_session_and_a_full_gate_rejects`: a held replica `ReadSession` keeps its permit. With the gate full, the next history read waits the full permit budget and is then rejected as `rejected/history_read_cap`, with no writer route recorded. Dropping the session frees the permit and the next read routes to the replica. - `history_reads::tests`: a cap sizing table, plus bounded-wait rejection and reopen under paused time. - `observability` label tests now require `operation` on both legacy families. - `api::db_read_error_tests::unavailable_database_maps_to_retryable_503`: pool timeout, closed pool and I/O map to 503 `database unavailable`. Other errors stay 500. - `postgres_tests::shed_history_read_maps_to_503_on_every_bridge_branch`: with the reader saturated, `/query` channel-window, mentions, needs_action, activity and thread-reply requests each answer 503 `database unavailable` (each serves 200 beforehand). - Mutation-checked on Blox: re-enabling the writer spill, forcing the saturation check either way, skipping the gate, dropping the permit early and removing the legacy label each fail a test. ## Residual - During a reader failover whose old sockets stall, both slots sit in the release check, so the pool reads as saturated. History reads are shed as retryable for up to the release bound (5s) instead of loading the writer. `stalled_reader_sessions_release_their_slots_for_new_connections` now retries through that window and still requires recovery onto the reader. - A reader that accepts no TCP at all can fill the pool with in-flight dials and look saturated. Its history reads are then rejected rather than moved to the writer, which is the same outcome as a full gate. - Shedding relies on clients backing off on a retryable `CLOSED` (RPR #9). WebSocket clients already treat `error: database error` as retryable. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 22:12:30 +00:00
Will Pfleger	9b1c4f98e2	🤖 perf: bound long-thread aux reads and make query deadlines terminal (#7854) ... Long threads (over ~100 replies) time out in desktop. The aux hop that loads reactions and other e-tag references builds its filter as one `tags @> …` term per id, joined with `OR`. Past about 100 ids the planner switches to walking each partition's primary key backward, expecting to stop early. When few rows actually match, it scans the whole community. With a dirty GIN pending list, a 188-reply thread never finishes. ### What changes - **Set-based e-tag reads (`buzz-db`).** The aux e-tag query now puts the id set behind a `MATERIALIZED` CTE and matches with `ANY`. Each e-tag read also runs under a transaction-local `statement_timeout` of 20 s. - **Terminal timeout contract (`buzz-relay`).** When a read hits the deadline, the client gets a terminal timeout (`CLOSED` for REQ, the timeout error for COUNT and search), not a retryable failure. - **Subscription lifecycle (`buzz-relay`).** The per-connection subscription map lock is now the lifecycle lock. Claims, replacement, client `CLOSE`, owner-checked timeout and search cleanup, revoke, and connection teardown all change the map, the registry, topic refcounts and terminal frames while holding it. As a result, a timed-out subscription's `CLOSED` goes out before any replacement with the same id can claim it, and claims made after teardown are refused. - **Dropped terminal frame disconnects (`buzz-relay`).** If the `CLOSED` for a retired subscription (owner-checked timeout or channel eviction) can't be queued, the relay cancels the connection instead of leaving the client holding a subscription that will never receive fan-out. A client-initiated `CLOSE` ack is exempt. - **COUNT under the same deadline (`buzz-db`).** The fast COUNT path runs e-tag counts under the same transaction-local 20 s cap as SELECT, so WS COUNT and HTTP `/count` produce the terminal timeout too. - **Replica reads surface the deadline (`buzz-db`).** When a replica-routed read is cancelled by `statement_timeout` (SQLSTATE `57014`), the error is returned instead of re-running the query on the writer, which would run the same slow statement. Other replica errors still fall back to the writer. `PLANS/REPLICA_FULL_READ_ROUTING_DESIGN.md` records the exception. - **Deadline handling (desktop).** Only the relay's `query timed out` counts as a query deadline. A client-side `request timed out` stays retryable. For one-shot reads a deadline is terminal: the global query retry (budget 1), thread-replies retry and reaction hydration don't re-send the same slow query. Reaction ids held back by a deadline are released when the channel is opened again, so reopening retries them while same-channel re-renders don't. Live subscriptions, in both the renderer and the native archive-sync socket, keep treating a deadline `CLOSED` as transient: they back off and resubscribe with the same filter instead of being dropped. - **Deadline handling (mobile).** Mobile recognizes the deadline on both transports: HTTP `503 {"error":"query timed out"}` and the WS `CLOSED` reason. - The root `ProviderScope` retry policy returns no retry for deadline errors and keeps Riverpod's default backoff for everything else. - Background thread-summary recounts don't retry a deadline. - Forum posts polling pauses while the last load settled on a deadline; reopening the forum retries. - A channel's newest-window load surfaces a deadline instead of falling back to legacy WS history, and the channel list's batched queries return "unavailable" on a deadline instead of fanning out to per-filter fallbacks. An unavailable batch is not treated as empty. Known last-message timestamps and unread state are kept, and unread catch-up moves `lastMessageAt` forward (never backward) the same way live events do, so a cold start with no known timestamp still gets one. - A live subscription that gets a deadline `CLOSED` backs off and resubscribes with the same filter. History requests still fail on it. - Settled load failures show a Retry control, matching desktop's thread-replies error card: a shared `LoadErrorView` on the channel timeline, forum thread and forum posts list, and an icon button beside the thread replies summary. While the thread query has no value, the thread page shows the route snapshot plus live, cached and optimistic replies, with the loading or error status still visible. - Other re-sends (reconnect, foreground resume, live events, the unread backstop, scroll) are not suppressed; each is bounded by the relay's 20 s statement deadline. - **Long-thread seeder.** `just seed-long-thread [replies]` posts one thread (default 187 replies) with some reactions to the local dev relay, for reproducing long threads in the apps. It uses a throwaway key, refuses non-loopback relays and ignores ambient `BUZZ_*` credentials. Writes are paced to fit the stock relay rate limit. **Lifecycle concurrency.** The lock serializes one connection's subscription changes. Other connections can still be delayed by pressure on the shared pubsub command queue, and by serial eviction loops waiting behind a blocked connection. This PR does not claim strict isolation between connections. **Recovery vs. load.** Clients stop automatic retries of the same one-shot read, but they don't remember deadlines across other triggers: reconnect, resume, live events and the unread backstop can each send the read again later, bounded by the 20 s statement deadline. A live subscription that hits the deadline resubscribes with its original filter under the existing 1–30 s backoff, and against a filter that keeps timing out it retries at the 30 s cap indefinitely, so the worst case is one bounded query per subscription every 30 s. `since` is deliberately not advanced to now on resubscribe, because that would skip events missed while the subscription was closed. Unchanged from `main`: when mobile's channel history sync fails with cached messages available, the cached messages are shown without the error or Retry control. ### Benchmark Local reproduction on an isolated bench stack (not committed): - PG16, migrations applied, one 188-reply thread plus 5M background events in the same community. - Each request replays the desktop thread-replies filter through the relay, with `include_aux` on. Times are total ms for 3 runs, with a 60 s client cap. GIN pending-list state is controlled with `VACUUM` (clean) or by refilling the pending list (dirty), and recorded per partition with `pgstatginindex`. Dirty runs held 402 pending pages on every partition's tags index at the start and end of each matrix. Clean runs held 0–1 pages (relay startup writes). | build | GIN pending state | limit 10 | 50 | 100 | 200 (188 ids) | |---|---|---|---|---|---| | `main` | clean | 26 / 16 / 13 | 56 / 51 / 52 | 92 / 93 / 95 | 662 / 639 / 641 | | `main` | dirty | 315 / 298 / 327 | 22141 / 15888 / 16769 | 28992 / 27819 / 28080 | timeout ×3 (>60 s) | | this PR | clean | 35 / 21 / 14 | 57 / 49 / 48 | 82 / 80 / 75 | 153 / 150 / 135 | | this PR | dirty | 290 / 254 / 251 | 1257 / 1204 / 1206 | 2432 / 2378 / 2382 | 4333 / 4706 / 5104 | Baseline `origin/main` `b37e47721`; this PR at `5f2650f5f`, whose e-tag read builders are identical to the current head. At head `1ad81d868` against `origin/main` `15d44dcce`, limit 200 measured 145–179 ms clean (`main` 683–715) and 4.41–4.51 s dirty (`main` timeout ×3). Every completed cell returns the same signed event set: 209 events, 187 replies at limit 200. With `include_aux` off, every cell is ≤66 ms. The thread-window loader (`select_aux`) is unchanged from `main`, and its timings and captured plans match `main`'s. Actual aux-hop statements captured from each relay binary with `auto_explain` (analyze + buffers), execution ms: | build | clean, 100 ids | clean, 188 ids | dirty, 100 ids | dirty, 188 ids | |---|---|---|---|---| | `main` (N-way `OR`) | 40 | 298 | 28785 | 62892 | | this PR (`MATERIALIZED` fence + `ANY`) | 23 | 45 | 1992 | 3539 | **What the fence changes** - On `main`, the planner combines `ORDER BY created_at … LIMIT` with an overestimated match count and walks each partition's primary key backward, expecting to stop early. With few real matches it scans the whole community: the dirty 188-id statement touches ~4.9M shared buffers on `main` vs ~0.6M with the fence. - The fence prevents that early-stop plan. Across the measured cases (1–188 ids, both pending states), the planner used the GIN tags index. The fence doesn't force GIN in general; a broad enough filter could still get a different plan. **Tradeoff:** the fence materializes every match before sorting. In this bench small limits stayed within run-to-run noise of `main`. A single hot id (5,000 reactions, limit 100) measured 30–34 ms unfenced vs 35 ms fenced. **Deadline:** each e-tag read (SELECT and COUNT) runs under a transaction-local `statement_timeout` of 20 s. It only tightens: a shorter operator timeout (`BUZZ_DB_STATEMENT_TIMEOUT_MS`) is kept. It's a per-statement limit, not a per-request one, so a request that runs several pages or hops can take longer than 20 s in total. The benchmark does not force the deadline. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz>	2026-09-28 16:36:40 -04:00
Will Pfleger	b6a26556ec	feat(canvas): add version history with atomic restore (#6780) ... 🤖 Adds append-only channel canvas history, restore, and relay-authoritative optimistic concurrency so stale edits cannot silently replace a newer canvas. ## What this adds - **Desktop history and restore** — browse revisions with author, timestamp, preview, and diff; restore an older revision as a new signed head after explicit confirmation. Persisted empty canvases remain reachable, mutation outcomes are announced accessibly, and focus moves to a stable destination after save or restore. - **CLI history and restore** — `buzz canvas history --channel …`, `buzz canvas get --channel … --revision <id>`, and `buzz canvas restore --channel … --revision <id>` use keyset pagination over the retained revision stream. - **SDK canvas builders** — optional `expected-revision` tags, bounded ancestry classification, and shared timestamp discipline keep first-party writes ahead of the head they read without propagating poisoned future timestamps. ## Authoritative concurrency contract Tagged canvas writes use compare-and-swap semantics in the relay and database: - `expected-revision=none` succeeds only when no live canvas exists. - An event ID succeeds only when it names the canonical live head. - A byte-identical replay succeeds only when that event is already the canonical live head. - A successful candidate must sort strictly ahead of the current head under `created_at DESC, id ASC`. - Malformed preconditions and stale or non-advancing writes are rejected before event rows, mentions, or fan-out are committed. The database serializes tagged writes, unconditional kind `40100` writes, and canvas soft deletion on the same `(community, kind, channel)` advisory key. Untagged `buzz canvas set` remains an unconditional write, but participates in that serialization boundary so it cannot race a tagged save or deletion. Deletion classification is derived inside the datastore transaction; unrelated event kinds do not take the canvas lock. ## Read and timestamp consistency Precondition and post-write ancestry reads request strong consistency from `/query`, pinning them to the writer when replica routing is enabled. Display-only history remains replica-eligible. First-party canvas writers stamp `created_at = max(now, head + 1)` with a conservative client skew limit. The relay applies a canvas-specific future-timestamp ceiling before persistence, preventing an accepted boundary head from making subsequent first-party writes invalid while preserving the broader ingest limit for other event kinds. ## Durable outcomes and recovery After an accepted publish, a failed verification read is reported as durable success with `verified: false` rather than a failed save. Desktop and CLI preserve the new event ID and direct users to History when verification is unavailable. A successful read walks the bounded, cycle-guarded `expected-revision` ancestry chain so transitive descendants are not misclassified as supersession. Canvas caches invalidate for every settled mutation outcome, including a retained but superseded revision. The 409 reconciliation path in `cmd_restore_canvas` now separately establishes persistence evidence from ancestry classification. `canvas_write_survived` checks reachability only; a secondary writer-pinned IDs lookup distinguishes "A absent" from "A present but superseded by a legacy write." The four-way outcome (survived / persisted-but-superseded / genuinely-absent / read-fail) is now correct for all lost-response shapes including unconditional legacy writes. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>	2026-09-23 11:03:54 -04:00
tlongwell-block	a13cb05685	Delete HUDDLES_IMPLEMENTATION.md (#520)	2026-05-09 03:39:21 +00:00
tlongwell-block	454781caae	Sprout speaks Nostr. Nothing else. (#475)	2026-05-05 12:03:06 -04:00