4 Commits

Author	SHA1	Message	Date
Luke Tornquist	0a362eef15	perf(db,relay): bound thread participant scans and coalesce live summaries (#8635) ... ## Summary A thread summary's `participants` (up to 10 recent authors, used for the facepile) was computed by aggregating **every** reply in the thread. The live `kind:39005` emit runs that aggregate on the writer after every reply insert and every threaded delete. Each summary therefore cost O(thread), recomputed after every reply, so one very large thread could saturate the writer. ### Scale and impact A single long-running thread can grow to hundreds of thousands of replies and take several new replies per second. Each of those replies then pays a whole-thread aggregate on the writer, so writer CPU follows the reply rate of the largest active threads, and the read-side participant queries (channel window, thread root summaries) slow down the same way. Synthetic reproduction (PostgreSQL 17, the repo's `schema/schema.sql`, default `shared_buffers`): one channel with 200k background messages and three threads of 450k, 6k and 300 replies from a few authors with ~800-byte bodies. `EXPLAIN (ANALYZE, BUFFERS)` of the participant query, warm cache (runs 2–3 of 3): | Thread | Before (whole-thread aggregate) | After (newest 1000 replies) | |---|---|---| | 450k replies | ~7.6 s, ~117k buffers read, ~16k temp blocks spilled | ~6.8 ms, ~4k buffers | | 6k replies | ~25 ms, ~24k buffers | ~7 ms, ~4k buffers | | 300 replies | ~2.6 ms, ~1.2k buffers | ~2.6 ms, ~1.2k buffers | Cost now stops growing at 1000 replies, and small threads cost the same as before. With coalescing, a busy thread runs this query at most about once per second per pod instead of once per reply. ### Change 1. **Bound the participant scan to the newest 1000 replies.** All three participant readers now share one SQL fragment (`recent_thread_participants_sql!`): - `get_thread_summary` (the live 39005 emit) - the channel-window batch - `resolve_thread_root_summaries` The fragment walks `idx_thread_metadata_window` newest-first with `LIMIT 1000`, then joins `events` only for those rows, to get `pubkey` and drop deleted replies. Deleted replies keep their `thread_metadata` row, so they count toward the 1000 but contribute no author. It aggregates by author and returns the 10 most recently active. Cost no longer grows with thread size. Desktop and mobile show the 3 most recent participants and the forum view shows up to 10, so an author who hasn't posted in the last 1000 replies is not visible in any client today. `NIP-CW`, `docs/bridge-channel-window.md` and the `ThreadSummary::participants` doc now state the bound, deleted replies included. 2. **Coalesce live summary emits per thread root.** `emit_live_thread_summary` goes through a new `CoalescedRefresh` keyed by `(community, root)`: - A quiet thread emits immediately, as before. - On a busy thread, emits are spaced at least 1s apart per pod. A trailing emit covers the rest of the burst and reads state committed after its last mutation. - Because emits are at least 1s apart, consecutive overlays from one pod always get strictly increasing `created_at`, so the client's newest-wins replacement is never tied. - A panicking or cancelled refresh releases its key, including a task dropped before its first poll: the release guard is built before the spawn. Each run is bounded by a 30s timeout so a hung refresh can't wedge it. ### What got simpler - Three hand-written copies of the participant aggregate are now one fragment. - `get_thread_summary` is one statement instead of two round trips. - The channel-window batch dropped its `ROW_NUMBER()` window function and the `GROUP BY root, pubkey` over every reply of every root on the page. ### Related issue None found. ### Testing - `participant_summaries_read_only_the_newest_replies` (buzz-db, Postgres) seeds a thread with 1001 replies: - an early author, pushed just outside the window - a filler author - a deleted reply - a latest author All three readers must return `[latest, filler]`. Mutant check: removing the `LIMIT` (whole-thread aggregate) turns it red. - `coalesced_refresh` unit tests (paused tokio clock): - A 100-request burst runs once immediately and once trailing, then frees the key. - Separate keys run independently. - A panicking refresh doesn't wedge its key. - A hung refresh times out and releases its key. Mutant check: removing the timeout turns it red. - A request that arrives while a run is in flight gets one trailing run, and that run reads the state written mid-run. Mutant check: dropping the rerun flag turns it red. - A task dropped before its first poll (runtime shut down) still releases its key. Mutant check: building the guard inside the task turns it red. These run in `just test-unit` (added to the buzz-relay `--lib` filter and `scripts/run-tests.sh`). - Blox gate: `cargo fmt --check`, clippy `-D warnings` (buzz-db, buzz-relay), and the full buzz-db and buzz-relay Postgres nextest lanes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-28 15:59:19 -04:00
Kalvin C	5a349db91c	feat(relay): add opt-in newest-first thread windows (#7823) ... ## Summary Add an opt-in, newest-first thread window to the authenticated HTTP query path. A filter with `thread_window: true` returns: - reply rows ordered by `(created_at DESC, id ASC)` with an exact composite cursor; - optional bounded reactions, edits, deletions, and delete-of-aux closure; - one relay-signed `kind:39007` bounds event bound to the normalized host, authenticated reader, channel, root, request cursor, kinds, depth, limit, and aux choice. Filters without the flag keep the existing oldest-first thread behavior. Channel windows and `kind:39006` are unchanged. No client opts into this mode in this PR. ### Why extend NIP-CW NIP-CW already defines relay-computed, cursor-paged conversation views over extended NIP-01 filters. Thread windows use the same protocol ideas: ordinary signed rows, composite keyset pagination, optional aux closure, and relay-signed bounds. Adding a thread mode keeps those related wire contracts together instead of creating a second NIP with duplicated terminology and compatibility rules. Extending a draft, optional NIP with an additive mode is normal protocol evolution. Existing fields and meanings are not reinterpreted: - `top_level: true` remains channel mode; - `kind:39006` remains channel-window bounds; - `thread_window: true` selects the new mode; - `kind:39007` has a distinct identity and root/request binding; - extension-unaware clients never send the selector; - legacy thread queries remain unchanged. NIP-CW documents only the wire contract and relay semantics. Database plans, replica internals, and rollout procedures remain implementation concerns. ### Why thread windows use independent modules The implementation deliberately keeps `thread_window.rs` separate in `buzz-core`, `buzz-db`, and `buzz-relay` rather than adding branches throughout the existing channel-window path. The modes share pagination concepts, but their invariants differ: - channel mode selects top-level rows; thread mode requires one root and depths `1..=depth_limit`; - thread mode accepts only conversation row kinds and rejects unknown/conflicting fields; - thread bounds bind the root and authenticated reader because cross-channel aux visibility can differ by reader; - thread aux includes the root and requires complete two-hop closure; - thread mode refreshes writer authorization before selection and before signing; - historical cursor pages may use a proved replica transaction, but fallback must restart both aux hops on the writer; - legacy thread traversal is oldest-first and must not inherit descending cursor semantics. Forcing these rules into channel-window code would create a mode flag across parsing, SQL predicates, bounds construction, authorization, replica fallback, and tests. That abstraction would not remove complexity. It would distribute it. Separate modules keep each contract local and leave the established channel and legacy paths untouched. ### Technical design - Strict request parsing normalizes UUIDs, event IDs, kinds, defaults, and cursors before computing the response binding. - SQL applies community, root, channel, depth, kind, deletion, and cursor predicates before `limit + 1`. - Ordered metadata scanning plus exact event-key lookup avoids the stale-statistics plan that previously exceeded the four-second SQL budget on a 10k-reply thread. - Aux work is bounded by target batches, query count, raw candidate count, response bytes, SQL timeouts, and one shared HTTP deadline. - Fresh authorization-set comparison catches both grants and revocations during closure. The relay returns retryable `503` rather than signing an incomplete page. - Replica failure permanently degrades the request to the writer and restarts closure from its original targets without resetting budgets. - Migration `0049_thread_window_index.sql` adds the mixed-direction index used by newest-first scans. It validates a prebuilt index and bounds startup lock/build time. ### Tradeoffs - The stale-statistics-safe plan is intentionally less clever. Measured warm 100k deep-cursor execution increased from roughly 0.30 ms to 4.80 ms, plus about 8 ms planning, in exchange for avoiding a reproduced multi-second timeout after rapid 10k-thread insertion. - Bounds are authoritative only for the served page. There is no cross-page snapshot guarantee, and replica insertion coverage does not imply latest edit/deletion/aux visibility. - Strict parsing rejects unsupported constraints instead of silently serving a different row set. - Required aux closure fails closed. The relay does not omit aux and sign plausible-but-incomplete bounds. - This PR ships relay capability only. Client rendering, retry UX, live-history overlap, and compatibility fallback remain separate work. ### Related issue No existing issue found. This work follows the thread-loading design and review discussion. ### Testing - `cargo test -p buzz-core`: 258 unit tests and 2 doctests passed. - `cargo check --workspace --all-targets`: passed at the rebased implementation head. - `cargo test -p buzz-db`: 131 non-PostgreSQL tests passed; 275 infrastructure tests were ignored as declared. - Focused PostgreSQL migration/runtime regressions: 3/3 passed against an isolated PostgreSQL container, including both corrected migration-ledger tests. - `cargo fmt --all --check`: passed. - Pre-rebase feature validation passed 398/398 PostgreSQL tests, signed-wire multi-tenant/auth/pagination/aux/revocation probes, clean Compose boot/reboot, and full core tests. - Independent HA validation exercised three relay pods and a physical PostgreSQL standby on the same runtime logic before the final documentation-only cleanup. - Independent final-delta review confirmed exactly two changed files, byte-identical executable migration SQL after comments/diagnostic normalization, matching prebuild/catalog/recovery instructions, and `git diff --check`; no runtime receipts were relabeled to the new SHA. - The pre-push hook ran repository checks but its Rust unit lane exited 100 without exposing an individual failed test in the captured multiplexed output. The independently run affected-package checks above passed; current-head repository CI remains required. Known unrelated baseline behavior remains: the mesh-demo echo test can return 504, and same-second archive → unarchive can deduplicate its membership notification. Neither path is changed here. --------- Signed-off-by: Kalvin Chau <kalvin@block.xyz> Signed-off-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz> Signed-off-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz> Co-authored-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz> Co-authored-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>	2026-09-23 12:01:45 -07:00
Tom Brow	a61239f0d8	fix(mobile): avoid opening empty threads on message tap (#7756) ... Tapping a mobile channel message with no replies currently opens an empty thread. Only enable message-row thread navigation when the existing thread summary reports replies, including replies known to the relay but not loaded locally. Long-press → Reply still starts a new thread. Added widget tests for zero replies, loaded replies, relay-only replies, and starting a thread through Reply. The zero-reply test failed before the fix and passes afterward. Validation: all 227 channel-page tests, the full mobile pre-push checks, and all PR CI checks passed. The full repository gate failed in unchanged Rust ACP timing tests (idle reset, keepalive, and steering deadlines); this PR changes only mobile Dart files. No duplicate open PR or issue found. --------- Signed-off-by: Tom Brow <tomb@block.xyz>	2026-09-21 08:51:05 -07:00
Tyler	62bb9fe8c8	GUI read-model overhaul: server-assembled channel windows (Correct™ pagination + relay-signed bounds) (#1500) ... Signed-off-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub1cc3ha7z055mu0rwwu7806t2wt8mj3pvu0uv5mfp2c50dahaqhczshdalg6 <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co>	2026-07-04 11:51:58 -04:00