3312 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Luke Tornquist	bd1b17dee8	perf(db): plan multi-value #d reads with their values (#9131) ... ## Summary A REQ with several `#d` values for a NIP-33 kind runs as a named, cached statement. A client resolving channel metadata for up to 128 channels sends this kind of REQ. After a few runs on a connection, Postgres can switch the statement to a generic plan, which ignores the bound values: - The `d` match stops being an index condition on `idx_events_parameterized`. - The generic plan instead walks the author's rows in `created_at` order and filters `d_tag` row by row. - The `LIMIT` is one row per requested value, so the walk never stops early. It reads every stored version of the kind, not one probe per value. This change makes three fixes: - Multi-value `#d` reads now run as unnamed statements, as tag-value matches already do (`plans_with_bound_values`), so Postgres plans every call with its values. This covers both the page `SELECT` and the `COUNT`. - The values are bound as one array (`d_tag = ANY($n)`) instead of one bind per value. The SQL text no longer changes with the number of values, which matches how channel scope is bound. - The page `SELECT` and the `COUNT` now share one `push_d_tag_filter` helper instead of two copies of the `d` predicate. **What got simpler:** - One helper replaces the duplicated `d` predicate blocks. - A `#d` read no longer produces a different statement for every value count. ### Related issue N/A. None found. ### Testing - New unit test `d_tag_sql_does_not_depend_on_value_count`: the SQL text is identical for 2 and 128 values, for both the page query and the count query. - Postgres test `value_matched_queries_are_not_cached_as_prepared_statements`, extended from the tag-value test. It runs each `#d` page and count query 6 times and asserts no `d_tag = ANY(` statement is left in `pg_prepared_statements`. It also checks the page and count results. - Mutants, each caught by the matching test: - dropping `d_tags` from `plans_with_bound_values` fails the Postgres test; - going back to a per-value `IN (...)` list fails the unit test. - Gates run at 5f71e7143b8bc4cfbecb6b4cf886be3827d54943: - `cargo fmt --check` - `cargo clippy -p buzz-db -p buzz-relay --all-targets -D warnings` - `buzz-db` unit tests: 215 passed - `scripts/postgres-test-run.sh -p buzz-relay -p buzz-db --lib --tests`: 1291 passed - `scripts/test-postgres-test-discovery.sh` - Local `EXPLAIN ANALYZE` on synthetic data (50k versions of one author's kind 39000 rows, 128 `#d` values): - With `force_custom_plan`, `d_tag = ANY(...)` is in the Index Cond on every partition. - With `force_generic_plan`, the partition holding the rows drops `d_tag` to a row filter. - On larger data the generic plan can switch to an ordered `created_at` walk. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 19:41:03 -05:00
Diem Nguyen	133c883b8c	fix(mobile): confirm before starting a Huddle from the header (#9155) ... Tapping the headphones button in a channel header used to start a Huddle straight away, which turned on the microphone and posted "Huddle started" to the channel. The button is easy to hit by accident. Now, when no Huddle is running there, the button asks "Start a Huddle?" first, with Cancel and Start Huddle. - `showConfirmation` in `shared/widgets/confirmation_dialog.dart` adds a non-destructive mode to the shared confirm helper. `showDestructiveConfirmation` now calls it with `destructive: true`, so existing callers behave the same. On iOS the native UIKit alert reads a new `destructive` argument and uses `.default` style instead of `.destructive` when it's false. On other platforms the confirm button keeps the default `FilledButton` style instead of the error color. - `_huddleNavigationAction` asks before calling `start`. That covers both the Material header button and the iOS native navigation action. Opening or joining a Huddle that's already running stays one tap, and so do the explicit "Start new" buttons inside the Huddle sheets. Companion to block/buzz-app#2687, which adds the same confirmation on desktop. buzz-app#2660 also moves the desktop button away from the members button, but there's no equivalent move on mobile: the headphones is the only action in the header. ### Screenshots Rendered by a scratch widget test from the real `ChannelDetailPage` (Android/Material path, Inter loaded) at 390×844. The scratch test has been removed. On iOS, the same confirmation shows as a native UIKit alert, which isn't captured here because this machine has no iOS Simulator. Before: tapping the headphones goes straight into a call. <img src="https://github.com/user-attachments/assets/adced280-2abd-44ad-a079-1c83749892a0" width="300" alt="Before"> After, light and dark, cropped to the new dialog: | Light | Dark | | --- | --- | |  |  | ### Validation - The existing journey "community transition cancels and awaits an in-flight Huddle start" now taps the button and checks that the dialog appears, that Cancel closes it without publishing anything, and that tapping again and choosing Start Huddle starts the call. With the confirm removed, this test fails. - `channel_member_profile_actions_test` checks that destructive confirmations still send `destructive: true` to the iOS bridge. - Full `flutter test`: 4,508 passed. `flutter analyze` and `dart format` are clean. - Not yet checked: the iOS native alert on a device or simulator, and a human trying it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Diem Nguyen <diem@squareup.com> Co-authored-by: Fizz <6be7be23ec5bba868f3e1db5ec86013d76bd1a293dce5b435a192158fec13641@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 16:33:07 -08:00
Luke Tornquist	e3a0f3d59d	feat(cli): send NIP-FI evidence via credential chain and Buzz desktop (#8999) ... ## Summary Relays that enforce federated identity (NIP-FI) reject every protected `buzz` CLI request with `401`. The CLI never sends `Nostr-Federated-Identity`. This PR gives the CLI a credential chain, lets it present NIP-FI evidence, and lets it take its identity from Buzz desktop. Relays without NIP-FI behave exactly as before. **Credential chain** (`crates/buzz-cli/src/credentials.rs`) - Raw `Keys` are replaced by `Credentials`, which provide the public key, signing through nostr's `NostrSigner`, NIP-44, and the NIP-OA auth tag. NIP-FI evidence comes through `evidence(relay)`. - The explicit key (`--private-key` / `BUZZ_PRIVATE_KEY`) is the first link. A later link can add a signer that holds no local key (a desktop provider, for example) without reshaping callers. - Operations that need key material rather than a signature are purpose-bound methods instead of a key accessor: - `engram_d_tag` (the NIP-AE HMAC over the conversation key) - the relay-scoped GIF customer id (random and fixed for the process when there is no local key) - `buzz mem` now encrypts, decrypts and derives the `d` tag through the provider. Invalid ciphertext skips an entry; a signer failure aborts, so an unavailable credential never reads as empty memory. `buzz_core::engram` gains the composable steps behind it: `encode_body`, `event_builder`, `validate_envelope` and `check_d_tag`. `build_event` and `validate_and_decrypt` are unchanged wrappers, so `buzz-acp` and desktop callers don't move. - Agent-management drafts (`agent_management.rs`) encrypt and sign through the provider. **Evidence on every protected request** (`crates/buzz-cli/src/client.rs`) - A single `authorize` helper gets the NIP-FI assertion first, then signs the proof just before sending, so a slow mint cannot age it past NIP-98's 60 s window. It sets these on every protected route: `Authorization` (NIP-98 or Blossom kind:24242), `x-auth-tag`, and `Nostr-Federated-Identity: Bearer …`. The routes are `/query`, `/count`, `/events` (stored and moderation), `/upload`, `/media/upload`, `/media/*`, `get_authed`, `post_json_authed` and `post_json_once_authed`. - Exempt routes (`get_public`) are unchanged. - The WebSocket publish sends the header on the upgrade via `connect_authenticated_with_headers`. NIP-42 AUTH signs through the signer (`buzz-ws-client` now takes `NostrSigner`; `&Keys` callers compile unchanged). - An upgrade denial is an HTTP response under NIP-FI. `WsClientError::upgrade_rejection` returns its status and body, which map like any other relay error, so a `403` still says whether evidence or authorization was refused. A denial after NIP-42 (such as a key mismatch) is a 1008 close; `WsClientError::PolicyClose` keeps its reason and the CLI reports it as an `auth_error`. - The WebSocket publish caps each network phase as a whole, writes included: connect, AUTH challenge (with any Pongs), AUTH send plus OK, EVENT send plus OK, and a best-effort close. NIP-42 signing between phases is bounded by the signer. The relay drops a socket not signed in within 5 s, so a sign-in that is throttled or comes back after the socket closed reconnects and signs the new challenge, at most 3 connections. Each connection fetches current NIP-FI evidence for its upgrade, and a throttled sign-in is waited out between connections, so a dead challenge is never signed. EVENT is signed before connecting and sent once, after AUTH succeeds. Running out is `unavailable` (exit 2, `retryable:true`). - Relay traffic follows no redirects: reqwest strips `Authorization` across origins but not `x-auth-tag` or `Nostr-Federated-Identity`, so a 3xx is returned as a relay error. Credential header values are marked sensitive. **Headless evidence source** (`crates/buzz-cli/src/federated_identity.rs`) - Set `BUZZ_FEDERATED_IDENTITY_TOKEN` (your own NIP-FA sign-in session: a human session for your key, not an agent grant) and `BUZZ_FEDERATED_IDENTITY_ADAPTER` (the adapter base) to enable it. Both are env only: command lines are visible to other local users. - The CLI requests an assertion and the adapter mints it, through `POST {adapter}/v1/identity/assertions` with a NIP-98 proof in `Nostr-Authorization` (NIP-FA §Relay assertion). - It checks the returned pubkey and expiry, parses the assertion as a compact JWS (three base64url segments, JSON header with `alg`, JSON payload), and requires its `exp` to equal `expires_at`. It caches one assertion per process, bound to the relay when credentials are resolved, and re-mints at the earlier of half-life and 30 s before expiry; the desktop link caches and refreshes its assertion by the same rule. Concurrent callers share one mint. An HTTP proof is signed again, after current evidence, when the assertion changed while it was signed or the evidence check aged it 30 s or more, up to 3 rounds, then fails as `unavailable` (retryable). - Compared with the separate rules before: a 300 s assertion now refreshes at 150 s (was 270 s) on both links, a 40 s adapter assertion at 10 s (was 20 s), and a 20 s desktop assertion is kept for 10 s instead of fetched on every lookup. Desktop assertions now need a readable JWS header with `alg` and a non-empty signature segment. An unsupported headless relay URL now fails when credentials are resolved. - The token goes only to the adapter origin (https, or http on loopback), never follows a redirect, and never appears in errors or `Debug`. - A mint retries network errors, `429` and any `5xx` up to three attempts with jittered backoff (or the adapter's `Retry-After` on `429`/`503`, as delta-seconds or any HTTP-date form, capped at 30 s), on every path (reads, post-once, moderation, stored events, WebSocket). It then fails as `CliError::Adapter` (`adapter_error`, exit 2, `retryable:true`). Relay retry loops never repeat it. A stored write keeps `delivery_unknown` when any earlier attempt may have been delivered, so a later adapter failure never invites a re-sign. Relay mutation retries are otherwise unchanged. - An explicit port 80 relay URL (read numerically from the authority, on any scheme) is a usage error: `wss://host` means port 443, and NIP-FA has no relay on port 80. - A refusal is never retried: a non-transient status with an unreadable or truncated body is still a refusal. - Each NIP-FA denial code maps to an `auth_error` with a specific message. - These settings belong to the explicit key: without one they are not read, and when the desktop link wins a warning names them. - If an enforcing relay answers `401` and no assertion was sent, the CLI returns an `auth_error` that names both settings, or, for a desktop user, says to sign in to the community in Buzz. A desktop user's `403` hint no longer mentions `BUZZ_AUTH_TAG`. **What got simpler** - About ten request sites built `Authorization` and `x-auth-tag` themselves. Five copies of the 401/403 error-mapping block are now one `relay_error`. - The `sign_nip98` / `sign_blossom_*` helpers are gone. Proofs are a three-case `Proof` enum consumed by `authorize`. - `BuzzClient::keys()` (raw secret access from every command) is removed. Commands use `public_key()`, `sign_event` or a purpose-bound credentials method. **Desktop credential provider** (`crates/buzz-cli/src/desktop_provider.rs`, `docs/local-credential-provider.md`) Buzz desktop is a credential provider in the chain, after an explicit key: 1. An inherited `BUZZ_CREDENTIAL_PROVIDER` token (set by the desktop for the agents it launches). It decides the identity: when it is revoked, not bound to the relay or unreachable, the command fails. A paired token, which acts as the human, is never tried. That keeps a well-behaved CLI from acting as the owner by accident; it is not a boundary against an agent that can run its own commands as the owner's OS user, and the contract says so with the mitigations. 2. Otherwise, a paired token stored in the OS keychain (`keyring`; macOS native keychain, Linux Secret Service; no file fallback). A desktop that answers `denied` or `unavailable` fails the command; only an unreachable or unconfirmed advertisement is skipped. Revoked entries are forgotten. New commands: `buzz auth desktop pair` and `buzz auth desktop unpair`. The help no longer marks `BUZZ_PRIVATE_KEY` as required. The CLI talks to the desktop over the loopback contract in `docs/local-credential-provider.md` (NIP-46-shaped envelopes, `Buzz-Credential-Provider` version header, same-user check on the listening socket for paired and inherited tokens, via `/usr/sbin/lsof` on macOS, given 5 s; a check that runs out of time is a retryable `unavailable`, a wrong owner is refused). Every response's `id` must match its request, error envelopes included. Paired tokens live only in the keychain, survive desktop restarts, and are harness-neutral for agents. - **Relay binding:** tokens bind the whole relay URL (`http`/`https` read as `ws`/`wss`, lowercase host, no default port, `/` for an empty path; path and query count). The CLI sends that URL in `pair` and `federated_identity`; NIP-FA audience canonicalization is the desktop's. - **Rate limits:** a `rate_limited` answer refuses before acting, so the CLI re-sends after `retry_after` (capped at 30 s), up to four times, except a NIP-42 sign-in, whose wait goes to the WebSocket reconnect. A re-sent NIP-42, NIP-98 or Blossom proof is re-dated (new id, Blossom `expiration` moved with it); content events are re-sent exactly. If it is still throttled, the command fails as `rate_limited` (exit 2, `retryable:true`), not `auth_error`, whether the throttled call was NIP-98, event signing, NIP-42 or encryption. Transport failures are never re-sent, and relay commands are never replayed. A `buzz mem` list that is throttled partway through completes. - **Memory:** a `nip44_decrypt` answered with `invalid_request` is invalid ciphertext, so `buzz mem` skips that entry (NIP-AE filtering); any other provider failure aborts. - **Keychain changes:** each change reloads the map under a short `flock` on `~/.buzz-foundation/cli-pairings.lock` (directory forced to `0700`, lock opened with `O_NOFOLLOW`) and changes only its own instance; the lock is never held across the approval wait. An unreadable entry is refused, never overwritten; `unpair` with no instance clears it. ### Related Related: [block/buzz-app#2260](https://github.com/block/buzz-app/pull/2260) (the desktop side of the provider contract) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Signed-off-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Will Pfleger <wpfleger@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>	2026-11-06 00:26:22 +00:00
Logan Johnson	c39ee507b2	fix(mobile): find verified people who have no profile (#9086) ... ## What changes Mobile now finds a person who has a relay-verified profile (NIP-FI, kind 30382) but never published a kind:0 profile. This happens, for example, when someone skips profile setup. The relay's `POST /profiles/search` already finds such people by their verified name. Each one comes back as a kind:30382 head with no kind:0. Mobile kept only kind:0 results, so these people did not appear. - `RelaySessionNotifier.searchProfiles` keeps kind:30382 heads from the endpoint. - `VerifiedProfilesNotifier.searched` turns a search page into people: - If the relay authority is not known yet, it first waits for NIP-11 discovery. Otherwise, a search that finished first would drop these people for that query. - It checks each head's signature with the directory's off-isolate proof checker. - Proven heads go into the verified directory, so the rows show verified names right away. - `searchedPeople` (in `verified_profile.dart`) builds one row per person, in the relay's order: - If the person has a kind:0, the newest kind:0 is used, as before. - If not, it uses the proven head that the authority signed, provided it parses as an active verified profile. - These cases add no row: - a head signed by another key; - a head that names the relay key but has a bad signature; - a retraction; - a malformed subject. - **Behavior change for all mobile mentions:** every verified person now matches in mention ranking by their verified full name and username, as well as by display name, including people who have a kind:0. The relay's search matches these same three names. Without this, a person whose display name is their username (`glilienfeld`, full name "Gabi Lilienfeld") came back from the relay for "gabi", and then the local filter dropped them. - Callers: - Mention search, which also serves Add members and New DM. - The user rows in global search, through `directoryUsersFromSearch`. The desktop app has the same fix for Manage members: https://github.com/block/buzz-app/pull/2554 ## Screenshots These come from the real Add members sheet on an Android emulator. Only the relay session, verified-profile discovery and the member actions are faked. The names are made up. Jordan Park has a kind:0. jrivera has only a verified profile, with the full name "Jordan Rivera". The search is "jordan". The row shows the verified display name, which here is the same as the username. | Search finds jrivera | After tapping + on jrivera | | --- | --- | | <img width="300" alt="jrivera is found next to Jordan Park" src="https://github.com/user-attachments/assets/d93b77b1-1bc7-4d09-afc1-f359616a192a" /> | <img width="300" alt="jrivera is selected to add" src="https://github.com/user-attachments/assets/ce6c7a71-b098-434e-8b0e-ed0de8d1a5a2" /> | ## Tests - `add_members_sheet_names_test.dart`: a journey in which Gabi has `display_name` = `glilienfeld` and `full_name` = "Gabi Lilienfeld", and NIP-11 discovery is slow. - While discovery is pending, typing "gabi" shows nothing yet. - When discovery finishes, Gabi appears with no further keystroke. - A head from another key does not appear, and neither does a head that names the relay key but has a bad signature. - **Add member** adds Gabi's key. - I removed each fix in turn: the full-name alias, the signature proof, and the wait for discovery. The journey fails each time, for the expected reason. - `verified_profile_test.dart`: a table of `searchedPeople` cases. - A kind:0 wins over a head, and the newest kind:0 counts. - These find nobody: another signer, a retraction, an uppercase subject, a non-profile kind, and any head when the authority is unknown. - `relay_session_test.dart` and `channel_management_provider_test.dart`: rows for the kept heads and the directory row. - `flutter analyze` is clean, and the full `flutter test` passes on 6efef79cc: 4,462 tests. **Note:** mobile on `main` does not compile at e558f15fa. `account_identity_store.dart` calls the removed `_keys`. That fix is https://github.com/block/buzz/pull/9085. The local runs above had that one-line fix applied but not committed. Until it lands, mobile CI on this PR will fail. ## Not covered - **Search during a reconnect:** discovery starts only when the relay session is connected. If an HTTP search succeeds while the session is reconnecting, that search leaves these people out. - **Discovery failure:** if NIP-11 discovery fails, the search also leaves these people out. - In both cases the mention cache keeps that partial page for the query, and the next search tries discovery again. People who have a kind:0 show as before. Showing the left-out people when the authority arrives later would need the cached results to be rebuilt; that is a possible follow-up. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 18:53:40 -05:00
Brad Seiler	66c3d6917d	feat(protocol): define compact membership discovery (BUZZ-615) (#9117) ... ## Summary Cold login currently downloads full channel rosters to learn the viewer's memberships. This defines compact discovery in NIP-CL and reserves relay-only kind 39008 for a signed, request-bound response. Full kind-39002 rosters remain compatible. The contract binds a complete snapshot to the community, viewer, NIP-98 request and requested channel types. It defines a primary database cut, overflow errors, lazy-roster precedence, reconnect repair and safe fallback across mixed relay versions. The next PR implements both query modes. This PR does not enable either capability. A separate `membership_public` capability uses relay-only kind 39009 for lookups of any subject’s listed-open forum, stream and workflow memberships. It preserves roles and archived state while excluding private, unlisted, DM and system channels. Signed answers bind requester, subject and public scope; they cannot grant membership or sending permissions. The self-mode contract is unchanged. ### Related issue [BUZZ-615](https://linear.app/squareup/issue/BUZZ-615). No duplicate compact-membership implementation found. Related work: [#8898](https://github.com/block/buzz/pull/8898) protects full-roster signing; [#9060](https://github.com/block/buzz/pull/9060) caches membership lookups. ### Testing McKeon independently reviewed the contract and the fixes to request binding, per-channel race fencing and mixed-version fallback. Content checks parse every JSON example and resolve relative links. Paul’s review fixes remove implementation details from the normative text and move the existing 39009 registration and regression test into this PR. Protocol behavior is unchanged. Runtime validation belongs to the implementation PR. This description does not attest human testing. ### Implementation and operator notes Both kinds are registered in `crates/buzz-core/src/kind.rs`, included in `ALL_KINDS`, and rejected as client-submitted events by the relay-only guard. The dependent implementation uses one primary SQL statement’s MVCC snapshot over active `channel_members` joined to nondeleted channels, supported by the partial `(community_id, pubkey) WHERE removed_at IS NULL` index. Public requests share the existing community/requester `ApiCalls` quota configured by `human_api_calls_per_min`. `BUZZ_MEMBERSHIP_SNAPSHOT_ENABLED` and `BUZZ_MEMBERSHIP_PUBLIC_ENABLED` independently default to false. Deploy support across the fleet with both flags off, then enable capabilities independently. Disable the affected capability to roll back; client fallback and back-off behavior remain specified in NIP-CL. Client opt-in and production-shaped throttled validation follow separately. Generated with Codex --------- Signed-off-by: Brad Seiler <seiler@squareup.com> Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz>	2026-11-05 23:52:07 +00:00
Jemiah Westerman	cec47c20a3	perf(db): hydrate search hits with exact event timestamps (#9126) ... ## Summary Keep each search hit's existing signed `created_at` alongside its event ID during canonical hydration. WebSocket search, HTTP search, and profile-name search now use a bounded exact-pair database lookup so PostgreSQL can prune unrelated event months. Also constrain channel-create replay detection by the timestamp of the already-available signed event. ## Scope / compatibility - Combined-tree verification also passed: repository-wide `just ci`, 220 unit/source-policy tests, and the full 722-test buzz-db/search PostgreSQL suite. All three temporary timestamp-constraint mutations were killed by functional/plan assertions and restored. - Independent of companion post-insert push and deletion PRs; based directly on `main`. - Deploy/code-only change. No schema/migrations/backfills, persistent or serialized formats, indexes, triggers, functions, procedures, rules, or foreign keys added. - Existing ID-only event APIs remain available for callers without timestamps. - Maximum 500 input pairs, validated before checkout; duplicate inputs do not multiply results; missing/deleted/wrong-community/wrong-timestamp hits are omitted. - Caller ordering, authorization, observability, bounded-only replica routing, proved connection use, non-cancel fallback, and cancellation propagation remain intact. A timestamp is not treated as a completeness proof. - Existing/new relays coexist against unchanged schema; no rollout/prebuild dependency. ## Verification - Compiled `buzz-db`, `buzz-search`, and `buzz-relay` library/tests. - 220 infrastructure-free unit/source-policy tests passed. - 35 focused PostgreSQL tests passed, including cross-month exact pairs, tenant/deleted/missing scope, same-second IDs, duplicate/empty/invalid/oversized/full batches, routing/fallback/cancellation, channel replay, and existing search regressions. - `EXPLAIN ANALYZE BUFFERS` of production hydration SQL verifies only requested event months execute under forced custom **and generic** plans, including the full **500-distinct-pair** shape. A wrong-timestamp decoy detects cross-product matching. - Targeted clippy with `-D warnings`, formatting, diff checks, PostgreSQL discovery, and differential file-size checks passed. - Live isolated relay smoke: exact channel-create replay, HTTP NIP-50 search hydration, profile-name search hydration, and authenticated WebSocket NIP-50 results/EOSE. All three hydration route labels observed in local metrics. - Independent review completed; optional maximum-batch plan coverage added and independently re-reviewed. Compile fixes for SQLx 0.9 API compatibility verified. No production latency improvement or workload-percentage claim is made. --------- Signed-off-by: Jemiah Westerman <jemiah@squareup.com> Co-authored-by: Son of Anton <8e0ef2ab02d69a7ac089ef5eca90d1a3fbbdc546f6322dc34d0c4f3deecea7f4@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 15:43:28 -08:00
Jemiah Westerman	997ce3a2c9	perf(db): prune post-commit push enqueue by signed event timestamp (#9128) ... ## Summary Carry each inserted event's signed `created_at` through the existing in-memory, post-commit push enqueue path. The worker now looks up the source event by community, ID **and exact timestamp**, allowing PostgreSQL to prune unrelated monthly partitions. All seven direct insert-followup hooks forward the timestamp already bound into their INSERT. No receipt-time or causal-order inference is used. ## Scope / compatibility - Combined-tree verification also passed: repository-wide `just ci`, 220 unit/source-policy tests, and the full 722-test buzz-db/search PostgreSQL suite. All three temporary timestamp-constraint mutations were killed by functional/plan assertions and restored. - Independent of the companion search-hydration and deletion PRs; based directly on `main`. - Deploy/code-only change. No migrations, backfills, schema changes, new indexes, triggers, functions, rules, procedures, or foreign keys. - No persistent/serialized payload changes: `push_match_queue` and `push_wake_outbox` remain unchanged. Their later ID-only lookups are outside this PR. - Admission/gate ordering, commit-only submission, savepoint rollback behavior, duplicate suppression, bounded queues, best-effort drops, and existing lock/statement/client deadlines are retained. - Existing and new relay versions can coexist against the unchanged schema; no rollout ordering or operator prebuild is required. ## Verification - Compiled `buzz-db`, `buzz-search`, and `buzz-relay` library/tests. - 218 infrastructure-free unit/source-policy tests passed. - 12 focused PostgreSQL tests passed, including new timestamp propagation and savepoint tests plus existing follow-up/admission/gate regressions. - `EXPLAIN ANALYZE` of the exact production enqueue INSERT verifies a single event month executes under forced custom and generic prepared plans. - Targeted clippy with `-D warnings`, formatting, diff checks, PostgreSQL discovery, and differential file-size checks passed. - Live isolated relay smoke: channel creation/replay, signed message ingestion/readback, and completed post-commit enqueue worker metric. Test lease had empty subscriptions; no push gateway was contacted. - Independent local review completed: no remaining blockers. No production latency improvement or workload percentage is claimed; this establishes the pruning mechanism and behavior preservation. --------- Signed-off-by: Jemiah Westerman <jemiah@squareup.com> Co-authored-by: Son of Anton <8e0ef2ab02d69a7ac089ef5eca90d1a3fbbdc546f6322dc34d0c4f3deecea7f4@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 15:43:02 -08:00
Jemiah Westerman	0cfa50ac70	perf(db): use exact event timestamps for transactional deletion (#9130) ... ## Summary Preserve the target event's already-known signed timestamp through deletion planning and use `(community_id, id, created_at)` for the canonical transactional read, soft delete, and retention-free purge. Known-target operations can prune unrelated monthly partitions instead of repeatedly probing by ID alone. Public ID-only deletion callers retain their discovery step and reuse the discovered timestamp for subsequent mutations. Operator deletion similarly reuses its existing discovery result. Canonical kind/channel classification and locking remain database-owned inside the transaction. ## Scope / compatibility - Combined-tree verification also passed: repository-wide `just ci`, 220 unit/source-policy tests, and the full 722-test buzz-db/search PostgreSQL suite. All three temporary timestamp-constraint mutations were killed by functional/plan assertions and restored. - Independent of companion search-hydration and push-enqueue PRs; based directly on `main`. - Deploy/code-only change. No schema, migrations, backfills, persistent/serialized data formats, indexes, triggers, functions, rules, procedures, or foreign keys added. - Runtime `DeletionTarget` now carries the target timestamp; no wire payload changes. - Community admission, canvas replacement locking, replay repair, atomicity, retention-free purge/mention cleanup, artifact restrictions, and once-only thread accounting remain intact. - Extract deletion helpers into a focused private module, preserving existing public reexports. - Existing/new relays coexist against the unchanged schema; no operator prebuild or rollout dependency. ## Verification - Compiled `buzz-db`, `buzz-search`, and `buzz-relay` library/tests. - 218 infrastructure-free unit/source-policy tests passed. - 23 focused PostgreSQL tests passed: cross-month/exact-key/tenant isolation, missing/wrong/deleted targets, rollback and concurrent replay/counters, retention-free cleanup in desired and migration schema modes, canvas locking, and existing deletion/admin regressions. - Actual production prepared SELECT/UPDATE/purge SQL prunes to one event month under custom and generic plans. - Targeted clippy with `-D warnings`, formatting, diff checks, PostgreSQL discovery, and differential file-size checks passed. - Live isolated relay smoke: signed message and reply ingestion, atomic deletion, deleted payload absent from queries, thread counters return to zero, and safe deletion replay. - Independent local review completed: no remaining blockers. No production latency/workload benchmark is claimed. --------- Signed-off-by: Jemiah Westerman <jemiah@squareup.com> Co-authored-by: Son of Anton <8e0ef2ab02d69a7ac089ef5eca90d1a3fbbdc546f6322dc34d0c4f3deecea7f4@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 15:42:41 -08:00
Luke Tornquist	b6d4403149	perf(redis): index mesh ready records and share one Redis connection (#9113)	2026-11-05 18:42:02 -05:00
Matt Wilcox	4aae6c22aa	fix(mobile): delay iOS message actions until a deliberate hold (#9049) ... ## Summary Increase the iOS message and reaction hold delay from 200 ms to 500 ms, giving users more time to start scrolling without opening a menu. Movement detection and other platforms remain unchanged. Apple documents a [0.5-second default for UILongPressGestureRecognizer](https://developer.apple.com/documentation/uikit/uilongpressgesturerecognizer/minimumpressduration). ### Related issue Fixes #8500. ### Testing - Regression checks fail with the old timeout and pass with the change: no recognition at 499 ms, recognition by 501 ms, and scrolling after a 300 ms pause cancels the hold. - Device testing pending; the local iOS simulator SDK is unavailable. Before review, verify deliberate holds, slow scroll starts, and reaction holds on an iPhone. 🤖 Generated with Goose Signed-off-by: Matt Wilcox <mwilcox@squareup.com> Co-authored-by: Gen <8a28d770b293a0b223e7c671e5985a35fd9bc1a3c7e3fa92ff715bd2fe61b253@squarewebplatform.communities.buzz.xyz>	2026-11-05 15:34:10 -08:00
Blake McAnally	b0f68a06ad	feat(cli): user status expiry and status read (#8777) ... > Replaces #8768, which was opened from a fork. ## Summary This change adds two things to `buzz-cli`. `buzz users set-status` now accepts `--expires`. The status then clears itself at that time. `buzz users status` is a new command. It reads the visible status of one or more users. ## Design `--expires` takes a unix timestamp or a duration. A duration is a number with a unit: `s`, `m`, `h`, `d`, or `w`. A bare number is a unix timestamp. The CLI had no duration parser to reuse. The only one is private to `buzz-workflow`. A small parser now lives in `users.rs`. The CLI rejects an expiry that is not in the future. It does this before it signs anything. The error text says that a bare number is a timestamp. `--expires` cannot combine with `--clear`. A status with no text and no emoji cannot have an expiry. The SDK rejects that shape. `buzz-sdk` has a new builder, `build_user_status_expiring`. It adds `["expiration", ts]` after the emoji tag. This is the tag order desktop uses. `build_user_status` now calls the new builder with no expiry. Its output is unchanged. Existing callers need no change. `users status` queries kind 30315 with `#d: general`. It picks the newest event for each author. A higher `created_at` wins. On a tie, the lower event id wins. Desktop uses the same rule. The CLI then drops a user when the newest event has no text and no emoji. It also drops a user when the status has expired. An expired status means `expiration <= now`. An older status never shows behind a newer cleared or expired one. The expiration tag is read like desktop's `parseInt`. A value such as `90abc` reads as 90. A value such as `soon` reads as no expiry. The output is an array: `[{pubkey, text, emoji, expires_at, updated_at}]`. `emoji` is an empty string when there is none. `expires_at` is `null` when the status does not expire. A user with no visible status has no row. `--pubkey` takes hex, npub, or an exact display name. It reuses the name resolver of `messages`. It is repeatable. The limit is 100, because each name costs one profile search. With no `--pubkey`, the command reads your own status. A relay response that is not valid JSON is an error. It is never an empty result. ## Testing What ran: - `cargo test -p buzz-cli -p buzz-sdk` passes. - `cargo clippy -p buzz-cli -p buzz-sdk --all-targets -- -D warnings` passes. - `cargo fmt --all --check` passes. - `just file-size-check` passes. The CLI tests use an axum fake relay. They call the production functions `publish_status` and `read_statuses`. These are the same functions that the commands call. The tests cover these cases: - One publish carries the right tags. - Bad expiries cause no publish. - Expired, cleared, and malformed statuses. - Newest-event choice and tie-break. - Hex, npub, and name input. - The 100-pubkey limit. - An invalid relay response. - A publish followed by a read. Interop evidence: I ran the real `buzz` binary against a local fake relay. No live relay was used. I fed the published event to desktop's own `parseUserStatusEvent` and `fetchUserStatusLookup` with node. Desktop showed the same text, emoji, and `expiresAt` that `buzz users status` printed. The check printed `INTEROP OK`. Mutation checks: I broke each guard once. Each break made a test fail. The guards were: - the expiry comparison, `<=` changed to `<` - the future-expiry check - the `d=general` filter - the newest-first order - the tie-break - the 100-pubkey limit - the author de-duplication - the invalid-response error - the hide-cleared-status check - the duration overflow check - the SDK expiration tag - the SDK empty-status check I restored each guard after the check. One note on process: I wrote the code and the tests in the same pass. I did not write the tests first. The mutation checks cover the gap. ## Manual test steps Use a test identity on a relay that you may write to. Set `BUZZ_RELAY_URL` and `BUZZ_PRIVATE_KEY` first. 1. Set a status that expires in two minutes. ```bash buzz users set-status --text "testing expiry" --emoji "🧪" --expires 2m ``` Success: the output has `"accepted":true`. 2. Read the status. ```bash buzz users status ``` Success: the output is one row. `text` is `testing expiry`. `emoji` is `🧪`. `expires_at` is about two minutes from now. 3. Open the desktop app as the same user. Success: your status shows on your profile. 4. Wait for the expiry. Read again. ```bash buzz users status ``` Success: the output is `[]`. Desktop also hides the status. 5. Try a bad expiry. ```bash buzz users set-status --text x --expires 3600 ``` Success: exit code 1. The error says that the value must be in the future. Nothing is published. 6. Try the clear conflict. ```bash buzz users set-status --clear --expires 1h ``` Success: exit code 1. The error says `--clear` cannot be used with `--expires`. 7. Read another user. ```bash buzz users status --pubkey <hex|npub|name> ``` Success: you get that user's visible status. You get `[]` if they have none. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: bmcanally <bmcanally@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 17:28:24 -06:00
Blake McAnally	303157f53d	feat(cli): send a thread message to the channel (#8778) ... > Replaces #8769, which was opened from a fork. ## Summary This change adds `buzz messages send --from-thread <root-event-id>`. The flag is the CLI form of desktop "Send to channel". It publishes a new top-level message in the channel. The message carries the tag `["buzz:sent-from-thread", <root-id>, <excerpt>]`. The message has no NIP-10 reply tags. ## Design Desktop puts the thread root id in the tag. It does not use the selected message id. Desktop publishes kind 9. The CLI does the same. The CLI checks the root event before it publishes anything. The root must exist. The root must have the same `h` tag as `--channel`. The root must be a thread root. If the id belongs to a reply, the CLI fails and prints the real root id. A missing root exits with a not-found error. Other failures exit with a usage error. No event is published after a failed check. The excerpt comes from the root content. It is a Rust port of `summarizeThreadRoot` in `sentFromThread.ts`. The port clips by code point, as desktop does. It breaks words by UTF-16 unit, as desktop does. `--from-thread` conflicts with `--reply-to` and `--broadcast`. It works only with kind 9. The relay does not validate this tag. Desktop's Tauri code validates it (`message_tags.rs`). The CLI output follows those rules: 64 hex characters, an excerpt of at most 64 characters, and no control characters. The change adds no dependency and no SDK change. The CLI appends the tag to the SDK `build_message` builder. The README describes the flag. This PR does not change `desktop/` (deprecated; see #8840). ## Testing What ran: - `cargo test -p buzz-cli`: 508 passed. - `cargo clippy -p buzz-cli --all-targets -- -D warnings`: clean. - `cargo fmt --all --check`: clean. - `just file-size-check`: passed. Test coverage: - One test calls `cmd_send_message` against a fake relay. It covers success, a reply id, another channel, a missing root, an invalid id, and a non-kind-9 send. The failure cases publish nothing. The success case publishes one event with the tag and without `e` or `broadcast` tags. - One test checks the clap conflicts. - Unit tests cover the excerpt boundaries (63, 64, 65 characters, the word-break threshold, and multi-byte and emoji text). - One test compares the Rust excerpt with 639 vectors. Desktop's own `summarizeThreadRoot` generated the vectors. The first run found an ordering bug (desktop replaces each pattern over the whole text in turn). The code now does the same. Interop evidence: - A node script runs the real `buzz` binary against a local fake relay. It is not a live relay. - The script passes the published tags to desktop's `getSentFromThreadReference` and `getSentFromThreadRootId`. - Desktop accepts the tag, returns the same root id, and returns the same excerpt as `summarizeThreadRoot`. - Output: `["buzz:sent-from-thread","abab…ab","Launch plan for the doc 😀 word word word word word word word…"]` then `INTEROP OK`. Mutation checks. Each guard was broken once. A test failed each time. Then the guard was restored. - Channel check removed: success test fails. - Reply-root check removed: success test fails. - Kind check removed: success test fails. - `--broadcast` conflict removed: clap test fails. - Word-break threshold `>` changed to `>=`: three excerpt tests fail. - UTF-16 break changed to code-point break: two excerpt tests fail. - Tag not attached: success test fails. ## Manual test steps Use a test channel. You need `BUZZ_RELAY_URL` and `BUZZ_PRIVATE_KEY` set. Build with `cargo build -p buzz-cli`. 1. Post a root message. Copy its event id as `ROOT`. `buzz messages send --channel <CHANNEL> --content "Launch plan: [doc](https://example.com)"` 2. Send it to the channel. `buzz messages send --channel <CHANNEL> --content "Decision: ship it" --from-thread $ROOT` Success: the output has `"accepted":true`. A new top-level message appears in the channel. 3. Read it back. `buzz --format json messages get --channel <CHANNEL> --limit 1` Success: the tags include `["buzz:sent-from-thread","<ROOT>","Launch plan: doc"]`. There are no `e` tags. 4. Open the channel in desktop. Success: the message shows the "sent from thread" line for that root. 5. Try a reply id. `buzz messages send --channel <CHANNEL> --content x --from-thread <REPLY_ID>` Success: exit code 1. The error names the root id. Nothing is published. 6. Try a root from another channel, and try `--reply-to` together with `--from-thread`. Success: both fail and publish nothing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: bmcanally <bmcanally@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 17:28:19 -06:00
Blake McAnally	5c91e0d6fe	feat(relay): add kind 30180 custom emoji entry (#8783) ... ## Summary This PR adds kind 30180, `KIND_EMOJI_ENTRY`. One event holds one custom emoji. The relay accepts, validates, stores, and replaces these events. The SDK gets `build_emoji_entry`. `docs/nips/NIP-CE.md` documents the kind. This is the relay part of the custom-emoji scale-up (Option C). It does not change kind 30030 sets, message `emoji` tags, or reaction `emoji` tags. It adds no migration and no schema change. **Read this before you deploy.** Prefix search for kind 30180 works only on a database whose `events.search_tsv` indexes that kind. See "Search index" below. ## Design - `crates/buzz-core/src/kind.rs`: `KIND_EMOJI_ENTRY = 30180`. It is in `ALL_KINDS`. A compile-time check proves it is parameterized replaceable. 30180 was free on `origin/main`. - `crates/buzz-relay/src/handlers/ingest.rs`: - The kind needs `UsersWrite`. - The kind is global-only. It is never channel-scoped. - `validate_emoji_entry` runs before storage. It requires exactly one `d` tag and exactly one `emoji` tag. - It rejects any `h` tag. - It requires `d` to equal the shortcode in the `emoji` tag, and both to equal the normalized shortcode. - It requires an `http://` or `https://` URL of at most 2048 bytes. It uses the SDK check `check_custom_emoji_url`, which is now public. - It caps `content` at 256 bytes (`MAX_EMOJI_ENTRY_CONTENT_LEN`). A shortcode is at most 64 bytes, so the cap leaves room. - Every rejection starts with `invalid:`. - `crates/buzz-sdk/src/builders.rs`: `build_emoji_entry(shortcode, url)` and `emoji_entry_search_text`. The content is the shortcode with `_` and `-` replaced by spaces. - Replace semantics use the existing parameterized-replaceable path. A second publish with the same `(pubkey, d)` leaves one live event. Two authors with the same `d` keep two events. The NIP tells clients to prefer the newest event and report the conflict. ### Decisions - **WebSocket `search_mode`: not changed.** Prefix search works through `POST /query`, which already reads `search_mode`. This is the smallest correct change. The `REQ` search path still uses full-word matching. The NIP says so. A client that needs typeahead uses `POST /query`. - **Search text.** Postgres `simple` already splits `party_parrot` at the underscore. It keeps `sad-parrot` as one token and also indexes `sad` and `parrot`. So the space replacement is not needed for matching today. It is kept because it is harmless and does not depend on parser details. - **Content cap of 256 bytes.** The cap is documented in the NIP and the code doc comment. ### Search index (important) The design doc says kind 30030 is not excluded from the full-text index. That is true for `schema/` (the desired state): it indexes every kind except a deny list. It is not true for a database that was created by `migrations/`: - Migration 0008 gives a fresh install a positive allowlist: kinds 0, 9, 40002, 45001, 45003. - `scripts/maintenance/nip_rs_search_allowlist.sql` moves a populated database to the same allowlist. On an allowlist database, kind 30180 is **stored and resolvable by `#d`, but not searchable**. I did not change this. A change to `search_tsv` rewrites the partitioned `events` table. That needs the reviewed out-of-band procedure in `docs/database.md` (rules 6 and 5), and this task forbids migrations and schema files. Before clients depend on search, a human must check the deployed expression: ```sql SELECT pg_get_expr(adbin, adrelid) FROM pg_attrdef d JOIN pg_attribute a ON a.attrelid = d.adrelid AND a.attnum = d.adnum WHERE a.attname = 'search_tsv' AND d.adrelid = 'events'::regclass; ``` If the result lists `kind IN (0, 9, ...)`, add 30180 to the allowlist through the out-of-band procedure first. ## Spike I ran this on a throwaway local Postgres 17 (Docker), desired-state schema from `pgschema`. The loader inserted 100,000 kind 30180 events from 200 authors through `Db::replace_parameterized_event`. It used 16 parallel tasks, and it did not use the network. The loader was a scratch example. It is not in this PR. Rows landed in `events_p_future` because the local partitions end at 2026-06. Real deployments have monthly partitions, so scans split across partitions. **Insert rate.** 100,000 inserts took 33.9 s (about 2,900 per second) at the DB layer. The relay adds rate limits on top (see the design doc, section 2.5). **`#d` batch of 50 names** (actual relay SQL shape, warm cache, `LIMIT 100`): | Case | Plan | Execution time | |------|------|----------------| | No `authors` | Index scan on `idx_events_parameterized`. It uses `community_id` and `kind` only. `d_tag` is not a leading key, so it scans the kind range. | 12 ms warm, 28 ms cold | | 50 names, all their authors (50 distinct) | Index scan with `pubkey` and `d_tag` in the index condition | 1.5 ms | | 50 names, one author | Same index, tight condition | 0.13 ms | Result: with `authors`, the lookup is fast. Without `authors`, the cost grows with the number of entries of the kind. At 100,000 rows it is still tens of milliseconds. No index change is needed now. Re-measure at 1,000,000 rows. **Prefix search**, `kinds=[30180]`, relevance order (the default), `LIMIT 50`: | Query | Matching rows | Execution time (warm) | |-------|---------------|-----------------------| | `parr` | 8,169 | 115 ms (124 ms cold) | | `p` | 15,990 | 219 ms | | `party par` | 8,169 | 266 ms (cold) | | `zap blob 999` (rare) | 1 | 0.4 ms | | `parr` with `search_sort: "newest"` | 8,169 | 0.8 ms | Result: the GIN index finds matches fast. Relevance order then scores every matching row (`ts_rank_cd`) and sorts them. The cost grows with the number of matches, not with `limit`. My synthetic words are few (24), so each prefix matches many rows. Real names are more varied, but a one- or two-letter query will still match many rows. The `search_sort: "newest"` option avoids the scoring and was 150 times faster here. Clients should use `search_sort: "newest"` for typeahead, or require at least 3 characters. This is a recommendation for the client work. It needs no relay change. **Side effects of bulk insert** (from reading `ingest.rs` and `handlers/event.rs`): - Audit log: each accepted event queues one `EventCreated` audit entry. The audit worker writes them one at a time under an advisory lock. The queue holds 1000 entries and uses `send().await`, so a bulk run slows the writer down instead of dropping entries. 96,000 events add 96,000 audit rows. - Live fan-out: each event is published to the global topic and matched against open subscriptions in `sub_registry`. A client with an open filter for kind 30180 receives every event. - Workflow triggers: each event spawns a task that calls `workflow_engine.on_event`. That is one trigger lookup per event. - Push: the DB push enqueue only matches kinds 9, 40002, 45001, and 45003 (`PUSH_MATCH_KINDS`). Kind 30180 does not wake push subscribers. - No `channel_id`, so the `events_created_at_floor_guard` trigger does not apply. ## Testing What ran (all on the final tree): - `cargo test -p buzz-core -p buzz-sdk -p buzz-relay -p buzz-search`: pass. - `cargo clippy -p buzz-core -p buzz-sdk -p buzz-relay -p buzz-search --all-targets -- -D warnings`: pass. - `cargo fmt --all --check`: pass. - `just file-size-check`: pass. - Postgres lane, desired-state schema (`BUZZ_TEST_SCHEMA_MODE=desired`) with a local Redis: `cargo test -p buzz-relay --lib handlers::ingest -- --include-ignored`: 205 passed, including the four new tests. New tests: - `emoji_entry_validation_table` (no infra): one row for each rule. - `emoji_entry_is_a_global_users_write_kind` (no infra). - `emoji_entry_ingest_validates_and_replaces` (Postgres): goes through `ingest_event`. It rejects a bad shortcode, `d` not equal to shortcode, an `h` tag, and a non-http URL. It checks that nothing is stored. It then publishes twice with the same `d` and checks that one live event remains and that it is the newer one. - `emoji_entry_prefix_search_finds_by_word_prefix` (Postgres): `parr` finds `party parrot` and `sad parrot`. `party par` finds one. `joy` finds `bufo joy`. It fails with a clear message on an allowlist database. - SDK: `emoji_entry_happy_path` and `emoji_entry_rejects_bad_input`. Not run: - `just test` was not run in full. I ran the relevant Postgres tests directly. - `cargo test -p buzz-search --test postgres_fts_integration` fails 23 of 23 on my shared local database. Those fixtures use fixed hostnames and need the fresh-database lane (`scripts/postgres-test-run.sh`). I did not change that crate. This is an environment limit, not a result of this change. - The new search test fails on a database built by `buzz-admin migrate` (allowlist). That is the finding above. It passes on the desired-state schema that the CI lane uses (`postgres-test-run.sh`). - I ran no live relay and no end-to-end client. Interop evidence: none for this item. The desktop and mobile constants are out of scope here. `/home/bloxer/ref/interop.mjs` was not needed because no TypeScript changed. Mutation checks (break the guard, confirm a test fails, restore): | Mutation | Result | |----------|--------| | Remove the `validate_emoji_entry` call | `emoji_entry_ingest_validates_and_replaces` fails | | Remove the `h` tag check | table test and ingest test fail | | Remove the `d == shortcode` check | table test and ingest test fail | | Remove the URL check | table test and ingest test fail | | Remove the content cap | table test fails | | Allow more than one `d` tag | table test fails | | Allow more than one `emoji` tag | table test fails | | Remove the kind from `required_scope_for_kind` | scope test, ingest test, and search test fail | | Remove the kind from `is_global_only_kind` | global test and ingest test fail | | SDK: skip shortcode normalization | both SDK tests fail | | SDK: skip the URL check | `emoji_entry_rejects_bad_input` fails | | SDK: skip the space replacement | `emoji_entry_happy_path` fails | One guard has no relay-level test: the space replacement does not change search results, because Postgres already splits `_`. The SDK test covers it. ## Manual test steps You need Docker and a local checkout. Use a local relay only. 1. Start the services and apply the desired-state schema, so that search indexes kind 30180: ```bash cd ~/buzz && . ./bin/activate-hermit docker run -d --name buzz-pg -e POSTGRES_USER=buzz -e POSTGRES_PASSWORD=buzz_dev -e POSTGRES_DB=buzz -p 127.0.0.1:5432:5432 postgres:17-alpine redis-server --daemonize yes --save "" PGHOST=localhost PGPORT=5432 PGUSER=buzz PGPASSWORD=buzz_dev PGDATABASE=buzz ./bin/pgschema apply --file schema/schema.sql --auto-approve docker exec -i buzz-pg psql -U buzz -d buzz -v ON_ERROR_STOP=1 < scripts/reconcile-schema-after-pgschema.sql ``` Success: `Changes applied successfully!`. 2. Run the new tests: ```bash export BUZZ_TEST_DATABASE_URL=postgres://buzz:buzz_dev@localhost:5432/buzz BUZZ_TEST_SCHEMA_MODE=desired cargo test -p buzz-relay --lib emoji_entry -- --include-ignored cargo test -p buzz-sdk --lib emoji_entry ``` Success: 4 relay tests and 2 SDK tests pass. 3. Optional, with a local relay (`just relay`) and a local key. Send these through `POST /events` with the SDK or a small script that calls `build_emoji_entry("party_parrot", "https://example.com/p.gif")`. Success: the relay returns `accepted: true`. - Publish a second entry with the same shortcode. Query `POST /query` with `{"kinds":[30180],"#d":["party_parrot"]}`. Success: one event, the newer one. - Query `{"kinds":[30180],"search":"parr","search_mode":"prefix","limit":50}`. Success: the entry is in the result. - Publish an entry that has an `h` tag or an `ftp://` URL. Success: the relay returns `invalid:`. 4. Check your target database before you rely on search (the SQL in "Search index"). Success: the expression does not list an allowlist such as `kind IN (0, 9, 40002, 45001, 45003)`. If it does, search does not find kind 30180 yet. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: bmcanally <bmcanally@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 17:28:05 -06:00
Blake McAnally	171100f308	feat(cli): read-only unread state (NIP-RS) (#8789) ... ## Summary This change adds `buzz channels unread`. The command is read-only. It shows how many unread messages each channel has. It uses the read state that desktop and mobile already sync (NIP-RS). It never publishes an event. It never marks anything read. Output: `[{channel_id, name, last_read_at, unread_count, capped}]`. The most unread channel is first. ## Design The command has two parts. **Part 1: read state** (`commands/read_state.rs`) - The command queries your own `kind:30078` events with the `read-state` tag. - It verifies each signature. It rejects events from other authors. - It keeps the newest version of each `d` tag. - It decrypts each slot with NIP-44 to yourself. It applies desktop's validation rules. - It merges all slots with `max`. - It ignores `ov_*` keys. Desktop and mobile do not read them either. **Part 2: counting** (`commands/channel_unread.rs`) The rules follow desktop's `communityUnreadObserver.ts`. - Without `--channel`, the command uses your non-archived channel memberships. - It reads message kinds 9, 40002, 45001, and 45003. For a DM it also reads huddle starts (48100). - It reads only events after the channel marker (`since = marker + 1`). - Your own messages do not count. - A message is read when its time is at or below the greatest of three markers: the channel marker, its `msg:<id>` marker, and its `thread:<root>` marker. A broadcast reply ignores the thread marker. **Bounds and signals** - At most 200 channels per run. `--limit` sets the number. The maximum is 200. - At most 1000 events per channel. A channel at that cap shows `"capped": true`. Its count is a lower bound. - If you have more channels than `--limit`, the extra channels are not counted. Stderr shows `{"warning":"channels_truncated","counted":N,"dropped":M}`. - More than `--limit` `--channel` values is an error. The command never drops a named channel. - An unknown named channel exits with code 1. **Failure rules** - A slot that cannot be verified, decrypted, or decoded fails the command with exit 4. Desktop skips such a slot. A skip would make the count too high, with no sign of it. - A slot with a newer `v` is skipped. Stderr shows `read_state_slots_skipped`. **Deviations from desktop** - The command has no 7-day fetch horizon. It reads all slots. This can only add markers. - It does not apply desktop's notification gate. That gate uses desktop-local data (followed threads, muted channels). So a count can be higher than the desktop badge. - It does not read mobile's `activity:` markers. Desktop does not read them. The README describes the output and the gotchas. The stable-inventory test lists the new subcommand. This PR does not change `desktop/` (deprecated; see #8840). ## Testing **What ran** - `cargo test -p buzz-cli`: 510 passed. - `cargo clippy -p buzz-cli --all-targets -- -D warnings`: clean. - `cargo fmt --all --check`: clean. - `just file-size-check`: passed. **Unit tests** The tests run the production functions against a fake axum relay. The fake relay applies the real filters (`#h`, `#d`, `#p`, `#t`, `since`, `limit`). - Merge: max across slots, newest version wins, bad values dropped, `ov_*` ignored. - Failures: bad JSON, wrong shape, bad `client_id`, wrong author, bad signature, and undecryptable content all fail. - Counting: own messages, `msg:` markers, `thread:` markers, broadcast replies, archived channels, DM kinds, and sort order. - Bounds: channel limit, event cap, `capped`, and explicit-channel limits. - Query shape: each channel query has `#h`, a bounded `limit`, and `since = marker + 1`. - A bad slot stops the command before any channel query. **Interop evidence** A node script runs the real `buzz` binary against a fake relay. Events are made and encrypted by `nostr-tools`. This is an independent NIP-44 code base. The script compares the CLI with desktop's own TypeScript: - `mergeReadStateEvents` gives the same frontier as `last_read_at` for six channels. - `fetchCommunityUnread` gives the same read/unread result as `unread_count > 0` for each channel. The six channels cover: read, unread, never read, own message only, two slots with max-merge, and a `msg:` marker. The script prints `INTEROP OK`. The script is not part of the repository. **Mutation checks** I broke each guard once and ran the tests. Each break made a test fail: - Own-message exclusion. - `msg:` marker, `thread:` marker, and broadcast rule. - `since = marker + 1`. - `capped` signal and channel limit. - Explicit channels over the limit. - Archived skip and DM kinds. - Max-merge, newest version wins, and the `t` tag requirement. - Signature check, author check, unknown `v`, and the 256-byte key limit. A mutation of the decrypt error path did not fail a test. The JSON step fails the same way, so the break changes nothing. I removed that check from the list. ## Manual test steps You need a relay and an identity that has used desktop or mobile on that relay. 1. Build the CLI: ```bash . ./bin/activate-hermit && cargo build -p buzz-cli ``` 2. Set the identity and the relay: ```bash export BUZZ_RELAY_URL=<relay-url> export BUZZ_PRIVATE_KEY=<your-key> ``` 3. Run the command: ```bash ./target/debug/buzz channels unread | jq . ``` Success: you see one object per channel. Each object has `channel_id`, `name`, `last_read_at`, `unread_count`, and `capped`. 4. Compare with the app. Open desktop or mobile on the same identity. A channel with a badge must show `unread_count` above 0. A channel with no badge must show 0. Counts can be higher than the badge for muted channels and followed threads. This is expected. 5. Read a channel in the app. Wait a few seconds. Run the command again. Success: the count for that channel is 0 and `last_read_at` is higher. 6. Test the channel limit: ```bash ./target/debug/buzz channels unread --limit 1 | jq length ``` Success: the command prints `1`. If you have more than one channel, stderr shows `channels_truncated`. 7. Test an unknown channel: ```bash ./target/debug/buzz channels unread --channel 00000000-0000-4000-8000-000000000000; echo $? ``` Success: the command prints a not-found error and the exit code is `1`. 8. Check that nothing was written. The command sends no events. The app badges stay the same after the run. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: bmcanally <bmcanally@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 17:27:47 -06:00
Blake McAnally	82bc5a47c5	feat(cli): report a message to relay moderators (#8829) ... ## Summary This change adds `buzz messages report`. The command sends a NIP-56 report (kind 1984) to the community moderators. You can report a message with `--event`. You can report a user with `--pubkey`. The `--reason` flag is required. It accepts one of seven values: `illegal`, `nudity`, `malware`, `spam`, `impersonation`, `profanity`, `other`. The `--note` flag is optional. The `--channel` flag is optional. When you give it, the CLI checks that the message is in that channel. The command prints the write envelope `{event_id, accepted, message}` and nothing else. ## Design The relay keeps a report in the moderation queue only. The relay does not store it as a normal event. The relay does not send it to subscribers. So the CLI prints no read-back. The tag shape is the same as desktop's `submitReport`: - Message report: `["p", <author>]` and `["e", <event id>, <reason>]`. - User report: `["p", <pubkey>, <reason>]`. The relay accepts this shape. - The note is trimmed and becomes the event content. The command adds no `h` tag. Desktop sends none. The relay reads the channel from the stored event. The task text mentioned an `h` tag. The real relay and desktop code do not use one. For `--event`, the CLI queries the event by id to learn the author's pubkey. That query uses an `ids` filter with `limit` 1. It names no kinds, because a message can have any kind. The existing `messages edit` and `messages delete` commands use the same lookup. `buzz-sdk` has a new `build_report` builder and a `REPORT_TYPES` list. The list matches the relay's `REPORT_TYPES`. The builder rejects unknown reasons, bad hex, and a note over 64 KiB. The builder allows self-tagging. Without it, the signer drops the `p` tag when you report your own message or key, and the relay rejects the report. Desktop opts in for the same reason. The CLI lowercases `--event` and `--pubkey` before it queries and signs. Clap also rejects unknown reasons, so the usage error appears before any network call. Exit codes: - Missing or unknown `--reason`, bad hex, channel mismatch: 1. - Relay rejection "invalid: report target ... not found" (exact text): 1. The relay gives this answer for a missing message and for a message you cannot read. - Other relay rejections (HTTP 400): 2. - HTTP 401 or 403: 3. The CLI README describes the new command. ## Testing What ran: - `cargo test -p buzz-cli -p buzz-sdk`: 510 CLI tests and 327 SDK tests pass. - `cargo clippy -p buzz-cli -p buzz-sdk --all-targets -- -D warnings`: clean. - `cargo fmt --all --check`: clean. - `just file-size-check`: passes. This PR does not change `desktop/` (deprecated; see #8840), so I did not run the Tauri crate tests. The CLI tests call the production `cmd_report_message` function against a local fake relay. They check the exact tags, the user-report shape, self-reports (the signer is the reported author), uppercase ids, argument parsing, the channel check, the not-found path, and the exit code for each relay rejection. Interop evidence: - I ran the real `buzz` binary against a local node HTTP server that stands in for a relay. - In the same node process, I ran desktop's own `submitReport` from `desktop/src/shared/api/moderation.ts` with a mocked signer. - Kind, tags, and content are equal for `spam` with a note and for `other` without a note. - The script prints `INTEROP OK`. - No shared relay was used. Mutation checks (break the guard, see a test fail, restore): - SDK reason check removed: fails. - SDK `e` tag without the reason: fails. - SDK `p` tag changed: fails. - SDK note trim removed: fails. - SDK note size cap removed: fails. - CLI channel check removed: fails. - CLI not-found mapping removed: fails. - CLI author read from the wrong field: fails. - CLI event-id hex check removed: fails. - SDK self-tagging removed: fails in the SDK test and in the CLI self-report test. - CLI lowercasing removed: fails. - Not-found mapping widened to any `invalid:` 400: fails. - Clap reason value parser removed: fails. - Clap required-target group removed: fails. - Clap `--channel` conflict with `--pubkey` removed: fails. One CLI `--pubkey` hex check was redundant with the builder. I removed it instead of keeping a guard no test can catch. ## Manual test steps You need a local relay and a running desktop or a second account to see the queue. Do not use a shared relay. 1. Start a local relay with `just relay`. 2. Send a message: `buzz messages send --channel <uuid> --content "test"`. Copy the event id. 3. Report it: `buzz messages report --channel <uuid> --event <event-id> --reason spam --note "test report"`. Success: the output is `{"event_id":"...","accepted":true,"message":""}` and the exit code is 0. 4. As a moderator, run `buzz moderation reports`. Success: the report appears with type `spam` and your note. 5. Run `buzz messages get --channel <uuid>`. Success: the report does not appear in the channel. 6. Report a user: `buzz messages report --pubkey <hex> --reason impersonation`. Success: `accepted` is true and the report appears in the queue. 7. Run `buzz messages report --event <event-id> --reason bogus`. Success: exit code 1 and a `user_error` message. 8. Run `buzz messages report --event <64 zeros> --reason spam`. Success: exit code 1 and a not-found message. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: bmcanally <bmcanally@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 17:27:33 -06:00
Luke Tornquist	0ada1d3707	buzz-db: read explicit #h sets one channel at a time (#9110) ... ## Summary Explicit multi-channel `#h` reads whose channels × page limit fit a budget of 8,192 now read each channel through the channel index and keep the newest `n` rows overall. Before, Postgres was left to plan `channel_id = ANY($set) ORDER BY created_at DESC LIMIT n` from a row estimate. Stacked on #9091 (both change `build_query_events_sql`). It retargets to `main` when that merges. ## Problem The desktop app's cold unread repair reads every joined channel in chunks, with `kinds [9, 40002, 40008]`, `include_aux`, `limit 500` and no `since`. It never fills its page. For that shape, Postgres's custom plans (the first executions of each prepared statement on a connection) overestimate how many rows a quiet channel set matches. They walk the partition's primary key backward and test every community row against the set. The inflated cost also turns on JIT. On a load-test database with about 940k events in one partition (`EXPLAIN ANALYZE`, custom plan): | Channels in the set | Plan | Rows returned | Time | |---|---|---|---| | 3 | channel index | 4 | 0.5 ms | | 6 | channel index, plus JIT | 254 | 54 ms (1.2 ms scan) | | 15 | pkey walk, 940k rows filtered | 427 | 546 ms | | 48 | pkey walk, 299k rows filtered | 863 | 197 ms | | 128 | channel index | 500 | 5 ms | Generic plans pick the channel index, but every connection runs custom plans first. A client-side `since` would not avoid the walk: it still walks the whole window. It would also change unread semantics, and it does nothing for app versions already shipped. ## Change `build_query_events_sql` handles plain explicit `#h` reads like this: ```sql SELECT e.* FROM (SELECT DISTINCT unnest($set::uuid[]) AS ch) c CROSS JOIN LATERAL ( SELECT … FROM events WHERE community_id = $1 … AND channel_id = c.ch … ORDER BY created_at DESC, id ASC LIMIT $n ) e ORDER BY e.created_at DESC, e.id ASC LIMIT $n OFFSET 0 ``` - **Same page as the set read:** every row of the overall top `n` is also in its own channel's top `n`. - **Bounded candidates:** at most distinct channels × `n` rows reach the outer sort, whatever the statistics say. How many index entries each channel scans still depends on the other predicates, such as `kinds`. - **Same SQL for any number of channels:** the channel set stays one array bind. - **Repeated ids:** `DISTINCT` reads a repeated id once. - **Per pass:** #9091's recent and older passes each use this shape. These keep today's SQL: scopes that include global events (`channel_id IS NULL OR ANY`), `#p` mention joins, e-tag fences, offset pages, the COUNT, and sets whose channels × page limit exceed the budget. ### Candidate budget A per-channel read fetches up to `n` rows from every channel, where the set read stops once the page fills. So busy channel sets cost more. On a synthetic community (920k events; 128 busy channels with 1,000 recent messages and 1,000 reactions each, 400 more busy channels, 128 quiet channels with 3 messages each), the repair shape (`kinds [9, 40002, 40008]`, recent pass), JIT off: | Set × limit | Set read, custom plan | Set read, generic plan | Per-channel | |---|---|---|---| | 128 busy × 500 | 2.3 ms | 238 ms | 114 ms | | 20 busy + 108 quiet × 500 | 19 ms | 33 ms | 18 ms | | 128 quiet × 500 | 473 ms | 0.7 ms | 1.3 ms | | 15 busy × 500 | 22 ms | 23 ms | 13 ms | | 15 quiet × 500 | 0.2 ms | 0.2 ms | 0.3 ms | | 128 busy × 100 | 0.7 ms | | 17 ms | | 128 quiet × 100 | 141 ms | | 1.2 ms | With JIT on (the default), the per-channel custom plan for 128 channels × 500 crossed the JIT thresholds: about 0.9 s for the busy set and 0.75 s for the quiet one, nearly all compile time. At 15 × 500 it stayed under them. So the per-channel read runs only while channels × page limit ≤ 8,192 (`PER_CHANNEL_CANDIDATE_BUDGET`, the clamped page limit). That bounds it at about 8k candidate rows, where it was still faster than the set read here. Larger sets keep the set read, as on `main`. Either way there are two SQL texts, so plan caching holds. The desktop repair's full 128-channel chunks at `limit 500` (64,000) are over the budget and keep the set read. Only its last, partial chunk (16 channels or fewer) and other small explicit sets read per channel. The load-rig numbers below predate the budget and don't show its effect. On the same data, the per-channel form took 0.5 ms (6 channels), 0.9 ms (15) and 4.2 ms (128). Load rig, cold desktop reconnect scenario (every client runs the unread repair), #9091 alone vs this PR on top: | | Pool default: #9091 | Pool default: + this PR | Pool 200: #9091 | Pool 200: + this PR | |---|---|---|---|---| | Repair read mean | 302 ms | 41 ms | 1,990 ms | 93 ms | | Repair read total DB time | 1,547 s | 245 s | 9,622 s | 561 s | | Repair reads served | 5,121 | 5,915 | 4,834 | 6,011 | | Client op failures | 23,475 | 10,966 | 28,797 | 10,829 | | Unread-load errors | 1,018 | 113 | 1,434 | 3 | | Relay pool-timeout log lines | 8,762 | 824 | 13,860 | 409 | The rig stays overloaded in every arm. Most remaining failures are the per-connection concurrent-request limit and connect/auth under load, not this read. ## Tests - `per_channel_reads_match_the_set_read` (Postgres) covers: - every limit from 1 to total+1, including a page held entirely by one channel - newer rows of another kind or another channel taking no slot - a repeated channel id and an id tie across channels - keyset paging across the tie, and `since` - rows below the recent-pass floor - a prepared-statement check that the read ran per channel and no `ANY` read ran - `per_channel_scope_eligibility` covers which reads keep the current SQL, including the budget edge (16 × 512 in, 17 × 512 and 128 × 500 out) and the clamped page limit. - `channel_scope_sql_does_not_depend_on_channel_count` now also asserts the per-channel shape for explicit `#h` pages. ## What got simpler Not much. This adds one SQL shape to the events query builder, behind one eligibility function and one budget. In exchange, the planner's row estimate no longer decides whether a small multi-channel read scans a whole partition. The channel predicate is decided in one place: `push_event_filters` no longer pushes the channel scope, and each caller (the page read and the thread route) pushes its own. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 18:25:56 -05:00
Blake McAnally	198c4c3e9b	feat(cli): list huddles and their participants (#8830) ... ## Summary This change adds two read-only commands to the Buzz CLI. - `buzz huddles list --channel <uuid> [--active] [--limit n]` lists the huddles of a channel. - `buzz huddles participants --channel <uuid> --huddle <id>` shows who is in a huddle now. The commands do not join, start, or end a huddle. They do not touch audio. They do not write to the relay. ## Design The CLI rebuilds huddle state from relay-signed lifecycle events (kinds 48100 to 48103). It uses a Rust port of desktop's `HuddlePresenceTracker`. The CLI and the app therefore agree on rosters. The port follows these desktop rules: - Only the relay can sign a join or a leave. - Only the relay or the huddle creator can end a huddle. - Events replay in desktop's order: time, then start, join/leave, end, then relay roster revision, then event id. - Each admission has an ID. A late leave for an old admission does not remove a rejoined participant. - A join from a newer generation replaces the older roster. A leave from an older generation is ignored. A huddle has one of three states: - `active`: no end event, and the start is less than one hour old. - `ended`: a valid end event exists. - `stale`: no end event, and the start is more than one hour old. Desktop shows this state as ended. The CLI keeps it separate because no end event proves it. Every query names the four lifecycle kinds and the `#h` channel. A command scans at most 5000 newest events. The output always has `complete`, `truncated`, `events_scanned`, and `event_bound`. - If the huddle start is inside the scan, the roster is exact. `complete` is `true`. - If the start is not inside the scan, `participants` returns `state: "unknown"` and `complete: false`. It does not return a partial roster. - A huddle that does not exist, with no truncation, returns exit code 1. Deviations from desktop and from CLI convention: - The HTTP `/query` endpoint does not serve liveness events (kind 48104). Only the WebSocket REQ path creates them. The CLI cannot check liveness. Every output has `liveness_checked: false`. Generations are never authoritative in the CLI. - Desktop keeps the roster in a Set, so its order is not defined. The CLI sorts by join time, then by pubkey. - Desktop also adds backing-channel members as a fallback roster. The CLI does not. It reports only what the lifecycle events show. - Desktop's roster view lists the creator from the start event. The CLI lists the creator only after the relay records their join. Desktop's roster view also uses a second event comparator. The CLI uses the presence tracker's comparator. - The commands return one JSON object, not a bare array. The object needs room for the truncation fields. Silent truncation would break Rule 1. The change also updates `crates/buzz-cli/README.md`. This PR does not change `desktop/` (deprecated; see #8840). ## Testing What ran: - `cargo test -p buzz-cli -p buzz-sdk`: 518 and 322 tests passed. - `cargo clippy -p buzz-cli -p buzz-sdk --all-targets -- -D warnings`: clean. - `cargo fmt --all --check`: clean. - `just file-size-check`: clean. The new tests call the production functions. A test also checks exit codes for relay-info failures: a 500 and an unreachable relay exit 2, and a 403 exits 3. They parse arguments, build output from events, and run `run_list` and `run_participants` against a local fake relay. The fake relay serves NIP-11 and `/query` with the same cursor paging as the real bridge. The tests cover rejoin, leave without join, ended huddle, stale generation, bad end signer, forged join, same-second ordering, and the event bound. Interop evidence: - `crates/buzz-cli/tests/fixtures/huddle_sequences.json` holds 14 recorded event sequences. - The Rust tests check each sequence against the expected roster and ended state. - `crates/buzz-cli/tests/huddle_interop.mjs` replays the same file through desktop's own `HuddlePresenceTracker` under node. All 14 sequences pass with `INTEROP OK`. - Run it with: `node --import ./desktop/test-loader.mjs --experimental-strip-types crates/buzz-cli/tests/huddle_interop.mjs` Mutation checks. I broke each guard once and confirmed a test failed. I then restored the code. - Relay-only author check on join and leave. - End signer check. - Ignore events after an end. - Roster revision comparison. - Ignore a leave from an older generation. - One-hour stale window. - Tie rule for the oldest scanned second. - Unknown state on truncation. - The truncation flag. - Kinds in the query filter. - `--active` filter. - Required relay `self` pubkey. All 12 mutations failed at least one test. A 13th mutation, which wraps relay-info errors as `Other`, also fails a test. No live relay was used. No event was written. ## Manual test steps You need a relay URL, a key, and a channel UUID where someone has run a huddle. 1. Build the CLI: `cargo build --release -p buzz-cli` 2. Set `BUZZ_RELAY_URL` and `BUZZ_PRIVATE_KEY`. 3. Run `./target/release/buzz huddles list --channel <uuid>`. - Success: the output is one JSON object with a `huddles` array. Newest huddle is first. Each huddle has `state`, `started_at`, and `ended_at`. `complete` is `true`. 4. Start a huddle in the desktop app. Run `./target/release/buzz huddles list --channel <uuid> --active`. - Success: the new huddle appears with `state: "active"`. 5. Copy its `huddle` value. Run `./target/release/buzz huddles participants --channel <uuid> --huddle <huddle-id>`. - Success: `participants` lists the same people as the desktop huddle window. 6. Have one person leave, then run the command again. - Success: that person is gone from `participants`. 7. End the huddle. Run the `participants` command again. - Success: `state` is `ended`, `ended_at` has a time, and `participants` is empty. 8. Run the `participants` command with a random UUID as `--huddle`. - Success: the command exits with code 1 and prints a not-found error. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: bmcanally <bmcanally@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 17:20:21 -06:00
Will Pfleger	f82944e318	docs(agents): add open-source boundary and drop private repo list (#9114) ... This PR strengthens the public-source boundary in `AGENTS.md` and removes internal deployment details from the contributor guide. - Adds an **Open-source boundary** section that prohibits Block-only identifiers and configuration, including internal community hosts, private repositories, organization-specific SSH users, internal infrastructure, tickets, Slack channels, and personal setups. It explicitly keeps public Builderlab supported. - Clarifies that the rule covers functionality as well as identifiers: OSS features must be generic and useful to any operator or distributor, with no knowledge of one company's internal services, workflows, or identity setup. Block-only functionality belongs in private repositories and may reach this code only through generic settings or documented general-purpose extension points that do not name, detect, or special-case an organization, plugin, or service. - Requires example values in tests, fixtures, docs, and examples, and makes boundary violations blocking. - Removes private build/deployment repository names and the internal deployment flow from the Ecosystem section. It retains the public repository descriptions and a general statement that internal builds and deployments live in private repositories. Related: [block/buzz-app#2649](https://github.com/block/buzz-app/pull/2649) (the corresponding boundary for block/buzz-app) --------- Signed-off-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Alia <d32955ad69077062930cc46cfe2df30ca9aaf6f8e76422681265e9e9af704d78@buzz.block.builderlab.xyz>	2026-11-05 14:59:31 -08:00
Matt Toohey	2bd4917825	fix(mobile): answer push snapshot bridge calls at user-initiated QoS; report hop latency on callback timeout (#8916) ... ## Summary `BuzzPushSnapshotEnrichmentTests` (iOS RunnerTests) timed out on two unrelated PR branches on 2026-11-01 and passed on re-run both times. The timings point to a starved test host, not a missed callback. This PR makes one production change and two test changes, in two files: - **`mobile/ios/Runner/PushSnapshotBridge.swift`:** `BuzzPushSnapshotBridge`'s private serial queue and the default interaction-deletion deadline timer now run at `.userInitiated` instead of `.utility`. Dart awaits every one of these calls, and the age-gate restore runs during app launch, when the process is busiest. User-initiated is the class for work a caller is waiting on. - **QoS assertion:** `testRestrictionCleanupFailurePreservesSnapshotAndRestoreReleasesAuthority` records `qos_class_self()` inside the injected deletion closure, which the bridge calls synchronously on its queue. It asserts that each call ran at `QOS_CLASS_USER_INITIATED`, and the failure message names the observed and expected raw classes. - **`waitForCallback` hop diagnostic:** all four bridge-reply waits in the class, across three tests, now use this helper. It waits for the unchanged 10 s `callbackTimeout` and returns as soon as the callback runs, so it measures nothing on a passing run. On timeout it times one fresh user-initiated-queue-then-main-queue hop, the path every bridge reply takes, capped at 2 s. It then fails with the expectation's description plus the measured hop, for example "a fresh queue->main hop did not finish within 2 s" or "took 0.4 ms". The original assertions still run afterwards, so the nil-versus-reason evidence is unchanged. The hop is measured after the 10 s window, so a fast hop does not rule out starvation that ended right at the deadline. No timeout changed. ### Related issue None found. #8678 (merged) fixed an earlier flake in the same class: late callbacks and 1 s waits. It introduced the 10 s `callbackTimeout` this PR keeps. ### Failing runs Both runs were in "Mobile Swift (Native)", step "Test native controls and Runner unit tests", attempt 1. Both passed on attempt 2. - [Run 41327410782, job 124910378184](https://github.com/block/buzz/actions/runs/41327410782/job/124910378184), on `mobile-avatar-thumb-followup` (#8896): `testRestrictionCleanupFailurePreservesSnapshotAndRestoreReleasesAuthority` timed out its 10 s "allowed without restoring snapshot" wait (19.3 s total). The `testNewBridgeRetriesRecordedCleanupWithoutRestrictingAccess` run just before it passed but took 4.3 s. - [Run 41325606910, job 124905175903](https://github.com/block/buzz/actions/runs/41325606910/job/124905175903), on `mobile-http2` (#8897): `testNewBridgeRetriesRecordedCleanupWithoutRestrictingAccess` timed out its "retry failure stays allowed" wait, getting nil instead of `interaction_cleanup_retry_failed`. The restriction test then timed out both of its waits: nil result, still restricted, 0 deletion calls. ### Diagnosis Every reply from `BuzzPushSnapshotBridge` goes through a private serial queue, previously created at `.utility` QoS, and then `DispatchQueue.main`. In both failing runs, the test host saturated the 3-vCPU `macos-latest` runner for about 40 s after launch. The test host is the full Flutter app, loaded as `TEST_HOST`. - `BrowserAuthBridgeTests.testAllowsOnlyBuilderlabLoginPages`, a synchronous main-thread URL-parsing test, took 2.65 s instead of 0.013 s. - `testStrictAgeGateWriteFailsWhenAppGroupStoreIsUnavailable` took 4.9 s. That test has since been deleted from `main`. - A trivial hop took 1.3 s, then more than 10 s, during that window. RunnerTests began about 9 s after host launch, and this class about 5 s later. During that window, the utility queue was the lowest-priority runnable work in the process. It got no CPU for 10–20 s, while main-thread work kept moving slowly. In both passing re-runs, every test in this class took under 0.3 s. The test logic doesn't explain a callback that never arrives: - The deletion mocks are synchronous. - The deadline scheduler is a no-op. - Each test uses its own temp directory and bridge. - No state is shared between tests. That rules out both a state leak and an expectation-ordering race. This diagnosis is inferred from CI timings. The starvation did not reproduce on a 16-core local host (see Testing). ### Tried and dropped - **A class `setUp` settle barrier (ff19edf0d, removed in 6cc43c9fd).** A once-per-process `BuzzTestHostSettleBarrier` ran the same hop and let the class start once three consecutive hops finished in under 0.5 s, capped at 60 s. Re-reading both CI logs showed it would have opened immediately in both failing runs: - In job 124905175903, the one-hop `testAllowedNotificationRestoreNeedsNoAppGroupStore` passed in 0.070 s at the start of the class. - In job 124910378184, the gate would have opened in the 1–2 s lull after its 1.287 s hop. Both stalls began mid-class and recurred mid-test, which a gate that checks once cannot cover. A version that held the gate for longer would just be a CI-only wait. - **Deriving `callbackTimeout` from a measured host-load factor.** This still waits inside each test and only scales the same timeout. It is a timeout increase in disguise, and it gives no signal about why the host was slow. ### Residual risk and remaining levers On a saturated 3-vCPU runner, `.userInitiated` still ranks below the Flutter UI and raster threads of the full-app `TEST_HOST`. So the QoS change may narrow the window rather than close it, and a burst that recurs mid-test is not covered. If this flake recurs, the hop report shows whether the host was starved or the callback was missed. The remaining levers are: - **The test host.** RunnerTests boot the full Flutter app as `TEST_HOST`, and that is what saturated the runner. #8678 tried keeping Flutter out of the host, and that did not work locally. - **The 10 s `callbackTimeout`.** ### Testing Local runs used Xcode 27.2 with the 27.2 SDK, the iOS 26.2 simulator runtime, and a 16-core host. CI's starvation did not reproduce locally. QoS change (4278e9944): - **Class in isolation:** 10/10 green, 0.1–0.4 s each. - **Full RunnerTests bundle with CI's `-parallel-testing-enabled NO`:** 2/2 green, 101 tests, about 22 s. - **Falsification:** with the queue temporarily set back to `.utility`, the new assertion fails. It observes `[17, 17]` (utility) instead of `[25, 25]`. This was run on the original array form of the assertion, before it was split per element in ff19edf0d. - **Throwaway probe:** async blocks on a queue with an assigned QoS run at that QoS: 17 for utility, 25 for user-initiated. So the change alters real scheduling. - **Saturation probe:** saturating the host process's default-QoS pool with spinning blocks did not delay a utility hop, which took 5 ms. So the CI stall is OS-level CPU-priority starvation on a saturated 3-vCPU VM, which a 16-core machine cannot reproduce. The diagnosis rests on the CI timings above. Hop diagnostic (6cc43c9fd, e5515f765): - **Forced timeout (temporary variants, not committed):** when the bridge's retry deletion never called back, the test failed with "'retry failure stays allowed' not fulfilled within 10 s; a fresh queue->main hop took 0.4 ms", followed by the existing nil-versus-reason assertion failure. Adding a 0.7 s sleep inside the hop produced "took 705.3 ms". - **Class in isolation:** 10/10 green, 0.1–0.5 s each. - **Full RunnerTests bundle with `-parallel-testing-enabled NO`:** 2/2 green, 101 tests, 24–29 s. - **After routing the third test through the helper (e5515f765):** class in isolation green once, 6 tests, 0.05 s. All commits: - `git diff --check` is clean. - The repo has no swiftformat or swiftlint configuration. - The branch was rebased onto `main` with no conflicts. `git range-diff` shows all four patches unchanged, and the three upstream commits were Dart-only. - The tests were not re-run after the rebase. - Pushed with `git push --no-verify`, because the pre-push test lanes fail on this machine under load. ### Readiness - **Hosted CI** had not run on this branch when the PR was opened. The "Mobile Swift (Native)" lane is the check that matters. - **Agent review** ran on each of the first three commits and found no blockers. Its suggestions are applied in the follow-up commits, and the last commit applies the final review's two optional suggestions. - **Human confirmation** (AGENTS.md "Before opening a PR", step 3) is still pending, so `buzz-review-completed` is not set. The four commits are kept as written for the squash merge. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-11-06 09:50:20 +11:00
Matt Toohey	46cdc52d23	test(mobile): lock in invite-page and bare-host community link entry (#9047) ... Both mobile community-link fields (Add community and the welcome/onboarding "Enter a community link" page) already accept a typed or pasted `https://<host>/invite/<code>` link and keep the code. They also read a bare host as `wss://`. This PR adds regression coverage for both. **It doesn't change any production code.** Companion: block/buzz-app#2412 fixes the same two inputs in buzz-app's Join a community. ## What's covered - **`community_link_input_test.dart`** - The address loop is now an input → relay table, with two new rows: `secondary.example` and `relay.example.com:8443` (the port is kept). - The invite case now uses `https://primary.example/invite/fixture`, and expects relay `wss://primary.example` with code `fixture`. - Two new rejected inputs: `ws://secondary.example`, and the invite path with a trailing slash, which the relay doesn't serve. - **`onboarding_entry_routes_test.dart`**: before the bare-host step, the journey pastes the invite link into the Add route's field. It expects the app-level `InviteJoinSheet` to show host `primary.example` and code `fixture`. ## Shown to catch a regression I removed the invite/add-community short-circuit in `parseCommunityLinkInput`. Both the invite unit case and the new journey step then failed. ## Not in this PR Two parser differences are left for a separate decision, along with the tests that depend on them: - `buzz://connect?relay=…`: mobile rejects it in the fields and silently drops it as an OS link, but buzz-app, desktop and the web repo browser's Connect button all use it. - An invite-page link with a query string: mobile accepts it and throws the query away, but buzz-app rejects it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 09:49:51 +11:00
Matt Toohey	e94b547f60	fix(desktop): keep an own send out of the frozen timeline tail (#9095) ... ## Summary The timeline freezes its live tail when Virtua reports that the reader has left the bottom, and a send arms a one-shot pin for its own row. A not-at-bottom report between the send and the sent row reaching the deferred snapshot froze the tail again, so the sent row waited behind a "1 new message" pill. That is the `messaging.spec.ts:483` flake in Desktop Smoke E2E. The first commit holds off that freeze while the send is pending. The other three are follow-ups from pre-PR review that bound the hold to the send itself: it ends when the row lands, it is disarmed when the send fails, and it is armed only for the channel the send belongs to. ### fix(desktop): keep an own send out of the frozen timeline tail #### Problem `messaging.spec.ts:483` "long autolink wraps without widening the timeline" flaked in Desktop Smoke E2E at :497: the timeline never showed the sent message. The composer was empty, and a "1 new message" pill sat at the bottom of a timeline that was already at the floor. - seiler/BUZZ-517/desktop-upgrade-guidance: - https://github.com/block/buzz/actions/runs/41751914119/job/126319694104 (retry 1 passed) The failure video shows the composer filling while the channel still showed its skeleton, the rows rendering under the tall composer, and the send landing about 120 ms later. #### Cause The timeline freezes its live tail when Virtua reports that the reader has left the bottom, and later rows wait behind the pill. A send lifts the freeze and arms a one-shot pin for the next append (`prepareForOwnMessage`). Nothing kept a not-at-bottom report between the send and the sent row reaching the deferred snapshot from freezing the tail again. The sent row then waited behind the pill, and the pin, which waits for that append, never fired. Only a later scroll to the floor would have released it. Virtua 0.49.3 makes that report while the reader sits at the floor. It applies a list length change to its store during render (Virtualizer.tsx). React discarded the deferred render that added the sent row, but the store kept the row's estimated height. When the cleared composer shrank the trailing spacer, the next scroll event put the reader 96 px from a floor the DOM didn't have: Virtua read 941 - 479 - 366 = 96, the DOM read 917 - 551 - 366 = 0. The freeze it queued renders at default priority, ahead of the deferred render, which then restarted against the frozen tail. #### Fix While the send's pin is armed, a not-at-bottom report doesn't freeze the tail. `useAnchoredScroll` exposes `isBottomPinArmed()`, which reads the pin's own flag, so the hold ends when the pin fires or the channel changes. `prepareForOwnMessage` also cancels a freeze queued before the send, which would otherwise land after its release. Bottom state still comes from Virtua. Reading it from the DOM would add false gaps of its own, because unmeasured rows overflow their slots: a passing run read 51 px from the DOM where Virtua read 0. A scroll up between the send and its row no longer freezes the tail. The pin brings the reader down to the sent row, as it already does when the row lands first. A send that adds no row leaves the pin armed until the next append, as before, and the hold follows it. #### Validation - New `MessageTimeline.ownSend.jsdom-test.mjs` renders the real MessageTimeline over Virtua with a modelled layout and manual frames. It sends with a gap reported after the send, and with a freeze already queued at the send. The sent row must render at the floor with no pill. Without the pin guard the first case fails, and without the cancel the second does, each showing "1 new message". - Repro, using temporary, uncommitted changes: the test with a 600 ms channel-head delay, filling during the skeleton, sending once rows render, with CPU throttled 4x over CDP. Without the fix: 3/20 failed with the CI signature, and instrumentation showed the 96 px report above. With the fix: 40/40 passed, and 6 of those runs logged a not-at-bottom report while the pin was armed. - `messaging.spec.ts --grep "long autolink"`: 30/30 passed at 4 workers, 10/10 serially. - messaging, virtualization, scroll-history, stream, smoke, thread-follow, file-attachment and message-target-highlight specs, run twice at 4 workers: 185 passed and 1 failed each time, a different test each run. Repeating those two fails as often without the fix: `:2785` (own avatar) 22/25 vs 20/25 at 4 workers and 8/10 vs 9/10 serially; `:3170` (thread panel width) 4/25 vs 8/25 and 1/10 vs 3/10. `:2785` fails on the avatar image after its message has rendered, and `:3170` never sends, so the change can't reach it. - `tsc --noEmit`, `biome check`, and the messages UI unit (337) and jsdom (9) tests pass. A green run doesn't prove the flake is gone. Hosted CI covers broad validation. ### fix(desktop): end an own send's freeze hold when its row lands #### Problem Review of f2c57afb5 ("fix(desktop): keep an own send out of the frozen timeline tail") raised two findings in MessageTimeline.tsx. - Warning: the hold on freezing the tail lasted as long as the send's bottom pin, not as long as the send. The pin fires only when the tail id changes (`newLatestArrived` in useAnchoredScroll), and only firing or a channel change cleared it. The window cache sorts rows by created_at, so the sent row doesn't become the tail when the last row has a later created_at (a sender whose clock runs ahead) or a same-second row sorts after it. The pin then stayed armed after the sent row rendered, and a reader who scrolled up never froze the tail. The next arrival to become the tail fired the pin and pulled them to the bottom instead of showing the pill. Before f2c57afb5 the freeze buffered that arrival, so a stale pin could not fire under a reader. - Nit: `prepareForOwnMessage` set `semanticAtBottomRef.current = true`, which the hold already made redundant, and cancelling the queued freeze left behind the `suppressNextSemanticBottomRef` it had set. #### Cause The hold read the bottom pin's own flag. The pin is cleared only when it fires on a new tail id (`newLatestArrived` in useAnchoredScroll) or when the channel changes. The window cache sorts rows by `created_at`, so a sent row that sorts above the current tail never becomes the tail. That happens when the last row's sender has a clock running ahead, or when a same-second row sorts after the sent one. The pin then stays armed after the sent row renders, and the hold stays with it. #### Fix `useAnchoredScroll` keeps a pending-send flag next to the pin. `scrollToBottomOnNextUpdate` sets both. The restoration effect clears the flag on the first append after arming, whether or not it moves the tail (`newLatestArrived` or an "append" delta), and the channel reset clears it. `isBottomPinArmed` becomes `isOwnSendPending`, which reads the flag, and MessageTimeline holds off freezing only while it is set. The pin itself is unchanged and still waits for a new tail. Once the hold ends, a reader who scrolls up freezes the tail as before, so later arrivals wait behind the pill and a stale pin can't fire under them. This keeps arming and disarming in the hook. Disarming from ChannelPane after `onSendMessage` settles would need a second call site (the welcome add-agent flow also arms the pin), and the send's promise isn't tied to when the sent row reaches the deferred snapshot. `prepareForOwnMessage` now clears `suppressNextSemanticBottomRef` instead of setting the semantic ref, so dropping the queued freeze also drops the suppression it armed. No test covers this: a leftover suppression only swallowed a bottom report while the send's release was already making the timeline semantically at the bottom. #### Validation - New case in `MessageTimeline.ownSend.jsdom-test.mjs`: the tail's createdAt is an hour ahead, the send lands above it, the reader scrolls up 200 px, and a row from the same fast clock arrives and becomes the tail. It expects "1 new message" and the reader's offset unchanged. On f2c57afb5 it fails: no pill, and the pin moved the reader from 756 to the floor at 1052 and rendered the arrival. With the fix the reader stays at 756 and the arrival waits behind the pill. - Mutations: clearing the flag only on a new tail, or never on an append, fails only the new case. Removing the hold check fails the original "gap after the send" case, as before. - Messages UI jsdom tests (10) and `src/features/messages` unit tests (1530) pass. Full desktop `pnpm test`: 6933 unit and 179 jsdom tests pass. - `tsc --noEmit`, `biome check` on the changed files, `check:file-sizes` and `git diff --check` are clean. No browser E2E was run for this follow-up. ### fix(desktop): disarm an own send's pin and hold when the send fails #### Problem Second review of the branch (warning, regression vs main) on c1767345e ("fix(desktop): end an own send's freeze hold when its row lands"), useAnchoredScroll.ts and ChannelPane.tsx. The hold can still go stale when a send never adds a row to this timeline. `ChannelPane.handleSendMessage` armed the pin and the hold before it awaited `onSendMessage`, for every main-composer send. That includes a DM send that `useMentionSendFlow`'s `onPrepareSendChannel` moves to another DM (the message mentions an agent who isn't in this one). Its optimistic row goes to that channel's window, and `goChannel` runs only after the send succeeds. If the publish is rejected or times out, the user stays on the original DM with both flags armed. A same-channel send whose `onMutate` can't resolve the channel also adds no row. Then the reader scrolls up, the hold skips the freeze, the next arrival from the other person becomes the tail, and the stale pin moves the reader to the bottom with no pill. The reviewer's jsdom repro: at HEAD, no pill and scrollTop 660 -> 956. At the merge base, "1 new message" and 660 stays. A failed send to the same channel was fine: its optimistic row lands in `onMutate`, an append that clears the hold, and is then reverted. #### Cause `ChannelPane.handleSendMessage` armed the pin and the hold before it awaited `onSendMessage`, for every main-composer send, and nothing disarmed them if the send then added no row to this timeline. A send that `onPrepareSendChannel` moves to another DM lands its optimistic row in that channel's window. A same-channel send whose `onMutate` can't resolve the channel adds no row either. When such a send fails, the user stays on the original timeline with both flags still armed. #### Fix - `ChannelPane.handleSendMessage` arms only when the send targets this channel (`!channelId || channelId === activeChannelId`). A send prepared into another channel lands its row there, and the pane navigates to it once the send succeeds. - `MessageTimelineHandle.cancelScrollToBottomOnNextUpdate` disarms a send. `useAnchoredScroll` clears the pin and the pending-send flag. The hold skipped every freeze while the send was pending, including one queued before the send that the arm cancelled. So if the send was still pending and Virtua's last report has the reader off the floor, the cancel queues that freeze, as the report would have without the hold. Later arrivals then wait behind the pill. If the send's row already landed, the hold had ended and the cancel only clears the pin. - `ChannelPane` arms through `armOwnSend`, which returns the disarm. The main composer calls it when `onSendMessage` rejects, and the Welcome "create in chat" flow when its `mutateAsync` rejects. Its `beforeSend` option now returns the disarm. A generation counter keeps a late failure from disarming a pin that a later send armed. #### Validation - `MessageTimeline.ownSend.jsdom-test.mjs` adds the reviewer's no-row case in three orders: fail before the reader scrolls up, after, and with the freeze queued before the send. Each expects "1 new message" and the reader's offset unchanged after a new tail arrives. With the cancel a no-op (HEAD's behavior), all three fail: no pill, and the reader moves from 660 to 956. With the cancel clearing the flags but not queueing the freeze, the two orders where the reader left the floor during the send fail with "Jump to latest". A fourth case, a same-channel send whose row lands and is then reverted, still shows "1 new message" and keeps the offset. It passes on HEAD as well, because it pins behavior that must not change. 7/7 pass. - `useAnchoredScroll.lifecycle.test.mjs` adds a hook case: arm, cancel, the reader leaves the floor, a new tail arrives without a jump, and a later send still pins. It fails if the cancel is a no-op, or if it clears only the pending flag. useAnchoredScroll unit tests: 45/45. - Full desktop `pnpm test`: 6934 unit and 183 jsdom tests pass. - `tsc --noEmit`, `biome check --error-on-warnings` on the changed files and `git diff --check` are clean. - No test renders `ChannelPane`, so its arm condition, the rejection handlers and the generation counter are covered only by tsc and review. No browser E2E was run. ### fix(desktop): arm an own send's pin only for the channel it belongs to #### Problem Third review of the branch (review 89cce157, warning) on c257ff169 ("fix(desktop): disarm an own send's pin and hold when the send fails"), ChannelPane.tsx. `handleSendMessage` armed the timeline when `!channelId || channelId === activeChannelId`, and `activeChannelId` was the value captured in its closure. A send whose slow upload spans a channel switch passes that check against the channel it was written in, then arms `messageTimelineRef.current`, the timeline now showing the other channel. That channel keeps its tail open and its pin armed until its next append, because the sent row lands in the original channel's window. A reader who scrolls up there doesn't freeze the tail, and the next arrival fires the pin and pulls them to the bottom with no pill. #### Cause `useMentionSendFlow` reads `onSendRef.current` before it starts a deferred upload and calls that `send` from the upload's `onComplete`. The arm carried no channel, so only the stale closure decided whether to arm, and the timeline accepted an arm for whatever channel it rendered. #### Fix - `MessageTimelineHandle.scrollToBottomOnNextUpdate(sendChannelId)` takes the send's channel. `useAnchoredScroll` keeps the channel its state belongs to in a ref that its channel reset updates, and its arm returns false without arming for any other channel. MessageTimeline releases the frozen tail only when the arm is accepted, so a stale send neither releases nor pins the channel shown now. - The arm and disarm move from ChannelPane into `useOwnSendArm(timelineRef, getTimelineChannelId)`. It skips an arm whose channel isn't the one `getTimelineChannelId` reads now (ChannelPane passes its `activeChannelIdRef`) and returns a no-op disarm. A skipped arm leaves the generation alone, so it can't stop a pending send in the shown channel from disarming when it fails. Otherwise the generation works as before: a later send's arm turns an earlier send's disarm into a no-op. - `handleSendMessage` arms `channelId ?? activeChannelId`. A send prepared into another DM is still skipped, now by the hook's live check instead of the closure. - The Welcome "create in chat" flow names the channel it sends to: `beforeSend` takes a `channelId`, and `useWelcomeAgentCreate` passes `activeChannel.id`. - `useAnchoredScroll` exposes `isVirtualizerAtBottom()`, and MessageTimeline's cancel reads it instead of keeping its own copy of Virtua's last report. The two agree once the channel has confirmed its bottom, which the cancel already requires. The hook's copy is also cleared by a target jump or a live row under a held target before Virtua reports, so a cancel in that window freezes as the report would. - The channel reset effect now reads `channelId`, so its exhaustive-deps suppression goes. "a cancelled own send leaves the next tail with the reader" in `useAnchoredScroll.lifecycle.test.mjs` (from c257ff169) failed 3 of 6 runs at HEAD on this workstation. The mount's settling frame, a `setTimeout(0)` in that file's shim, could run after the test cleared its recorded pins and add a "jump". The test now waits for that frame first, as other cases in the file do: 0 of 15 runs failed. It still fails when the cancel is a no-op or clears only the pending flag. #### Validation - New `useOwnSendArm.test.mjs` (node:test, like the rest of desktop's unit tests): a send to the shown channel arms it, a send that settles after a switch isn't armed and leaves the pending send's disarm working, a failed send disarms, and a later arm outlives an earlier failure. Each mutation fails a case: dropping the live check, bumping the generation before it, dropping the generation check, and a no-op disarm. - `useAnchoredScroll.lifecycle.test.mjs` adds an 11-row table for the two gates the cancel reads, `isOwnSendPending()` and `isVirtualizerAtBottom()`: no send, a send whose row hasn't landed, landed as the tail, landed above the tail, a send to another channel, a send then a channel change, a channel change then a send there, and the reader off the floor, back at it, after a channel change, and with a send pending. Dropping the arm's channel check, not updating the channel ref on reset, and a constant `isVirtualizerAtBottom` each fail rows. - `MessageTimeline.ownSend.jsdom-test.mjs`, 9/9: - New: with the reader scrolled up and an arrival behind the pill, an arm for another channel and a second arrival leave "2 new messages" and the offset unchanged. It fails if useAnchoredScroll accepts the arm, or if MessageTimeline releases the tail when it is refused. - New: a send that fails at the floor leaves the next arrival rendered at once. Without the off-floor gate the cancel freezes and the arrival is missing right after it renders. - Extended: after a same-channel send lands and fails, the reader returns to the floor and the arrival renders with the pill gone. Without the pending gate the cancel re-arms the freeze's bottom-report suppression and the arrival stays behind the pill. - The existing cases now pass the channel. - Desktop unit tests under `src/features/messages` and `src/features/channels`: 2006/2006. jsdom tests there: 21/21. - Playwright smoke after `pnpm build:e2e`, `--workers=4`: messaging, send-channel-binding and welcome-agent-modal-screenshots, run twice. First run 99 passed and 1 failed, second run 98 passed and 2 failed: - `messaging.spec.ts:2785` (own avatar) fails on the avatar image after its message renders: 5/6 failed with this change and 6/6 at HEAD (`--workers=2 --repeat-each=6`). f2c57afb5 recorded the same local failure. - `messaging.spec.ts:3170` (thread panel width) never sends. It passed 5/5 on rerun. - `send-channel-binding.spec.ts`, `--repeat-each=5`: 10/10. Its mid-send case switches channels while a send is in flight. welcome-agent-modal-screenshots (its chat-first case clicks "create in chat" and finds the message in the timeline) plus nine messaging cases including "long autolink" and the channel-switch cases, `--repeat-each=5`: 80/80. - `tsc --noEmit`, `biome check --error-on-warnings` on the changed files, `check:file-sizes` and `git diff --check` are clean. - No test renders ChannelPane, so its getter and its `armOwnSend(channelId ?? activeChannelId)` call are covered only by tsc, the hook's tests and the specs above. **Rebased onto origin/main `64d12e297` (this branch, all four commits):** Linux workstation. This branch doesn't include the Virtua patch change from #9094. `pnpm install --frozen-lockfile` installs main's virtua patch (`30a7a92b9480...`), so every result below runs against the Virtua build on main. - `pnpm typecheck` and `biome check --error-on-warnings` on the 11 changed desktop files are clean. - Unit tests under `src/features/messages` and `src/features/channels`: 2006/2006. jsdom tests there: 16/16, including all 9 `MessageTimeline.ownSend.jsdom-test.mjs` cases. Full desktop `pnpm test`: 6949 unit and 180 jsdom tests passed. - Playwright smoke after `pnpm build:e2e`, `--workers=4 --retries=0`: messaging, send-channel-binding, scroll-history, virtualization, timeline-no-shift, overscroll-boundary, live-broadcast-reply-timeline, message-target-highlight and welcome-agent-modal-screenshots, 142 passed and 2 failed. The failures are `messaging.spec.ts:2785` (own-message avatar), which fails the same way on main here as noted above, and `messaging.spec.ts:3170` (thread panel width), which never sends and passed 5/5 on rerun. - `messaging.spec.ts --grep "long autolink"`, `--repeat-each=30`: 30/30 passed. `send-channel-binding.spec.ts`, `--repeat-each=5`: 10/10 passed. ## Coordination - #6695 ("feat: show thread replies in channel") gains a textual conflict in `desktop/src/features/channels/ui/ChannelPane.tsx`. It already conflicts with main in four other files, so it needs a rebase anyway. - #8563, #8565, #8030, #7604, #7657, #7499, #5563 and #4663 touch `MessageTimeline.tsx`, `ChannelScreen.tsx`, `ChannelPane.tsx` or `useChannelIntro.tsx` and merge cleanly with this branch. - Older open PRs that touch these files already conflict with main, and none gains a new conflict from this branch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: mtoohey <mtoohey@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 09:49:32 +11:00
Matt Toohey	4c7c15a9c2	fix(db): end each table's maintenance transaction before the next (#9096) ... ## Summary Partition maintenance runs one transaction per table in a pass. When a table's attempt failed on the server, the transaction was dropped as soon as the error arrived, which can be before the backend has aborted and released its schema-scoped advisory lock. The next table could then find the lock held and report `skipped_locked`, which is the `parent_contention_times_out_without_blocking_the_other_table` flake and a mis-reported outcome in production. `apply_change` now owns the transaction and commits or rolls it back before it returns, so the next table starts only after the previous one's locks are free. ### fix(db): finish each table's maintenance transaction before the next starts #### Problem `store::partition::maintenance::tests::postgres_tests::parent_contention_times_out_without_blocking_the_other_table` failed once on main at maintenance_tests.rs:480 (`assert_advanced`). `events` reported `lock_timeout` as expected, but `delivery_log` was left on its old catch-all with no error. - main: https://github.com/block/buzz/actions/runs/41669713108/job/126035790322 Production has the same race. When one table's attempt fails on the server (lock_timeout, statement_timeout, deadlock), the next table in the same pass can report `skipped_locked` and skip its DDL. The next interval repairs it, but that pass reports the wrong outcome and metric. #### Cause `delivery_log` can only stay un-advanced without an error through `skipped_locked`: its `pg_try_advisory_xact_lock` returned false. The key is per schema, so the holder was `events`'s own failed transaction. PostgreSQL sends a statement's ErrorResponse (`EmitErrorReport` flushes it) before the backend calls `AbortCurrentTransaction`, which releases its locks (postgres.c, PG 16 and 17). sqlx returns the error as soon as the ErrorResponse arrives. `apply_change`'s `?` then dropped the transaction without waiting for that backend again, and `maintain_audited` started `delivery_log` on another pooled connection. The abort normally finishes within microseconds. If the `events` backend is descheduled in that gap, `delivery_log`'s probe finds the lock still held. The first suspect was that a dropped transaction only queues its ROLLBACK, which the pool's return task flushes later. That does not explain this failure. A backend left "idle in transaction (aborted)" already holds no locks, and keeping the queued ROLLBACK unflushed for 500 ms did not reproduce it. It does apply to client-side errors, where the transaction is still live on the server. #### Fix `apply_change` now owns the transaction. It begins it, runs the attempt in a new `attempt_change`, commits a `Committed` result and rolls back every other result, errors included. The early-return arms no longer roll back by hand. ROLLBACK first drains the failed statement's ReadyForQuery, which the backend sends only after its abort, so `apply_change` returns only once the lock is free. If the rollback fails, it is logged and the original error is returned. The test is unchanged. #### Validation Local Linux workstation, PostgreSQL 17.11 (docker compose): - Deterministic: gdb stopped the `events` backend at `AbortCurrentTransaction`, after its error reached the client, for 1 s. Without the fix: 6/6 failed at :480 with `delivery_log` reporting `skipped_locked`, the CI signature. With the fix: 6/6 passed. - Natural contention: 24 parallel loops of the test, with Postgres pinned to 2 CPUs shared with 4 busy loops. Without the fix: 6/1440 failed, all at :480 with `delivery_log` `skipped_locked`. With the fix: 0/1440. - `cargo test -p buzz-db --lib store::partition -- --include-ignored --test-threads=1`: 77/77 passed. Run in parallel against one shared local database, another test's database-wide event trigger also fails `cluster_global_parent_readers_follow_monthly_creation_and_catch_all_advancement` on main. CI gives each test its own database. - The test alone: 30/30 passed. `cargo test -p buzz-db --lib`, `cargo clippy -p buzz-db --all-targets -- -D warnings` and `cargo fmt --check` are clean. No regression test was added. Only a debugger can hold the window open, and the only remaining client-side error that leaves the transaction live is a concurrent-drop audit error, itself timing-dependent. **Rebased onto origin/main `64d12e297` (this branch):** Linux workstation, private `postgres:16` container with the desired-state schema applied as in the PostgreSQL lane (pgschema plus the reconcile script). - `cargo test -p buzz-db`: 191 lib, 21 `observability_source` and 5 doc tests passed. The 673 Postgres-backed tests are `#[ignore]`d there. - `cargo test -p buzz-db --lib store::partition -- --ignored --test-threads=1`: 56/56 Postgres-backed partition tests passed. The other 21 of the 77 counted above aren't Postgres-backed and ran with the lib tests. - `parent_contention_times_out_without_blocking_the_other_table` alone: 20/20 passed. - `cargo clippy -p buzz-db --all-targets -- -D warnings`, `cargo fmt -p buzz-db -- --check` and `git diff --check` are clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: mtoohey <mtoohey@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 09:49:06 +11:00
Tom Brow	cec918ae7a	perf(mobile): show a static placeholder for posterless videos (#9138) ... Video messages without a NIP-71 poster (older events, or uploads where poster generation failed) used to start a network `VideoPlayerController` for each bubble, seek to 100 ms and show that paused frame. Nothing cached the frame, and the decoder was disposed whenever the bubble left the tree, so every scroll back opened a new stream and decoded again. Those bubbles now show the existing static "Video attachment" placeholder with the play button. Nothing loads until the user taps, and the full-screen viewer behaves as before. Videos that have a poster are unchanged. ### Validation - Full mobile Flutter test suite passes, and `flutter analyze` is clean. - The change is almost pure deletion: the frame loader, its provider and test seam, and the tests that covered it are removed. - Limitation: no simulator run. A cloned golden sim built from this branch signed in to the internal community, but its channel list never finished loading, so I couldn't open a channel with a posterless video. buzz-review-completed 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Tom Brow <tomb@squareup.com> Co-authored-by: Tom Brow <tomb@squareup.com>	2026-11-05 14:36:05 -08:00
Matt Toohey	bdd76baea5	fix(desktop): compensate head trims from the unclamped scroll offset (#9094) ... ## Summary Re-entering a channel trims older pages from the head of the cached timeline, and our Virtua patch compensates for each trim from the live DOM `scrollTop`. When the reader was near the floor, the browser had already clamped `scrollTop` to the shorter list, so the removal was counted twice and the reader could be thrown to the top of history. That is the `message-target-highlight.spec.ts` flake in Desktop Smoke E2E. The patched scroller now remembers the last offset it observed or wrote, and compensates from that offset when the live one was clamped to the floor. ### fix(desktop): compensate head trims from the unclamped scroll offset #### Problem `message-target-highlight.spec.ts:79` ("message target flashes: ...") flaked in Desktop Smoke E2E, always at :179: the linked row had left the viewport 2.2 s after it flashed. - main: - https://github.com/block/buzz/actions/runs/41568206043/job/125695139845 - https://github.com/block/buzz/actions/runs/41669669478/job/126035103096 - https://github.com/block/buzz/actions/runs/41669740241/job/126035480466 - https://github.com/block/buzz/actions/runs/41694874145/job/126123865714 - https://github.com/block/buzz/actions/runs/41728489172/job/126240193479 - https://github.com/block/buzz/actions/runs/41737019256/job/126269365379 - branches: - https://github.com/block/buzz/actions/runs/41575774131/job/125719148353 - https://github.com/block/buzz/actions/runs/41636043464/job/125918005733 - https://github.com/block/buzz/actions/runs/41751914119/job/126319694104 - https://github.com/block/buzz/actions/runs/41544910958/job/125618626645 - https://github.com/block/buzz/actions/runs/41714110046/job/126190515793 All eleven runs predate #8742, and each failed in a case that picks `mounted[2]`: the bottom-settle mode #8742 fixed by holding the target. Run 41694874145 also failed once in "other channel, row near the bottom", and that mode survives #8742: on current main it fails 40 of 60 local runs with tracing on, as CI's retries run. #### Cause Re-entering a channel refreshes its newest window, which drops the older pages from the head of the cached timeline. #8660 passes that head trim to Virtua as a `shift`, and Virtua compensates by the removed height in a layout effect. Our Virtua patch (#2855) applies shift compensation to the live DOM `scrollTop` so that corrections landing before the next scroll event accumulate. But by then the browser has laid out the shorter list and clamped `scrollTop` to the new floor whenever the reader was within the removed height of it, so the removal is counted twice. In a failing run the view sat 6px above the floor: trimming the 412px channel intro moved it 406px too far, and trimming 3054px of older rows then wrote `scrollTop` -907, pinning it to the top. `onStartReached` loaded older history and the target never came back. A reader scrolled slightly above the bottom loses their place the same way. #### Fix The patched Virtua scroller remembers the last offset it observed in a scroll event or wrote itself. When a shift compensation finds the live offset below that and pinned to the floor, the browser clamped it, so the compensation starts from the remembered offset. Otherwise it keeps #2855's live-offset base. The ESM and CJS builds change identically, and the lockfile carries the new patch hash. #### Validation - New `virtuaShiftClampPatch.jsdom-test.mjs` drives the real patched `VList` over a scroller that clamps like a browser. Head trims far from, near and at the floor keep the reader's rows in place, a trim keeps a scroll whose event has not fired yet, and a prepend then a trim before any scroll event accumulate. Each part of the new condition fails a case when removed: the previous patch line fails the near-floor, at-floor and accumulation cases (3500 for 3900 and 4000, 3000 for 4000); upstream's store-offset base fails the unreported-scroll and accumulation cases; always using the remembered offset, or dropping the floor check, fails the unreported-scroll case (2500 for 1900). - "other channel, row near the bottom" with `--repeat-each=30 --workers=4 --trace=retain-on-failure`: 40/60 failed at :179 on current main, 0/60 failed (60/60 passed) with the fix, and 25/30 failed again with the previous patch reinstalled. - All five cases, 20 traced runs each: 100/100 passed. - scroll-history, virtualization, timeline-no-shift, overscroll-boundary, dm-history-live-regression and live-broadcast-reply-timeline, 3 runs each: 117/117 passed. - Full smoke project: 1485 passed, 3 skipped, 2 failed. `mentions.spec.ts:807` passed 5/5 on rerun. `messaging.spec.ts:2785` (own-message avatar) fails the same way on a current-main build here (3/3). - `pnpm test`: 6933 unit and 176 jsdom tests passed. **Rebased onto origin/main `64d12e297` (this branch):** Linux workstation. - `pnpm install --frozen-lockfile` installs virtua with the new patch hash (`f4df71c39f6c...`). - `pnpm typecheck` and `biome check --error-on-warnings` on the new jsdom test are clean. - Unit tests under `src/features/messages` and `src/features/channels`: 1990/1990. jsdom tests there: 12/12. Full desktop `pnpm test`: 6933 unit and 176 jsdom tests passed. - Playwright smoke after `pnpm build:e2e`, `--workers=4 --retries=0`: messaging, send-channel-binding, scroll-history, virtualization, timeline-no-shift, overscroll-boundary, live-broadcast-reply-timeline and message-target-highlight, 137 passed and 1 failed. The failure is `messaging.spec.ts:2785` (own-message avatar), which fails the same way on main here, as noted above. `dm-history-live-regression.spec.ts` isn't in the smoke project, so this run left it out. - `message-target-highlight.spec.ts`, `--repeat-each=10`: 50/50 passed. "other channel, row near the bottom", `--repeat-each=30`: 30/30 passed. ## Coordination No open PR edits `patches/virtua@0.49.3.patch`. The overlaps are all in `pnpm-lock.yaml`: - #5547 ("fix(desktop): ship a bundle that parses on WebKit without lookbehind (Intel/macOS 12)") and #7074 ("chore(deps): update react monorepo (patch)") merge cleanly with main but conflict with this branch on the `virtua@0.49.3` patch-hash lines. #5547 adds a patched dependency next to the virtua entry, and #7074 changes the React peer versions in the same lines. Whichever lands second resolves it by taking both sides' package changes and re-running `pnpm install` so the lockfile records `patch_hash=f4df71c39f6c...` for virtua. - #7073, #7080, #7633, #8218 and #8643 also touch `pnpm-lock.yaml` and merge cleanly with this branch. Older open PRs that touch it already conflict with main, and none gains a new conflict from this branch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: mtoohey <mtoohey@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 09:35:10 +11:00
Matt Toohey	1a42b5c39c	fix(relay): read kind:39000/39001 inputs behind the replacement lock (#9035) ... ## Problem `test_nip29_announce_channel_wire` flaked in CI (9/30 local solo runs failed). `emit_group_discovery_events` read the channel, its members and the latest live `created_at` *before* taking the coordinate's replacement lock, and only locked for the write. Two publishers for one channel could therefore store in the reverse order of their reads, or land on the same second where NIP-16's lower-id tie-break picks a random winner. Either way a kind:39000/39001 built from stale state could stay live. In the test, the relay owner reconnects to send an `auto_join=false` edit, gets auto-joined into the channel, and the join's kind:39000 — built from the old `auto_join=true` state — replaces the edit's. ## Fix - **buzz-db:** `lock_group_state_snapshot` takes the replacement lock first, then reads the channel, the members the kind actually publishes (full roster for a DM's kind:39000, owners/admins for kind:39001, none otherwise) and the latest timestamp; `LockedGroupStateSnapshot::replace` stores on the same transaction. This is the fence kind:39002 already had via `LockedMemberSnapshot`. The lock is taken once per write (`take_replacement_lock`), and the write keeps its `ReplaceAddressableEvent` transaction timer and writer population so metrics stay continuous. - **buzz-relay:** `emit_addressable_discovery_event` builds tags and `created_at` (live + 1s) from that snapshot. ## Tests - buzz-db: `group_state_snapshot_reads_behind_the_replacement_lock`, `group_state_snapshot_reads_the_members_its_kind_publishes`, `group_state_replacement_takes_its_lock_once`. - buzz-relay: `a_publisher_that_read_before_an_edit_cannot_replace_it`, with `BeforeLock` and `AfterRead` legs driven by new `nip_fi_test_hooks`. - Every new assertion was confirmed to fail when its guard is reverted. - `test_nip29_announce_channel_wire` solo against a relay built from this tree: 30/30 passed (9/30 failed before). Merged with current `main`; the only conflict kept main's bounded `channel state write failed: {kind}` error mapping on the new snapshot write. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-11-06 09:34:33 +11:00
Matt Toohey	580caca667	fix(db): unlock the storage-accounting lease before its session closes (#9097) ... ## Summary `StorageAccountingLeader` holds a session-level advisory lock on a detached connection. Dropping it only closes the socket, and PostgreSQL frees the lock when the backend exits, which can be after the next lock attempt. `release(self)` now unlocks on the leader's own session before closing it. The relay storage-sweep test releases its worker between iterations instead of dropping it, which fixes the `worker_publication_activates_reader_without_restart_or_mode_change` flake. The follow-up commit makes `release` report success only when the unlock actually ran on the session that held the lease. ### fix(db): unlock the storage-accounting lease before its session closes #### Problem The PostgreSQL lane's `storage_sweep::postgres_tests::worker_publication_activates_reader_without_restart_or_mode_change` failed with "called `Option::unwrap()` on a `None` value" at crates/buzz-relay/src/storage_sweep/postgres_tests.rs:66:74 after 5.53 s, on the unrelated branch elrond/cli-credential-chain: - https://github.com/block/buzz/actions/runs/41730818977/job/126253029211 #### Cause Line 66 is `db.try_lock_storage_accounting().await.unwrap().unwrap()` inside the loop over three metrics modes. `Ok(None)` means `pg_try_advisory_lock` returned false. A passing run takes about 15 s, roughly 5 s per iteration, so the failure is the start of iteration 2. Each test process gets its own database and advisory locks are per-database, so the holder is iteration 1's own worker. `StorageAccountingLeader` holds a session-level advisory lock on a detached connection. The test ended each iteration with `drop(worker)`, which only closes the socket. PostgreSQL frees session locks during backend exit, after it notices EOF, so the lock can outlive the drop on a loaded server. Iteration 2's lock attempt raced that exit. The db-crate test `lost_lock_session_cannot_overwrite_successor_snapshot` already polls for this release. The relay test didn't. #### Fix Add `StorageAccountingLeader::release(self)`. It runs `pg_advisory_unlock` on the leader's own session and then closes it, so the lease is free when it returns. This follows the unlock-then-close pattern in `migration.rs`. The relay test now releases the worker at the end of each iteration instead of dropping it. No assertions, timeouts or retries change. buzz-admin's `storage-snapshot` is left as is. It runs once per pod, and its CronJob uses `concurrencyPolicy: Forbid`, `backoffLimit: 0` and `restartPolicy: Never`, so the next lock attempt comes from a later scheduled run, not from the same process. `released_lease_admits_successor_before_old_session_exits` covers the new method. A closing backend drops its temp tables before it frees session locks. So the test locks a temp table owned by the leader's session from another transaction, which keeps the old backend alive after release. It then asserts that a fresh `try_lock` succeeds at once and that the old backend has not exited. Open PRs #9040 (edits this relay test and keeps `drop(worker)`) and #9043 (changes the `sqlx` import in storage_accounting.rs) will see small textual conflicts. #### Validation Local Linux workstation, throwaway `postgres:16-alpine` (CI uses `postgres:16`), desired-state schema applied as in the PostgreSQL lane: - Adversarial ordering, using temporary, uncommitted changes: the leader's connection was made public and the relay test held the worker backend's exit for 1 s with the same temp-table lock. Without the fix: 15/15 failed at postgres_tests.rs:66:74 after 5.1-5.3 s, the CI signature. With the fix: 10/10 passed. - Natural contention: Postgres, the test processes and busy loops were pinned to 4 CPUs. With 8 parallel runs and 4 busy loops, the unmodified test passed 24/24 without the fix. With 16 parallel runs and 16 busy loops, it passed 48/48 without the fix and 48/48 with it. Load alone did not open the window locally. - New db test: 50/50 passed. With the unlock removed from `release`, leaving only `close`, it fails with "a released lease must admit a successor at once". All four storage_accounting Postgres tests pass. - `cargo fmt --all -- --check`, `cargo clippy -p buzz-db -p buzz-relay -p buzz-admin --all-targets -- -D warnings`, `just file-size-check` and `git diff --check` are clean. A green run doesn't prove the flake is gone. Hosted CI covers broad validation. **Rebased onto origin/main `64d12e297` (this branch):** the relay test, run in a fresh database cloned from the desired-state template each time, passed 8/8 at the branch head, which includes this commit. The full rebased results are under the next commit. #### Rebase note #9040, which this commit's message lists as an open PR, has since merged to main. It removed the per-community storage metric, including the `buzz_community_storage_bytes` assertion just above `drop(worker)` in `crates/buzz-relay/src/storage_sweep/postgres_tests.rs`. The cherry-pick onto current main conflicted on that hunk. The resolution keeps main's version of the test and changes only `drop(worker)` to `worker.release().await.expect("release worker lease")` with its comment. Nothing else in the commit changed, and no semantic adjustment was needed. The commit message is unchanged, so it still describes #9040 as open. ### fix(db): report a storage-accounting release only once its unlock succeeded #### Problem Second review of the branch (nit) on 51c27212a ("fix(db): unlock the storage-accounting lease before its session closes"), `StorageAccountingLeader::release` in storage_accounting.rs. That commit cites the unlock-then-close pattern in runtime/migration.rs but handles errors differently: - A failed unlock returned early with `?`, skipping the graceful close. The dropped connection then closed only the socket. - A failed close after a successful unlock was returned as an error, although the lease was already free. - `pg_advisory_unlock` returning false, which means this session didn't hold the lock, was ignored and reported as success. So the doc comment's "Once this returns, the lease is already free" did not hold for every `Ok`. #### Cause `release` propagated the unlock's error with `?` before the close, propagated the close's error after a successful unlock, and ignored the boolean that `pg_advisory_unlock` returns. So a failed unlock skipped the graceful close, a failed close was reported although the lease was already free, and an unlock that returned false because this session no longer held the lock was reported as success. #### Fix Follow migration.rs's `with_schema_destruction_lock`. Run the unlock as `query_scalar::<bool>` and keep its result, always attempt the close and ignore its error, then return the unlock's error, or `InvalidData` if the unlock returned false. The doc comment now says that `Ok` means the unlock ran on the lease's own session, so the lease is free even if the close failed, and that an error means the unlock failed or the session no longer held the lease. `released_lease_admits_successor_before_old_session_exits` gains a final step: it unlocks the successor's lease on the successor's own session, then expects `release` to fail. Its successor check is now an `expect` with the same message. #### Validation Local Linux workstation, the `postgres:16` container used earlier on this branch, `BUZZ_TEST_DATABASE_URL` pointing at it: - `cargo test -p buzz-db --lib storage_accounting -- --ignored`: 4/4 storage_accounting Postgres tests pass. - With the unlock's boolean ignored, as before this change, the new step fails with "releasing a lease the session lost must fail". - `cargo fmt -p buzz-db -- --check`, `cargo clippy -p buzz-db --all-targets -- -D warnings` and `git diff --check` are clean. **Rebased onto origin/main `64d12e297` (this branch, both commits):** Linux workstation, private `postgres:16` container with the desired-state schema applied as in the PostgreSQL lane (pgschema plus the reconcile script). - `cargo test -p buzz-db`: 191 lib, 21 `observability_source` and 5 doc tests passed. The 674 Postgres-backed tests are `#[ignore]`d there. - `cargo test -p buzz-db --lib storage_accounting -- --ignored --test-threads=1`: 4/4 passed. `released_lease_admits_successor_before_old_session_exits` alone: 20/20 passed. - `storage_sweep::postgres_tests::worker_publication_activates_reader_without_restart_or_mode_change`, each run in a fresh database cloned from the desired-state template: 8/8 passed. - `cargo clippy -p buzz-db --all-targets -- -D warnings`, `cargo clippy -p buzz-relay --all-targets -- -D warnings`, `cargo fmt -p buzz-db -p buzz-relay -- --check` and `git diff --check` are clean. ## Coordination - #9043 (draft, "feat(relay): serve storage usage as an operator pull feed") edits `crates/buzz-db/src/store/storage_accounting.rs`. `git merge-tree` against its current head shows two small textual conflicts. First, both sides rewrite the `sqlx` import to the same `use sqlx::{postgres::PgConnection, Connection};`, and #9043 also drops `use uuid::Uuid;`. Second, #9043 replaces `usage_delivery_reads_snapshot_time_and_community_states` at the point where this branch adds `released_lease_admits_successor_before_old_session_exits`. Keeping both sides resolves each one. - #7899 also touches `storage_accounting.rs`. It already conflicts with main, and this branch adds no new conflicted file to it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: mtoohey <mtoohey@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 09:28:55 +11:00
Matt Toohey	a22ff133b4	test(relay,db): remove three timing races from Postgres and E2E tests (#9098) ... ## Summary Three tests asserted on outcomes that depended on how fast something unordered happened. The first depended on dial latency inside a 150 ms acquire budget, the second on whether two aux-page replies landed in one poll, and the third on whether NIP-42 admission auto-joined the joiner before the invite claim did. Each test now sets up the state it asserts before the racing actor can run: it fills the reader warm floor first, pauses the writer's proxy after the first aux page, or claims the invite before the joiner connects. Only tests change. No assertions are loosened, and no sleeps, retries or timeouts are added. ### test(db): fill the reader warm floor outside the acquire budget first #### Problem `reader_warm_at_production_budget_ends_partial_when_dials_stall` failed on main at reader_pool_postgres_tests.rs:177, "one partial wave, then one wave that opens nothing": left (15, 0, 5), right (15, 3, 7). Postgres lanes run with no retries. - main (d3a4909cc): - https://github.com/block/buzz/actions/runs/41693681008/job/126127000284 The test is unchanged on current main. #### Cause The test builds the reader pool with the production constructor, so every warm acquire, dial included, gets the 150 ms `READER_ACQUIRE_TIMEOUT`. It relayed the proxy's first 3 sessions and stalled the rest, so it needed the first wave's 3 relayed dials (TCP through the proxy, backend fork, SCRAM, `after_connect`) to finish inside that budget. On a loaded runner (4 vCPUs, nextest test-threads=8) none did. All 5 first-wave acquires missed, and the pass ended after one empty wave as the product should, which gives (15, 0, 5). The warm pass was correct, but the test's outcome depended on dial latency. The session counts had a second race. The proxy accepts connections on the test's current-thread runtime. If that thread is descheduled past the budget, the timed-out acquires and the listener's readiness are handled in the same tick. So `proxy.sessions() == 10`, checked right after the pass, could miss a stalled dial still waiting to be accepted. #### Fix Fill the warm floor before stalling anything, as boot does. `Db::read_pool_boot_ping_once` closes its probe connection, and sqlx's floor maintenance then dials the 3 floor connections under its own 300 s deadline, outside the acquire budget. The test waits for them with `await_idle_floor`, then stalls every session from the recorded baseline (`proxy.stall_from(proxy.sessions())`) instead of a hardcoded 3. Wave 1 now holds the 3 idle connections without dialing, and its 2 dials stall. Wave 2's 5 dials stall, so it makes no progress and the pass ends. That gives (15, 3, 7) at any dial latency. The session count is now relative to the baseline and asserted once, after the existing settle. That single check covers both "the pass stops dialing" and "nothing re-dials". The report assertion, the `elapsed < 2s` bound and the production pool are unchanged. Only the test changes. The setup still needs one of the boot ping's five attempts (spaced over about 4 s by `READER_BOOT_PROBE_BACKOFF`) to dial within 150 ms, the same exposure as `reader_boot_ping_warms_pool_to_min_connections`. Before, all 3 concurrent first-wave dials had to make it on their first try. #### Validation Local Linux workstation, on a private postgres:17-alpine. The test processes and Postgres shared 4 pinned CPUs, with 8 parallel workers. "Before" is the test as on main. - CPU stress, no busy loops: before, 2/40 failed with (15, 0, 5) at :177, the CI signature. After, 120/120 passed. - Plus 4 busy loops on those CPUs: before, 80/80 failed with (15, 0, 5). After, 80/80 passed. In 80 more instrumented runs (all passed), the boot ping retried 16 times: 15 runs needed 2 attempts and 1 needed 3. - Adversarial latency, using a temporary, uncommitted proxy hook that delays chosen sessions' first server reply by 200 ms (past the budget), 10 runs each: - every relayed session slow: before, 10/10 (15, 0, 5). After, 10/10 failed at "boot ping must reach the reader", the bound above. - the first 4 sessions slow: before, 10/10 (15, 0, 5). After, 10/10 passed, with the ping succeeding on its 5th attempt. - every session but the first slow: before, 10/10 (15, 1, 9). After, 10/10 passed. - Production mutations, each of which fails the test: dropping the no-progress break gives (15, 3, 27). A 2 s acquire budget fails the elapsed bound at 4.0 s. Closing the warmed connections instead of releasing them leaves the floor at 0 idle. - All 7 `reader_pool_postgres_tests` pass. `cargo fmt --check`, `cargo clippy -p buzz-db --all-targets -- -D warnings` and `git diff --check` are clean. A green run doesn't prove the flake is gone. Hosted CI covers broad validation. **Rebased onto origin/main `64d12e297` (this branch):** Linux workstation, private `postgres:16` container with the desired-state schema applied as in the PostgreSQL lane (pgschema plus the reconcile script). - `cargo test -p buzz-db`: 191 lib, 21 `observability_source` and 5 doc tests passed. - `cargo test -p buzz-db --lib reader_pool_postgres_tests -- --ignored --test-threads=1`: 7/7 passed. `reader_warm_at_production_budget_ends_partial_when_dials_stall` alone: 20/20 passed. - `cargo clippy -p buzz-db -p buzz-relay -p buzz-test-client --all-targets -- -D warnings`, `cargo fmt -p buzz-db -p buzz-relay -p buzz-test-client -- --check` and `git diff --check` are clean. ### test(relay): hold the writer's second aux page too #### Problem `thread_window_batch_discards_earlier_windows_after_revocation` failed on main at failure_postgres_tests.rs:55, "barrier must precede closure completion": left 2, right 1. Postgres lanes run with no retries. - main (d3a4909cc): - https://github.com/block/buzz/actions/runs/41693681008/job/126127000284 The test and the code it covers are unchanged on current main. #### Cause This is the race #8740 (322fdbf08) fixed for the replica test only. `pause_after_aux_page` polls the request until a `thread_window_aux` span closes, then expects exactly one closed. `query` runs the closure's two aux hops (reactions and edits, then their deletions) back to back, one span each. On the writer, each hop is a pool acquire with its idle ping, BEGIN, two SET LOCALs, the SELECT and ROLLBACK. #8740 left the writer callers on `None`, so only the request returning Pending on a reply that had not arrived yet held hop 2. sqlx tries each read before it waits for readiness, and tokio leaves the socket marked readable after the previous reply, so a reply Postgres has already sent is consumed in the same poll. If each of hop 2's replies lands before the test thread reads it, for example when a loaded runner deschedules that thread after its writes, hop 2 closes in the poll that closed hop 1 and the barrier counts 2. `assert_aux_access_change`, behind the grant and revocation tests, uses the same `None` path. #### Fix `proxied_state` gives the paused request a copy of the fixture's state whose production writer connects through a `PausingProxy`. `production_db_at` is `production_db` with a URL override. `AuxPages` pauses the proxy when hop 1's span closes. That runs inside the request's poll, before hop 2's first bytes are written, so hop 2 stays pending however fast Postgres answers. Each test resumes the proxy after its access change, just before `request.await`. Only the request uses the proxied state: `assert_aux_access_change` reads through the fixture's direct `state.db` while the request is held, and through the proxy that read would deadlock. No caller passes `None` any more, so the `Option` goes. Only tests change. No assertions are loosened, and no sleeps, retries or timeouts are added. Draft #8991 (BUZZ-512) also edits `AuxPages` and `pause_after_aux_page`, adding a waker for its detached HTTP handler. It also moves these two callers onto a proxy, through its own `proxy_request_pool`, which replaces `f.state.db` with a raw pool. Rebasing it onto this keeps its waker on the non-`Option` proxy field and can use `proxied_state` for those two callers. #### Validation Local Linux workstation, Postgres 17 in docker, one test per process as nextest runs them. "Before" is the test as on main. - Adversarial ordering, using a temporary, uncommitted change as in #8740: after the barrier, ten more polls of the request over 500 ms, then the barrier's count assertion. Before, the batch, grant and revocation tests failed 30/30 at that assertion with left 2, right 1, the CI signature, and the replica test passed 10/10. After, 40/40 passed. - Natural contention: the three tests and a private postgres:17-alpine shared 2 pinned CPUs with 4 busy loops, 3 processes at a time. Before, 450/450 passed, so the same-poll burst did not occur locally. After, 450/450 passed. - Production mutation: skipping `query_batch`'s final implicit access-set check fails all three tests with 200 instead of 503. - All 19 `api::bridge::thread_window` Postgres tests pass, run serially against one database. `cargo fmt --check`, `cargo clippy -p buzz-relay --all-targets -- -D warnings` (1.98.1), `git diff --check` and `check-postgres-test-discovery.py` are clean. Running several of these tests in one libtest process can fail them at :41 with no aux span seen, on main too. tracing-core builds a new callsite's interest from the registering thread's default while only one scoped dispatcher exists. nextest runs one test per process, so CI is not exposed. The runs above use one process per test. A green run doesn't prove the flake is gone. Hosted CI covers broad validation. **Rebased onto origin/main `64d12e297` (this branch):** same database setup as above. - Each of the 19 `api::bridge::thread_window` Postgres tests ran in its own process and its own database cloned from the desired-state template, as nextest runs them: 19/19 passed. - The batch, grant, revocation and replica tests, 5 more runs each in the same way: 20/20 passed. ### test(relay): claim the invite before the joiner connects #### Problem `test_invite_claim_auto_joins_default_channel` (crates/buzz-test-client/tests/e2e_relay.rs) failed in "Relay and PostgreSQL / Relay E2E" at :363 with "recv kind:44100 notification for the default channel: Timeout". Two of the three reruns passed. - run 41568808045 attempt 1 (rerun passed): - https://github.com/block/buzz/actions/runs/41568808045/job/125701098947 - run 41657074260: - https://github.com/block/buzz/actions/runs/41657074260/job/125997465078 - run 41716543775 attempt 1 (rerun passed): - https://github.com/block/buzz/actions/runs/41716543775/job/126206390776 #### Cause The e2e relay runs without `BUZZ_REQUIRE_RELAY_MEMBERSHIP`, so every human NIP-42 admission spawns `auto_join_never_joined_channels` (handlers/auth.rs). The test connected the joiner and sent its kind:44100 REQ before minting and claiming the invite. The admission therefore joined the joiner to the flagged channel before the claim did. The claim's `auto_join_default_channels` then got `AlreadyMember` and emitted nothing, so the admission notice was the only kind:44100. All three relay logs show this. The joiner's "auto-joined default channel on admission" appears 39, 52 and 106 ms after its NIP-42 auth, before "relay member added via v2 invite". The claim logs no join for the joiner. When the admission notice reached the client before EOSE, `collect_until_eose` consumed it, and the test discarded that result. The test then waited 5 s for a live event that never came. When the notice arrived after EOSE, the test passed, but on the admission path. Either way, the test never covered the claim-time auto-join it is named for. #### Fix Only the test changes. The joiner opens no WebSocket until the claim returns. Over NIP-98, the claim awaits the join, the kind:39002 roster and the kind:44100 before its 200. The test now: - checks the roster before the joiner connects, so only the claim can have joined them; - then connects and REQs kind:44100 `#p=joiner` with `since` taken before the claim (minus the file's usual 5 s margin); - asserts that the events `collect_until_eose` returns hold exactly one notice for the channel, signed by the relay's NIP-11 `self` key. Nothing depends on EOSE-vs-live ordering any more. The admission that follows skips the channel, because the claim already added a member row. No timeouts change, no retries are added, and no assertions are loosened. The repeat-claim and cleanup steps are unchanged. No open PR touches this test. Draft #9045 (default-channel recovery) changes invites.rs, but not this test or this ordering. #### Validation Local Linux workstation, fresh database, relay started by scripts/start-relay-for-tests.sh. Redis was the host's own, and `BUZZ_METRICS_PORT` was moved off 9102, which bloxlet owns. Each iteration ran CI's `e2e_relay invite` selection (8 tests in parallel). - Unmodified test: 12/60 failed at :363 with the CI message. - Adversarial ordering, using temporary, uncommitted changes: a 500 ms sleep between connect and REQ, plus logging of what `collect_until_eose` returned. 20/20 failed with the CI signature, and each time `collect_until_eose` had returned the channel's notice. In the relay log, the admission join came 28 ms after auth, the REQ queued one event, and the claim emitted nothing. - Fixed test: 100/100 passed. With the same 500 ms sleep: 30/30 passed, each with exactly one notice, from the claim. Four concurrent loops of 25: 100/100 passed. - Falsifiability, using a temporary relay build without the claim-time `auto_join_default_channels` call: the fixed test failed 10/10 at the roster assertion (`None`). The old test passed 18/20 against that build; its two failures were its own :363 flake. - The full CI "Relay E2E tests" step list passed once. - `cargo fmt --check`, `cargo clippy -p buzz-test-client --tests -D warnings`, `just file-size-check` and `git diff --check` are clean. A green run doesn't prove the flake is gone. Hosted CI covers broad validation. **Rebased onto origin/main `64d12e297` (this branch):** the relay was built from this branch with `--profile ci` (its relay code is identical to main) and run with CI's environment. It used a fresh desired-state database with the deployment community seeded, the host's Redis, and a private MinIO-compatible container (the image docker-compose pins) for the startup git probe. `BUZZ_METRICS_PORT` was moved off 9102. - CI's `e2e_relay invite` selection: 30/30 iterations passed, 8 tests each. - CI's full "Relay E2E tests" step list, once: 10 of 11 steps passed. `test_nip29_announce_channel_wire` failed at e2e_relay.rs:3862: the kind:39000 metadata still carried `auto_join` right after the relay owner cleared it. This branch doesn't touch that test or any relay code. Run alone against the same relay, that test failed 8 of 20 times. Re-running the whole step against one database isn't meaningful, because `test_nip29_announce_channel_wire_overlong_name_is_refused_before_storage` reuses a fixed channel name. ## Coordination - #8991 ("BUZZ-512: Track session application and drain") edits `AuxPages` and `pause_after_aux_page` in `crates/buzz-relay/src/api/bridge/thread_window/postgres_tests/failure_postgres_tests.rs`, and `git merge-tree` shows a conflict there. #8991 already conflicts with main in 15 files, so it needs a rebase anyway. The second commit's message describes that rebase: keep #8991's waker on the non-`Option` proxy field, and use `proxied_state` for the two callers it moves onto a proxy. - #6642 and #7569 touch `crates/buzz-test-client/tests/e2e_relay.rs` and merge cleanly with this branch. - #7852 touches `thread_window/postgres_tests.rs` and `e2e_relay.rs`. It already conflicts with main, and this branch adds no new conflicted file to it. Older open PRs that touch `e2e_relay.rs` also already conflict with main, and none gains a new conflict from this branch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: mtoohey <mtoohey@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 09:28:38 +11:00
Luke Tornquist	47be2e2d3f	fix(db): keep busy-reader history reads off the writer and cap them per pod (#9132) ... ## Problem A routed read that misses the reader's 150ms acquire budget (`reader_acquire_timeout`) falls back to the writer, whatever the read is. Under a burst of subscription-history reads (REQ history, channel and thread windows, feeds, search hydration, COUNT), the reader pool fills, and every miss moves a heavy history query onto the writer. There it competes with the authentication, tenant-resolution and moderation lookups that only the writer serves, and their latency climbs with it. Nothing bounds how many history reads a pod sends to the reader at once, so the reader queue grows without limit as well. Separately, the compatibility families `buzz_db_pool_acquire_wait_seconds` and `buzz_db_pool_acquisitions_total` have no `operation` label, so a breakdown by operation reports every sample as unlabelled. ## Changes 1. **History reads no longer spill onto the writer when the reader is busy** (`Db::route_read`). `route_read` now returns `Result<RouteDecision>`. For `ReaderOperation::SubscriptionHistory`, a `reader_acquire_timeout` on a saturated reader pool rejects the read with `sqlx::Error::PoolTimedOut`. Saturated means every connection the pool may hold exists and none is idle (`Db::reader_saturated`). The check is needed because sqlx also reports a reader that refuses dials as `PoolTimedOut`, and that reader is down, not busy. The relay already answers that with its generic retryable `CLOSED` `error: database error`, so there is no new client contract. A reader that is wrong rather than busy (unreachable, validation failure, proof timeout, stale token) and prechecks that never reach the reader still fall back to the writer. Authorization reads (`is_relay_member`) keep the writer fallback unchanged. 2. **Per-pod cap on reader-served history reads** (`runtime/history_reads.rs`). A FIFO `tokio::sync::Semaphore` admits at most the reader pool size less one fifth (at least one) history reads at once. For example, the default pool of 20 admits 16. The remaining connections are normally free for authorization reads. A read waits at most 1s for a permit and is otherwise rejected the same way. The permit is owned by the reader transaction (`ReaderTx`) and released when the transaction is dropped; routed reads drop it before re-running a failed replica read on the writer. That holds whether the transaction is a per-call routed read or a `ReadSession` the relay keeps for follow-up queries. The cap bounds history transactions, not connections: sqlx returns a dropped transaction's connection asynchronously through the reader release check. Released connections can therefore briefly occupy the free slots, for one round trip normally and at most the 5s release timeout when sockets stall. In that window, authorization reads fall back to the writer as before, and history reads on the saturated pool are rejected, not spilled. 3. **Rejections are observable**: `buzz_db_route_decision{decision="rejected",reason="history_read_cap"|"reader_acquire_timeout"}`. 4. **Legacy pool families gain `operation`.** The population is unchanged, and summing over `operation` gives the original series. 5. **HTTP reads answer 503 `database unavailable` when the database can't be reached** (`api::db_read_error`). This covers a pool timeout, closed pool or socket failure on `POST /query`, `/count` and every bridge `/query` branch that serves a history read (channel window, the three feeds, thread replies), including a shed history read. These used to return a generic 500. The thread-window path already answered 503 for pool timeouts; this applies the same mapping to the shared helper. Pool sizes are unchanged on purpose. A larger pool adds lock contention on the writer rather than relieving it. Docs: `deploy/charts/buzz/README.md` (operator contract) and `PLANS/REPLICA_FULL_READ_ROUTING_DESIGN.md` (fail-closed exceptions). ## What got simpler The concept count is roughly flat. Instead of a caller-side permit, the permit lives in the one type every replica transaction already flows through (`RouteDecision::Replica` / `ReadSessionInner::Replica`). The 11 routed call sites change only by a `?`, and the compiler enforces that change: a call site cannot silently fall through to the writer on a rejection. No new error variant, CLOSED reason or env var. ## Tests - `postgres_tests::saturated_reader_spends_one_acquire_budget_and_rejects_history_reads` (rewrites the former `routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold`). A size-1 reader is held saturated with a cold Aurora cache. A history read (`count_events_routed`) is rejected with `PoolTimedOut` within one acquire budget, records `rejected/reader_acquire_timeout`, and releases its permit. An authorization read on the same reader still falls back as `writer/reader_acquire_timeout`. - `postgres_tests::match_nothing_reads_take_no_connection` (unchanged): its control still requires a history read against an unreachable reader to fall back to the writer. It caught the first cut of this change, which rejected that read. - `postgres_tests::history_permit_follows_the_read_session_and_a_full_gate_rejects`: a held replica `ReadSession` keeps its permit. With the gate full, the next history read waits the full permit budget and is then rejected as `rejected/history_read_cap`, with no writer route recorded. Dropping the session frees the permit and the next read routes to the replica. - `history_reads::tests`: a cap sizing table, plus bounded-wait rejection and reopen under paused time. - `observability` label tests now require `operation` on both legacy families. - `api::db_read_error_tests::unavailable_database_maps_to_retryable_503`: pool timeout, closed pool and I/O map to 503 `database unavailable`. Other errors stay 500. - `postgres_tests::shed_history_read_maps_to_503_on_every_bridge_branch`: with the reader saturated, `/query` channel-window, mentions, needs_action, activity and thread-reply requests each answer 503 `database unavailable` (each serves 200 beforehand). - Mutation-checked on Blox: re-enabling the writer spill, forcing the saturation check either way, skipping the gate, dropping the permit early and removing the legacy label each fail a test. ## Residual - During a reader failover whose old sockets stall, both slots sit in the release check, so the pool reads as saturated. History reads are shed as retryable for up to the release bound (5s) instead of loading the writer. `stalled_reader_sessions_release_their_slots_for_new_connections` now retries through that window and still requires recovery onto the reader. - A reader that accepts no TCP at all can fill the pool with in-flight dials and look saturated. Its history reads are then rejected rather than moved to the writer, which is the same outcome as a full gate. - Shedding relies on clients backing off on a retryable `CLOSED` (RPR #9). WebSocket clients already treat `error: database error` as retryable. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 22:12:30 +00:00
Tom Brow	7ac16b2917	perf(mobile): memoize npub labels (#9137) ... Opening a big channel on mobile stalls while member labels resolve. On an iPhone 12 mini (wired Profile build of main 120d75d976), opening a 5,892-member channel had frames of 217, 113 and 101 ms. The channel identity scope recomputed 13 times during the open as profiles arrived, 25 to 35 ms each: 235 ms of 1,755 ms UI CPU, 190 ms of it bech32 encoding. Every recompute labels each member without a profile name by npub (`missingOwnerProfiles` → `IdentityNameSources.factFor` → `shortPubkey` → `fullNpub`), and encoded the same keys again. The same chain cost 152 ms in a 15 s resume with the channel open. `fullNpub` now memoizes encodings by hex key, bounded at 8,192 entries. Once full it keeps what it has, so a roster larger than the limit still reuses most of its encodings. Labels are unchanged. ### Validation - Added tests: `fullNpub reuses the encoding of a key it has already encoded` fails without the memo; `a roster larger than the cache still reuses its encodings` fails if the cache clears when full. - Host benchmark, `shortPubkey` over 5,900 random keys: 53.5 ms on the first pass, about 3.1 ms on later passes. - Full mobile suite passes except the 10 known notification/push failures and `push_lease_revocation_outbox_test`, which also fail on main. Analyze clean. - Automated simulator run of this branch (debug build, Builderlab community): searched "studio", opened a channel result and came back; people results show the same short npub labels as main (for example `npub194z...aq3k`). - No post-change device capture. The scope still recomputes as profiles arrive; this removes the encoding cost from each recompute. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Tom Brow <tomb@squareup.com> Co-authored-by: Tom Brow <tomb@squareup.com> Co-authored-by: studio <dc2fc481343672df31a689840c09299972e704dac592e0da6f18486646a7ee3e@buzz.block.builderlab.xyz>	2026-11-05 13:49:09 -08:00
Tom Brow	1ce831645c	mobile: stop re-parsing messages on every resume (#9140) ... Resuming the app with a busy channel open stalled for several frames (149, 136, 109 and 102 ms on an iPhone 12 mini), because every visible message re-parsed its markdown, repeatedly. The app widget rebuilds on every lifecycle change, and each rebuild created new light and dark `ThemeData`. A new `ThemeData` never equals the previous one, so `MaterialApp` animated between two identical themes for its full theme transition. Each frame of that animation produced a new markdown theme, and every mounted message re-parsed its markdown in response (`_MdWidgetState.didChangeDependencies` and `didUpdateWidget`). In a 15 s resume on a large channel that re-parse took 373 ms of main-isolate time. The themes are now built only when the community accent changes, so a lifecycle rebuild keeps the same theme objects and starts no transition. ### Validation - Added test: `an app rebuild keeps its themes and markdown theme` rebuilds the app and checks that the theme and the markdown theme stay identical. It fails without the change. - Full mobile suite: only the known failures that also fail on main (notification toggle, push bootstrap, push bridge, push lease revocation outbox). - Automated simulator run (iPhone 17 Pro, debug build): opened a channel with markdown messages, backgrounded and resumed the app several times, and switched between light and dark appearance while the app was open. Messages render, resume works, and the app follows appearance changes live. A channel's long-press action sheet keeps its light background after a switch to dark mode on both this build and main, so that is unrelated to this change. - Profile evidence (iPhone 12 mini, Profile build, before the change): the stalls and re-parse samples above came from a resume with a large channel open. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Tom Brow <tomb@squareup.com> Co-authored-by: Tom Brow <tomb@squareup.com> Co-authored-by: studio <dc2fc481343672df31a689840c09299972e704dac592e0da6f18486646a7ee3e@buzz.block.builderlab.xyz>	2026-11-05 13:48:23 -08:00
Tom Brow	1b9b307176	fix(mobile): renew expired assertions for native media and drop retired-session avatars (#9134) ... Follow-up to [#9069 media auth](https://github.com/block/buzz/pull/9069), which merged at its approved head. This PR carries the fixes for the last review round on that PR. ### Changes - **Native media reads renew an expired assertion** (Carl's [recommended blocker](https://github.com/block/buzz/pull/9069#discussion_r4485183400)). Voice notes and posterless video previews hand headers to native code. They now use `freshNativeMediaHeadersProvider`, which awaits `HostedRelayAuth.headersForUrl` and so asks for a new assertion once the cached one has expired. Before, while the socket stayed up they sent none and the relay refused the read. If the renewal fails, the read falls back to cached headers and the relay decides. The voice-note player's `headers` callback is now `FutureOr`. Both the iOS authenticated download and the Android native load await it, then recheck their generation, so a cancel during a slow renewal never starts the stale load. - **Notification avatars drop a retired organization session** ([review thread](https://github.com/block/buzz/pull/9069#discussion_r4485125609)). When an avatar download finishes after the organization session that issued its assertion was replaced, BuzzPushKit removes the cached response and refuses the avatar. - **Test:** cached reads never present a signed-out or replaced account's assertion ([optional note](https://github.com/block/buzz/pull/9069#discussion_r4485183412)). Not changed: native reaction-detail avatars still use the cached assertion. After it expires they fall back to initials. ### Validation - Each new test was checked to fail with the code it guards reverted: - `replacedOrganizationSessionRefusesAvatar` (BuzzPushKit): a session replaced mid-download must be refused. - `fresh native media headers renew an expired assertion`: covers renewal and the fallback when renewal fails. - `remote voice notes renew an expired relay assertion`: a journey through `VoiceNoteAttachment`; it fails when the attachment is switched back to the synchronous provider. - `Android defers remote auth until playback starts`: now cancels during a slow renewal and expects no native load. - `an issued assertion serves cached reads for the relay`: now signs out and restores a replacement account. It fails without the account check. - `swift test` (69 tests) passes. The full mobile suite (`just mobile-install mobile-check mobile-test`) passes with these commits on main as of [#9069 media auth](https://github.com/block/buzz/pull/9069). - The automated full-app simulator run from [#9069 media auth](https://github.com/block/buzz/pull/9069) covers the surrounding flows. This change is limited to the header source for voice notes and video previews, and none of the test communities enforce the hosted assertion, so a simulator run would not show the difference. No end-to-end run against a relay that enforces the assertion. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Tom Brow <tomb@squareup.com> Co-authored-by: Tom Brow <tomb@squareup.com> Co-authored-by: studio <dc2fc481343672df31a689840c09299972e704dac592e0da6f18486646a7ee3e@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 13:44:54 -08:00
Luke Tornquist	74abe2f0fe	perf(db): split newest-first events reads at a month floor so Postgres prunes partitions (#9091) ... ## Summary Newest-first `events` page reads are now split at the start of the previous UTC month, so Postgres can prune the older partitions whenever the newest ones fill the page. Results are unchanged. **Why.** `events` is range-partitioned by month, and every partition carries the full index set. A page read with no `created_at` lower bound can't prune, so Postgres plans and locks every partition and every partition index: about 200 relations per read with 19 partitions. A backend has 16 fast-path lock slots, and every lock past those goes through the shared lock table. On a production-shaped load rig at pool 200, active backends spent 36% of their time waiting on `LWLock:LockManager` and 1.6% on disk. **What changes.** `query_events_on` (REQ history, bridge `/query`, mentions, rosters, notices) and `get_channel_window_on` (channel windows) now read in up to two passes on the same connection, through one helper (`store/recent_pass.rs`): 1. **Recent pass:** the same read plus `created_at >= <floor>`, where the floor is the start of the previous UTC month. When it returns a full page, that page is the answer, because every row at or above the floor sorts before every row below it. 2. **Older pass,** only when the recent pass comes up short: `created_at < <floor>`, limited to the rows still missing, appended after the recent rows. The ranges don't overlap, so no row is read twice and the page is exact. For channel windows the limit includes the +1 probe row, so `has_more` stays exact. **One snapshot, one deadline.** The two passes run in one `REPEATABLE READ, READ ONLY` transaction (`ReadTransaction`), so both read the same snapshot. Under `READ COMMITTED`, a replacement committing between the passes (a replaceable event, an artifact head) could move the live row from below the floor to above it, and it would land in neither pass. Inside a transaction the passes run in a savepoint instead; the only callers that read inside a transaction are routed replica reads, which are already `REPEATABLE READ`. Each pass after the first runs under whatever is left of the 20 s read deadline, so a split read never gets more time than an unsplit one. Single-statement reads (COUNT, unsplit pages) keep their old path, now through the same type. Channel windows now take the read deadline too. On a routed reader that is the reader's session deadline. On the writer they used to run under the writer's own session timeout (2 minutes), and now get the 20 s read deadline, like every other event read there. The floor is a SQL literal, not a bind parameter, so Postgres prunes when it plans the statement, cached generic plans included. It changes once a month, so the statement cache holds one statement per pass and shape. `#p` reads put the same bound on `event_mentions.event_created_at`. These reads are not split: - Reads whose own bounds already prune (`since` at or after the floor) or fall entirely before the floor (`until` before it). - `OFFSET` reads, since the offset spans both sides of the floor. - Lookups that match a fixed set of rows wherever they sit: `ids`, `#e`, `#d`, artifact `d` tags. - Fetch-everything reads (`max_limit`, the COUNT fallback). - Reads limited to replaceable or addressable kinds. Those are latest-state rows, often old and few, so a recent pass would rarely fill the page. No protocol, schema, index or client change. **Not covered:** COUNT and the per-row aux `#e` reads, neither of which has a page to fill. The recent pass also has no upper bound, because rows dated in the future must stay visible. So partitions ahead of the current month (the manager keeps 6) and the catch-all are still planned. ## Evidence A production-shaped load rig (6,600 desktop clients connecting cold at 100/s, against a 19-partition `events` table), comparing `main` with this branch. Each row is one arm, so expect run-to-run noise of about ±10%. | Arm | Postgres exec time (top statements) | Errors | Relation locks outside fast path (avg) | |---|---|---|---| | Pool 200 + channel-first index, `main` | 13,217 s | 4.75% | 9,481 | | Pool 200 + channel-first index, this PR | 10,931 s | 3.68% | 4,988 | | Default pool, no extra index, `main` | 6,705 s | 5.96% | — | | Default pool, no extra index, this PR | 6,145 s | 5.15% | — | | Default pool + index, `main` (repeat) | 1,070 s | 0.99% | — | | Default pool + index, this PR (repeat) | 1,013 s | 0.93% | — | Pages that fill above the floor stop planning old partitions, and that halves the locks taken outside the fast path. `LWLock:LockManager` stays at about 41% of active backend samples at pool 200, though. The dominant remaining cost there is the cold unread repair: a multi-channel read with no `since` and a page that never fills. That read needs its own fix (a lookback bound or a better plan) and is out of scope here. An earlier version re-ran the whole read after a short recent pass. That doubled the work on non-filling reads: errors went from 4.75% to 11.88% at pool 200. This version reads only the missing older rows. ## Tests - `recent_pass::tests`: floor arithmetic (including the year wrap), the predicate literals, an eligibility table, and an in-memory exactness test that drives the passes at every limit and pins which passes run with which limits. - `e_tag_filter_shape_tests::split_passes_bound_created_at_with_a_literal` / `recent_pass_skips_lookups`: SQL shape (both the `e.` and `m.` sides for `#p`, no bound on unsplit reads or COUNT) and the exclusions, including offset. - `postgres_tests::split_page_reads_match_the_unsplit_read` (buzz-db event): rows on both sides of the floor. A full recent page is answered alone; short pages at limits 4, 5 and 6 add exactly the missing older rows; offset reads across the floor are not split; a cursor below the floor works. It reads `pg_prepared_statements` to show which passes ran. With another session locking a partition only the older pass plans (and a short `lock_timeout`), a full recent page still answers and a short one returns `Err` rather than a partial page. - `postgres_tests::channel_window_split_matches_the_unsplit_read` (buzz-db thread): the same checks for channel windows, including `has_more` when the recent pass has no probe row, and an `Err` when the older pass fails. - `split_read_passes_share_one_snapshot` (event, both deadline modes) and `channel_window_passes_share_one_snapshot` (thread): another session holds a lock that only the older pass needs, so the read stops between its passes. A replacement then commits that deletes a row below the floor and makes its replacement live above it. The page must still be the first pass's snapshot: the replaced row is present and its replacement is absent. - `read_statements_share_one_deadline`: per-read, session and inside-a-transaction reads. The second statement runs under less than the full deadline, and finishing the read restores the connection's own timeout. - `channel_window_passes_share_one_deadline` (thread): through the production writer entry. The recent pass waits 8 s on a lock and the older pass is blocked for good. The window must be cancelled within the one shared deadline, not a full deadline after its older pass starts. - `channel_scoped_mention_reads_share_one_prepared_statement` now expects one recent and one older statement, shared by every channel count. Mutation-checked. Each of these mutants fails a new Postgres test: dropping the older pass, giving the older pass the full limit, removing the split, splitting offset reads, dropping the window's probe row from the split limit, removing the window's split, swallowing an older-pass error in either path, opening the split read `READ COMMITTED` (events and windows), opening it with a plain `BEGIN`, giving every pass the full deadline, and running a window with no read deadline. ## What got simpler Nothing got simpler overall. This adds one small module (`store/recent_pass.rs`) shared by both read paths. The single-statement deadline helper is now a thin wrapper over `ReadTransaction`, so the begin, savepoint and tighten logic lives in one place. The alternative, a `since` in every client plus a new index per read shape, means more code in more places and does nothing for app versions already shipped. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 16:03:15 -05:00
Luke Tornquist	9b4b73eb08	fix(relay): bound mesh dials and keep failing pods out of the mesh (#8752) ... ## Problem A rolling deploy can stall relay startup for 30s per dead mesh peer. - A pod publishes its mesh ready record during `mesh_boot` and clears it only on graceful shutdown. If a later fatal startup check kills the pod, the record lingers until it expires. - Dialing that address never gets an answer, so `MeshEndpoint::connect` waits out QUIC's default 30s idle timeout. `reconcile_once` dials candidates one at a time, so every dead peer adds another 30s, both in the boot-time `mesh_initial_reconcile` step and in every later reconcile pass. - The fatal check that killed those pods was the git conformance probe. A transient S3 `500 InternalError` on one racer was treated as a conformance failure. ## Changes 1. **Bounded, concurrent mesh dials** (`buzz-relay-mesh/src/runtime.rs`). Each connect attempt is wrapped in `DIAL_TIMEOUT` (2s; in-cluster handshakes take milliseconds), and a reconcile pass dials its candidates concurrently. A pass with dead peers now costs about one dial timeout instead of 30s per peer. The candidate set is bounded by the membership table. 2. **Conformance probe before mesh boot** (`buzz-relay/src/main.rs`). This only moves an existing block, so a pod that fails the probe exits before it publishes a ready record peers would dial. 3. **5xx racers are unknown outcomes** (`buzz-relay/src/api/git/store.rs`). The race phases already drop socket/send failures from the observer set as "outcome unknown". A 5xx is the backend declining to classify the write, so it now joins them through one shared `racer_outcome_unknown` helper. 4xx answers, including 412, still count as evidence. A backend that 5xxs every racer still fails closed on the two-observer floor. Because a dropped racer's write may still have landed, every `if_match_race` round now reads the pointer back and requires the winner's exact body and ETag. Otherwise a backend that commits a stale write and then answers 5xx would pass in the last round. §Conformance steps 2–3 in `docs/git-on-object-storage.md` now state this drop-and-floor rule. ## What got simpler The two duplicated drop-classification match arms in the probe now share one helper. The probe/mesh fix is a reorder of existing startup steps, not a new mechanism or rollback path. ## Tests - `runtime::tests::reconcile_pass_bounds_dials_to_unresponsive_peers`: three black-hole peers (bound UDP sockets that never read) plus one live peer. The pass must finish within `2 × DIAL_TIMEOUT` (a serial pass would take at least `3 × DIAL_TIMEOUT`, and without the timeout about 30s), the live peer must connect, and each dead peer must end `Disconnected`. - `store::tests::probe_drops_5xx_racers_but_keeps_4xx_as_evidence`: pins the classification boundary (5xx and I/O are unknown; 400/403/412/499 are evidence). - `store::fault_probe::*`: runs `run_conformance_probe` against an in-process S3 fake that injects racer faults at the HTTP boundary, so responses pass through rust-s3's `fail-on-err` mapping and retry. One 5xx racer in either race phase is dropped and admitted (`transport_drops == 1`). Too few classified observers fail closed: one observer in either phase, and all-5xx in `if_none_match_race`. An unexpected 4xx racer fails closed in both phases. A racer whose stale write lands and is then answered 5xx fails closed on the post-round readback (`dropped_racer_whose_stale_write_lands_fails_closed`). A clean fake passes with zero drops. - CI lane: no `api::store` test ran in any CI lane before, including the existing ones. The `just test-unit` nextest filter and the `scripts/run-tests.sh` fallback now select `api::store::tests` and `::fault_probe`. The live-MinIO `::probe` module stays opt-in. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 15:56:07 -05:00
Tom Brow	97ede701c0	fix(mobile): send hosted relay assertion from native media and tombstone paths (#9069) ... Follow-up to https://github.com/block/buzz/pull/9056, from an audit for other places that miss the hosted relay assertion. Hosted and organization relays require `Nostr-Federated-Identity` on every non-exempt HTTP request and WebSocket upgrade. The Dart `HostedRelayHttpClient` adds it, but anything that downloads or connects outside that client has to add it itself. The notification service extension and the iOS emoji picker both missed it, and these places do too: - **Voice notes** (`just_audio`, mainly Android) and **video preview first frames** fetch media natively with only the Blossom `Authorization` header. Both now use `nativeMediaHeadersProvider` from https://github.com/block/buzz/pull/9056. - **Push-lease tombstones**: removing a community, and the durable revocation outbox, open a one-off relay socket with no headers. A hosted relay refuses the upgrade, so the lease is never revoked and the outbox keeps retrying. Both now ask `HostedRelayAuth` for that community's own assertion, because the community being removed may not be the active session. - **Notification avatars** (iOS notification service extension): relay-hosted sender avatars were fetched with Blossom auth only. The loader now takes the same `BuzzRelayAssertionIssuer` the resolver uses for `/query`. Signed requests already refuse redirects. The iOS Me-chat agent picker also fetched avatars with no headers, but https://github.com/block/buzz/pull/9079 removed that native screen, so this PR no longer touches it. ### Validation - Added tests, each checked to fail when the code it guards is reverted: - A queued push-lease revocation and a community-removal tombstone both send `Nostr-Federated-Identity` on the WebSocket upgrade, as background requests. Runs against a local relay socket that handles NIP-42 auth. - A notification avatar on an organization relay is downloaded with the issued assertion (BuzzPushKit). - Issuing for a second community keeps the first one's cached assertion. A background request records no failure, never backs up the key, and its failure never reaches the active community's reconnect waiting on it. - Background issuance never binds an unbound account to its key, bypasses the shared assertion cache (each attempt gets a fresh assertion, and foreground use never inherits one that skipped the key backup), and its failure is never handed to a foreground caller waiting on it. When it can't get an assertion it sends none, so a non-enforcing relay still accepts the tombstone. - A notification avatar whose organization session is replaced mid-download is refused and dropped from the cache (BuzzPushKit). - Voice notes and posterless video previews renew an expired assertion before the native fetch, and fall back to cached headers if renewal fails. A voice-note attachment journey binds the production provider. - Cached reads never present a signed-out or replaced account's assertion. - `flutter analyze` is clean. `swift test` passes for the BuzzPushKit avatar loader tests, and `flutter build ios --simulator --debug` succeeds. - Full mobile suite (`just mobile-check` and `just mobile-test`) passes at 2c84940832. - Automated full-app run on an iOS simulator (debug build of this PR before it was rebuilt on main, signed in to the internal Builderlab community with two more communities saved). Passed: launch and channel list, image attachments and avatars in a channel, New chat sheet kept open for about 80 seconds before the agent list was opened (all agent avatars loaded, so headers are requested per download), dismiss and reopen, background and resume, switching to a second community (its icon and avatars loaded), and removing a community. The native New chat sheet is unreachable in the normal flow on this branch (a new chat opens the Flutter draft page; the native sheet only opens when resuming an unfinished chat), so that check ran with a temporary local bypass that was reverted. Limits: none of these communities requires the hosted relay assertion in this build, so the run shows the Swift-to-Dart header bridge, the per-relay cache and the tombstone path cause no regression, not that the assertion is sent or accepted. No voice note, video preview or deleted message was reachable. Screenshots are omitted because they show internal channel content. - Limitations: no end-to-end run against a hosted relay. Simulator sign-in to a hosted relay needs SSO, so a hosted-relay run needs a device build. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Tom Brow <tomb@squareup.com> Co-authored-by: Tom Brow <tomb@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: studio <dc2fc481343672df31a689840c09299972e704dac592e0da6f18486646a7ee3e@buzz.block.builderlab.xyz>	2026-11-05 12:32:18 -08:00
Matt Dickenson	120d75d976	Expose strict code-entry pairing and live authorization bindings (#9061) ... Native clients using Rust pairing can now receive the existing `desktop-code-v1` flow: accept its challenge, submit a user-entered six-digit code, handle correlated rejections, and require successful code submission before accepting the source proof. A failed durable import can explicitly acknowledge failure. Adds a public, live-session channel binding (session ID, ordered ephemeral peers and transcript commitment) so applications can authorize the exact exchange using each device's independently authenticated account session. The binding does not itself authorize secret export. Legacy APIs and the wire format remain compatible. Validation: 79 pairing unit tests and seven public API journeys pass, including wrong-code retries, five-attempt lockout, replay/peer rejection, failed import, and identical live bindings on both devices. Focused `cargo clippy -p buzz-core --all-targets -- -D warnings` and formatting pass. Repository formatting/mobile checks and desktop, web, and Flutter build/test lanes pass after installing the pinned Flutter dependencies. Three initial normal push-hook attempts stopped at different unrelated ACP deadlines. The final normal push passed all applicable hooks, including desktop Tauri Clippy/tests and the full Rust unit lane, with existing hook lanes serialized and four Rust test threads. No checks were disabled and no ACP source was changed. Independent review found no blocker in this API delta. App and human acceptance remain outstanding; this PR is a draft. --------- Signed-off-by: Matt Dickenson <mdickenson@squareup.com>	2026-11-05 20:18:14 +00:00
Tom Brow	6793cbd25b	fix(mobile): focus the search field when search opens (#9120) ... Opening search from Home now focuses the search field, so the keyboard comes up right away and you can start typing without tapping the field first. This applies when `SearchPage` opens as its own route; the embedded variant is unchanged. Closing with X and reopening works as before (the previous query is kept and the field is focused again). ### Validation - Extended the existing open-and-close journey in `search_page_test.dart`: it now opens search without tapping the field and asserts the field has focus and the keyboard is shown. The step fails with the production change reverted. - Full mobile test suite and checks passed in the pre-push hook. - Automated run on an iOS 27 simulator (iPhone 17 Pro) signed in to a real community: opened search from Home (keyboard up, caret in the field), typed a query and got channel and message results, closed with X, reopened (keyboard up again). The recording contains internal community content, so it was shared internally rather than here. - Human testing waived under the author's standing exception; the automated simulator run above covers it. buzz-review-completed 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Tom Brow <tomb@squareup.com> Co-authored-by: Tom Brow <tomb@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 11:49:48 -08:00
Tom Brow	9aa70124ba	perf(mobile): validate each verified profile once, not on every publish (#9101) ... Opening or resuming a big channel spent about 210 ms of UI-thread CPU validating verified profiles, measured in Dart CPU samples on an iPhone 12 mini (Profile build) across two separate channel-resume captures (213 ms and 211 ms in `VerifiedProfilesNotifier._snapshot`, nearly all of it in `parseVerifiedProfile`). Every publish rebuilt the profile map by validating every known head again, and a channel with many members publishes once per subscription batch as each lands, so the work grew with members times batches. Each subject's parse is now kept while its head and the authority stand. Only `checked` changes between publishes (rebatching unchecks and rechecks subjects), so it is applied to the kept parse without validating again. ### Validation - Added `a later batch reuses profiles already parsed`: after a second subject rebatches the first, each head is validated exactly once. It fails when the kept parse is ignored. - `flutter analyze` is clean and the full mobile suite passes. - Same phone and scenario after this change: `parseVerifiedProfile` no longer appears in the resume CPU samples, and main-isolate message handling over the 15 s after resume dropped from 604 ms to 491 ms (one run each). Frame build and layout time on resume is unchanged; that is separate work. - Automated app run on the same phone with this change: Home, search, the large channel and its resume after the reconnect grace period. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Tom Brow <tomb@squareup.com> Co-authored-by: Tom Brow <tomb@squareup.com> Co-authored-by: studio <dc2fc481343672df31a689840c09299972e704dac592e0da6f18486646a7ee3e@buzz.block.builderlab.xyz>	2026-11-05 11:31:38 -08:00
John Matthew Tennant	c848b6dcad	fix(mobile): offer a new identity beside saved keys on unclaimed accounts (#9107) ... ## Summary This fixes a dead end in mobile Builderlab sign-in. A new account (unbound, with no backups) on a device that already had saved identities could only pick one of those identities. Any saved key forced the picker, and creating a key required the device to have none. If the service then definitively rejected binding the chosen key, the app showed "Let's try that again" with no way forward: the chosen key stayed in the account's slot, so every retry and every launch re-sent the same rejected bind. For an unclaimed account, the picker now also offers **Create a new identity**: - Creation writes only this account's empty identity slot, so every saved and device key is kept. - Creation is offered only when every current device key was among the choices shown, so a new identity never silently passes over a key the user didn't see. - Bound accounts, accounts with backups, pairing and pending import never offer it. When binding a chosen key is definitively rejected, the blocked screen offers **Choose a different identity**: - It appears only for a 400, 404, 409 or 422 from `/verify` in the current attempt, for the current account and pubkey, and never during pairing, whose transferred key is the only choice. Evidence from an earlier attempt is cleared before each bind. A challenge rejection, a 403, a 503 or any other ambiguous failure offers nothing. - Before releasing the slot, it rechecks the account: still unbound (`/current` and status), no backups, the same subject, and a slot that still holds that pubkey without a receipt. - It releases the slot only when another durable copy of the same key exists (the device inventory or another account slot), then verifies the slot is empty. No key is deleted: only the account's pointer to the rejected choice is cleared. A failed or refused release keeps the slot and the choice, so it can be retried in place or after a restart. - After release, the picker runs again and offers the saved keys plus creation. The recovery code lives in `hosted_flow_identity_recovery.dart`, a part of `hosted_flow.dart`, to stay within the mobile file-size limit. ### Related issue None found. ### Testing **Physical iPhone, a signed release build of `1fb633d55` installed over the existing app (keychain and saved identities kept):** a saved identity was rejected, then Choose a different identity, the picker, Create a new identity and Join your community. Joining then completed. This head differs from `1fb633d55` only in the guards for leaving, pairing and a failed or refused release listed below, which this journey does not reach, and in no UI code. The status of that phone's original rejection was not captured, so it is not confirmed to be one of the rejections above. The 409 case is covered by tests. **Tests (red on `main`, green here):** - Identity service and store: creation beside offered keys; refusal for unseen or partly offered device keys; release when another copy exists; refusal for a sole copy, a binding, a backup, a receipt, another account or an unusable recheck; failed delete and retry; a cancelled attempt. - Onboarding journeys: create beside saved keys; select, rejected bind, restart, choose a different identity, create and bind; ambiguous 503 and 403 failures; challenge rejection and stale evidence. - Release boundaries: an account switch at `/current`, at status, or between the recheck and the delete; cancelling between the recheck and the delete; leaving the picker before a late creation; an unreadable device inventory; leaving while a release is in flight; a failed delete retried in place and after a restart; a release refused because the inventory is unreadable, then retried in place; a rejected paired key. Each keeps every key, leaves the new account's slot untouched and leaves a cancelled flow idle, and a retry converges. - Mutating the creation guard, the generation check, the binding recheck, the store's receipt check, the account and cancellation checks before the delete, or the device-inventory error handling fails a test. #### Screenshots Physical iPhone at `1fb633d55` (frames from a screen recording; saved public keys redacted): | Rejected bind | After Choose a different identity | After Create a new identity | | --- | --- | --- | |  |  |  | Widget-test renders of the same screens (the real `HostedFlow` against a mocked account service, with a fixture key; the square at the top left is the test renderer's placeholder for an asset): | Picker on an unclaimed account | Rejected bind | | --- | --- | |  |  | 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: John Tennant <jtennant@squareup.com> Co-authored-by: Xander <ff30c859b96bd5233ffaf5d071200458f1274bd0dab03589214cf1055198491d@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 14:23:59 -05:00
Brad Seiler	8a113ed99e	fix(desktop): show upgrade guidance (BUZZ-517) (#9033) ... ## Summary Old Buzz desktop clients now check a community’s public requirement before each connection. An explicit `requires_nip_fi: true` shows an upgrade screen with a fixed https://buzz.xyz download action and manual fallback. Missing or false requirements, and failed discovery, preserve normal connection behavior. Retry, community navigation and address editing preserve keys and saved communities. Changing an address cancels the old onboarding transaction and fences pending profile checks. Native discovery sends no credentials, follows no redirects, and limits requests to five seconds and 256 KiB. Wire fixtures are exported from main `6b8a72e835` with reproducible provenance. ### Related issue [BUZZ-517](https://linear.app/squareup/issue/BUZZ-517/v0-clients-show-an-upgrade-screen-for-communities-requiring-nip-fi). Related: [#8984](https://github.com/block/buzz/pull/8984) contains mobile parser preparation; [#8920](https://github.com/block/buzz/pull/8920) repairs relay rejection messages. ### Rollout There is no client flag. The upgrade screen appears only when the relay advertises `requires_nip_fi: true`. Public auto-update delivery requires a desktop release containing this change and promotion to the [OSS auto-update feed](aa749a862e/RELEASING.md (promote-an-oss-desktop-release-to-auto-update)). In Shadow mode, requiring a community activates the screen; making it optional restores normal connections. Global Enforce requires every community regardless of that setting. Rollback must restore an effective non-required policy. The optional and required outcomes are demonstrated by Brad’s two manual cases below. ### Testing At `71689b4708`, Honor’s isolated browser journeys passed 3/3 with zero retries and verified served assets, including the fixed `https://buzz.xyz` opener and manual fallback. McKeon independently built and passed the same three journeys at the previous head `8caa25260b` with the repository command: `CI=1 pnpm --dir desktop build:e2e`, then `CI=1 pnpm --dir desktop exec playwright test relay-entry-guidance.spec.ts --project=smoke`. Native IPC was mocked. Brad [tested the running pre-rebase app locally](https://github.com/block/buzz/pull/9033#issuecomment-6662960875): an optional bb-public community in Shadow connected without a message; required bb-block staging showed the upgrade message. These results apply to that running app, not the rebased head. Before:  After:  Generated with Codex --------- Signed-off-by: Brad Seiler <seiler@squareup.com> Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz>	2026-11-05 19:22:01 +00:00
Tom Brow	dce37f74d7	fix(mobile): skip the read-state snapshot when nothing needs publishing (#9093) ... Every mobile resume re-syncs read state and then schedules a read-state publish, whether or not anything changed. Each publish starts by downloading the full read-state snapshot (about 1 MB compressed on a heavy account) and only then finds nothing to send. Together with the re-sync's own snapshot, that is two full downloads per resume, and a third when a publish from before backgrounding retries. The publish now checks first, using the same packing it would publish, and returns before the snapshot when nothing is pending and no slot differs from what was last published. When something did change it behaves exactly as before, snapshot included. Today the snapshot fails before downloading on gzip-serving relays, so this only pays off once [#9089 read-state gzip fix](https://github.com/block/buzz/pull/9089) lands; without this PR, [#9089 read-state gzip fix](https://github.com/block/buzz/pull/9089) roughly doubles resume download on heavy accounts. It has no merge-order dependency. ### Validation - Added test: `a publish with nothing new skips the snapshot`. It publishes a change (one snapshot), publishes again with nothing new (no snapshot, no event), then publishes another change (snapshot and an event carrying both marks). It fails without the early return. - Existing read-state suite passes. - iPhone 12 mini, Profile build, Home backgrounded 12 s and resumed, all socket bytes received in the 15 s after resume: | Build | Download per resume | Snapshot fetches | | --- | --- | --- | | main (snapshot broken) | 3.07 MB | 0 completed | | [#9089 read-state gzip fix](https://github.com/block/buzz/pull/9089) | 6.85 MB | 3 | | [#9089 read-state gzip fix](https://github.com/block/buzz/pull/9089) + [#9090 no read-state replay](https://github.com/block/buzz/pull/9090) | 6.08 MB | 3 | | [#9089 read-state gzip fix](https://github.com/block/buzz/pull/9089) + [#9090 no read-state replay](https://github.com/block/buzz/pull/9090) + this | 2.71 MB | 1 | These are single runs on one account with 28 read-state slots, so treat the numbers as indicative. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Tom Brow <tomb@squareup.com> Co-authored-by: Tom Brow <tomb@squareup.com> Co-authored-by: studio <dc2fc481343672df31a689840c09299972e704dac592e0da6f18486646a7ee3e@buzz.block.builderlab.xyz>	2026-11-05 11:21:33 -08:00
Tom Brow	cf8fbbba55	fix(mobile): keep verified names on screen in channels with over 1,024 members (#9111) ... In a channel with more members than the verified-name interest limit (1,024), returning to the app made member names flip: a verified name changed to a profile name with a collision suffix (for example "Ankit · ykn9") or to an npub, then back once a refetch landed. Seen on an iPhone 12 mini in a 2,004-member channel; instrumented logging showed the verified-name map dropping from 956 entries to 0 on opening that channel. Tracking a large roster evicts the oldest-tracked subjects, and eviction dropped their records too. On resume, member and profile preloads track the evicted people again, which evicts the ones on screen, so their verified names disappear until the next fetch. Evicted records now stay as unchecked cache (up to 4,096, oldest untracked dropped first) and are rechecked when tracked again. Only tracked subjects accept new records, so the cache never grows from relay traffic. ### Validation - Added `a subject evicted by a large roster keeps its last known name`: it fails when eviction drops the record. - `flutter analyze` is clean. The full mobile suite passes except 10 push and notification-toggle tests that fail identically on unchanged main on this machine. - Same phone, 2,004-member channel, after this change: the verified-name map held steady (978 entries, growing to 1,828) through opening the channel and two resumes after 12 s and 20 s in the background, and the "ankit, lhoward" and "sfoo, Dani" rows never changed in the screen recording. Before, both flipped (to "Ankit · ykn9" and an npub) and back after each resume. - Automated app run on the same phone with this change: Home, search, the large channel, and its resume after the reconnect grace period. This touches the same `_snapshot` line as [#9101](https://github.com/block/buzz/pull/9101), so whichever lands second needs a one-line rebase. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Tom Brow <tomb@squareup.com> Co-authored-by: Tom Brow <tomb@squareup.com> Co-authored-by: studio <dc2fc481343672df31a689840c09299972e704dac592e0da6f18486646a7ee3e@buzz.block.builderlab.xyz>	2026-11-05 11:15:41 -08:00
Tom Brow	244038b495	mobile: reuse channel message rows that did not change (#9115) ... Opening the keyboard in a busy channel janked: the composer's bottom inset changes on every keyboard animation frame, and each change rebuilt the message list, which rebuilt every visible message row. On an iPhone 12 mini that came to 8 frames over 16 ms per keyboard open, with 290 to 360 ms spent in build. The same cascade runs on every page rebuild, such as the dozen or so while a large channel settles after returning to the app. Flutter skips a rebuild only when it gets the identical widget object back, so the message list now keeps each row it built and returns it while nothing the row is built from has changed. Shared inputs (the message list, membership, channel names, send animations, the focus callback, today's date and so on) clear the cache when they change. Each row also compares its neighbors, thread summary, typing and working state, and send-animation and handoff flags. Rows still rebuild through their own provider watches. The composer's focus callback is now stable so the page's rebuilds don't clear the cache. ### Validation - Added `a growing composer reuses the message rows on screen`: it types a four-line message and checks that the visible rows are the same widget objects. It fails without the cache. - `flutter analyze` is clean. The full mobile suite passes except the 10 push and notification-toggle tests that fail identically on unchanged main on this machine. - iPhone 12 mini, Profile build, a 2,000-member channel, one matched run each before and after (the after column is an earlier revision of this change with the same mechanism): | | Before | After | | --- | --- | --- | | Keyboard open, build time | 359 ms, 290 ms | 64 ms, 43 ms | | Keyboard open, frames over 16 ms | 8, 4 | 1, 0 | | Return to the app, build time | 953 ms | 881 ms | | Return to the app, frames over 16 ms | 21 | 18 | Most of what remains on return comes from rows' own watches (for example the agent display-name map is a new object on every agent directory reload), which this change doesn't address. - Automated app run on the same phone with this exact branch: Home, opening the large channel, opening the keyboard twice (58 and 69 ms of build, one frame over 16 ms each), scrolling back through several days of history, and returning to the app after 12 s in the background. Day dividers, reply counts, reactions and system rows rendered correctly throughout. Typing is covered by the widget test rather than the device run, to avoid drafting in a real channel. - Simulator QA of this branch (Debug build, a private test channel): typing a four-line message, sending, scrolling to the top and back, a reaction, an edit, a thread reply, a delete, sending again, and backgrounding and returning. Every row rendered and updated correctly, with no replayed send animations. The jump-to-latest button didn't respond to synthetic simulator taps before or after resume; it's a native iOS view this change doesn't touch. https://github.com/user-attachments/assets/04909261-81fe-4514-98aa-8d7fdfdcace5 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Tom Brow <tomb@squareup.com> Co-authored-by: Tom Brow <tomb@squareup.com> Co-authored-by: studio <dc2fc481343672df31a689840c09299972e704dac592e0da6f18486646a7ee3e@buzz.block.builderlab.xyz>	2026-11-05 11:15:32 -08:00
Tom Brow	3ff4154ca6	fix(mobile): back off push bootstrap retries (#9092) ... When push setup fails, the app retried every 5 seconds forever. On an iPhone 12 mini whose push gateway enrollment is refused (a CDN 403 on the installation challenge), each retry refetched the relay's NIP-11 document over a new TLS connection and logged the whole HTML refusal. That is about 720 attempts and roughly 14 MB of traffic an hour for as long as the app stays open, with no way to succeed. Push registration, lease publication and tombstone retries now back off: each consecutive failure of the same attempt doubles the delay from 5 seconds up to 15 minutes, with jitter between half and all of it so devices that failed together don't retry together. A success (including a renewal), a different attempt, or a channel-grammar retry (which keeps its 5-minute delay) ends the streak. The first retry still comes within 5 seconds, so a brief blip recovers as quickly as before. ### Validation - Added tests: `repeated failures back off exponentially up to a cap` checks every delay's bounds at both jitter extremes over 12 failures, `a success ends the failure streak`, and `a different attempt starts its own backoff`. The first fails when the gate goes back to a fixed delay. - Existing push suite passes. - On the 12 mini before this change: `Push lease bootstrap failed` with the 403 page and a fresh NIP-11 fetch every 5 seconds for the whole session. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Tom Brow <tomb@squareup.com> Co-authored-by: Tom Brow <tomb@squareup.com> Co-authored-by: studio <dc2fc481343672df31a689840c09299972e704dac592e0da6f18486646a7ee3e@buzz.block.builderlab.xyz>	2026-11-05 11:15:22 -08:00
Tom Brow	e34acb0ac5	fix(mobile): keep reply counts on screen while a reconnect catches up (#9100) ... Every time the app comes back from the background, the "N replies" links under messages in an open channel vanish and reappear a moment later. On an iPhone 12 mini in a busy channel this happened on nearly every resume, since the session drops its socket five seconds into the background and the channel reconnects. A reconnect discards the channel window and refetches its newest page. Until that page lands, thread summaries came only from replies already loaded, which in a windowed channel is usually none. The discarded window's summaries now stand, below every fresher source (live summaries, thread queries, overflow floors), until the replacement history lands, which then decides, including dropping a summary the relay no longer reports. ### Validation - Added `reply counts stay on screen while a reconnect catches up`, which fails when the retained summaries are left out. - `flutter analyze` is clean and the full mobile suite passes. - iPhone 12 mini, Profile build, a 2,000-member channel backgrounded 12 s and resumed (same scenario, one run each). Before: the "6 replies" link vanished for about a second and the timeline shifted up, then back. After: the link and the scroll position stay put through the catch-up. - Automated app run on the same phone with this change: Home, search, the large channel and its resume after the reconnect grace period. ### Accepted gap If you had scrolled back past the first page, those older rows still lose their reply counts once the reconnect catch-up finishes, as they do on main today. Their counts return when pagination refetches their page or the channel is reopened. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Tom Brow <tomb@squareup.com> Co-authored-by: Tom Brow <tomb@squareup.com> Co-authored-by: studio <dc2fc481343672df31a689840c09299972e704dac592e0da6f18486646a7ee3e@buzz.block.builderlab.xyz>	2026-11-05 11:14:52 -08:00
Tom Brow	5e36d65c9e	fix(mobile): inflate gzip read-state snapshot responses (#9089) ... On relays behind a CDN that gzips responses (the Builderlab relay does), mobile read state never synced. Every read-state snapshot fetch, and therefore every read-mark publish, failed with `FormatException: Unexpected extension byte`, so read marks made on the phone never reached other devices and each reconnect retried and failed again. The relay query transport deliberately leaves gzip bodies compressed so they can be inflated off the UI isolate (see `RelayHttpQueryClient`), but the snapshot path decoded the raw bytes with `utf8.decode`. It now uses `relayResponseText`, like the other query callers, for both the discovery and the snapshot responses. Heads-up for reviewers: with the snapshot working, a resume also pays for the full snapshot download on top of the existing read-state replay on the live subscription. A follow-up PR removes that replay. ### Validation - Added test: `inflates gzip discovery and snapshot bodies` serves both responses gzipped. Without the fix it fails with the same `Unexpected extension byte` error seen on the device. - Full mobile suite: all pass except the known stale `fullscreen_onboarding_test` case, which fails on main too. - iPhone 12 mini (Profile build, Builderlab community): before, every resume logged `snapshot failed` and `publish failed (journal retained)`. After, the snapshot on resume and the snapshot inside a publish both completed with no errors. - Automated app run on the same phone with this change: Home backgrounded past the reconnect grace period and resumed, then search and a large channel opened. No read-state errors logged. Read marks were not compared across devices. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Tom Brow <tomb@squareup.com> Co-authored-by: Tom Brow <tomb@squareup.com> Co-authored-by: studio <dc2fc481343672df31a689840c09299972e704dac592e0da6f18486646a7ee3e@buzz.block.builderlab.xyz>	2026-11-05 11:14:39 -08:00
Luke Tornquist	50c10dc7b2	perf(relay): cache the community-active check on socket admission (#9109) ... ## What Socket admission (root and audio) and the ephemeral write fence now share one cache of "is this community active?". An entry is served for 1 s after its read checked out a pool connection, and concurrent misses for one community share one read. Archive and deletion drop the entry before they close the community's sockets. ## Why Every new socket runs `is_community_active` before it can authenticate. In a reconnect storm that means thousands of identical reads for one community. A read that times out in the pool refuses the socket (`community active check failed; refusing the socket`), and the client then reconnects and adds more load. On a local cold-start reconnect rig, this refusal count roughly tripled once #9012 freed pool capacity for other reads. It is now one of the top causes of connect failures. ## How - `buzz_db::Db::is_community_active_read_at` returns the answer together with the instant its connection was checked out, following the #9012 `ReadAt` pattern. `is_community_active` now calls it, and the datastore span stays under its existing name. - `community_active::CommunityActiveCache` holds an entry per community and a `SharedReads` gate. An entry is served for `COMMUNITY_ACTIVE_MAX_AGE` (1 s) after its read. Both answers are cached. A cached "inactive" fails safe and lasts at most 1 s after an unarchive. Errors are not cached. - **The archive sweep stays exact.** Admission registers a socket before it checks, so the sweep closes every socket registered before it. A socket that registers after the sweep must see the community as inactive. `apply_community_disconnect` invalidates the entry before the fence and sweep. Invalidation bumps an epoch, and a fill whose read began before the bump is not published. A post-sweep socket therefore reads afresh. It cannot take a pre-archive answer from the cache or from a shared in-flight read. Host invalidations (archive and unarchive writes, local and received) also drop the entry. - **Bound when the notification is missed.** A pod that misses the Redis disconnect admits sockets from a cached "active" for at most 1 s after the archive commits. Lifecycle revalidation then closes them. This matches the 1 s host-routing bound in `docs/community-auth-upgrades.md` ("archive and deletion … cached tenant binding can lag those changes by up to one second"). Deletion keeps its own write fence (`guard_community_write`). - Lifecycle revalidation and persistent ingest still read uncached. ## What got simpler The ephemeral fence's separate `serving_active_cache` is gone. It ran the same SQL predicate through `is_serving_active`, used a 10 s TTL and had no invalidation. The connect path, the WebSocket ephemeral path and HTTP typing now share one cache with one 1 s bound and one invalidation path. That leaves one fewer cache concept, and the ephemeral fence goes from 10 s stale to at most 1 s. ## Tests - `community_active::tests` (unit): - 16 concurrent misses cost one read, and an entry past its max age is re-read. - An invalidation during an in-flight read fences that fill. A waiter that joined the read reads afresh and gets "inactive". - Errors are not cached. - `connection::tests::bootstrap_admits_from_the_cached_community_answer`: with the database unreachable, a cached "active" admits a socket through the real `handle_connection`, and the socket gets its AUTH challenge. - `postgres_tests::cache_misses_behind_a_held_pool_share_one_fresh_read` now also covers this cache. With a one-connection pool held past every max age, eight concurrent admission checks cost one DB read, and the entry lands fresh. - `postgres_tests::archived_community_is_refused_after_invalidation_or_max_age`: after an applied archive, the next check refuses at once. An archive this pod never hears about refuses once the entry passes 1 s. - The ephemeral and typing fence tests now seed the new cache. - Mutation-checked on Blox. Each of these turns a test red: dropping the epoch fence, the max-age filter, the single-flight gate, the connect path's cached read, or the disconnect invalidation; stamping `read_at` before the pool wait; and calling the uncached read on the ephemeral path. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 14:09:14 -05:00
Luke Tornquist	ff15c347c2	fix(db): auto-migrate prebuild-guarded indexes on small databases (#8802) ... ## Problem Migrations `0076`, `0078` and `0081` raise an exception on any table with at least one row unless its index was prebuilt with `CREATE INDEX CONCURRENTLY`. A relay that runs migrations at startup (`BUZZ_AUTO_MIGRATE`, on by default in the Helm chart) and has data in those tables therefore crash-loops on upgrade: ``` migration error: while executing migration 81: error returned from database: prebuild idx_nip_fi_profile_heads_names concurrently; see docs/profile-search.md ``` `0081` (#9019) hits every auto-migrating relay that has published NIP-FI verified profiles. `0076`/`0078` (#8523) hit any relay with artifact data or channel members. Each of these tables is small enough on such relays that a plain index build takes milliseconds. The root cause is `docs/database.md` Rule 6. It told authors that an index on a populated table MUST be built concurrently by an operator ahead of time, so migrations were written to refuse populated tables. With auto-migration that refusal is a crash loop, not a safe stop. ## Fix The migrations are already applied on some databases, so they can't be edited: sqlx would reject the changed checksums. Instead, `run_migrations_locked` builds their missing indexes first. For each entry in `INLINE_INDEX_BUILDS` (`0076`, `0078`, `0081`), it runs migrations through the version before it, then builds any missing index. Each build is a plain `CREATE INDEX` in its own transaction, under the same 1s `lock_timeout` and 5s `statement_timeout` the migrations use. That is the bounded inline build `0049` already does: - **Small populated databases** upgrade the same way empty ones do. - **Large databases** still fail closed. Only a lock or statement timeout (`55P03` / `57014`) gets the "prebuild concurrently" hint, pointing at that migration's own doc. Other errors pass through unchanged. The blocked migration is not recorded. - **Prebuilt indexes** are found with `to_regclass` and skipped before `CREATE INDEX`, so deployments that prebuilt concurrently never request the writer-conflicting lock. - **Invalid prebuilt indexes** are left for the migration to reject, as before. `CONCURRENTLY` is not used in this path. The runner holds the migration advisory lock, and another replica blocked on that lock holds a snapshot that a concurrent build would wait on. ### Rules - `docs/database.md` Rule 6: with `BUZZ_AUTO_MIGRATE` on, every migration must apply its own change on a populated database and must never refuse one because it has rows. A new index is built inline under the migration's budgets. Only deployments that apply migrations out of band (auto-migration off) prebuild concurrently. - Root `AGENTS.md` (step 4) and `migrations/AGENTS.md` repeat the never-refuse rule. - `docs/artifact-sync-deployment.md` and `docs/profile-search.md` describe the startup behavior first and the concurrent prebuild as the out-of-band path. ## Tests - `populated_database_auto_migrates_through_every_migration` (Postgres) migrates to `0075`, seeds artifact rows, and checks four phases: 1. **Lock held:** while a writer lock is held, the `0076` build fails closed. 2. **Index prebuilt:** with the `0076` index prebuilt, the build is skipped. 3. **Partial `0078`:** a blocked `0078` build keeps the builds before it committed. 4. **Verified profile seeded:** with a verified profile head inserted before `0081`, a final startup succeeds. It asserts that the applied versions equal every embedded migration and that all six indexes are valid, ready and live. - `frozen_prebuild_guards_have_inline_builds` (no database) scans `migrations/`. It fails if any migration refuses a populated table (`RAISE EXCEPTION 'prebuild …`) without an inline build, and if a new guard is added. - `migration_execution_cannot_bypass_schema_destruction_lock` allows exactly one production `run_to`, inside `run_migrations_locked`. ## What got simpler - Auto-migrating relays no longer need a manual, operator-only prebuild step for small deployments. That restores the chart README's contract that `helm upgrade` is the whole upgrade procedure. - Rule 6 no longer steers authors toward refusing populated tables, and a test enforces that. - The change adds one table-driven step for three frozen migrations and no new configuration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 13:52:06 -05:00
Blake McAnally	ab399231c2	docs: mark desktop/ as the deprecated Buzz 0.X app (#8840) ... ## Summary `desktop/` is the Buzz 0.X desktop app. Buzz 1.0 lives in [block/buzz-app](https://github.com/block/buzz-app), and its README already says so. Nothing in this repo says it, so contributors and agents keep opening feature PRs against `desktop/`. This PR adds the rule in three places: - **`AGENTS.md`**: a notice at the top of "Desktop App" and a note in the repo layout. Agents read this file first. The notice also says that CLI PRs do not update the 0.X agent guidance in `nest_skill.md`. - **`CONTRIBUTING.md`**: a new item in "PRs We're Unlikely to Merge". - **`desktop/README.md`**: a notice at the top. The rule: build desktop features in block/buzz-app, or as a Buzz plugin. Change `desktop/` here only to fix a 0.X bug, or to keep it compiling after a shared crate changes. ## Testing - This PR changes only documentation. I checked the diff and the two block/buzz-app links. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Blake McAnally <bmcanally@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 12:11:22 -06:00
Luke Tornquist	bc99726f5c	docs(agents): add Review-Proven Rule 9, design client paths for the whole fleet (#9106) ... ## What Adds Review-Proven Rule 9 to `AGENTS.md`: **design every client-driven path for the whole fleet running it at once.** Before an author adds or changes a connect, reconnect, renewal, poll, or catch-up path, they: - count the relay reads it costs per client, and keep reconnect to a delta from the client's cursor - jitter its start and back off with a cap - prefer an event-driven refresh with a long backstop over a short poll - on the relay, check every cache on a connect or auth path at a 0% hit rate, with single-flight misses that survive a saturated pool ## Why Relay load tests keep finding the same class of defect. A path is cheap for one client, but when a rollout drain, node eviction, relay restart, or network blip makes every client run it in the same few seconds, it collapses the relay. Each bullet cites a merged PR that fixed an instance: #4542, #8846, #9012, block/buzz-app#2215 and #2308. Unlike rules 1–8, this one comes from load tests rather than review mining, and the rule text says so. The section intro and the Reviewing bullet now say the rules list includes non-review sources. ## What got simpler Nothing in code. This is one rule added to the existing list. It turns a recurring load-test finding into a check made at planning and review time, so it doesn't keep resurfacing as a late load-test result. ## Validation Docs-only. I checked the diff, the Markdown rendering, and every cited PR number and title. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> mobile-v0.28.0-rc.3	2026-11-05 11:57:01 -05:00
Luke Tornquist	aa749a862e	perf(db): serve single-root thread reply reads from thread_metadata (#9075) ... ## Summary A plain thread-reply read, `{kinds:[9], #h:[channel], #e:[root], limit:100}`, goes through the generic `query_events` path. That path gathers every event tagging the root into a `MATERIALIZED` CTE, then sorts it to keep one page. On a root with 500k replies, each page costs about 514k buffer reads plus a temp-file spill. This PR serves single-root reply reads from `thread_metadata` instead. It walks `idx_thread_metadata_window` newest-first, fetches each reply by primary key, and applies every filter predicate before `LIMIT`. Page cost scales with the page size, not with the thread. ## What changed - **Routing (`crates/buzz-db/src/store/event.rs`).** A `query_events` read takes the new route when it has: - one channel, - one lowercase-hex `#e` root, - kinds limited to 9 and/or 40002, - no `#p` filter (any number of values; the relay marks such filters `exact_match`), - no `authors`, `pubkey` or `ids` selector. The route tests those per reply, so an author with no reply in a hot thread would walk the whole thread; the generic read narrows by author first. Every other `#e` read stays on the fenced path from #7854. - **Root check before routing.** `#e` must name a thread root. A cheap probe looks up the target in `thread_metadata` first. If the target is itself a reply, the read takes the generic path, which returns its direct children the NIP-01 way. A reply counts if its row has a root, or if it is a rootless legacy stub whose own NIP-10 markers resolve to a reply. - **COUNT unchanged.** COUNT post-filter fallbacks (`count.rs`, `bridge.rs`) set the new `EventQuery::exact_match` flag through `prepare_count_fallback`. Routing is skipped for them, so COUNT still counts every event that matches under NIP-01. - **Same predicates on both paths.** The predicate block moved out of `build_query_events_sql` into `push_event_filters`, which both paths call. Filters on the route therefore match the generic path, including the `#e` tag match, `since`/`until`/`before_id` and deletion. - **Unnamed statements.** Routed reads run as unnamed statements, so Postgres plans each one with its root and never caches a generic plan. The benchmark below shows the routed SQL is also fast under a forced generic plan, so this is a second safeguard. - **Narrowed semantics, documented.** Messages that name the root only in a bare `e` tag, a `mention` tag or a lone `root` marker are top-level under NIP-10. They have no `thread_metadata` row, so the routed read omits them. Live subscriptions still deliver them. `docs/nips/NIP-CW.md` gains a "Plain Thread-reply Filters" section describing this. A client that needs those messages can add another kind to the filter, which sends the read down the generic path. ## Benchmark (synthetic, Postgres 16, 500k-reply thread) Median of 5, newest page of 100. "Rare" is a root with 3 replies. "Generic" forces `plan_cache_mode = force_generic_plan`. | | main | this PR | |---|---|---| | hot root, custom plan | 2802 ms (513k buffers, 46,750 temp blocks written) | 1.6 ms (407 buffers) | | hot root, generic plan | 1779 ms | 0.6 ms | | rare root, custom plan | 1.2 ms | 0.8 ms | | rare root, generic plan | 61 ms | 0.08 ms | Author-filtered reads (`authors:[x], limit:1`) on the hot root: an author with no reply there takes 1.4 ms on the generic path but 1.7 s on the route, so author and id selectors stay generic. On both roots, the PR's SQL returns the same rows as main. ## Tests - `e_tag_filter_shape_tests::thread_reply_route_admits_only_single_root_message_reads` checks which reads take the route. Two roots, mixed kinds, no kinds, no channel, an uppercase root, a `#p` join, an `exact_match` read, and `authors`/`pubkey`/`ids` selectors all stay generic. The routed SQL is checked for the `thread_metadata` walk, the `#e` match and the window ordering, with no `MATERIALIZED`. - `postgres_tests::thread_reply_reads_use_thread_metadata` (Postgres lane) covers: - mixed 9/40002 replies, a nested reply and a soft-deleted reply, - a bare-`e` message that the route omits and the generic read (kinds `[9,40002,7]`) still returns, - a lone-`root` message, which the route omits too, - an `exact_match` read, which returns the same rows as the generic read, - a nested-parent target, which gets its direct child the generic way, - root stubs and a legacy parent with no row of its own, checked through the probe, - keyset paging, - no named prepared statement left behind after repeated runs. - `handlers::req::tests::postgres_tests::multi_p_thread_filter_keeps_nip01_results` (Postgres lane) runs a two-value `#p` thread filter through the real filter-to-query adapter and checks that a bare reference to the root still comes back. - `handlers::req::tests::count_fallback_fetches_one_extra_candidate` checks that COUNT fallbacks set `exact_match`. It is DB-free and was missing from the `buzz-relay --lib` allowlist, so this PR adds it to `just test-unit`. - Mutation-checked: the Postgres test fails if any of these is broken: - routing disabled, - persistent statements, - the nested-target probe dropped, - stubs treated as all replies or as no replies, - COUNT allowed to route, - the deleted filter dropped. - Blox gate: `cargo fmt`, clippy (`-p buzz-db -p buzz-relay --all-targets`), buzz-db lib (nextest and plain parallel `cargo test`), buzz-relay lib, and the buzz-db and buzz-relay Postgres lanes. One unrelated artifact-sync scale test timed out under load; it passes alone and in CI. ## What got simpler - One predicate builder replaces the inline block, so the two read shapes cannot drift apart. - One targeted route replaces the planned extra fence variant (a keys-only `MATERIALIZED` CTE). That variant was dropped because it reduced the spill but still scanned every reply. ## Identified debt (not in this PR) - The bridge `page` extension (`crates/buzz-relay/src/api/bridge.rs`, `extract_page_offset`, built for the kind:0 directory listing) still maps to SQL `OFFSET`. The routed read honours it too, for parity. Deep pages there should move to keyset (`until` + `before_id`). buzz-review-completed 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 11:14:12 -05:00