901 Commits

Author	SHA1	Message	Date
Luke Tornquist	1bb44ae9ef	fix(relay): close a WS search whose hit hydration fails instead of sending EOSE (#9148)	2026-11-05 20:23:46 -05:00
Luke Tornquist	8ae35c5c92	perf(db): stop the events recent pass at a ceiling for regular-kind reads (#9157)	2026-11-05 20:22:57 -05:00
Brad Seiler	bb3c5f7175	feat(relay): project compact viewer memberships (BUZZ-615) (#9118) ... ## Summary Cold login downloads other people's full rosters just to find the viewer's channels. This adds a compact projection from canonical memberships through the existing Nostr `/query` bridge. One primary SQL statement reads the viewer's rows and channel state, then the relay signs that exact result. Merged spec PR #9117 owns both relay-only kind registrations. Self mode (`membership_snapshot`, kind 39008) requires the viewer's own NIP-98 key. Public mode (`membership_public`, kind 39009) lets an admitted requester look up any subject's listed, open forum/stream/workflow memberships. It excludes private/unlisted channels, DMs, system channels and deleted or removed memberships, including shared private channels. Both modes require payload-bound NIP-98 and return signed request-bound envelopes; overflow or backend errors fail the whole request. Legacy kind-39002 reads and signatures stay intact. No schema migration or persisted per-user list is added. ### Related issue [BUZZ-615](https://linear.app/squareup/issue/BUZZ-615). Implements the compact discovery contract merged in #9117. Related: [#8898](https://github.com/block/buzz/pull/8898), [#9060](https://github.com/block/buzz/pull/9060). Client lazy-loading follows separately; this does not include [buzz-app#2627](https://github.com/block/buzz-app/pull/2627). ### Testing Current head `5c0c35400a` rebases the five implementation patches unchanged onto main `c39ee507`, then repairs the removed Redis test API. McKeon independently verified the rebase and reviewed that repair. Full core/database/relay package tests, PostgreSQL membership tests (9/9), Clippy and every mandatory publication hook pass at this head. Old-head CI run 42132655155 failed six jobs on that same removed API; four summary checks propagated those failures. New-head hosted CI is pending. Main advanced to `e3a0f3d59d` during publication; a local merge-tree check found no conflicts. Historical runtime evidence below remains attributed to its tested heads. Prior self-mode validation at `6d2ffceeee` (historical evidence, not validation of the public-mode change): - `just ci`, pre-push hooks and all 2,221 core/DB/relay package tests passed. - All six new Postgres protocol tests passed. All five guard mutations failed assertions: mandatory auth, flag gate, single-statement cut, overflow probe and tenant predicate. A/B are older runs against production files unchanged from `9784df51`; C/D/E were rerun on final SQL/fixtures. The earlier surviving tenant mutation is superseded by the final duplicate-ID witness. - The full isolated Postgres run finished 1,288/1,294 green; six existing push/media/FI/startup cases failed. All six passed when rerun serially after the concurrent local gates stopped. The failed full run is retained and is not represented as a clean full-suite pass. The preceding correction `1c47dfc9` had passed all 1,294 tests. - `just test` passed its unit steps but integration setup was blocked by Docker Desktop organization sign-in enforcement. The waiting wrapper was stopped with SIGTERM; this is not a complete pass. - A real local relay rebuilt at this head returned 67,896 bytes for 468 memberships in 33 ms unthrottled. Signatures, request/viewer/tenant binding, content identity, wrong-viewer refusal, removal and role changes verified. The scale fixture adds five large rosters and 20,000 unrelated channels. Compact bytes stay unchanged as other members grow; EXPLAIN checks 468 indexed viewer rows and bounded channel-ID lookups. A same-UUID cross-tenant fixture verifies fields and duplicate rejection independently of row order. Durable local evidence: `WORK_LOGS/BUZZ_615_20261105/MANIFEST.md` and `MANIFEST.json` map commands/results to commits or verified mutant-file hashes and index the exported logs. Concrete human-test steps and the signed-request helper are included. Human testing remains pending; ready state follows the author’s explicit request. Native/browser 140/205 KB/s acceptance belongs to the client PRs. Public-mode validation at `c6e269170e`: McKeon independently ran all nine isolated PostgreSQL membership tests and five mutations. Every mutation failed its expected assertion: public selection predicate, subject tag, public routing, identity domain and self serialization order. The full core/DB/relay package run, all nine isolated PostgreSQL membership tests and the relay build passed at c6. A real signed HTTP workflow verified signatures, bindings, identity hashes, all visibility exclusions, archived guest roles, requester-independent content identity and malformed-request refusals. After a channel became private, the public list shrank from three entries to two; a role change appeared as admin. The local test relay was stopped afterward. The restacked child `8c56f97d6409` has the same code, tests and configuration as c6; only the spec differs. A new full core/DB/relay package run passed at that exact child head. The parent `17a481a89716` separately passed the full core package check. The first child attempt used the wrong local database and failed five existing tests on a missing table; its log is retained separately from the successful configured run. These checks do not attest human testing. ### Rollout and rollback This PR does not deploy or enable the feature. Both `BUZZ_MEMBERSHIP_SNAPSHOT_ENABLED` and `BUZZ_MEMBERSHIP_PUBLIC_ENABLED` default to false and independently gate advertisement and execution. An authorized operator sets them in the relay Helm environment managed by `squareup/block-coder-tf-stacks`. First deploy the image to every staging pod with both flags off; then enable each capability separately for staging validation. Enabling either mode does not require or enable the other. Repeat that dark-then-enable order for bb-block after staging passes. Opt-in client rollout follows the relay contract and the throttled desktop/browser workflow. Watch `buzz_membership_snapshot_query_seconds`, `buzz_membership_snapshot_encode_sign_seconds`, `buzz_membership_snapshot_response_bytes`, split by `mode=membership_snapshot` or `mode=membership_public`, plus `/query` 413/503 responses and primary-pool wait time for the Authorization operation. Confirm zero unexpected full-roster discovery downloads on supported clients and correct roles/membership against the canonical cut. Abort enablement for any binding/signature mismatch, access leakage, incorrect complete result, stale-response regrant, unexpected overflow for the 468-channel account, sustained 503s or a material regression in primary-pool wait/foreground reads. Turn the affected flag off to roll back. Self-mode clients recheck support and start a new legacy authority generation; public-mode clients recheck support and back off without treating public absence as membership loss. Keep cached display/user data. The first named usable channel target of five seconds at 140 KB/s remains a client acceptance target, not a result established by this relay PR. The 119-event approximately 5 MB read-state contention and native/browser throttled tests are required before broad client rollout. Generated with Codex --------- Signed-off-by: Brad Seiler <seiler@squareup.com> Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz>	2026-11-05 20:14:08 -05:00
Will Pfleger	fba8a7a94f	feat(relay): log WebSocket EVENT writes on the access target (#9154) ... Logs every WebSocket `EVENT` on the access target (`buzz_relay::access`). Before this, accepted writes logged only `Event ingested` on the handler target, and refused or ephemeral writes logged nothing. ### What's logged Each `EVENT` frame writes one `event=write` line when its handler finishes. The line has `conn_id`, `community`, `pubkey` (the authenticated login, if any), `kind`, `event_id`, and `outcome`/`reason`, taken from the OK the relay sent: - `accepted` - `rejected` (INFO), with `reason` set to the OK's NIP-01 prefix (`invalid`, `restricted`, `auth-required`, `duplicate`, `rate-limited`, …). Any other message is logged as `other`. - `error` (WARN), when the OK starts with `error` - `aborted`, when the handler ended before deciding an OK `outcome` is always the relay's decision. If the OK couldn't be queued because the outbound channel was full or closed, the line adds `ok_queued=false`; the field is left off otherwise. `EVENT`s refused before the handler runs are logged too, where their OK is sent: admission rate limits, with admission's reason, and a full handler pool, as `rate-limited`. The OK text after the prefix, the event's content and its tag values are never logged. Lines join to `auth`/`close` lines (and so to `peer_addr`) through `conn_id`, the same way `req` lines do. Rejected writes are capped per connection by the same refusal-budget mechanism as refused reads, on their own counter: the first 16 rejections are logged and the rest are only counted, in a new `writes_suppressed` field on the `close` line. Accepted writes are never capped. ### How `access::WriteLog` sends the OK and records it in one call (`write.ok(...)`). That call replaces every `conn.send(RelayMessage::ok(...))` in `handle_event` and the observer-frame handler, so the line always shows what the client was told. ### Tests - `access::tests`: the reason comes from the OK prefix only; each outcome logs at the right level and without content; rejected lines are capped by the budget and accepted ones are not (draining the channel so every OK is queued); an OK refused by a full or a closed outbound channel keeps its outcome and adds `ok_queued=false`. - `handlers::event::tests::fanout_access`: an unauthenticated `EVENT` going through `handle_event` writes one `rejected`/`auth-required` line, with no content or tag values. --------- Signed-off-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>	2026-11-06 00:46:24 +00:00
Luke Tornquist	bd1b17dee8	perf(db): plan multi-value #d reads with their values (#9131) ... ## Summary A REQ with several `#d` values for a NIP-33 kind runs as a named, cached statement. A client resolving channel metadata for up to 128 channels sends this kind of REQ. After a few runs on a connection, Postgres can switch the statement to a generic plan, which ignores the bound values: - The `d` match stops being an index condition on `idx_events_parameterized`. - The generic plan instead walks the author's rows in `created_at` order and filters `d_tag` row by row. - The `LIMIT` is one row per requested value, so the walk never stops early. It reads every stored version of the kind, not one probe per value. This change makes three fixes: - Multi-value `#d` reads now run as unnamed statements, as tag-value matches already do (`plans_with_bound_values`), so Postgres plans every call with its values. This covers both the page `SELECT` and the `COUNT`. - The values are bound as one array (`d_tag = ANY($n)`) instead of one bind per value. The SQL text no longer changes with the number of values, which matches how channel scope is bound. - The page `SELECT` and the `COUNT` now share one `push_d_tag_filter` helper instead of two copies of the `d` predicate. **What got simpler:** - One helper replaces the duplicated `d` predicate blocks. - A `#d` read no longer produces a different statement for every value count. ### Related issue N/A. None found. ### Testing - New unit test `d_tag_sql_does_not_depend_on_value_count`: the SQL text is identical for 2 and 128 values, for both the page query and the count query. - Postgres test `value_matched_queries_are_not_cached_as_prepared_statements`, extended from the tag-value test. It runs each `#d` page and count query 6 times and asserts no `d_tag = ANY(` statement is left in `pg_prepared_statements`. It also checks the page and count results. - Mutants, each caught by the matching test: - dropping `d_tags` from `plans_with_bound_values` fails the Postgres test; - going back to a per-value `IN (...)` list fails the unit test. - Gates run at 5f71e7143b8bc4cfbecb6b4cf886be3827d54943: - `cargo fmt --check` - `cargo clippy -p buzz-db -p buzz-relay --all-targets -D warnings` - `buzz-db` unit tests: 215 passed - `scripts/postgres-test-run.sh -p buzz-relay -p buzz-db --lib --tests`: 1291 passed - `scripts/test-postgres-test-discovery.sh` - Local `EXPLAIN ANALYZE` on synthetic data (50k versions of one author's kind 39000 rows, 128 `#d` values): - With `force_custom_plan`, `d_tag = ANY(...)` is in the Index Cond on every partition. - With `force_generic_plan`, the partition holding the rows drops `d_tag` to a row filter. - On larger data the generic plan can switch to an ordered `created_at` walk. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 19:41:03 -05:00
Luke Tornquist	e3a0f3d59d	feat(cli): send NIP-FI evidence via credential chain and Buzz desktop (#8999) ... ## Summary Relays that enforce federated identity (NIP-FI) reject every protected `buzz` CLI request with `401`. The CLI never sends `Nostr-Federated-Identity`. This PR gives the CLI a credential chain, lets it present NIP-FI evidence, and lets it take its identity from Buzz desktop. Relays without NIP-FI behave exactly as before. **Credential chain** (`crates/buzz-cli/src/credentials.rs`) - Raw `Keys` are replaced by `Credentials`, which provide the public key, signing through nostr's `NostrSigner`, NIP-44, and the NIP-OA auth tag. NIP-FI evidence comes through `evidence(relay)`. - The explicit key (`--private-key` / `BUZZ_PRIVATE_KEY`) is the first link. A later link can add a signer that holds no local key (a desktop provider, for example) without reshaping callers. - Operations that need key material rather than a signature are purpose-bound methods instead of a key accessor: - `engram_d_tag` (the NIP-AE HMAC over the conversation key) - the relay-scoped GIF customer id (random and fixed for the process when there is no local key) - `buzz mem` now encrypts, decrypts and derives the `d` tag through the provider. Invalid ciphertext skips an entry; a signer failure aborts, so an unavailable credential never reads as empty memory. `buzz_core::engram` gains the composable steps behind it: `encode_body`, `event_builder`, `validate_envelope` and `check_d_tag`. `build_event` and `validate_and_decrypt` are unchanged wrappers, so `buzz-acp` and desktop callers don't move. - Agent-management drafts (`agent_management.rs`) encrypt and sign through the provider. **Evidence on every protected request** (`crates/buzz-cli/src/client.rs`) - A single `authorize` helper gets the NIP-FI assertion first, then signs the proof just before sending, so a slow mint cannot age it past NIP-98's 60 s window. It sets these on every protected route: `Authorization` (NIP-98 or Blossom kind:24242), `x-auth-tag`, and `Nostr-Federated-Identity: Bearer …`. The routes are `/query`, `/count`, `/events` (stored and moderation), `/upload`, `/media/upload`, `/media/*`, `get_authed`, `post_json_authed` and `post_json_once_authed`. - Exempt routes (`get_public`) are unchanged. - The WebSocket publish sends the header on the upgrade via `connect_authenticated_with_headers`. NIP-42 AUTH signs through the signer (`buzz-ws-client` now takes `NostrSigner`; `&Keys` callers compile unchanged). - An upgrade denial is an HTTP response under NIP-FI. `WsClientError::upgrade_rejection` returns its status and body, which map like any other relay error, so a `403` still says whether evidence or authorization was refused. A denial after NIP-42 (such as a key mismatch) is a 1008 close; `WsClientError::PolicyClose` keeps its reason and the CLI reports it as an `auth_error`. - The WebSocket publish caps each network phase as a whole, writes included: connect, AUTH challenge (with any Pongs), AUTH send plus OK, EVENT send plus OK, and a best-effort close. NIP-42 signing between phases is bounded by the signer. The relay drops a socket not signed in within 5 s, so a sign-in that is throttled or comes back after the socket closed reconnects and signs the new challenge, at most 3 connections. Each connection fetches current NIP-FI evidence for its upgrade, and a throttled sign-in is waited out between connections, so a dead challenge is never signed. EVENT is signed before connecting and sent once, after AUTH succeeds. Running out is `unavailable` (exit 2, `retryable:true`). - Relay traffic follows no redirects: reqwest strips `Authorization` across origins but not `x-auth-tag` or `Nostr-Federated-Identity`, so a 3xx is returned as a relay error. Credential header values are marked sensitive. **Headless evidence source** (`crates/buzz-cli/src/federated_identity.rs`) - Set `BUZZ_FEDERATED_IDENTITY_TOKEN` (your own NIP-FA sign-in session: a human session for your key, not an agent grant) and `BUZZ_FEDERATED_IDENTITY_ADAPTER` (the adapter base) to enable it. Both are env only: command lines are visible to other local users. - The CLI requests an assertion and the adapter mints it, through `POST {adapter}/v1/identity/assertions` with a NIP-98 proof in `Nostr-Authorization` (NIP-FA §Relay assertion). - It checks the returned pubkey and expiry, parses the assertion as a compact JWS (three base64url segments, JSON header with `alg`, JSON payload), and requires its `exp` to equal `expires_at`. It caches one assertion per process, bound to the relay when credentials are resolved, and re-mints at the earlier of half-life and 30 s before expiry; the desktop link caches and refreshes its assertion by the same rule. Concurrent callers share one mint. An HTTP proof is signed again, after current evidence, when the assertion changed while it was signed or the evidence check aged it 30 s or more, up to 3 rounds, then fails as `unavailable` (retryable). - Compared with the separate rules before: a 300 s assertion now refreshes at 150 s (was 270 s) on both links, a 40 s adapter assertion at 10 s (was 20 s), and a 20 s desktop assertion is kept for 10 s instead of fetched on every lookup. Desktop assertions now need a readable JWS header with `alg` and a non-empty signature segment. An unsupported headless relay URL now fails when credentials are resolved. - The token goes only to the adapter origin (https, or http on loopback), never follows a redirect, and never appears in errors or `Debug`. - A mint retries network errors, `429` and any `5xx` up to three attempts with jittered backoff (or the adapter's `Retry-After` on `429`/`503`, as delta-seconds or any HTTP-date form, capped at 30 s), on every path (reads, post-once, moderation, stored events, WebSocket). It then fails as `CliError::Adapter` (`adapter_error`, exit 2, `retryable:true`). Relay retry loops never repeat it. A stored write keeps `delivery_unknown` when any earlier attempt may have been delivered, so a later adapter failure never invites a re-sign. Relay mutation retries are otherwise unchanged. - An explicit port 80 relay URL (read numerically from the authority, on any scheme) is a usage error: `wss://host` means port 443, and NIP-FA has no relay on port 80. - A refusal is never retried: a non-transient status with an unreadable or truncated body is still a refusal. - Each NIP-FA denial code maps to an `auth_error` with a specific message. - These settings belong to the explicit key: without one they are not read, and when the desktop link wins a warning names them. - If an enforcing relay answers `401` and no assertion was sent, the CLI returns an `auth_error` that names both settings, or, for a desktop user, says to sign in to the community in Buzz. A desktop user's `403` hint no longer mentions `BUZZ_AUTH_TAG`. **What got simpler** - About ten request sites built `Authorization` and `x-auth-tag` themselves. Five copies of the 401/403 error-mapping block are now one `relay_error`. - The `sign_nip98` / `sign_blossom_*` helpers are gone. Proofs are a three-case `Proof` enum consumed by `authorize`. - `BuzzClient::keys()` (raw secret access from every command) is removed. Commands use `public_key()`, `sign_event` or a purpose-bound credentials method. **Desktop credential provider** (`crates/buzz-cli/src/desktop_provider.rs`, `docs/local-credential-provider.md`) Buzz desktop is a credential provider in the chain, after an explicit key: 1. An inherited `BUZZ_CREDENTIAL_PROVIDER` token (set by the desktop for the agents it launches). It decides the identity: when it is revoked, not bound to the relay or unreachable, the command fails. A paired token, which acts as the human, is never tried. That keeps a well-behaved CLI from acting as the owner by accident; it is not a boundary against an agent that can run its own commands as the owner's OS user, and the contract says so with the mitigations. 2. Otherwise, a paired token stored in the OS keychain (`keyring`; macOS native keychain, Linux Secret Service; no file fallback). A desktop that answers `denied` or `unavailable` fails the command; only an unreachable or unconfirmed advertisement is skipped. Revoked entries are forgotten. New commands: `buzz auth desktop pair` and `buzz auth desktop unpair`. The help no longer marks `BUZZ_PRIVATE_KEY` as required. The CLI talks to the desktop over the loopback contract in `docs/local-credential-provider.md` (NIP-46-shaped envelopes, `Buzz-Credential-Provider` version header, same-user check on the listening socket for paired and inherited tokens, via `/usr/sbin/lsof` on macOS, given 5 s; a check that runs out of time is a retryable `unavailable`, a wrong owner is refused). Every response's `id` must match its request, error envelopes included. Paired tokens live only in the keychain, survive desktop restarts, and are harness-neutral for agents. - **Relay binding:** tokens bind the whole relay URL (`http`/`https` read as `ws`/`wss`, lowercase host, no default port, `/` for an empty path; path and query count). The CLI sends that URL in `pair` and `federated_identity`; NIP-FA audience canonicalization is the desktop's. - **Rate limits:** a `rate_limited` answer refuses before acting, so the CLI re-sends after `retry_after` (capped at 30 s), up to four times, except a NIP-42 sign-in, whose wait goes to the WebSocket reconnect. A re-sent NIP-42, NIP-98 or Blossom proof is re-dated (new id, Blossom `expiration` moved with it); content events are re-sent exactly. If it is still throttled, the command fails as `rate_limited` (exit 2, `retryable:true`), not `auth_error`, whether the throttled call was NIP-98, event signing, NIP-42 or encryption. Transport failures are never re-sent, and relay commands are never replayed. A `buzz mem` list that is throttled partway through completes. - **Memory:** a `nip44_decrypt` answered with `invalid_request` is invalid ciphertext, so `buzz mem` skips that entry (NIP-AE filtering); any other provider failure aborts. - **Keychain changes:** each change reloads the map under a short `flock` on `~/.buzz-foundation/cli-pairings.lock` (directory forced to `0700`, lock opened with `O_NOFOLLOW`) and changes only its own instance; the lock is never held across the approval wait. An unreadable entry is refused, never overwritten; `unpair` with no instance clears it. ### Related Related: [block/buzz-app#2260](https://github.com/block/buzz-app/pull/2260) (the desktop side of the provider contract) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Signed-off-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Will Pfleger <wpfleger@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>	2026-11-06 00:26:22 +00:00
Brad Seiler	66c3d6917d	feat(protocol): define compact membership discovery (BUZZ-615) (#9117) ... ## Summary Cold login currently downloads full channel rosters to learn the viewer's memberships. This defines compact discovery in NIP-CL and reserves relay-only kind 39008 for a signed, request-bound response. Full kind-39002 rosters remain compatible. The contract binds a complete snapshot to the community, viewer, NIP-98 request and requested channel types. It defines a primary database cut, overflow errors, lazy-roster precedence, reconnect repair and safe fallback across mixed relay versions. The next PR implements both query modes. This PR does not enable either capability. A separate `membership_public` capability uses relay-only kind 39009 for lookups of any subject’s listed-open forum, stream and workflow memberships. It preserves roles and archived state while excluding private, unlisted, DM and system channels. Signed answers bind requester, subject and public scope; they cannot grant membership or sending permissions. The self-mode contract is unchanged. ### Related issue [BUZZ-615](https://linear.app/squareup/issue/BUZZ-615). No duplicate compact-membership implementation found. Related work: [#8898](https://github.com/block/buzz/pull/8898) protects full-roster signing; [#9060](https://github.com/block/buzz/pull/9060) caches membership lookups. ### Testing McKeon independently reviewed the contract and the fixes to request binding, per-channel race fencing and mixed-version fallback. Content checks parse every JSON example and resolve relative links. Paul’s review fixes remove implementation details from the normative text and move the existing 39009 registration and regression test into this PR. Protocol behavior is unchanged. Runtime validation belongs to the implementation PR. This description does not attest human testing. ### Implementation and operator notes Both kinds are registered in `crates/buzz-core/src/kind.rs`, included in `ALL_KINDS`, and rejected as client-submitted events by the relay-only guard. The dependent implementation uses one primary SQL statement’s MVCC snapshot over active `channel_members` joined to nondeleted channels, supported by the partial `(community_id, pubkey) WHERE removed_at IS NULL` index. Public requests share the existing community/requester `ApiCalls` quota configured by `human_api_calls_per_min`. `BUZZ_MEMBERSHIP_SNAPSHOT_ENABLED` and `BUZZ_MEMBERSHIP_PUBLIC_ENABLED` independently default to false. Deploy support across the fleet with both flags off, then enable capabilities independently. Disable the affected capability to roll back; client fallback and back-off behavior remain specified in NIP-CL. Client opt-in and production-shaped throttled validation follow separately. Generated with Codex --------- Signed-off-by: Brad Seiler <seiler@squareup.com> Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz>	2026-11-05 23:52:07 +00:00
Jemiah Westerman	cec47c20a3	perf(db): hydrate search hits with exact event timestamps (#9126) ... ## Summary Keep each search hit's existing signed `created_at` alongside its event ID during canonical hydration. WebSocket search, HTTP search, and profile-name search now use a bounded exact-pair database lookup so PostgreSQL can prune unrelated event months. Also constrain channel-create replay detection by the timestamp of the already-available signed event. ## Scope / compatibility - Combined-tree verification also passed: repository-wide `just ci`, 220 unit/source-policy tests, and the full 722-test buzz-db/search PostgreSQL suite. All three temporary timestamp-constraint mutations were killed by functional/plan assertions and restored. - Independent of companion post-insert push and deletion PRs; based directly on `main`. - Deploy/code-only change. No schema/migrations/backfills, persistent or serialized formats, indexes, triggers, functions, procedures, rules, or foreign keys added. - Existing ID-only event APIs remain available for callers without timestamps. - Maximum 500 input pairs, validated before checkout; duplicate inputs do not multiply results; missing/deleted/wrong-community/wrong-timestamp hits are omitted. - Caller ordering, authorization, observability, bounded-only replica routing, proved connection use, non-cancel fallback, and cancellation propagation remain intact. A timestamp is not treated as a completeness proof. - Existing/new relays coexist against unchanged schema; no rollout/prebuild dependency. ## Verification - Compiled `buzz-db`, `buzz-search`, and `buzz-relay` library/tests. - 220 infrastructure-free unit/source-policy tests passed. - 35 focused PostgreSQL tests passed, including cross-month exact pairs, tenant/deleted/missing scope, same-second IDs, duplicate/empty/invalid/oversized/full batches, routing/fallback/cancellation, channel replay, and existing search regressions. - `EXPLAIN ANALYZE BUFFERS` of production hydration SQL verifies only requested event months execute under forced custom **and generic** plans, including the full **500-distinct-pair** shape. A wrong-timestamp decoy detects cross-product matching. - Targeted clippy with `-D warnings`, formatting, diff checks, PostgreSQL discovery, and differential file-size checks passed. - Live isolated relay smoke: exact channel-create replay, HTTP NIP-50 search hydration, profile-name search hydration, and authenticated WebSocket NIP-50 results/EOSE. All three hydration route labels observed in local metrics. - Independent review completed; optional maximum-batch plan coverage added and independently re-reviewed. Compile fixes for SQLx 0.9 API compatibility verified. No production latency improvement or workload-percentage claim is made. --------- Signed-off-by: Jemiah Westerman <jemiah@squareup.com> Co-authored-by: Son of Anton <8e0ef2ab02d69a7ac089ef5eca90d1a3fbbdc546f6322dc34d0c4f3deecea7f4@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 15:43:28 -08:00
Jemiah Westerman	997ce3a2c9	perf(db): prune post-commit push enqueue by signed event timestamp (#9128) ... ## Summary Carry each inserted event's signed `created_at` through the existing in-memory, post-commit push enqueue path. The worker now looks up the source event by community, ID **and exact timestamp**, allowing PostgreSQL to prune unrelated monthly partitions. All seven direct insert-followup hooks forward the timestamp already bound into their INSERT. No receipt-time or causal-order inference is used. ## Scope / compatibility - Combined-tree verification also passed: repository-wide `just ci`, 220 unit/source-policy tests, and the full 722-test buzz-db/search PostgreSQL suite. All three temporary timestamp-constraint mutations were killed by functional/plan assertions and restored. - Independent of the companion search-hydration and deletion PRs; based directly on `main`. - Deploy/code-only change. No migrations, backfills, schema changes, new indexes, triggers, functions, rules, procedures, or foreign keys. - No persistent/serialized payload changes: `push_match_queue` and `push_wake_outbox` remain unchanged. Their later ID-only lookups are outside this PR. - Admission/gate ordering, commit-only submission, savepoint rollback behavior, duplicate suppression, bounded queues, best-effort drops, and existing lock/statement/client deadlines are retained. - Existing and new relay versions can coexist against the unchanged schema; no rollout ordering or operator prebuild is required. ## Verification - Compiled `buzz-db`, `buzz-search`, and `buzz-relay` library/tests. - 218 infrastructure-free unit/source-policy tests passed. - 12 focused PostgreSQL tests passed, including new timestamp propagation and savepoint tests plus existing follow-up/admission/gate regressions. - `EXPLAIN ANALYZE` of the exact production enqueue INSERT verifies a single event month executes under forced custom and generic prepared plans. - Targeted clippy with `-D warnings`, formatting, diff checks, PostgreSQL discovery, and differential file-size checks passed. - Live isolated relay smoke: channel creation/replay, signed message ingestion/readback, and completed post-commit enqueue worker metric. Test lease had empty subscriptions; no push gateway was contacted. - Independent local review completed: no remaining blockers. No production latency improvement or workload percentage is claimed; this establishes the pruning mechanism and behavior preservation. --------- Signed-off-by: Jemiah Westerman <jemiah@squareup.com> Co-authored-by: Son of Anton <8e0ef2ab02d69a7ac089ef5eca90d1a3fbbdc546f6322dc34d0c4f3deecea7f4@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 15:43:02 -08:00
Jemiah Westerman	0cfa50ac70	perf(db): use exact event timestamps for transactional deletion (#9130) ... ## Summary Preserve the target event's already-known signed timestamp through deletion planning and use `(community_id, id, created_at)` for the canonical transactional read, soft delete, and retention-free purge. Known-target operations can prune unrelated monthly partitions instead of repeatedly probing by ID alone. Public ID-only deletion callers retain their discovery step and reuse the discovered timestamp for subsequent mutations. Operator deletion similarly reuses its existing discovery result. Canonical kind/channel classification and locking remain database-owned inside the transaction. ## Scope / compatibility - Combined-tree verification also passed: repository-wide `just ci`, 220 unit/source-policy tests, and the full 722-test buzz-db/search PostgreSQL suite. All three temporary timestamp-constraint mutations were killed by functional/plan assertions and restored. - Independent of companion search-hydration and push-enqueue PRs; based directly on `main`. - Deploy/code-only change. No schema, migrations, backfills, persistent/serialized data formats, indexes, triggers, functions, rules, procedures, or foreign keys added. - Runtime `DeletionTarget` now carries the target timestamp; no wire payload changes. - Community admission, canvas replacement locking, replay repair, atomicity, retention-free purge/mention cleanup, artifact restrictions, and once-only thread accounting remain intact. - Extract deletion helpers into a focused private module, preserving existing public reexports. - Existing/new relays coexist against the unchanged schema; no operator prebuild or rollout dependency. ## Verification - Compiled `buzz-db`, `buzz-search`, and `buzz-relay` library/tests. - 218 infrastructure-free unit/source-policy tests passed. - 23 focused PostgreSQL tests passed: cross-month/exact-key/tenant isolation, missing/wrong/deleted targets, rollback and concurrent replay/counters, retention-free cleanup in desired and migration schema modes, canvas locking, and existing deletion/admin regressions. - Actual production prepared SELECT/UPDATE/purge SQL prunes to one event month under custom and generic plans. - Targeted clippy with `-D warnings`, formatting, diff checks, PostgreSQL discovery, and differential file-size checks passed. - Live isolated relay smoke: signed message and reply ingestion, atomic deletion, deleted payload absent from queries, thread counters return to zero, and safe deletion replay. - Independent local review completed: no remaining blockers. No production latency/workload benchmark is claimed. --------- Signed-off-by: Jemiah Westerman <jemiah@squareup.com> Co-authored-by: Son of Anton <8e0ef2ab02d69a7ac089ef5eca90d1a3fbbdc546f6322dc34d0c4f3deecea7f4@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 15:42:41 -08:00
Luke Tornquist	b6d4403149	perf(redis): index mesh ready records and share one Redis connection (#9113)	2026-11-05 18:42:02 -05:00
Blake McAnally	b0f68a06ad	feat(cli): user status expiry and status read (#8777) ... > Replaces #8768, which was opened from a fork. ## Summary This change adds two things to `buzz-cli`. `buzz users set-status` now accepts `--expires`. The status then clears itself at that time. `buzz users status` is a new command. It reads the visible status of one or more users. ## Design `--expires` takes a unix timestamp or a duration. A duration is a number with a unit: `s`, `m`, `h`, `d`, or `w`. A bare number is a unix timestamp. The CLI had no duration parser to reuse. The only one is private to `buzz-workflow`. A small parser now lives in `users.rs`. The CLI rejects an expiry that is not in the future. It does this before it signs anything. The error text says that a bare number is a timestamp. `--expires` cannot combine with `--clear`. A status with no text and no emoji cannot have an expiry. The SDK rejects that shape. `buzz-sdk` has a new builder, `build_user_status_expiring`. It adds `["expiration", ts]` after the emoji tag. This is the tag order desktop uses. `build_user_status` now calls the new builder with no expiry. Its output is unchanged. Existing callers need no change. `users status` queries kind 30315 with `#d: general`. It picks the newest event for each author. A higher `created_at` wins. On a tie, the lower event id wins. Desktop uses the same rule. The CLI then drops a user when the newest event has no text and no emoji. It also drops a user when the status has expired. An expired status means `expiration <= now`. An older status never shows behind a newer cleared or expired one. The expiration tag is read like desktop's `parseInt`. A value such as `90abc` reads as 90. A value such as `soon` reads as no expiry. The output is an array: `[{pubkey, text, emoji, expires_at, updated_at}]`. `emoji` is an empty string when there is none. `expires_at` is `null` when the status does not expire. A user with no visible status has no row. `--pubkey` takes hex, npub, or an exact display name. It reuses the name resolver of `messages`. It is repeatable. The limit is 100, because each name costs one profile search. With no `--pubkey`, the command reads your own status. A relay response that is not valid JSON is an error. It is never an empty result. ## Testing What ran: - `cargo test -p buzz-cli -p buzz-sdk` passes. - `cargo clippy -p buzz-cli -p buzz-sdk --all-targets -- -D warnings` passes. - `cargo fmt --all --check` passes. - `just file-size-check` passes. The CLI tests use an axum fake relay. They call the production functions `publish_status` and `read_statuses`. These are the same functions that the commands call. The tests cover these cases: - One publish carries the right tags. - Bad expiries cause no publish. - Expired, cleared, and malformed statuses. - Newest-event choice and tie-break. - Hex, npub, and name input. - The 100-pubkey limit. - An invalid relay response. - A publish followed by a read. Interop evidence: I ran the real `buzz` binary against a local fake relay. No live relay was used. I fed the published event to desktop's own `parseUserStatusEvent` and `fetchUserStatusLookup` with node. Desktop showed the same text, emoji, and `expiresAt` that `buzz users status` printed. The check printed `INTEROP OK`. Mutation checks: I broke each guard once. Each break made a test fail. The guards were: - the expiry comparison, `<=` changed to `<` - the future-expiry check - the `d=general` filter - the newest-first order - the tie-break - the 100-pubkey limit - the author de-duplication - the invalid-response error - the hide-cleared-status check - the duration overflow check - the SDK expiration tag - the SDK empty-status check I restored each guard after the check. One note on process: I wrote the code and the tests in the same pass. I did not write the tests first. The mutation checks cover the gap. ## Manual test steps Use a test identity on a relay that you may write to. Set `BUZZ_RELAY_URL` and `BUZZ_PRIVATE_KEY` first. 1. Set a status that expires in two minutes. ```bash buzz users set-status --text "testing expiry" --emoji "🧪" --expires 2m ``` Success: the output has `"accepted":true`. 2. Read the status. ```bash buzz users status ``` Success: the output is one row. `text` is `testing expiry`. `emoji` is `🧪`. `expires_at` is about two minutes from now. 3. Open the desktop app as the same user. Success: your status shows on your profile. 4. Wait for the expiry. Read again. ```bash buzz users status ``` Success: the output is `[]`. Desktop also hides the status. 5. Try a bad expiry. ```bash buzz users set-status --text x --expires 3600 ``` Success: exit code 1. The error says that the value must be in the future. Nothing is published. 6. Try the clear conflict. ```bash buzz users set-status --clear --expires 1h ``` Success: exit code 1. The error says `--clear` cannot be used with `--expires`. 7. Read another user. ```bash buzz users status --pubkey <hex|npub|name> ``` Success: you get that user's visible status. You get `[]` if they have none. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: bmcanally <bmcanally@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 17:28:24 -06:00
Blake McAnally	303157f53d	feat(cli): send a thread message to the channel (#8778) ... > Replaces #8769, which was opened from a fork. ## Summary This change adds `buzz messages send --from-thread <root-event-id>`. The flag is the CLI form of desktop "Send to channel". It publishes a new top-level message in the channel. The message carries the tag `["buzz:sent-from-thread", <root-id>, <excerpt>]`. The message has no NIP-10 reply tags. ## Design Desktop puts the thread root id in the tag. It does not use the selected message id. Desktop publishes kind 9. The CLI does the same. The CLI checks the root event before it publishes anything. The root must exist. The root must have the same `h` tag as `--channel`. The root must be a thread root. If the id belongs to a reply, the CLI fails and prints the real root id. A missing root exits with a not-found error. Other failures exit with a usage error. No event is published after a failed check. The excerpt comes from the root content. It is a Rust port of `summarizeThreadRoot` in `sentFromThread.ts`. The port clips by code point, as desktop does. It breaks words by UTF-16 unit, as desktop does. `--from-thread` conflicts with `--reply-to` and `--broadcast`. It works only with kind 9. The relay does not validate this tag. Desktop's Tauri code validates it (`message_tags.rs`). The CLI output follows those rules: 64 hex characters, an excerpt of at most 64 characters, and no control characters. The change adds no dependency and no SDK change. The CLI appends the tag to the SDK `build_message` builder. The README describes the flag. This PR does not change `desktop/` (deprecated; see #8840). ## Testing What ran: - `cargo test -p buzz-cli`: 508 passed. - `cargo clippy -p buzz-cli --all-targets -- -D warnings`: clean. - `cargo fmt --all --check`: clean. - `just file-size-check`: passed. Test coverage: - One test calls `cmd_send_message` against a fake relay. It covers success, a reply id, another channel, a missing root, an invalid id, and a non-kind-9 send. The failure cases publish nothing. The success case publishes one event with the tag and without `e` or `broadcast` tags. - One test checks the clap conflicts. - Unit tests cover the excerpt boundaries (63, 64, 65 characters, the word-break threshold, and multi-byte and emoji text). - One test compares the Rust excerpt with 639 vectors. Desktop's own `summarizeThreadRoot` generated the vectors. The first run found an ordering bug (desktop replaces each pattern over the whole text in turn). The code now does the same. Interop evidence: - A node script runs the real `buzz` binary against a local fake relay. It is not a live relay. - The script passes the published tags to desktop's `getSentFromThreadReference` and `getSentFromThreadRootId`. - Desktop accepts the tag, returns the same root id, and returns the same excerpt as `summarizeThreadRoot`. - Output: `["buzz:sent-from-thread","abab…ab","Launch plan for the doc 😀 word word word word word word word…"]` then `INTEROP OK`. Mutation checks. Each guard was broken once. A test failed each time. Then the guard was restored. - Channel check removed: success test fails. - Reply-root check removed: success test fails. - Kind check removed: success test fails. - `--broadcast` conflict removed: clap test fails. - Word-break threshold `>` changed to `>=`: three excerpt tests fail. - UTF-16 break changed to code-point break: two excerpt tests fail. - Tag not attached: success test fails. ## Manual test steps Use a test channel. You need `BUZZ_RELAY_URL` and `BUZZ_PRIVATE_KEY` set. Build with `cargo build -p buzz-cli`. 1. Post a root message. Copy its event id as `ROOT`. `buzz messages send --channel <CHANNEL> --content "Launch plan: [doc](https://example.com)"` 2. Send it to the channel. `buzz messages send --channel <CHANNEL> --content "Decision: ship it" --from-thread $ROOT` Success: the output has `"accepted":true`. A new top-level message appears in the channel. 3. Read it back. `buzz --format json messages get --channel <CHANNEL> --limit 1` Success: the tags include `["buzz:sent-from-thread","<ROOT>","Launch plan: doc"]`. There are no `e` tags. 4. Open the channel in desktop. Success: the message shows the "sent from thread" line for that root. 5. Try a reply id. `buzz messages send --channel <CHANNEL> --content x --from-thread <REPLY_ID>` Success: exit code 1. The error names the root id. Nothing is published. 6. Try a root from another channel, and try `--reply-to` together with `--from-thread`. Success: both fail and publish nothing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: bmcanally <bmcanally@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 17:28:19 -06:00
Blake McAnally	5c91e0d6fe	feat(relay): add kind 30180 custom emoji entry (#8783) ... ## Summary This PR adds kind 30180, `KIND_EMOJI_ENTRY`. One event holds one custom emoji. The relay accepts, validates, stores, and replaces these events. The SDK gets `build_emoji_entry`. `docs/nips/NIP-CE.md` documents the kind. This is the relay part of the custom-emoji scale-up (Option C). It does not change kind 30030 sets, message `emoji` tags, or reaction `emoji` tags. It adds no migration and no schema change. **Read this before you deploy.** Prefix search for kind 30180 works only on a database whose `events.search_tsv` indexes that kind. See "Search index" below. ## Design - `crates/buzz-core/src/kind.rs`: `KIND_EMOJI_ENTRY = 30180`. It is in `ALL_KINDS`. A compile-time check proves it is parameterized replaceable. 30180 was free on `origin/main`. - `crates/buzz-relay/src/handlers/ingest.rs`: - The kind needs `UsersWrite`. - The kind is global-only. It is never channel-scoped. - `validate_emoji_entry` runs before storage. It requires exactly one `d` tag and exactly one `emoji` tag. - It rejects any `h` tag. - It requires `d` to equal the shortcode in the `emoji` tag, and both to equal the normalized shortcode. - It requires an `http://` or `https://` URL of at most 2048 bytes. It uses the SDK check `check_custom_emoji_url`, which is now public. - It caps `content` at 256 bytes (`MAX_EMOJI_ENTRY_CONTENT_LEN`). A shortcode is at most 64 bytes, so the cap leaves room. - Every rejection starts with `invalid:`. - `crates/buzz-sdk/src/builders.rs`: `build_emoji_entry(shortcode, url)` and `emoji_entry_search_text`. The content is the shortcode with `_` and `-` replaced by spaces. - Replace semantics use the existing parameterized-replaceable path. A second publish with the same `(pubkey, d)` leaves one live event. Two authors with the same `d` keep two events. The NIP tells clients to prefer the newest event and report the conflict. ### Decisions - **WebSocket `search_mode`: not changed.** Prefix search works through `POST /query`, which already reads `search_mode`. This is the smallest correct change. The `REQ` search path still uses full-word matching. The NIP says so. A client that needs typeahead uses `POST /query`. - **Search text.** Postgres `simple` already splits `party_parrot` at the underscore. It keeps `sad-parrot` as one token and also indexes `sad` and `parrot`. So the space replacement is not needed for matching today. It is kept because it is harmless and does not depend on parser details. - **Content cap of 256 bytes.** The cap is documented in the NIP and the code doc comment. ### Search index (important) The design doc says kind 30030 is not excluded from the full-text index. That is true for `schema/` (the desired state): it indexes every kind except a deny list. It is not true for a database that was created by `migrations/`: - Migration 0008 gives a fresh install a positive allowlist: kinds 0, 9, 40002, 45001, 45003. - `scripts/maintenance/nip_rs_search_allowlist.sql` moves a populated database to the same allowlist. On an allowlist database, kind 30180 is **stored and resolvable by `#d`, but not searchable**. I did not change this. A change to `search_tsv` rewrites the partitioned `events` table. That needs the reviewed out-of-band procedure in `docs/database.md` (rules 6 and 5), and this task forbids migrations and schema files. Before clients depend on search, a human must check the deployed expression: ```sql SELECT pg_get_expr(adbin, adrelid) FROM pg_attrdef d JOIN pg_attribute a ON a.attrelid = d.adrelid AND a.attnum = d.adnum WHERE a.attname = 'search_tsv' AND d.adrelid = 'events'::regclass; ``` If the result lists `kind IN (0, 9, ...)`, add 30180 to the allowlist through the out-of-band procedure first. ## Spike I ran this on a throwaway local Postgres 17 (Docker), desired-state schema from `pgschema`. The loader inserted 100,000 kind 30180 events from 200 authors through `Db::replace_parameterized_event`. It used 16 parallel tasks, and it did not use the network. The loader was a scratch example. It is not in this PR. Rows landed in `events_p_future` because the local partitions end at 2026-06. Real deployments have monthly partitions, so scans split across partitions. **Insert rate.** 100,000 inserts took 33.9 s (about 2,900 per second) at the DB layer. The relay adds rate limits on top (see the design doc, section 2.5). **`#d` batch of 50 names** (actual relay SQL shape, warm cache, `LIMIT 100`): | Case | Plan | Execution time | |------|------|----------------| | No `authors` | Index scan on `idx_events_parameterized`. It uses `community_id` and `kind` only. `d_tag` is not a leading key, so it scans the kind range. | 12 ms warm, 28 ms cold | | 50 names, all their authors (50 distinct) | Index scan with `pubkey` and `d_tag` in the index condition | 1.5 ms | | 50 names, one author | Same index, tight condition | 0.13 ms | Result: with `authors`, the lookup is fast. Without `authors`, the cost grows with the number of entries of the kind. At 100,000 rows it is still tens of milliseconds. No index change is needed now. Re-measure at 1,000,000 rows. **Prefix search**, `kinds=[30180]`, relevance order (the default), `LIMIT 50`: | Query | Matching rows | Execution time (warm) | |-------|---------------|-----------------------| | `parr` | 8,169 | 115 ms (124 ms cold) | | `p` | 15,990 | 219 ms | | `party par` | 8,169 | 266 ms (cold) | | `zap blob 999` (rare) | 1 | 0.4 ms | | `parr` with `search_sort: "newest"` | 8,169 | 0.8 ms | Result: the GIN index finds matches fast. Relevance order then scores every matching row (`ts_rank_cd`) and sorts them. The cost grows with the number of matches, not with `limit`. My synthetic words are few (24), so each prefix matches many rows. Real names are more varied, but a one- or two-letter query will still match many rows. The `search_sort: "newest"` option avoids the scoring and was 150 times faster here. Clients should use `search_sort: "newest"` for typeahead, or require at least 3 characters. This is a recommendation for the client work. It needs no relay change. **Side effects of bulk insert** (from reading `ingest.rs` and `handlers/event.rs`): - Audit log: each accepted event queues one `EventCreated` audit entry. The audit worker writes them one at a time under an advisory lock. The queue holds 1000 entries and uses `send().await`, so a bulk run slows the writer down instead of dropping entries. 96,000 events add 96,000 audit rows. - Live fan-out: each event is published to the global topic and matched against open subscriptions in `sub_registry`. A client with an open filter for kind 30180 receives every event. - Workflow triggers: each event spawns a task that calls `workflow_engine.on_event`. That is one trigger lookup per event. - Push: the DB push enqueue only matches kinds 9, 40002, 45001, and 45003 (`PUSH_MATCH_KINDS`). Kind 30180 does not wake push subscribers. - No `channel_id`, so the `events_created_at_floor_guard` trigger does not apply. ## Testing What ran (all on the final tree): - `cargo test -p buzz-core -p buzz-sdk -p buzz-relay -p buzz-search`: pass. - `cargo clippy -p buzz-core -p buzz-sdk -p buzz-relay -p buzz-search --all-targets -- -D warnings`: pass. - `cargo fmt --all --check`: pass. - `just file-size-check`: pass. - Postgres lane, desired-state schema (`BUZZ_TEST_SCHEMA_MODE=desired`) with a local Redis: `cargo test -p buzz-relay --lib handlers::ingest -- --include-ignored`: 205 passed, including the four new tests. New tests: - `emoji_entry_validation_table` (no infra): one row for each rule. - `emoji_entry_is_a_global_users_write_kind` (no infra). - `emoji_entry_ingest_validates_and_replaces` (Postgres): goes through `ingest_event`. It rejects a bad shortcode, `d` not equal to shortcode, an `h` tag, and a non-http URL. It checks that nothing is stored. It then publishes twice with the same `d` and checks that one live event remains and that it is the newer one. - `emoji_entry_prefix_search_finds_by_word_prefix` (Postgres): `parr` finds `party parrot` and `sad parrot`. `party par` finds one. `joy` finds `bufo joy`. It fails with a clear message on an allowlist database. - SDK: `emoji_entry_happy_path` and `emoji_entry_rejects_bad_input`. Not run: - `just test` was not run in full. I ran the relevant Postgres tests directly. - `cargo test -p buzz-search --test postgres_fts_integration` fails 23 of 23 on my shared local database. Those fixtures use fixed hostnames and need the fresh-database lane (`scripts/postgres-test-run.sh`). I did not change that crate. This is an environment limit, not a result of this change. - The new search test fails on a database built by `buzz-admin migrate` (allowlist). That is the finding above. It passes on the desired-state schema that the CI lane uses (`postgres-test-run.sh`). - I ran no live relay and no end-to-end client. Interop evidence: none for this item. The desktop and mobile constants are out of scope here. `/home/bloxer/ref/interop.mjs` was not needed because no TypeScript changed. Mutation checks (break the guard, confirm a test fails, restore): | Mutation | Result | |----------|--------| | Remove the `validate_emoji_entry` call | `emoji_entry_ingest_validates_and_replaces` fails | | Remove the `h` tag check | table test and ingest test fail | | Remove the `d == shortcode` check | table test and ingest test fail | | Remove the URL check | table test and ingest test fail | | Remove the content cap | table test fails | | Allow more than one `d` tag | table test fails | | Allow more than one `emoji` tag | table test fails | | Remove the kind from `required_scope_for_kind` | scope test, ingest test, and search test fail | | Remove the kind from `is_global_only_kind` | global test and ingest test fail | | SDK: skip shortcode normalization | both SDK tests fail | | SDK: skip the URL check | `emoji_entry_rejects_bad_input` fails | | SDK: skip the space replacement | `emoji_entry_happy_path` fails | One guard has no relay-level test: the space replacement does not change search results, because Postgres already splits `_`. The SDK test covers it. ## Manual test steps You need Docker and a local checkout. Use a local relay only. 1. Start the services and apply the desired-state schema, so that search indexes kind 30180: ```bash cd ~/buzz && . ./bin/activate-hermit docker run -d --name buzz-pg -e POSTGRES_USER=buzz -e POSTGRES_PASSWORD=buzz_dev -e POSTGRES_DB=buzz -p 127.0.0.1:5432:5432 postgres:17-alpine redis-server --daemonize yes --save "" PGHOST=localhost PGPORT=5432 PGUSER=buzz PGPASSWORD=buzz_dev PGDATABASE=buzz ./bin/pgschema apply --file schema/schema.sql --auto-approve docker exec -i buzz-pg psql -U buzz -d buzz -v ON_ERROR_STOP=1 < scripts/reconcile-schema-after-pgschema.sql ``` Success: `Changes applied successfully!`. 2. Run the new tests: ```bash export BUZZ_TEST_DATABASE_URL=postgres://buzz:buzz_dev@localhost:5432/buzz BUZZ_TEST_SCHEMA_MODE=desired cargo test -p buzz-relay --lib emoji_entry -- --include-ignored cargo test -p buzz-sdk --lib emoji_entry ``` Success: 4 relay tests and 2 SDK tests pass. 3. Optional, with a local relay (`just relay`) and a local key. Send these through `POST /events` with the SDK or a small script that calls `build_emoji_entry("party_parrot", "https://example.com/p.gif")`. Success: the relay returns `accepted: true`. - Publish a second entry with the same shortcode. Query `POST /query` with `{"kinds":[30180],"#d":["party_parrot"]}`. Success: one event, the newer one. - Query `{"kinds":[30180],"search":"parr","search_mode":"prefix","limit":50}`. Success: the entry is in the result. - Publish an entry that has an `h` tag or an `ftp://` URL. Success: the relay returns `invalid:`. 4. Check your target database before you rely on search (the SQL in "Search index"). Success: the expression does not list an allowlist such as `kind IN (0, 9, 40002, 45001, 45003)`. If it does, search does not find kind 30180 yet. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: bmcanally <bmcanally@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 17:28:05 -06:00
Blake McAnally	171100f308	feat(cli): read-only unread state (NIP-RS) (#8789) ... ## Summary This change adds `buzz channels unread`. The command is read-only. It shows how many unread messages each channel has. It uses the read state that desktop and mobile already sync (NIP-RS). It never publishes an event. It never marks anything read. Output: `[{channel_id, name, last_read_at, unread_count, capped}]`. The most unread channel is first. ## Design The command has two parts. **Part 1: read state** (`commands/read_state.rs`) - The command queries your own `kind:30078` events with the `read-state` tag. - It verifies each signature. It rejects events from other authors. - It keeps the newest version of each `d` tag. - It decrypts each slot with NIP-44 to yourself. It applies desktop's validation rules. - It merges all slots with `max`. - It ignores `ov_*` keys. Desktop and mobile do not read them either. **Part 2: counting** (`commands/channel_unread.rs`) The rules follow desktop's `communityUnreadObserver.ts`. - Without `--channel`, the command uses your non-archived channel memberships. - It reads message kinds 9, 40002, 45001, and 45003. For a DM it also reads huddle starts (48100). - It reads only events after the channel marker (`since = marker + 1`). - Your own messages do not count. - A message is read when its time is at or below the greatest of three markers: the channel marker, its `msg:<id>` marker, and its `thread:<root>` marker. A broadcast reply ignores the thread marker. **Bounds and signals** - At most 200 channels per run. `--limit` sets the number. The maximum is 200. - At most 1000 events per channel. A channel at that cap shows `"capped": true`. Its count is a lower bound. - If you have more channels than `--limit`, the extra channels are not counted. Stderr shows `{"warning":"channels_truncated","counted":N,"dropped":M}`. - More than `--limit` `--channel` values is an error. The command never drops a named channel. - An unknown named channel exits with code 1. **Failure rules** - A slot that cannot be verified, decrypted, or decoded fails the command with exit 4. Desktop skips such a slot. A skip would make the count too high, with no sign of it. - A slot with a newer `v` is skipped. Stderr shows `read_state_slots_skipped`. **Deviations from desktop** - The command has no 7-day fetch horizon. It reads all slots. This can only add markers. - It does not apply desktop's notification gate. That gate uses desktop-local data (followed threads, muted channels). So a count can be higher than the desktop badge. - It does not read mobile's `activity:` markers. Desktop does not read them. The README describes the output and the gotchas. The stable-inventory test lists the new subcommand. This PR does not change `desktop/` (deprecated; see #8840). ## Testing **What ran** - `cargo test -p buzz-cli`: 510 passed. - `cargo clippy -p buzz-cli --all-targets -- -D warnings`: clean. - `cargo fmt --all --check`: clean. - `just file-size-check`: passed. **Unit tests** The tests run the production functions against a fake axum relay. The fake relay applies the real filters (`#h`, `#d`, `#p`, `#t`, `since`, `limit`). - Merge: max across slots, newest version wins, bad values dropped, `ov_*` ignored. - Failures: bad JSON, wrong shape, bad `client_id`, wrong author, bad signature, and undecryptable content all fail. - Counting: own messages, `msg:` markers, `thread:` markers, broadcast replies, archived channels, DM kinds, and sort order. - Bounds: channel limit, event cap, `capped`, and explicit-channel limits. - Query shape: each channel query has `#h`, a bounded `limit`, and `since = marker + 1`. - A bad slot stops the command before any channel query. **Interop evidence** A node script runs the real `buzz` binary against a fake relay. Events are made and encrypted by `nostr-tools`. This is an independent NIP-44 code base. The script compares the CLI with desktop's own TypeScript: - `mergeReadStateEvents` gives the same frontier as `last_read_at` for six channels. - `fetchCommunityUnread` gives the same read/unread result as `unread_count > 0` for each channel. The six channels cover: read, unread, never read, own message only, two slots with max-merge, and a `msg:` marker. The script prints `INTEROP OK`. The script is not part of the repository. **Mutation checks** I broke each guard once and ran the tests. Each break made a test fail: - Own-message exclusion. - `msg:` marker, `thread:` marker, and broadcast rule. - `since = marker + 1`. - `capped` signal and channel limit. - Explicit channels over the limit. - Archived skip and DM kinds. - Max-merge, newest version wins, and the `t` tag requirement. - Signature check, author check, unknown `v`, and the 256-byte key limit. A mutation of the decrypt error path did not fail a test. The JSON step fails the same way, so the break changes nothing. I removed that check from the list. ## Manual test steps You need a relay and an identity that has used desktop or mobile on that relay. 1. Build the CLI: ```bash . ./bin/activate-hermit && cargo build -p buzz-cli ``` 2. Set the identity and the relay: ```bash export BUZZ_RELAY_URL=<relay-url> export BUZZ_PRIVATE_KEY=<your-key> ``` 3. Run the command: ```bash ./target/debug/buzz channels unread | jq . ``` Success: you see one object per channel. Each object has `channel_id`, `name`, `last_read_at`, `unread_count`, and `capped`. 4. Compare with the app. Open desktop or mobile on the same identity. A channel with a badge must show `unread_count` above 0. A channel with no badge must show 0. Counts can be higher than the badge for muted channels and followed threads. This is expected. 5. Read a channel in the app. Wait a few seconds. Run the command again. Success: the count for that channel is 0 and `last_read_at` is higher. 6. Test the channel limit: ```bash ./target/debug/buzz channels unread --limit 1 | jq length ``` Success: the command prints `1`. If you have more than one channel, stderr shows `channels_truncated`. 7. Test an unknown channel: ```bash ./target/debug/buzz channels unread --channel 00000000-0000-4000-8000-000000000000; echo $? ``` Success: the command prints a not-found error and the exit code is `1`. 8. Check that nothing was written. The command sends no events. The app badges stay the same after the run. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: bmcanally <bmcanally@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 17:27:47 -06:00
Blake McAnally	82bc5a47c5	feat(cli): report a message to relay moderators (#8829) ... ## Summary This change adds `buzz messages report`. The command sends a NIP-56 report (kind 1984) to the community moderators. You can report a message with `--event`. You can report a user with `--pubkey`. The `--reason` flag is required. It accepts one of seven values: `illegal`, `nudity`, `malware`, `spam`, `impersonation`, `profanity`, `other`. The `--note` flag is optional. The `--channel` flag is optional. When you give it, the CLI checks that the message is in that channel. The command prints the write envelope `{event_id, accepted, message}` and nothing else. ## Design The relay keeps a report in the moderation queue only. The relay does not store it as a normal event. The relay does not send it to subscribers. So the CLI prints no read-back. The tag shape is the same as desktop's `submitReport`: - Message report: `["p", <author>]` and `["e", <event id>, <reason>]`. - User report: `["p", <pubkey>, <reason>]`. The relay accepts this shape. - The note is trimmed and becomes the event content. The command adds no `h` tag. Desktop sends none. The relay reads the channel from the stored event. The task text mentioned an `h` tag. The real relay and desktop code do not use one. For `--event`, the CLI queries the event by id to learn the author's pubkey. That query uses an `ids` filter with `limit` 1. It names no kinds, because a message can have any kind. The existing `messages edit` and `messages delete` commands use the same lookup. `buzz-sdk` has a new `build_report` builder and a `REPORT_TYPES` list. The list matches the relay's `REPORT_TYPES`. The builder rejects unknown reasons, bad hex, and a note over 64 KiB. The builder allows self-tagging. Without it, the signer drops the `p` tag when you report your own message or key, and the relay rejects the report. Desktop opts in for the same reason. The CLI lowercases `--event` and `--pubkey` before it queries and signs. Clap also rejects unknown reasons, so the usage error appears before any network call. Exit codes: - Missing or unknown `--reason`, bad hex, channel mismatch: 1. - Relay rejection "invalid: report target ... not found" (exact text): 1. The relay gives this answer for a missing message and for a message you cannot read. - Other relay rejections (HTTP 400): 2. - HTTP 401 or 403: 3. The CLI README describes the new command. ## Testing What ran: - `cargo test -p buzz-cli -p buzz-sdk`: 510 CLI tests and 327 SDK tests pass. - `cargo clippy -p buzz-cli -p buzz-sdk --all-targets -- -D warnings`: clean. - `cargo fmt --all --check`: clean. - `just file-size-check`: passes. This PR does not change `desktop/` (deprecated; see #8840), so I did not run the Tauri crate tests. The CLI tests call the production `cmd_report_message` function against a local fake relay. They check the exact tags, the user-report shape, self-reports (the signer is the reported author), uppercase ids, argument parsing, the channel check, the not-found path, and the exit code for each relay rejection. Interop evidence: - I ran the real `buzz` binary against a local node HTTP server that stands in for a relay. - In the same node process, I ran desktop's own `submitReport` from `desktop/src/shared/api/moderation.ts` with a mocked signer. - Kind, tags, and content are equal for `spam` with a note and for `other` without a note. - The script prints `INTEROP OK`. - No shared relay was used. Mutation checks (break the guard, see a test fail, restore): - SDK reason check removed: fails. - SDK `e` tag without the reason: fails. - SDK `p` tag changed: fails. - SDK note trim removed: fails. - SDK note size cap removed: fails. - CLI channel check removed: fails. - CLI not-found mapping removed: fails. - CLI author read from the wrong field: fails. - CLI event-id hex check removed: fails. - SDK self-tagging removed: fails in the SDK test and in the CLI self-report test. - CLI lowercasing removed: fails. - Not-found mapping widened to any `invalid:` 400: fails. - Clap reason value parser removed: fails. - Clap required-target group removed: fails. - Clap `--channel` conflict with `--pubkey` removed: fails. One CLI `--pubkey` hex check was redundant with the builder. I removed it instead of keeping a guard no test can catch. ## Manual test steps You need a local relay and a running desktop or a second account to see the queue. Do not use a shared relay. 1. Start a local relay with `just relay`. 2. Send a message: `buzz messages send --channel <uuid> --content "test"`. Copy the event id. 3. Report it: `buzz messages report --channel <uuid> --event <event-id> --reason spam --note "test report"`. Success: the output is `{"event_id":"...","accepted":true,"message":""}` and the exit code is 0. 4. As a moderator, run `buzz moderation reports`. Success: the report appears with type `spam` and your note. 5. Run `buzz messages get --channel <uuid>`. Success: the report does not appear in the channel. 6. Report a user: `buzz messages report --pubkey <hex> --reason impersonation`. Success: `accepted` is true and the report appears in the queue. 7. Run `buzz messages report --event <event-id> --reason bogus`. Success: exit code 1 and a `user_error` message. 8. Run `buzz messages report --event <64 zeros> --reason spam`. Success: exit code 1 and a not-found message. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: bmcanally <bmcanally@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 17:27:33 -06:00
Luke Tornquist	0ada1d3707	buzz-db: read explicit #h sets one channel at a time (#9110) ... ## Summary Explicit multi-channel `#h` reads whose channels × page limit fit a budget of 8,192 now read each channel through the channel index and keep the newest `n` rows overall. Before, Postgres was left to plan `channel_id = ANY($set) ORDER BY created_at DESC LIMIT n` from a row estimate. Stacked on #9091 (both change `build_query_events_sql`). It retargets to `main` when that merges. ## Problem The desktop app's cold unread repair reads every joined channel in chunks, with `kinds [9, 40002, 40008]`, `include_aux`, `limit 500` and no `since`. It never fills its page. For that shape, Postgres's custom plans (the first executions of each prepared statement on a connection) overestimate how many rows a quiet channel set matches. They walk the partition's primary key backward and test every community row against the set. The inflated cost also turns on JIT. On a load-test database with about 940k events in one partition (`EXPLAIN ANALYZE`, custom plan): | Channels in the set | Plan | Rows returned | Time | |---|---|---|---| | 3 | channel index | 4 | 0.5 ms | | 6 | channel index, plus JIT | 254 | 54 ms (1.2 ms scan) | | 15 | pkey walk, 940k rows filtered | 427 | 546 ms | | 48 | pkey walk, 299k rows filtered | 863 | 197 ms | | 128 | channel index | 500 | 5 ms | Generic plans pick the channel index, but every connection runs custom plans first. A client-side `since` would not avoid the walk: it still walks the whole window. It would also change unread semantics, and it does nothing for app versions already shipped. ## Change `build_query_events_sql` handles plain explicit `#h` reads like this: ```sql SELECT e.* FROM (SELECT DISTINCT unnest($set::uuid[]) AS ch) c CROSS JOIN LATERAL ( SELECT … FROM events WHERE community_id = $1 … AND channel_id = c.ch … ORDER BY created_at DESC, id ASC LIMIT $n ) e ORDER BY e.created_at DESC, e.id ASC LIMIT $n OFFSET 0 ``` - **Same page as the set read:** every row of the overall top `n` is also in its own channel's top `n`. - **Bounded candidates:** at most distinct channels × `n` rows reach the outer sort, whatever the statistics say. How many index entries each channel scans still depends on the other predicates, such as `kinds`. - **Same SQL for any number of channels:** the channel set stays one array bind. - **Repeated ids:** `DISTINCT` reads a repeated id once. - **Per pass:** #9091's recent and older passes each use this shape. These keep today's SQL: scopes that include global events (`channel_id IS NULL OR ANY`), `#p` mention joins, e-tag fences, offset pages, the COUNT, and sets whose channels × page limit exceed the budget. ### Candidate budget A per-channel read fetches up to `n` rows from every channel, where the set read stops once the page fills. So busy channel sets cost more. On a synthetic community (920k events; 128 busy channels with 1,000 recent messages and 1,000 reactions each, 400 more busy channels, 128 quiet channels with 3 messages each), the repair shape (`kinds [9, 40002, 40008]`, recent pass), JIT off: | Set × limit | Set read, custom plan | Set read, generic plan | Per-channel | |---|---|---|---| | 128 busy × 500 | 2.3 ms | 238 ms | 114 ms | | 20 busy + 108 quiet × 500 | 19 ms | 33 ms | 18 ms | | 128 quiet × 500 | 473 ms | 0.7 ms | 1.3 ms | | 15 busy × 500 | 22 ms | 23 ms | 13 ms | | 15 quiet × 500 | 0.2 ms | 0.2 ms | 0.3 ms | | 128 busy × 100 | 0.7 ms | | 17 ms | | 128 quiet × 100 | 141 ms | | 1.2 ms | With JIT on (the default), the per-channel custom plan for 128 channels × 500 crossed the JIT thresholds: about 0.9 s for the busy set and 0.75 s for the quiet one, nearly all compile time. At 15 × 500 it stayed under them. So the per-channel read runs only while channels × page limit ≤ 8,192 (`PER_CHANNEL_CANDIDATE_BUDGET`, the clamped page limit). That bounds it at about 8k candidate rows, where it was still faster than the set read here. Larger sets keep the set read, as on `main`. Either way there are two SQL texts, so plan caching holds. The desktop repair's full 128-channel chunks at `limit 500` (64,000) are over the budget and keep the set read. Only its last, partial chunk (16 channels or fewer) and other small explicit sets read per channel. The load-rig numbers below predate the budget and don't show its effect. On the same data, the per-channel form took 0.5 ms (6 channels), 0.9 ms (15) and 4.2 ms (128). Load rig, cold desktop reconnect scenario (every client runs the unread repair), #9091 alone vs this PR on top: | | Pool default: #9091 | Pool default: + this PR | Pool 200: #9091 | Pool 200: + this PR | |---|---|---|---|---| | Repair read mean | 302 ms | 41 ms | 1,990 ms | 93 ms | | Repair read total DB time | 1,547 s | 245 s | 9,622 s | 561 s | | Repair reads served | 5,121 | 5,915 | 4,834 | 6,011 | | Client op failures | 23,475 | 10,966 | 28,797 | 10,829 | | Unread-load errors | 1,018 | 113 | 1,434 | 3 | | Relay pool-timeout log lines | 8,762 | 824 | 13,860 | 409 | The rig stays overloaded in every arm. Most remaining failures are the per-connection concurrent-request limit and connect/auth under load, not this read. ## Tests - `per_channel_reads_match_the_set_read` (Postgres) covers: - every limit from 1 to total+1, including a page held entirely by one channel - newer rows of another kind or another channel taking no slot - a repeated channel id and an id tie across channels - keyset paging across the tie, and `since` - rows below the recent-pass floor - a prepared-statement check that the read ran per channel and no `ANY` read ran - `per_channel_scope_eligibility` covers which reads keep the current SQL, including the budget edge (16 × 512 in, 17 × 512 and 128 × 500 out) and the clamped page limit. - `channel_scope_sql_does_not_depend_on_channel_count` now also asserts the per-channel shape for explicit `#h` pages. ## What got simpler Not much. This adds one SQL shape to the events query builder, behind one eligibility function and one budget. In exchange, the planner's row estimate no longer decides whether a small multi-channel read scans a whole partition. The channel predicate is decided in one place: `push_event_filters` no longer pushes the channel scope, and each caller (the page read and the thread route) pushes its own. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 18:25:56 -05:00
Blake McAnally	198c4c3e9b	feat(cli): list huddles and their participants (#8830) ... ## Summary This change adds two read-only commands to the Buzz CLI. - `buzz huddles list --channel <uuid> [--active] [--limit n]` lists the huddles of a channel. - `buzz huddles participants --channel <uuid> --huddle <id>` shows who is in a huddle now. The commands do not join, start, or end a huddle. They do not touch audio. They do not write to the relay. ## Design The CLI rebuilds huddle state from relay-signed lifecycle events (kinds 48100 to 48103). It uses a Rust port of desktop's `HuddlePresenceTracker`. The CLI and the app therefore agree on rosters. The port follows these desktop rules: - Only the relay can sign a join or a leave. - Only the relay or the huddle creator can end a huddle. - Events replay in desktop's order: time, then start, join/leave, end, then relay roster revision, then event id. - Each admission has an ID. A late leave for an old admission does not remove a rejoined participant. - A join from a newer generation replaces the older roster. A leave from an older generation is ignored. A huddle has one of three states: - `active`: no end event, and the start is less than one hour old. - `ended`: a valid end event exists. - `stale`: no end event, and the start is more than one hour old. Desktop shows this state as ended. The CLI keeps it separate because no end event proves it. Every query names the four lifecycle kinds and the `#h` channel. A command scans at most 5000 newest events. The output always has `complete`, `truncated`, `events_scanned`, and `event_bound`. - If the huddle start is inside the scan, the roster is exact. `complete` is `true`. - If the start is not inside the scan, `participants` returns `state: "unknown"` and `complete: false`. It does not return a partial roster. - A huddle that does not exist, with no truncation, returns exit code 1. Deviations from desktop and from CLI convention: - The HTTP `/query` endpoint does not serve liveness events (kind 48104). Only the WebSocket REQ path creates them. The CLI cannot check liveness. Every output has `liveness_checked: false`. Generations are never authoritative in the CLI. - Desktop keeps the roster in a Set, so its order is not defined. The CLI sorts by join time, then by pubkey. - Desktop also adds backing-channel members as a fallback roster. The CLI does not. It reports only what the lifecycle events show. - Desktop's roster view lists the creator from the start event. The CLI lists the creator only after the relay records their join. Desktop's roster view also uses a second event comparator. The CLI uses the presence tracker's comparator. - The commands return one JSON object, not a bare array. The object needs room for the truncation fields. Silent truncation would break Rule 1. The change also updates `crates/buzz-cli/README.md`. This PR does not change `desktop/` (deprecated; see #8840). ## Testing What ran: - `cargo test -p buzz-cli -p buzz-sdk`: 518 and 322 tests passed. - `cargo clippy -p buzz-cli -p buzz-sdk --all-targets -- -D warnings`: clean. - `cargo fmt --all --check`: clean. - `just file-size-check`: clean. The new tests call the production functions. A test also checks exit codes for relay-info failures: a 500 and an unreachable relay exit 2, and a 403 exits 3. They parse arguments, build output from events, and run `run_list` and `run_participants` against a local fake relay. The fake relay serves NIP-11 and `/query` with the same cursor paging as the real bridge. The tests cover rejoin, leave without join, ended huddle, stale generation, bad end signer, forged join, same-second ordering, and the event bound. Interop evidence: - `crates/buzz-cli/tests/fixtures/huddle_sequences.json` holds 14 recorded event sequences. - The Rust tests check each sequence against the expected roster and ended state. - `crates/buzz-cli/tests/huddle_interop.mjs` replays the same file through desktop's own `HuddlePresenceTracker` under node. All 14 sequences pass with `INTEROP OK`. - Run it with: `node --import ./desktop/test-loader.mjs --experimental-strip-types crates/buzz-cli/tests/huddle_interop.mjs` Mutation checks. I broke each guard once and confirmed a test failed. I then restored the code. - Relay-only author check on join and leave. - End signer check. - Ignore events after an end. - Roster revision comparison. - Ignore a leave from an older generation. - One-hour stale window. - Tie rule for the oldest scanned second. - Unknown state on truncation. - The truncation flag. - Kinds in the query filter. - `--active` filter. - Required relay `self` pubkey. All 12 mutations failed at least one test. A 13th mutation, which wraps relay-info errors as `Other`, also fails a test. No live relay was used. No event was written. ## Manual test steps You need a relay URL, a key, and a channel UUID where someone has run a huddle. 1. Build the CLI: `cargo build --release -p buzz-cli` 2. Set `BUZZ_RELAY_URL` and `BUZZ_PRIVATE_KEY`. 3. Run `./target/release/buzz huddles list --channel <uuid>`. - Success: the output is one JSON object with a `huddles` array. Newest huddle is first. Each huddle has `state`, `started_at`, and `ended_at`. `complete` is `true`. 4. Start a huddle in the desktop app. Run `./target/release/buzz huddles list --channel <uuid> --active`. - Success: the new huddle appears with `state: "active"`. 5. Copy its `huddle` value. Run `./target/release/buzz huddles participants --channel <uuid> --huddle <huddle-id>`. - Success: `participants` lists the same people as the desktop huddle window. 6. Have one person leave, then run the command again. - Success: that person is gone from `participants`. 7. End the huddle. Run the `participants` command again. - Success: `state` is `ended`, `ended_at` has a time, and `participants` is empty. 8. Run the `participants` command with a random UUID as `--huddle`. - Success: the command exits with code 1 and prints a not-found error. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: bmcanally <bmcanally@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 17:20:21 -06:00
Matt Toohey	4c7c15a9c2	fix(db): end each table's maintenance transaction before the next (#9096) ... ## Summary Partition maintenance runs one transaction per table in a pass. When a table's attempt failed on the server, the transaction was dropped as soon as the error arrived, which can be before the backend has aborted and released its schema-scoped advisory lock. The next table could then find the lock held and report `skipped_locked`, which is the `parent_contention_times_out_without_blocking_the_other_table` flake and a mis-reported outcome in production. `apply_change` now owns the transaction and commits or rolls it back before it returns, so the next table starts only after the previous one's locks are free. ### fix(db): finish each table's maintenance transaction before the next starts #### Problem `store::partition::maintenance::tests::postgres_tests::parent_contention_times_out_without_blocking_the_other_table` failed once on main at maintenance_tests.rs:480 (`assert_advanced`). `events` reported `lock_timeout` as expected, but `delivery_log` was left on its old catch-all with no error. - main: https://github.com/block/buzz/actions/runs/41669713108/job/126035790322 Production has the same race. When one table's attempt fails on the server (lock_timeout, statement_timeout, deadlock), the next table in the same pass can report `skipped_locked` and skip its DDL. The next interval repairs it, but that pass reports the wrong outcome and metric. #### Cause `delivery_log` can only stay un-advanced without an error through `skipped_locked`: its `pg_try_advisory_xact_lock` returned false. The key is per schema, so the holder was `events`'s own failed transaction. PostgreSQL sends a statement's ErrorResponse (`EmitErrorReport` flushes it) before the backend calls `AbortCurrentTransaction`, which releases its locks (postgres.c, PG 16 and 17). sqlx returns the error as soon as the ErrorResponse arrives. `apply_change`'s `?` then dropped the transaction without waiting for that backend again, and `maintain_audited` started `delivery_log` on another pooled connection. The abort normally finishes within microseconds. If the `events` backend is descheduled in that gap, `delivery_log`'s probe finds the lock still held. The first suspect was that a dropped transaction only queues its ROLLBACK, which the pool's return task flushes later. That does not explain this failure. A backend left "idle in transaction (aborted)" already holds no locks, and keeping the queued ROLLBACK unflushed for 500 ms did not reproduce it. It does apply to client-side errors, where the transaction is still live on the server. #### Fix `apply_change` now owns the transaction. It begins it, runs the attempt in a new `attempt_change`, commits a `Committed` result and rolls back every other result, errors included. The early-return arms no longer roll back by hand. ROLLBACK first drains the failed statement's ReadyForQuery, which the backend sends only after its abort, so `apply_change` returns only once the lock is free. If the rollback fails, it is logged and the original error is returned. The test is unchanged. #### Validation Local Linux workstation, PostgreSQL 17.11 (docker compose): - Deterministic: gdb stopped the `events` backend at `AbortCurrentTransaction`, after its error reached the client, for 1 s. Without the fix: 6/6 failed at :480 with `delivery_log` reporting `skipped_locked`, the CI signature. With the fix: 6/6 passed. - Natural contention: 24 parallel loops of the test, with Postgres pinned to 2 CPUs shared with 4 busy loops. Without the fix: 6/1440 failed, all at :480 with `delivery_log` `skipped_locked`. With the fix: 0/1440. - `cargo test -p buzz-db --lib store::partition -- --include-ignored --test-threads=1`: 77/77 passed. Run in parallel against one shared local database, another test's database-wide event trigger also fails `cluster_global_parent_readers_follow_monthly_creation_and_catch_all_advancement` on main. CI gives each test its own database. - The test alone: 30/30 passed. `cargo test -p buzz-db --lib`, `cargo clippy -p buzz-db --all-targets -- -D warnings` and `cargo fmt --check` are clean. No regression test was added. Only a debugger can hold the window open, and the only remaining client-side error that leaves the transaction live is a concurrent-drop audit error, itself timing-dependent. **Rebased onto origin/main `64d12e297` (this branch):** Linux workstation, private `postgres:16` container with the desired-state schema applied as in the PostgreSQL lane (pgschema plus the reconcile script). - `cargo test -p buzz-db`: 191 lib, 21 `observability_source` and 5 doc tests passed. The 673 Postgres-backed tests are `#[ignore]`d there. - `cargo test -p buzz-db --lib store::partition -- --ignored --test-threads=1`: 56/56 Postgres-backed partition tests passed. The other 21 of the 77 counted above aren't Postgres-backed and ran with the lib tests. - `parent_contention_times_out_without_blocking_the_other_table` alone: 20/20 passed. - `cargo clippy -p buzz-db --all-targets -- -D warnings`, `cargo fmt -p buzz-db -- --check` and `git diff --check` are clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: mtoohey <mtoohey@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 09:49:06 +11:00
Matt Toohey	1a42b5c39c	fix(relay): read kind:39000/39001 inputs behind the replacement lock (#9035) ... ## Problem `test_nip29_announce_channel_wire` flaked in CI (9/30 local solo runs failed). `emit_group_discovery_events` read the channel, its members and the latest live `created_at` *before* taking the coordinate's replacement lock, and only locked for the write. Two publishers for one channel could therefore store in the reverse order of their reads, or land on the same second where NIP-16's lower-id tie-break picks a random winner. Either way a kind:39000/39001 built from stale state could stay live. In the test, the relay owner reconnects to send an `auto_join=false` edit, gets auto-joined into the channel, and the join's kind:39000 — built from the old `auto_join=true` state — replaces the edit's. ## Fix - **buzz-db:** `lock_group_state_snapshot` takes the replacement lock first, then reads the channel, the members the kind actually publishes (full roster for a DM's kind:39000, owners/admins for kind:39001, none otherwise) and the latest timestamp; `LockedGroupStateSnapshot::replace` stores on the same transaction. This is the fence kind:39002 already had via `LockedMemberSnapshot`. The lock is taken once per write (`take_replacement_lock`), and the write keeps its `ReplaceAddressableEvent` transaction timer and writer population so metrics stay continuous. - **buzz-relay:** `emit_addressable_discovery_event` builds tags and `created_at` (live + 1s) from that snapshot. ## Tests - buzz-db: `group_state_snapshot_reads_behind_the_replacement_lock`, `group_state_snapshot_reads_the_members_its_kind_publishes`, `group_state_replacement_takes_its_lock_once`. - buzz-relay: `a_publisher_that_read_before_an_edit_cannot_replace_it`, with `BeforeLock` and `AfterRead` legs driven by new `nip_fi_test_hooks`. - Every new assertion was confirmed to fail when its guard is reverted. - `test_nip29_announce_channel_wire` solo against a relay built from this tree: 30/30 passed (9/30 failed before). Merged with current `main`; the only conflict kept main's bounded `channel state write failed: {kind}` error mapping on the new snapshot write. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-11-06 09:34:33 +11:00
Matt Toohey	580caca667	fix(db): unlock the storage-accounting lease before its session closes (#9097) ... ## Summary `StorageAccountingLeader` holds a session-level advisory lock on a detached connection. Dropping it only closes the socket, and PostgreSQL frees the lock when the backend exits, which can be after the next lock attempt. `release(self)` now unlocks on the leader's own session before closing it. The relay storage-sweep test releases its worker between iterations instead of dropping it, which fixes the `worker_publication_activates_reader_without_restart_or_mode_change` flake. The follow-up commit makes `release` report success only when the unlock actually ran on the session that held the lease. ### fix(db): unlock the storage-accounting lease before its session closes #### Problem The PostgreSQL lane's `storage_sweep::postgres_tests::worker_publication_activates_reader_without_restart_or_mode_change` failed with "called `Option::unwrap()` on a `None` value" at crates/buzz-relay/src/storage_sweep/postgres_tests.rs:66:74 after 5.53 s, on the unrelated branch elrond/cli-credential-chain: - https://github.com/block/buzz/actions/runs/41730818977/job/126253029211 #### Cause Line 66 is `db.try_lock_storage_accounting().await.unwrap().unwrap()` inside the loop over three metrics modes. `Ok(None)` means `pg_try_advisory_lock` returned false. A passing run takes about 15 s, roughly 5 s per iteration, so the failure is the start of iteration 2. Each test process gets its own database and advisory locks are per-database, so the holder is iteration 1's own worker. `StorageAccountingLeader` holds a session-level advisory lock on a detached connection. The test ended each iteration with `drop(worker)`, which only closes the socket. PostgreSQL frees session locks during backend exit, after it notices EOF, so the lock can outlive the drop on a loaded server. Iteration 2's lock attempt raced that exit. The db-crate test `lost_lock_session_cannot_overwrite_successor_snapshot` already polls for this release. The relay test didn't. #### Fix Add `StorageAccountingLeader::release(self)`. It runs `pg_advisory_unlock` on the leader's own session and then closes it, so the lease is free when it returns. This follows the unlock-then-close pattern in `migration.rs`. The relay test now releases the worker at the end of each iteration instead of dropping it. No assertions, timeouts or retries change. buzz-admin's `storage-snapshot` is left as is. It runs once per pod, and its CronJob uses `concurrencyPolicy: Forbid`, `backoffLimit: 0` and `restartPolicy: Never`, so the next lock attempt comes from a later scheduled run, not from the same process. `released_lease_admits_successor_before_old_session_exits` covers the new method. A closing backend drops its temp tables before it frees session locks. So the test locks a temp table owned by the leader's session from another transaction, which keeps the old backend alive after release. It then asserts that a fresh `try_lock` succeeds at once and that the old backend has not exited. Open PRs #9040 (edits this relay test and keeps `drop(worker)`) and #9043 (changes the `sqlx` import in storage_accounting.rs) will see small textual conflicts. #### Validation Local Linux workstation, throwaway `postgres:16-alpine` (CI uses `postgres:16`), desired-state schema applied as in the PostgreSQL lane: - Adversarial ordering, using temporary, uncommitted changes: the leader's connection was made public and the relay test held the worker backend's exit for 1 s with the same temp-table lock. Without the fix: 15/15 failed at postgres_tests.rs:66:74 after 5.1-5.3 s, the CI signature. With the fix: 10/10 passed. - Natural contention: Postgres, the test processes and busy loops were pinned to 4 CPUs. With 8 parallel runs and 4 busy loops, the unmodified test passed 24/24 without the fix. With 16 parallel runs and 16 busy loops, it passed 48/48 without the fix and 48/48 with it. Load alone did not open the window locally. - New db test: 50/50 passed. With the unlock removed from `release`, leaving only `close`, it fails with "a released lease must admit a successor at once". All four storage_accounting Postgres tests pass. - `cargo fmt --all -- --check`, `cargo clippy -p buzz-db -p buzz-relay -p buzz-admin --all-targets -- -D warnings`, `just file-size-check` and `git diff --check` are clean. A green run doesn't prove the flake is gone. Hosted CI covers broad validation. **Rebased onto origin/main `64d12e297` (this branch):** the relay test, run in a fresh database cloned from the desired-state template each time, passed 8/8 at the branch head, which includes this commit. The full rebased results are under the next commit. #### Rebase note #9040, which this commit's message lists as an open PR, has since merged to main. It removed the per-community storage metric, including the `buzz_community_storage_bytes` assertion just above `drop(worker)` in `crates/buzz-relay/src/storage_sweep/postgres_tests.rs`. The cherry-pick onto current main conflicted on that hunk. The resolution keeps main's version of the test and changes only `drop(worker)` to `worker.release().await.expect("release worker lease")` with its comment. Nothing else in the commit changed, and no semantic adjustment was needed. The commit message is unchanged, so it still describes #9040 as open. ### fix(db): report a storage-accounting release only once its unlock succeeded #### Problem Second review of the branch (nit) on 51c27212a ("fix(db): unlock the storage-accounting lease before its session closes"), `StorageAccountingLeader::release` in storage_accounting.rs. That commit cites the unlock-then-close pattern in runtime/migration.rs but handles errors differently: - A failed unlock returned early with `?`, skipping the graceful close. The dropped connection then closed only the socket. - A failed close after a successful unlock was returned as an error, although the lease was already free. - `pg_advisory_unlock` returning false, which means this session didn't hold the lock, was ignored and reported as success. So the doc comment's "Once this returns, the lease is already free" did not hold for every `Ok`. #### Cause `release` propagated the unlock's error with `?` before the close, propagated the close's error after a successful unlock, and ignored the boolean that `pg_advisory_unlock` returns. So a failed unlock skipped the graceful close, a failed close was reported although the lease was already free, and an unlock that returned false because this session no longer held the lock was reported as success. #### Fix Follow migration.rs's `with_schema_destruction_lock`. Run the unlock as `query_scalar::<bool>` and keep its result, always attempt the close and ignore its error, then return the unlock's error, or `InvalidData` if the unlock returned false. The doc comment now says that `Ok` means the unlock ran on the lease's own session, so the lease is free even if the close failed, and that an error means the unlock failed or the session no longer held the lease. `released_lease_admits_successor_before_old_session_exits` gains a final step: it unlocks the successor's lease on the successor's own session, then expects `release` to fail. Its successor check is now an `expect` with the same message. #### Validation Local Linux workstation, the `postgres:16` container used earlier on this branch, `BUZZ_TEST_DATABASE_URL` pointing at it: - `cargo test -p buzz-db --lib storage_accounting -- --ignored`: 4/4 storage_accounting Postgres tests pass. - With the unlock's boolean ignored, as before this change, the new step fails with "releasing a lease the session lost must fail". - `cargo fmt -p buzz-db -- --check`, `cargo clippy -p buzz-db --all-targets -- -D warnings` and `git diff --check` are clean. **Rebased onto origin/main `64d12e297` (this branch, both commits):** Linux workstation, private `postgres:16` container with the desired-state schema applied as in the PostgreSQL lane (pgschema plus the reconcile script). - `cargo test -p buzz-db`: 191 lib, 21 `observability_source` and 5 doc tests passed. The 674 Postgres-backed tests are `#[ignore]`d there. - `cargo test -p buzz-db --lib storage_accounting -- --ignored --test-threads=1`: 4/4 passed. `released_lease_admits_successor_before_old_session_exits` alone: 20/20 passed. - `storage_sweep::postgres_tests::worker_publication_activates_reader_without_restart_or_mode_change`, each run in a fresh database cloned from the desired-state template: 8/8 passed. - `cargo clippy -p buzz-db --all-targets -- -D warnings`, `cargo clippy -p buzz-relay --all-targets -- -D warnings`, `cargo fmt -p buzz-db -p buzz-relay -- --check` and `git diff --check` are clean. ## Coordination - #9043 (draft, "feat(relay): serve storage usage as an operator pull feed") edits `crates/buzz-db/src/store/storage_accounting.rs`. `git merge-tree` against its current head shows two small textual conflicts. First, both sides rewrite the `sqlx` import to the same `use sqlx::{postgres::PgConnection, Connection};`, and #9043 also drops `use uuid::Uuid;`. Second, #9043 replaces `usage_delivery_reads_snapshot_time_and_community_states` at the point where this branch adds `released_lease_admits_successor_before_old_session_exits`. Keeping both sides resolves each one. - #7899 also touches `storage_accounting.rs`. It already conflicts with main, and this branch adds no new conflicted file to it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: mtoohey <mtoohey@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 09:28:55 +11:00
Matt Toohey	a22ff133b4	test(relay,db): remove three timing races from Postgres and E2E tests (#9098) ... ## Summary Three tests asserted on outcomes that depended on how fast something unordered happened. The first depended on dial latency inside a 150 ms acquire budget, the second on whether two aux-page replies landed in one poll, and the third on whether NIP-42 admission auto-joined the joiner before the invite claim did. Each test now sets up the state it asserts before the racing actor can run: it fills the reader warm floor first, pauses the writer's proxy after the first aux page, or claims the invite before the joiner connects. Only tests change. No assertions are loosened, and no sleeps, retries or timeouts are added. ### test(db): fill the reader warm floor outside the acquire budget first #### Problem `reader_warm_at_production_budget_ends_partial_when_dials_stall` failed on main at reader_pool_postgres_tests.rs:177, "one partial wave, then one wave that opens nothing": left (15, 0, 5), right (15, 3, 7). Postgres lanes run with no retries. - main (d3a4909cc): - https://github.com/block/buzz/actions/runs/41693681008/job/126127000284 The test is unchanged on current main. #### Cause The test builds the reader pool with the production constructor, so every warm acquire, dial included, gets the 150 ms `READER_ACQUIRE_TIMEOUT`. It relayed the proxy's first 3 sessions and stalled the rest, so it needed the first wave's 3 relayed dials (TCP through the proxy, backend fork, SCRAM, `after_connect`) to finish inside that budget. On a loaded runner (4 vCPUs, nextest test-threads=8) none did. All 5 first-wave acquires missed, and the pass ended after one empty wave as the product should, which gives (15, 0, 5). The warm pass was correct, but the test's outcome depended on dial latency. The session counts had a second race. The proxy accepts connections on the test's current-thread runtime. If that thread is descheduled past the budget, the timed-out acquires and the listener's readiness are handled in the same tick. So `proxy.sessions() == 10`, checked right after the pass, could miss a stalled dial still waiting to be accepted. #### Fix Fill the warm floor before stalling anything, as boot does. `Db::read_pool_boot_ping_once` closes its probe connection, and sqlx's floor maintenance then dials the 3 floor connections under its own 300 s deadline, outside the acquire budget. The test waits for them with `await_idle_floor`, then stalls every session from the recorded baseline (`proxy.stall_from(proxy.sessions())`) instead of a hardcoded 3. Wave 1 now holds the 3 idle connections without dialing, and its 2 dials stall. Wave 2's 5 dials stall, so it makes no progress and the pass ends. That gives (15, 3, 7) at any dial latency. The session count is now relative to the baseline and asserted once, after the existing settle. That single check covers both "the pass stops dialing" and "nothing re-dials". The report assertion, the `elapsed < 2s` bound and the production pool are unchanged. Only the test changes. The setup still needs one of the boot ping's five attempts (spaced over about 4 s by `READER_BOOT_PROBE_BACKOFF`) to dial within 150 ms, the same exposure as `reader_boot_ping_warms_pool_to_min_connections`. Before, all 3 concurrent first-wave dials had to make it on their first try. #### Validation Local Linux workstation, on a private postgres:17-alpine. The test processes and Postgres shared 4 pinned CPUs, with 8 parallel workers. "Before" is the test as on main. - CPU stress, no busy loops: before, 2/40 failed with (15, 0, 5) at :177, the CI signature. After, 120/120 passed. - Plus 4 busy loops on those CPUs: before, 80/80 failed with (15, 0, 5). After, 80/80 passed. In 80 more instrumented runs (all passed), the boot ping retried 16 times: 15 runs needed 2 attempts and 1 needed 3. - Adversarial latency, using a temporary, uncommitted proxy hook that delays chosen sessions' first server reply by 200 ms (past the budget), 10 runs each: - every relayed session slow: before, 10/10 (15, 0, 5). After, 10/10 failed at "boot ping must reach the reader", the bound above. - the first 4 sessions slow: before, 10/10 (15, 0, 5). After, 10/10 passed, with the ping succeeding on its 5th attempt. - every session but the first slow: before, 10/10 (15, 1, 9). After, 10/10 passed. - Production mutations, each of which fails the test: dropping the no-progress break gives (15, 3, 27). A 2 s acquire budget fails the elapsed bound at 4.0 s. Closing the warmed connections instead of releasing them leaves the floor at 0 idle. - All 7 `reader_pool_postgres_tests` pass. `cargo fmt --check`, `cargo clippy -p buzz-db --all-targets -- -D warnings` and `git diff --check` are clean. A green run doesn't prove the flake is gone. Hosted CI covers broad validation. **Rebased onto origin/main `64d12e297` (this branch):** Linux workstation, private `postgres:16` container with the desired-state schema applied as in the PostgreSQL lane (pgschema plus the reconcile script). - `cargo test -p buzz-db`: 191 lib, 21 `observability_source` and 5 doc tests passed. - `cargo test -p buzz-db --lib reader_pool_postgres_tests -- --ignored --test-threads=1`: 7/7 passed. `reader_warm_at_production_budget_ends_partial_when_dials_stall` alone: 20/20 passed. - `cargo clippy -p buzz-db -p buzz-relay -p buzz-test-client --all-targets -- -D warnings`, `cargo fmt -p buzz-db -p buzz-relay -p buzz-test-client -- --check` and `git diff --check` are clean. ### test(relay): hold the writer's second aux page too #### Problem `thread_window_batch_discards_earlier_windows_after_revocation` failed on main at failure_postgres_tests.rs:55, "barrier must precede closure completion": left 2, right 1. Postgres lanes run with no retries. - main (d3a4909cc): - https://github.com/block/buzz/actions/runs/41693681008/job/126127000284 The test and the code it covers are unchanged on current main. #### Cause This is the race #8740 (322fdbf08) fixed for the replica test only. `pause_after_aux_page` polls the request until a `thread_window_aux` span closes, then expects exactly one closed. `query` runs the closure's two aux hops (reactions and edits, then their deletions) back to back, one span each. On the writer, each hop is a pool acquire with its idle ping, BEGIN, two SET LOCALs, the SELECT and ROLLBACK. #8740 left the writer callers on `None`, so only the request returning Pending on a reply that had not arrived yet held hop 2. sqlx tries each read before it waits for readiness, and tokio leaves the socket marked readable after the previous reply, so a reply Postgres has already sent is consumed in the same poll. If each of hop 2's replies lands before the test thread reads it, for example when a loaded runner deschedules that thread after its writes, hop 2 closes in the poll that closed hop 1 and the barrier counts 2. `assert_aux_access_change`, behind the grant and revocation tests, uses the same `None` path. #### Fix `proxied_state` gives the paused request a copy of the fixture's state whose production writer connects through a `PausingProxy`. `production_db_at` is `production_db` with a URL override. `AuxPages` pauses the proxy when hop 1's span closes. That runs inside the request's poll, before hop 2's first bytes are written, so hop 2 stays pending however fast Postgres answers. Each test resumes the proxy after its access change, just before `request.await`. Only the request uses the proxied state: `assert_aux_access_change` reads through the fixture's direct `state.db` while the request is held, and through the proxy that read would deadlock. No caller passes `None` any more, so the `Option` goes. Only tests change. No assertions are loosened, and no sleeps, retries or timeouts are added. Draft #8991 (BUZZ-512) also edits `AuxPages` and `pause_after_aux_page`, adding a waker for its detached HTTP handler. It also moves these two callers onto a proxy, through its own `proxy_request_pool`, which replaces `f.state.db` with a raw pool. Rebasing it onto this keeps its waker on the non-`Option` proxy field and can use `proxied_state` for those two callers. #### Validation Local Linux workstation, Postgres 17 in docker, one test per process as nextest runs them. "Before" is the test as on main. - Adversarial ordering, using a temporary, uncommitted change as in #8740: after the barrier, ten more polls of the request over 500 ms, then the barrier's count assertion. Before, the batch, grant and revocation tests failed 30/30 at that assertion with left 2, right 1, the CI signature, and the replica test passed 10/10. After, 40/40 passed. - Natural contention: the three tests and a private postgres:17-alpine shared 2 pinned CPUs with 4 busy loops, 3 processes at a time. Before, 450/450 passed, so the same-poll burst did not occur locally. After, 450/450 passed. - Production mutation: skipping `query_batch`'s final implicit access-set check fails all three tests with 200 instead of 503. - All 19 `api::bridge::thread_window` Postgres tests pass, run serially against one database. `cargo fmt --check`, `cargo clippy -p buzz-relay --all-targets -- -D warnings` (1.98.1), `git diff --check` and `check-postgres-test-discovery.py` are clean. Running several of these tests in one libtest process can fail them at :41 with no aux span seen, on main too. tracing-core builds a new callsite's interest from the registering thread's default while only one scoped dispatcher exists. nextest runs one test per process, so CI is not exposed. The runs above use one process per test. A green run doesn't prove the flake is gone. Hosted CI covers broad validation. **Rebased onto origin/main `64d12e297` (this branch):** same database setup as above. - Each of the 19 `api::bridge::thread_window` Postgres tests ran in its own process and its own database cloned from the desired-state template, as nextest runs them: 19/19 passed. - The batch, grant, revocation and replica tests, 5 more runs each in the same way: 20/20 passed. ### test(relay): claim the invite before the joiner connects #### Problem `test_invite_claim_auto_joins_default_channel` (crates/buzz-test-client/tests/e2e_relay.rs) failed in "Relay and PostgreSQL / Relay E2E" at :363 with "recv kind:44100 notification for the default channel: Timeout". Two of the three reruns passed. - run 41568808045 attempt 1 (rerun passed): - https://github.com/block/buzz/actions/runs/41568808045/job/125701098947 - run 41657074260: - https://github.com/block/buzz/actions/runs/41657074260/job/125997465078 - run 41716543775 attempt 1 (rerun passed): - https://github.com/block/buzz/actions/runs/41716543775/job/126206390776 #### Cause The e2e relay runs without `BUZZ_REQUIRE_RELAY_MEMBERSHIP`, so every human NIP-42 admission spawns `auto_join_never_joined_channels` (handlers/auth.rs). The test connected the joiner and sent its kind:44100 REQ before minting and claiming the invite. The admission therefore joined the joiner to the flagged channel before the claim did. The claim's `auto_join_default_channels` then got `AlreadyMember` and emitted nothing, so the admission notice was the only kind:44100. All three relay logs show this. The joiner's "auto-joined default channel on admission" appears 39, 52 and 106 ms after its NIP-42 auth, before "relay member added via v2 invite". The claim logs no join for the joiner. When the admission notice reached the client before EOSE, `collect_until_eose` consumed it, and the test discarded that result. The test then waited 5 s for a live event that never came. When the notice arrived after EOSE, the test passed, but on the admission path. Either way, the test never covered the claim-time auto-join it is named for. #### Fix Only the test changes. The joiner opens no WebSocket until the claim returns. Over NIP-98, the claim awaits the join, the kind:39002 roster and the kind:44100 before its 200. The test now: - checks the roster before the joiner connects, so only the claim can have joined them; - then connects and REQs kind:44100 `#p=joiner` with `since` taken before the claim (minus the file's usual 5 s margin); - asserts that the events `collect_until_eose` returns hold exactly one notice for the channel, signed by the relay's NIP-11 `self` key. Nothing depends on EOSE-vs-live ordering any more. The admission that follows skips the channel, because the claim already added a member row. No timeouts change, no retries are added, and no assertions are loosened. The repeat-claim and cleanup steps are unchanged. No open PR touches this test. Draft #9045 (default-channel recovery) changes invites.rs, but not this test or this ordering. #### Validation Local Linux workstation, fresh database, relay started by scripts/start-relay-for-tests.sh. Redis was the host's own, and `BUZZ_METRICS_PORT` was moved off 9102, which bloxlet owns. Each iteration ran CI's `e2e_relay invite` selection (8 tests in parallel). - Unmodified test: 12/60 failed at :363 with the CI message. - Adversarial ordering, using temporary, uncommitted changes: a 500 ms sleep between connect and REQ, plus logging of what `collect_until_eose` returned. 20/20 failed with the CI signature, and each time `collect_until_eose` had returned the channel's notice. In the relay log, the admission join came 28 ms after auth, the REQ queued one event, and the claim emitted nothing. - Fixed test: 100/100 passed. With the same 500 ms sleep: 30/30 passed, each with exactly one notice, from the claim. Four concurrent loops of 25: 100/100 passed. - Falsifiability, using a temporary relay build without the claim-time `auto_join_default_channels` call: the fixed test failed 10/10 at the roster assertion (`None`). The old test passed 18/20 against that build; its two failures were its own :363 flake. - The full CI "Relay E2E tests" step list passed once. - `cargo fmt --check`, `cargo clippy -p buzz-test-client --tests -D warnings`, `just file-size-check` and `git diff --check` are clean. A green run doesn't prove the flake is gone. Hosted CI covers broad validation. **Rebased onto origin/main `64d12e297` (this branch):** the relay was built from this branch with `--profile ci` (its relay code is identical to main) and run with CI's environment. It used a fresh desired-state database with the deployment community seeded, the host's Redis, and a private MinIO-compatible container (the image docker-compose pins) for the startup git probe. `BUZZ_METRICS_PORT` was moved off 9102. - CI's `e2e_relay invite` selection: 30/30 iterations passed, 8 tests each. - CI's full "Relay E2E tests" step list, once: 10 of 11 steps passed. `test_nip29_announce_channel_wire` failed at e2e_relay.rs:3862: the kind:39000 metadata still carried `auto_join` right after the relay owner cleared it. This branch doesn't touch that test or any relay code. Run alone against the same relay, that test failed 8 of 20 times. Re-running the whole step against one database isn't meaningful, because `test_nip29_announce_channel_wire_overlong_name_is_refused_before_storage` reuses a fixed channel name. ## Coordination - #8991 ("BUZZ-512: Track session application and drain") edits `AuxPages` and `pause_after_aux_page` in `crates/buzz-relay/src/api/bridge/thread_window/postgres_tests/failure_postgres_tests.rs`, and `git merge-tree` shows a conflict there. #8991 already conflicts with main in 15 files, so it needs a rebase anyway. The second commit's message describes that rebase: keep #8991's waker on the non-`Option` proxy field, and use `proxied_state` for the two callers it moves onto a proxy. - #6642 and #7569 touch `crates/buzz-test-client/tests/e2e_relay.rs` and merge cleanly with this branch. - #7852 touches `thread_window/postgres_tests.rs` and `e2e_relay.rs`. It already conflicts with main, and this branch adds no new conflicted file to it. Older open PRs that touch `e2e_relay.rs` also already conflict with main, and none gains a new conflict from this branch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: mtoohey <mtoohey@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 09:28:38 +11:00
Luke Tornquist	47be2e2d3f	fix(db): keep busy-reader history reads off the writer and cap them per pod (#9132) ... ## Problem A routed read that misses the reader's 150ms acquire budget (`reader_acquire_timeout`) falls back to the writer, whatever the read is. Under a burst of subscription-history reads (REQ history, channel and thread windows, feeds, search hydration, COUNT), the reader pool fills, and every miss moves a heavy history query onto the writer. There it competes with the authentication, tenant-resolution and moderation lookups that only the writer serves, and their latency climbs with it. Nothing bounds how many history reads a pod sends to the reader at once, so the reader queue grows without limit as well. Separately, the compatibility families `buzz_db_pool_acquire_wait_seconds` and `buzz_db_pool_acquisitions_total` have no `operation` label, so a breakdown by operation reports every sample as unlabelled. ## Changes 1. **History reads no longer spill onto the writer when the reader is busy** (`Db::route_read`). `route_read` now returns `Result<RouteDecision>`. For `ReaderOperation::SubscriptionHistory`, a `reader_acquire_timeout` on a saturated reader pool rejects the read with `sqlx::Error::PoolTimedOut`. Saturated means every connection the pool may hold exists and none is idle (`Db::reader_saturated`). The check is needed because sqlx also reports a reader that refuses dials as `PoolTimedOut`, and that reader is down, not busy. The relay already answers that with its generic retryable `CLOSED` `error: database error`, so there is no new client contract. A reader that is wrong rather than busy (unreachable, validation failure, proof timeout, stale token) and prechecks that never reach the reader still fall back to the writer. Authorization reads (`is_relay_member`) keep the writer fallback unchanged. 2. **Per-pod cap on reader-served history reads** (`runtime/history_reads.rs`). A FIFO `tokio::sync::Semaphore` admits at most the reader pool size less one fifth (at least one) history reads at once. For example, the default pool of 20 admits 16. The remaining connections are normally free for authorization reads. A read waits at most 1s for a permit and is otherwise rejected the same way. The permit is owned by the reader transaction (`ReaderTx`) and released when the transaction is dropped; routed reads drop it before re-running a failed replica read on the writer. That holds whether the transaction is a per-call routed read or a `ReadSession` the relay keeps for follow-up queries. The cap bounds history transactions, not connections: sqlx returns a dropped transaction's connection asynchronously through the reader release check. Released connections can therefore briefly occupy the free slots, for one round trip normally and at most the 5s release timeout when sockets stall. In that window, authorization reads fall back to the writer as before, and history reads on the saturated pool are rejected, not spilled. 3. **Rejections are observable**: `buzz_db_route_decision{decision="rejected",reason="history_read_cap"|"reader_acquire_timeout"}`. 4. **Legacy pool families gain `operation`.** The population is unchanged, and summing over `operation` gives the original series. 5. **HTTP reads answer 503 `database unavailable` when the database can't be reached** (`api::db_read_error`). This covers a pool timeout, closed pool or socket failure on `POST /query`, `/count` and every bridge `/query` branch that serves a history read (channel window, the three feeds, thread replies), including a shed history read. These used to return a generic 500. The thread-window path already answered 503 for pool timeouts; this applies the same mapping to the shared helper. Pool sizes are unchanged on purpose. A larger pool adds lock contention on the writer rather than relieving it. Docs: `deploy/charts/buzz/README.md` (operator contract) and `PLANS/REPLICA_FULL_READ_ROUTING_DESIGN.md` (fail-closed exceptions). ## What got simpler The concept count is roughly flat. Instead of a caller-side permit, the permit lives in the one type every replica transaction already flows through (`RouteDecision::Replica` / `ReadSessionInner::Replica`). The 11 routed call sites change only by a `?`, and the compiler enforces that change: a call site cannot silently fall through to the writer on a rejection. No new error variant, CLOSED reason or env var. ## Tests - `postgres_tests::saturated_reader_spends_one_acquire_budget_and_rejects_history_reads` (rewrites the former `routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold`). A size-1 reader is held saturated with a cold Aurora cache. A history read (`count_events_routed`) is rejected with `PoolTimedOut` within one acquire budget, records `rejected/reader_acquire_timeout`, and releases its permit. An authorization read on the same reader still falls back as `writer/reader_acquire_timeout`. - `postgres_tests::match_nothing_reads_take_no_connection` (unchanged): its control still requires a history read against an unreachable reader to fall back to the writer. It caught the first cut of this change, which rejected that read. - `postgres_tests::history_permit_follows_the_read_session_and_a_full_gate_rejects`: a held replica `ReadSession` keeps its permit. With the gate full, the next history read waits the full permit budget and is then rejected as `rejected/history_read_cap`, with no writer route recorded. Dropping the session frees the permit and the next read routes to the replica. - `history_reads::tests`: a cap sizing table, plus bounded-wait rejection and reopen under paused time. - `observability` label tests now require `operation` on both legacy families. - `api::db_read_error_tests::unavailable_database_maps_to_retryable_503`: pool timeout, closed pool and I/O map to 503 `database unavailable`. Other errors stay 500. - `postgres_tests::shed_history_read_maps_to_503_on_every_bridge_branch`: with the reader saturated, `/query` channel-window, mentions, needs_action, activity and thread-reply requests each answer 503 `database unavailable` (each serves 200 beforehand). - Mutation-checked on Blox: re-enabling the writer spill, forcing the saturation check either way, skipping the gate, dropping the permit early and removing the legacy label each fail a test. ## Residual - During a reader failover whose old sockets stall, both slots sit in the release check, so the pool reads as saturated. History reads are shed as retryable for up to the release bound (5s) instead of loading the writer. `stalled_reader_sessions_release_their_slots_for_new_connections` now retries through that window and still requires recovery onto the reader. - A reader that accepts no TCP at all can fill the pool with in-flight dials and look saturated. Its history reads are then rejected rather than moved to the writer, which is the same outcome as a full gate. - Shedding relies on clients backing off on a retryable `CLOSED` (RPR #9). WebSocket clients already treat `error: database error` as retryable. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 22:12:30 +00:00
Luke Tornquist	74abe2f0fe	perf(db): split newest-first events reads at a month floor so Postgres prunes partitions (#9091) ... ## Summary Newest-first `events` page reads are now split at the start of the previous UTC month, so Postgres can prune the older partitions whenever the newest ones fill the page. Results are unchanged. **Why.** `events` is range-partitioned by month, and every partition carries the full index set. A page read with no `created_at` lower bound can't prune, so Postgres plans and locks every partition and every partition index: about 200 relations per read with 19 partitions. A backend has 16 fast-path lock slots, and every lock past those goes through the shared lock table. On a production-shaped load rig at pool 200, active backends spent 36% of their time waiting on `LWLock:LockManager` and 1.6% on disk. **What changes.** `query_events_on` (REQ history, bridge `/query`, mentions, rosters, notices) and `get_channel_window_on` (channel windows) now read in up to two passes on the same connection, through one helper (`store/recent_pass.rs`): 1. **Recent pass:** the same read plus `created_at >= <floor>`, where the floor is the start of the previous UTC month. When it returns a full page, that page is the answer, because every row at or above the floor sorts before every row below it. 2. **Older pass,** only when the recent pass comes up short: `created_at < <floor>`, limited to the rows still missing, appended after the recent rows. The ranges don't overlap, so no row is read twice and the page is exact. For channel windows the limit includes the +1 probe row, so `has_more` stays exact. **One snapshot, one deadline.** The two passes run in one `REPEATABLE READ, READ ONLY` transaction (`ReadTransaction`), so both read the same snapshot. Under `READ COMMITTED`, a replacement committing between the passes (a replaceable event, an artifact head) could move the live row from below the floor to above it, and it would land in neither pass. Inside a transaction the passes run in a savepoint instead; the only callers that read inside a transaction are routed replica reads, which are already `REPEATABLE READ`. Each pass after the first runs under whatever is left of the 20 s read deadline, so a split read never gets more time than an unsplit one. Single-statement reads (COUNT, unsplit pages) keep their old path, now through the same type. Channel windows now take the read deadline too. On a routed reader that is the reader's session deadline. On the writer they used to run under the writer's own session timeout (2 minutes), and now get the 20 s read deadline, like every other event read there. The floor is a SQL literal, not a bind parameter, so Postgres prunes when it plans the statement, cached generic plans included. It changes once a month, so the statement cache holds one statement per pass and shape. `#p` reads put the same bound on `event_mentions.event_created_at`. These reads are not split: - Reads whose own bounds already prune (`since` at or after the floor) or fall entirely before the floor (`until` before it). - `OFFSET` reads, since the offset spans both sides of the floor. - Lookups that match a fixed set of rows wherever they sit: `ids`, `#e`, `#d`, artifact `d` tags. - Fetch-everything reads (`max_limit`, the COUNT fallback). - Reads limited to replaceable or addressable kinds. Those are latest-state rows, often old and few, so a recent pass would rarely fill the page. No protocol, schema, index or client change. **Not covered:** COUNT and the per-row aux `#e` reads, neither of which has a page to fill. The recent pass also has no upper bound, because rows dated in the future must stay visible. So partitions ahead of the current month (the manager keeps 6) and the catch-all are still planned. ## Evidence A production-shaped load rig (6,600 desktop clients connecting cold at 100/s, against a 19-partition `events` table), comparing `main` with this branch. Each row is one arm, so expect run-to-run noise of about ±10%. | Arm | Postgres exec time (top statements) | Errors | Relation locks outside fast path (avg) | |---|---|---|---| | Pool 200 + channel-first index, `main` | 13,217 s | 4.75% | 9,481 | | Pool 200 + channel-first index, this PR | 10,931 s | 3.68% | 4,988 | | Default pool, no extra index, `main` | 6,705 s | 5.96% | — | | Default pool, no extra index, this PR | 6,145 s | 5.15% | — | | Default pool + index, `main` (repeat) | 1,070 s | 0.99% | — | | Default pool + index, this PR (repeat) | 1,013 s | 0.93% | — | Pages that fill above the floor stop planning old partitions, and that halves the locks taken outside the fast path. `LWLock:LockManager` stays at about 41% of active backend samples at pool 200, though. The dominant remaining cost there is the cold unread repair: a multi-channel read with no `since` and a page that never fills. That read needs its own fix (a lookback bound or a better plan) and is out of scope here. An earlier version re-ran the whole read after a short recent pass. That doubled the work on non-filling reads: errors went from 4.75% to 11.88% at pool 200. This version reads only the missing older rows. ## Tests - `recent_pass::tests`: floor arithmetic (including the year wrap), the predicate literals, an eligibility table, and an in-memory exactness test that drives the passes at every limit and pins which passes run with which limits. - `e_tag_filter_shape_tests::split_passes_bound_created_at_with_a_literal` / `recent_pass_skips_lookups`: SQL shape (both the `e.` and `m.` sides for `#p`, no bound on unsplit reads or COUNT) and the exclusions, including offset. - `postgres_tests::split_page_reads_match_the_unsplit_read` (buzz-db event): rows on both sides of the floor. A full recent page is answered alone; short pages at limits 4, 5 and 6 add exactly the missing older rows; offset reads across the floor are not split; a cursor below the floor works. It reads `pg_prepared_statements` to show which passes ran. With another session locking a partition only the older pass plans (and a short `lock_timeout`), a full recent page still answers and a short one returns `Err` rather than a partial page. - `postgres_tests::channel_window_split_matches_the_unsplit_read` (buzz-db thread): the same checks for channel windows, including `has_more` when the recent pass has no probe row, and an `Err` when the older pass fails. - `split_read_passes_share_one_snapshot` (event, both deadline modes) and `channel_window_passes_share_one_snapshot` (thread): another session holds a lock that only the older pass needs, so the read stops between its passes. A replacement then commits that deletes a row below the floor and makes its replacement live above it. The page must still be the first pass's snapshot: the replaced row is present and its replacement is absent. - `read_statements_share_one_deadline`: per-read, session and inside-a-transaction reads. The second statement runs under less than the full deadline, and finishing the read restores the connection's own timeout. - `channel_window_passes_share_one_deadline` (thread): through the production writer entry. The recent pass waits 8 s on a lock and the older pass is blocked for good. The window must be cancelled within the one shared deadline, not a full deadline after its older pass starts. - `channel_scoped_mention_reads_share_one_prepared_statement` now expects one recent and one older statement, shared by every channel count. Mutation-checked. Each of these mutants fails a new Postgres test: dropping the older pass, giving the older pass the full limit, removing the split, splitting offset reads, dropping the window's probe row from the split limit, removing the window's split, swallowing an older-pass error in either path, opening the split read `READ COMMITTED` (events and windows), opening it with a plain `BEGIN`, giving every pass the full deadline, and running a window with no read deadline. ## What got simpler Nothing got simpler overall. This adds one small module (`store/recent_pass.rs`) shared by both read paths. The single-statement deadline helper is now a thin wrapper over `ReadTransaction`, so the begin, savepoint and tighten logic lives in one place. The alternative, a `since` in every client plus a new index per read shape, means more code in more places and does nothing for app versions already shipped. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 16:03:15 -05:00
Luke Tornquist	9b4b73eb08	fix(relay): bound mesh dials and keep failing pods out of the mesh (#8752) ... ## Problem A rolling deploy can stall relay startup for 30s per dead mesh peer. - A pod publishes its mesh ready record during `mesh_boot` and clears it only on graceful shutdown. If a later fatal startup check kills the pod, the record lingers until it expires. - Dialing that address never gets an answer, so `MeshEndpoint::connect` waits out QUIC's default 30s idle timeout. `reconcile_once` dials candidates one at a time, so every dead peer adds another 30s, both in the boot-time `mesh_initial_reconcile` step and in every later reconcile pass. - The fatal check that killed those pods was the git conformance probe. A transient S3 `500 InternalError` on one racer was treated as a conformance failure. ## Changes 1. **Bounded, concurrent mesh dials** (`buzz-relay-mesh/src/runtime.rs`). Each connect attempt is wrapped in `DIAL_TIMEOUT` (2s; in-cluster handshakes take milliseconds), and a reconcile pass dials its candidates concurrently. A pass with dead peers now costs about one dial timeout instead of 30s per peer. The candidate set is bounded by the membership table. 2. **Conformance probe before mesh boot** (`buzz-relay/src/main.rs`). This only moves an existing block, so a pod that fails the probe exits before it publishes a ready record peers would dial. 3. **5xx racers are unknown outcomes** (`buzz-relay/src/api/git/store.rs`). The race phases already drop socket/send failures from the observer set as "outcome unknown". A 5xx is the backend declining to classify the write, so it now joins them through one shared `racer_outcome_unknown` helper. 4xx answers, including 412, still count as evidence. A backend that 5xxs every racer still fails closed on the two-observer floor. Because a dropped racer's write may still have landed, every `if_match_race` round now reads the pointer back and requires the winner's exact body and ETag. Otherwise a backend that commits a stale write and then answers 5xx would pass in the last round. §Conformance steps 2–3 in `docs/git-on-object-storage.md` now state this drop-and-floor rule. ## What got simpler The two duplicated drop-classification match arms in the probe now share one helper. The probe/mesh fix is a reorder of existing startup steps, not a new mechanism or rollback path. ## Tests - `runtime::tests::reconcile_pass_bounds_dials_to_unresponsive_peers`: three black-hole peers (bound UDP sockets that never read) plus one live peer. The pass must finish within `2 × DIAL_TIMEOUT` (a serial pass would take at least `3 × DIAL_TIMEOUT`, and without the timeout about 30s), the live peer must connect, and each dead peer must end `Disconnected`. - `store::tests::probe_drops_5xx_racers_but_keeps_4xx_as_evidence`: pins the classification boundary (5xx and I/O are unknown; 400/403/412/499 are evidence). - `store::fault_probe::*`: runs `run_conformance_probe` against an in-process S3 fake that injects racer faults at the HTTP boundary, so responses pass through rust-s3's `fail-on-err` mapping and retry. One 5xx racer in either race phase is dropped and admitted (`transport_drops == 1`). Too few classified observers fail closed: one observer in either phase, and all-5xx in `if_none_match_race`. An unexpected 4xx racer fails closed in both phases. A racer whose stale write lands and is then answered 5xx fails closed on the post-round readback (`dropped_racer_whose_stale_write_lands_fails_closed`). A clean fake passes with zero drops. - CI lane: no `api::store` test ran in any CI lane before, including the existing ones. The `just test-unit` nextest filter and the `scripts/run-tests.sh` fallback now select `api::store::tests` and `::fault_probe`. The live-MinIO `::probe` module stays opt-in. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 15:56:07 -05:00
Matt Dickenson	120d75d976	Expose strict code-entry pairing and live authorization bindings (#9061) ... Native clients using Rust pairing can now receive the existing `desktop-code-v1` flow: accept its challenge, submit a user-entered six-digit code, handle correlated rejections, and require successful code submission before accepting the source proof. A failed durable import can explicitly acknowledge failure. Adds a public, live-session channel binding (session ID, ordered ephemeral peers and transcript commitment) so applications can authorize the exact exchange using each device's independently authenticated account session. The binding does not itself authorize secret export. Legacy APIs and the wire format remain compatible. Validation: 79 pairing unit tests and seven public API journeys pass, including wrong-code retries, five-attempt lockout, replay/peer rejection, failed import, and identical live bindings on both devices. Focused `cargo clippy -p buzz-core --all-targets -- -D warnings` and formatting pass. Repository formatting/mobile checks and desktop, web, and Flutter build/test lanes pass after installing the pinned Flutter dependencies. Three initial normal push-hook attempts stopped at different unrelated ACP deadlines. The final normal push passed all applicable hooks, including desktop Tauri Clippy/tests and the full Rust unit lane, with existing hook lanes serialized and four Rust test threads. No checks were disabled and no ACP source was changed. Independent review found no blocker in this API delta. App and human acceptance remain outstanding; this PR is a draft. --------- Signed-off-by: Matt Dickenson <mdickenson@squareup.com>	2026-11-05 20:18:14 +00:00
Luke Tornquist	50c10dc7b2	perf(relay): cache the community-active check on socket admission (#9109) ... ## What Socket admission (root and audio) and the ephemeral write fence now share one cache of "is this community active?". An entry is served for 1 s after its read checked out a pool connection, and concurrent misses for one community share one read. Archive and deletion drop the entry before they close the community's sockets. ## Why Every new socket runs `is_community_active` before it can authenticate. In a reconnect storm that means thousands of identical reads for one community. A read that times out in the pool refuses the socket (`community active check failed; refusing the socket`), and the client then reconnects and adds more load. On a local cold-start reconnect rig, this refusal count roughly tripled once #9012 freed pool capacity for other reads. It is now one of the top causes of connect failures. ## How - `buzz_db::Db::is_community_active_read_at` returns the answer together with the instant its connection was checked out, following the #9012 `ReadAt` pattern. `is_community_active` now calls it, and the datastore span stays under its existing name. - `community_active::CommunityActiveCache` holds an entry per community and a `SharedReads` gate. An entry is served for `COMMUNITY_ACTIVE_MAX_AGE` (1 s) after its read. Both answers are cached. A cached "inactive" fails safe and lasts at most 1 s after an unarchive. Errors are not cached. - **The archive sweep stays exact.** Admission registers a socket before it checks, so the sweep closes every socket registered before it. A socket that registers after the sweep must see the community as inactive. `apply_community_disconnect` invalidates the entry before the fence and sweep. Invalidation bumps an epoch, and a fill whose read began before the bump is not published. A post-sweep socket therefore reads afresh. It cannot take a pre-archive answer from the cache or from a shared in-flight read. Host invalidations (archive and unarchive writes, local and received) also drop the entry. - **Bound when the notification is missed.** A pod that misses the Redis disconnect admits sockets from a cached "active" for at most 1 s after the archive commits. Lifecycle revalidation then closes them. This matches the 1 s host-routing bound in `docs/community-auth-upgrades.md` ("archive and deletion … cached tenant binding can lag those changes by up to one second"). Deletion keeps its own write fence (`guard_community_write`). - Lifecycle revalidation and persistent ingest still read uncached. ## What got simpler The ephemeral fence's separate `serving_active_cache` is gone. It ran the same SQL predicate through `is_serving_active`, used a 10 s TTL and had no invalidation. The connect path, the WebSocket ephemeral path and HTTP typing now share one cache with one 1 s bound and one invalidation path. That leaves one fewer cache concept, and the ephemeral fence goes from 10 s stale to at most 1 s. ## Tests - `community_active::tests` (unit): - 16 concurrent misses cost one read, and an entry past its max age is re-read. - An invalidation during an in-flight read fences that fill. A waiter that joined the read reads afresh and gets "inactive". - Errors are not cached. - `connection::tests::bootstrap_admits_from_the_cached_community_answer`: with the database unreachable, a cached "active" admits a socket through the real `handle_connection`, and the socket gets its AUTH challenge. - `postgres_tests::cache_misses_behind_a_held_pool_share_one_fresh_read` now also covers this cache. With a one-connection pool held past every max age, eight concurrent admission checks cost one DB read, and the entry lands fresh. - `postgres_tests::archived_community_is_refused_after_invalidation_or_max_age`: after an applied archive, the next check refuses at once. An archive this pod never hears about refuses once the entry passes 1 s. - The ephemeral and typing fence tests now seed the new cache. - Mutation-checked on Blox. Each of these turns a test red: dropping the epoch fence, the max-age filter, the single-flight gate, the connect path's cached read, or the disconnect invalidation; stamping `read_at` before the pool wait; and calling the uncached read on the ephemeral path. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 14:09:14 -05:00
Luke Tornquist	ff15c347c2	fix(db): auto-migrate prebuild-guarded indexes on small databases (#8802) ... ## Problem Migrations `0076`, `0078` and `0081` raise an exception on any table with at least one row unless its index was prebuilt with `CREATE INDEX CONCURRENTLY`. A relay that runs migrations at startup (`BUZZ_AUTO_MIGRATE`, on by default in the Helm chart) and has data in those tables therefore crash-loops on upgrade: ``` migration error: while executing migration 81: error returned from database: prebuild idx_nip_fi_profile_heads_names concurrently; see docs/profile-search.md ``` `0081` (#9019) hits every auto-migrating relay that has published NIP-FI verified profiles. `0076`/`0078` (#8523) hit any relay with artifact data or channel members. Each of these tables is small enough on such relays that a plain index build takes milliseconds. The root cause is `docs/database.md` Rule 6. It told authors that an index on a populated table MUST be built concurrently by an operator ahead of time, so migrations were written to refuse populated tables. With auto-migration that refusal is a crash loop, not a safe stop. ## Fix The migrations are already applied on some databases, so they can't be edited: sqlx would reject the changed checksums. Instead, `run_migrations_locked` builds their missing indexes first. For each entry in `INLINE_INDEX_BUILDS` (`0076`, `0078`, `0081`), it runs migrations through the version before it, then builds any missing index. Each build is a plain `CREATE INDEX` in its own transaction, under the same 1s `lock_timeout` and 5s `statement_timeout` the migrations use. That is the bounded inline build `0049` already does: - **Small populated databases** upgrade the same way empty ones do. - **Large databases** still fail closed. Only a lock or statement timeout (`55P03` / `57014`) gets the "prebuild concurrently" hint, pointing at that migration's own doc. Other errors pass through unchanged. The blocked migration is not recorded. - **Prebuilt indexes** are found with `to_regclass` and skipped before `CREATE INDEX`, so deployments that prebuilt concurrently never request the writer-conflicting lock. - **Invalid prebuilt indexes** are left for the migration to reject, as before. `CONCURRENTLY` is not used in this path. The runner holds the migration advisory lock, and another replica blocked on that lock holds a snapshot that a concurrent build would wait on. ### Rules - `docs/database.md` Rule 6: with `BUZZ_AUTO_MIGRATE` on, every migration must apply its own change on a populated database and must never refuse one because it has rows. A new index is built inline under the migration's budgets. Only deployments that apply migrations out of band (auto-migration off) prebuild concurrently. - Root `AGENTS.md` (step 4) and `migrations/AGENTS.md` repeat the never-refuse rule. - `docs/artifact-sync-deployment.md` and `docs/profile-search.md` describe the startup behavior first and the concurrent prebuild as the out-of-band path. ## Tests - `populated_database_auto_migrates_through_every_migration` (Postgres) migrates to `0075`, seeds artifact rows, and checks four phases: 1. **Lock held:** while a writer lock is held, the `0076` build fails closed. 2. **Index prebuilt:** with the `0076` index prebuilt, the build is skipped. 3. **Partial `0078`:** a blocked `0078` build keeps the builds before it committed. 4. **Verified profile seeded:** with a verified profile head inserted before `0081`, a final startup succeeds. It asserts that the applied versions equal every embedded migration and that all six indexes are valid, ready and live. - `frozen_prebuild_guards_have_inline_builds` (no database) scans `migrations/`. It fails if any migration refuses a populated table (`RAISE EXCEPTION 'prebuild …`) without an inline build, and if a new guard is added. - `migration_execution_cannot_bypass_schema_destruction_lock` allows exactly one production `run_to`, inside `run_migrations_locked`. ## What got simpler - Auto-migrating relays no longer need a manual, operator-only prebuild step for small deployments. That restores the chart README's contract that `helm upgrade` is the whole upgrade procedure. - Rule 6 no longer steers authors toward refusing populated tables, and a test enforces that. - The change adds one table-driven step for three frozen migrations and no new configuration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 13:52:06 -05:00
Luke Tornquist	aa749a862e	perf(db): serve single-root thread reply reads from thread_metadata (#9075) ... ## Summary A plain thread-reply read, `{kinds:[9], #h:[channel], #e:[root], limit:100}`, goes through the generic `query_events` path. That path gathers every event tagging the root into a `MATERIALIZED` CTE, then sorts it to keep one page. On a root with 500k replies, each page costs about 514k buffer reads plus a temp-file spill. This PR serves single-root reply reads from `thread_metadata` instead. It walks `idx_thread_metadata_window` newest-first, fetches each reply by primary key, and applies every filter predicate before `LIMIT`. Page cost scales with the page size, not with the thread. ## What changed - **Routing (`crates/buzz-db/src/store/event.rs`).** A `query_events` read takes the new route when it has: - one channel, - one lowercase-hex `#e` root, - kinds limited to 9 and/or 40002, - no `#p` filter (any number of values; the relay marks such filters `exact_match`), - no `authors`, `pubkey` or `ids` selector. The route tests those per reply, so an author with no reply in a hot thread would walk the whole thread; the generic read narrows by author first. Every other `#e` read stays on the fenced path from #7854. - **Root check before routing.** `#e` must name a thread root. A cheap probe looks up the target in `thread_metadata` first. If the target is itself a reply, the read takes the generic path, which returns its direct children the NIP-01 way. A reply counts if its row has a root, or if it is a rootless legacy stub whose own NIP-10 markers resolve to a reply. - **COUNT unchanged.** COUNT post-filter fallbacks (`count.rs`, `bridge.rs`) set the new `EventQuery::exact_match` flag through `prepare_count_fallback`. Routing is skipped for them, so COUNT still counts every event that matches under NIP-01. - **Same predicates on both paths.** The predicate block moved out of `build_query_events_sql` into `push_event_filters`, which both paths call. Filters on the route therefore match the generic path, including the `#e` tag match, `since`/`until`/`before_id` and deletion. - **Unnamed statements.** Routed reads run as unnamed statements, so Postgres plans each one with its root and never caches a generic plan. The benchmark below shows the routed SQL is also fast under a forced generic plan, so this is a second safeguard. - **Narrowed semantics, documented.** Messages that name the root only in a bare `e` tag, a `mention` tag or a lone `root` marker are top-level under NIP-10. They have no `thread_metadata` row, so the routed read omits them. Live subscriptions still deliver them. `docs/nips/NIP-CW.md` gains a "Plain Thread-reply Filters" section describing this. A client that needs those messages can add another kind to the filter, which sends the read down the generic path. ## Benchmark (synthetic, Postgres 16, 500k-reply thread) Median of 5, newest page of 100. "Rare" is a root with 3 replies. "Generic" forces `plan_cache_mode = force_generic_plan`. | | main | this PR | |---|---|---| | hot root, custom plan | 2802 ms (513k buffers, 46,750 temp blocks written) | 1.6 ms (407 buffers) | | hot root, generic plan | 1779 ms | 0.6 ms | | rare root, custom plan | 1.2 ms | 0.8 ms | | rare root, generic plan | 61 ms | 0.08 ms | Author-filtered reads (`authors:[x], limit:1`) on the hot root: an author with no reply there takes 1.4 ms on the generic path but 1.7 s on the route, so author and id selectors stay generic. On both roots, the PR's SQL returns the same rows as main. ## Tests - `e_tag_filter_shape_tests::thread_reply_route_admits_only_single_root_message_reads` checks which reads take the route. Two roots, mixed kinds, no kinds, no channel, an uppercase root, a `#p` join, an `exact_match` read, and `authors`/`pubkey`/`ids` selectors all stay generic. The routed SQL is checked for the `thread_metadata` walk, the `#e` match and the window ordering, with no `MATERIALIZED`. - `postgres_tests::thread_reply_reads_use_thread_metadata` (Postgres lane) covers: - mixed 9/40002 replies, a nested reply and a soft-deleted reply, - a bare-`e` message that the route omits and the generic read (kinds `[9,40002,7]`) still returns, - a lone-`root` message, which the route omits too, - an `exact_match` read, which returns the same rows as the generic read, - a nested-parent target, which gets its direct child the generic way, - root stubs and a legacy parent with no row of its own, checked through the probe, - keyset paging, - no named prepared statement left behind after repeated runs. - `handlers::req::tests::postgres_tests::multi_p_thread_filter_keeps_nip01_results` (Postgres lane) runs a two-value `#p` thread filter through the real filter-to-query adapter and checks that a bare reference to the root still comes back. - `handlers::req::tests::count_fallback_fetches_one_extra_candidate` checks that COUNT fallbacks set `exact_match`. It is DB-free and was missing from the `buzz-relay --lib` allowlist, so this PR adds it to `just test-unit`. - Mutation-checked: the Postgres test fails if any of these is broken: - routing disabled, - persistent statements, - the nested-target probe dropped, - stubs treated as all replies or as no replies, - COUNT allowed to route, - the deleted filter dropped. - Blox gate: `cargo fmt`, clippy (`-p buzz-db -p buzz-relay --all-targets`), buzz-db lib (nextest and plain parallel `cargo test`), buzz-relay lib, and the buzz-db and buzz-relay Postgres lanes. One unrelated artifact-sync scale test timed out under load; it passes alone and in CI. ## What got simpler - One predicate builder replaces the inline block, so the two read shapes cannot drift apart. - One targeted route replaces the planned extra fence variant (a keys-only `MATERIALIZED` CTE). That variant was dropped because it reduced the spill but still scanned every reply. ## Identified debt (not in this PR) - The bridge `page` extension (`crates/buzz-relay/src/api/bridge.rs`, `extract_page_offset`, built for the kind:0 directory listing) still maps to SQL `OFFSET`. The routed read honours it too, for parity. Deep pages there should move to keyset (`until` + `before_id`). buzz-review-completed 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 11:14:12 -05:00
Logan Johnson	700e690193	feat(db): enforce unique listed channel names with an index (#8787) ... ## Summary Stacked on https://github.com/block/buzz/pull/8782 (merged), which makes the relay refuse a second open channel with a name that another channel in the community already has. That check lives in relay code, so an older relay, a direct database write or a missed code path can still create a duplicate. This PR adds the same rule to the database: a unique index on channel name, ignoring case, for the channels that show in the channel list. With the index in place, a duplicate fails at the write, whatever wrote it. The relay's own name check from https://github.com/block/buzz/pull/8782 is no longer needed, so this PR removes it. Clients still get the same refusal, `invalid: an open channel named "…" already exists`. The migration builds the index itself, also on a populated `channels` table, so an auto-migrating relay upgrades with no operator index step (the rule from https://github.com/block/buzz/pull/8802). It still refuses to run while duplicate names remain, because the index cannot exist over them. `docs/channel-name-uniqueness-deployment.md` gives the full rollout. ## Details - **The index** (migration `0083_channels_listed_name_unique.sql`): ```sql CREATE UNIQUE INDEX idx_channels_listed_name ON public.channels (community_id, lower(name)) WHERE visibility = 'open' AND NOT unlisted AND channel_type IN ('stream', 'forum') AND deleted_at IS NULL; ``` It covers open, not unlisted streams and forums that are not deleted, archived ones included. Private channels, DMs and unlisted channels may still repeat names. - **The migration builds the index inline, as `0049` does.** When `to_regclass` finds no index, it runs a plain `CREATE UNIQUE INDEX` under `lock_timeout` 1s and `statement_timeout` 5s. On a busy or very large table it fails on those budgets, records nothing, and the relay retries at its next start. When the index already exists (a concurrent prebuild), it skips `CREATE` so it never requests the write-blocking lock. In both cases it then checks that the index is valid, ready and has the exact expected definition. - **One refusal stays: duplicate names.** The migration refuses to run while listed duplicates remain, and its error names the rename script. The renames must go through the relay so clients get the new kind:39000, and a migration must not change rows (`docs/database.md` Rule 3). This is the "fails closed until its repair has run" case in `migrations/AGENTS.md`, so the writer release from https://github.com/block/buzz/pull/8782 and the rename are a required stop. - **Relay.** A write that takes the name of a concurrent, uncommitted write waits for it, and fails if that write commits. So the post-write check and the per-name lock from https://github.com/block/buzz/pull/8782 are removed. The store already maps the index violation to the same "name taken" error. - **Desired-state schema.** `schema/tables/public/channels.sql` has the same index, and the schema parity test compares its live shape between the desired-state schema and the migrations. - **Rollout** (`docs/channel-name-uniqueness-deployment.md`), in order: 1. Deploy https://github.com/block/buzz/pull/8782 and wait until no older relay pod serves. 2. Rename the existing duplicates in each community with that PR's script, while relays keep serving. The script needs `psql` and `python3`, and the operator key must be a member of each community. The doc covers archived communities and communities being deleted. 3. Confirm that no duplicate remains. 4. Only for a deployment that migrates out of band and is too large for the budgets: build the index with `CREATE UNIQUE INDEX CONCURRENTLY` and verify it. 5. Deploy this release. With auto-migration on, the relay applies 0083 at startup. With it off, run `buzz-admin migrate` first, never the file through `psql`. Rollback goes to the https://github.com/block/buzz/pull/8782 release only. Keep the index, and keep auto-migration off. - **Test fixtures.** Tests that inserted two open channels with one name now use distinct names. Tests that migrate a populated database through 0083 no longer prebuild the index. ### Related issue None found. Follows https://github.com/block/buzz/pull/8782. ### Testing - Migration tests (`listed_channel_name_postgres_tests.rs`) start from a database migrated through 0082: - With duplicates, 0083 fails and changes nothing. - After the rename plan is applied, a channel writer that holds its lock makes the inline build fail on the 1s lock budget, with nothing recorded. - With no writer, the populated table migrates with no prebuild, and a raw duplicate insert fails on the index. - An index with a different definition, and an invalid one, are refused. A valid prebuilt index is accepted, and not replaced, while another transaction is writing to `channels`. - The store and relay name tests from https://github.com/block/buzz/pull/8782 pass with the relay check removed: the index refuses the duplicate, and the client still gets `invalid:`. - The rename plan and script tests from https://github.com/block/buzz/pull/8782 run on a database at 0082, the state the script runs in. - Local Postgres runs on `108737dc0`, one fresh database per test as the nextest wrapper does: 38 of 40 buzz-db migration-lane tests pass (both listed-name tests, the migration parity test, the thread-window and invite-upgrade tests that migrate populated databases through 0083). The 2 `event_follow_up` contract tests fail locally the same way on main. All 49 buzz-db channel and channel-member store tests pass. ### Database review checklist - [x] Phase: contract. Before it: https://github.com/block/buzz/pull/8782 (writer release, required stop) and its rename script. Nothing after it. - [x] No trigger, function, procedure, rule or foreign key. - [x] No migration mutates rows. The renames happen before it, through the relay, with the script in https://github.com/block/buzz/pull/8782. - [x] Never refuses a table because it has rows. Fails closed only with duplicates (repair not run) or with an invalid or different index; then 0083 raises and the relay does not start. The release notes must name the release that contains https://github.com/block/buzz/pull/8782 as a required stop. - [x] Rule 4: `migrations/`, `schema/tables/public/channels.sql` and the parity test. A live catalog comparison covers the index. - [x] Deployed relay: the https://github.com/block/buzz/pull/8782 relay answers `invalid:` beside the index (its tests pass with the index present). A relay older than that still writes, but gets a database error, not `invalid:`, for a duplicate. That is why https://github.com/block/buzz/pull/8782 is a required stop. - [x] Timeouts and index build: `lock_timeout` 1s, `statement_timeout` 5s. The migration builds the index inline (plain `CREATE UNIQUE INDEX`, skipped when `to_regclass` finds a prebuilt one). Only out-of-band deployments too large for the budgets prebuild it with `CREATE UNIQUE INDEX CONCURRENTLY` (doc step 4). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 22:47:23 -05:00
Brad Seiler	64d12e2975	BUZZ-511: Honor relay mode for community NIP-FI enforcement (#8988) ... ## Summary Apply relay mode consistently across protected HTTP, root/audio WebSocket, Git, media and V1 admission. Off requires no NIP-FI, including for stored required communities. Shadow enforces only stored community requirements and observes other communities. Enforce requires NIP-FI for every community. Durable requirements survive all mode changes. Off retains ordinary tenant routing and authentication. It skips FI policy/dependency reads for admission, uses cached WebSocket tenant routing, and loads no FI binding snapshot at startup. Discovery reports `requires_nip_fi=false` and no FI capability in Off. Evaluating modes retain same-row WebSocket binding and immutable HTTP policy tenants, including host remaps, pending deletion and unarchive races. Global Enforce discovery retains its existing compatibility behavior. Host-specific root NIP-11 and `/info` expose the additive `requires_nip_fi` field with no-store. In evaluating modes, an unavailable policy returns 503 and an unknown or non-serving host returns 404. Off uses ordinary routing; a discovery lookup failure returns 404. Stored requirements in Shadow require usable FI dependencies. Admission does not establish activation. BUZZ-512 owns existing-session application, drain and mode-aware readback; its separate revision must report stored requirements as suspended in Off. Production baseline verification and human acceptance remain open. Community status now shares the bounded host cache: 10,000 entries, one-second leases measured before checkout, coalesced fills, and no cached misses or errors. Git reuses one request policy. Root/audio retain the final primary binding check: two reads on a cold successful upgrade, one on a cache hit. Off creates no verifier or JWKS source and reads no FI policy. Proof metrics count stored-required Shadow enforcement. Acceptance locks and validates the community row, takes the matching host advisory lock, and drains optional leases for 1.1 seconds before commit. Reads begun after commit see required across participating relay instances, even without Redis invalidation. The row lock closes single and double remap races. Admissions already in flight remain subject to BUZZ-512 activation fencing. ### Related issue [BUZZ-511](https://linear.app/squareup/issue/BUZZ-511/relay-enforce-per-community-nip-fi-policy-and-publish-compatible). [BUZZ-510 / #8985](https://github.com/block/buzz/pull/8985) is merged; this PR now targets main. The restack retains main’s #8974 denial telemetry and artifact capabilities. Related work includes #8526 tenant lookup errors and #8444 media authorization. ### Rollout and containment Keep `BUZZ_NIP_FI_UPGRADES_ENABLED=false`. [The rollout plan](docs/community-auth-upgrades.md#staging-and-containment) starts in staging with zero required rows and a 30-minute before/after traffic comparison. Initial abort limits include any policy timeout/503, policy-read p99 above 50 ms, writer acquisition p99 above 20 ms or twice baseline, and a five-minute 503 increase above 0.1 percentage point. These are initial limits to validate against measured staging results, not production capacity claims. Verify signed/unsigned free controls, real required root/audio upgrades, dependency failures, restart behavior and discovery before expansion. Also watch binding timeouts/errors and their current generic 404 responses on known serving hosts alongside writer-pool pressure; evaluating root/audio binding has a 500 ms budget; Off uses ordinary cached routing. In Shadow, a pre-511 rollback loses enforcement of accepted community requirements. Stop new upgrade calls, retain accepted policy and receipts, and repair with an enforcing binary or operator-approved containment. Never clear durable policy to restore access. Off deliberately suspends FI enforcement and must be an explicit operator mode choice. No production activation is claimed. All acceptance paths must implement the lease protocol before upgrades are enabled. Keep upgrades disabled through a mixed-version rollout. For rollback, disable new acceptance before removing the protocol while caching pods remain. A cold policy fill uses a six-statement transaction and a 400 ms server timeout within the 500 ms lookup budget. A contended policy fill returns 503. During the drain, archive, deletion, ownership, issuer-binding and competing acceptance writers for that community may wait; ordinary serving reads and writes retain their existing behavior. Host-binding changes retain the existing one-second routing bound. Local and Redis invalidation accelerate refresh; safety does not depend on delivery. Deploy the verified Shadow revocation behavior from #8991 before accepting required communities in Shadow. ### Testing Published review-fix head: `39dfe79b1fc04f10ee3eb94d6ea42cf686bafe1e`, tree `1e2e8d3abd39ee1d6821defe8c8d34de73726e08`. McKeon cleared this cache delta. Full affected auth (291), DB (192), pubsub (31) and relay (1,682) package tests passed, including integration binaries. Two previously documented hardcoded-Redis audit tests remained excluded. All-target clippy, all 1,297 native PostgreSQL tests and the full pubsub Redis suite passed. The full ACP package and `just ci` passed after rebuilding cached test executables that embedded a removed checkout path. No ACP source changed. Tauri's generated lockfile delta was archived and restored; it is excluded from this candidate. Both positive lease regressions passed. Four deliberately broken variants failed at the intended assertions: removing the drain left an optional lease alive after 202.935667 ms; skipping the fill lock failed contention refusal; moving the timestamp after the read failed the anchor check; taking the host lock before the row failed the remap-blocking check. Exact source was restored after each mutation. These are local validation results, not a deployed-fleet acceptance claim. All applicable normal push hooks passed. GitHub reports the exact tested head and tree; the published code has no delta from the reviewed candidate. Earlier mode-correction evidence follows for context. Current mode-correction head: `aa9abf07284b56a88eb78a2014e590f39d705288`, on main `af54258f19ac32667c027f889652bdeb479c8e13`. Its code equals reviewed `f1e7da5fc5`, plus the independently reviewed operator documentation follow-up; all 19 earlier remote patches are preserved. McKeon verified the correction and its mapping to the tested candidate. The mode-correction candidate passed full auth (291) and relay (1,679) package tests, clippy, and four explicit native PostgreSQL policy workflows. There were 623 ignored relay tests and two excluded Redis tests. Real PostgreSQL checks cover an ordinary unsigned caller on a stored-required host in Off with no FI token or dependencies, discovery false, durable policy retained, root/audio upgrades, host remaps and the unarchive race in evaluating modes. The `STORED_OFF` mutation fails the enum truth table only; `OFF_PRIMARY` fails cached upgrade routing. Restored source hashes match. Earlier failed compile/fixture attempts remain recorded. The first normal push hook attempt passed Tauri checks but failed the gateway startup verifier test (82/83 gateway tests passed). The remote remained unchanged. The retry rebuilt gateway artifacts with separate root and Tauri caches, then passed every normal push hook and published this head. No gateway source changes or hook bypasses were needed. Current-head hosted CI is tracked separately. Human acceptance and the staging binding-timeout/404 workflow remain open. The following retained validation and controls are historical and describe behavior before the mode correction; their Off-enforcement statements are superseded by the Summary above. Published head: `9a717fa89725f9430ff4a8bb78cd600affb8836d`, on main `1055f79d048ef22c878d02a7afb04f49c753564f`. CI, three schema checks, 2,136 ordinary tests, all 291 auth tests, 12 Redis tests, all 1,300 native PostgreSQL tests and discovery passed. The required wrapper passed units, then exited 1 at unavailable Docker bootstrap. BUZZ-510 is complete at `95c8d31c835d6f7ff6e432fb1ed248771d4f06dd`: CI, three schema checks, 2,127 ordinary tests, 12 Redis tests, all 1,295 native Postgres tests and discovery passed; its wrapper passed units and failed at unavailable Docker bootstrap. The first native run at historical BUZZ-510 `5380cb72` passed 1,291 tests and failed two event-follow-up queue assertions and the scale-sync case. All three passed serially, then the full 1,294-test run passed with normal concurrency. The original cause remains unresolved; failed runs and resource samples are retained. Standalone BUZZ-511 checks exposed access-slot callers, a Clippy warning and early HTTP 503 attribution that had remained in the child. All are now corrected in the parent. The unchanged real-router refusal regression passed before the latest commit. BUZZ-512's final tree is unchanged by moving these parent prerequisites. The remaining 46-file child delta was audited; no additional parent prerequisite was found. Earlier compile and native failures remain archived. The following earlier-head results and controls are historical; the current-head validation is reported above. At `48388834f99b8d045e243b619904cb3e27b47146`, based on BUZZ-510 `130f14a675edd7cbe6a69d1c9d7818bd1a51f46c` and reconciled main `152441ae0a56b204b42b6faa56a36ad2b98218b1`: - `just ci`: passed. - Four-package ordinary suite: 2,069 passed; full `buzz-auth` package: 291 passed. - Native Redis: 12 passed. Native PostgreSQL: 1,261 passed. PostgreSQL discovery checks passed. - Required `just test`: units passed, then exited 1 because Docker service bootstrap was unavailable. Native service results above are separate. - Real-router remap checks reject an A-only member under B policy, avoid A data after B admission, and advertise B in discovery despite a stale A cache. Removing each tenant pin separately fails its intended witness; restored source passes. An unmapped host cannot reuse a cached tenant. - Deadline witnesses advance their clock during real HTTP signature verification and retain socket primary-lookup barriers. Removing either final enforced or shadow HTTP deadline check separately fails its intended witness; restored source passes both deadlines and the single-resolution control. - Telemetry regression: unsigned dev proofs now expect the earlier NIP-98 `missing` denial; signed mismatched-key proofs retain pairing denial and signer attribution. Removing unsigned-proof rejection fails the exact two-counter assertion. Restored full relay package: 1,654 passed. - Preserved failed runs: two prior bottom-layer desktop gates failed an immediate inventory assertion, repaired with a bounded wait for the same observable arrays and verified with delayed-notification and frozen-provider controls. An earlier BUZZ-511 native run passed 1,242 tests and failed two deadline fixtures that relied on the tenant lookup removed by the policy pin; the test-only adaptation above replaces that seam. These failures remain archived. The first native run on bottom-layer `130f14a6` passed 1,255 tests and failed partition contention; its focused original test and fresh full 1,256-test run passed without source changes. The cause remains unresolved. The first CI run at `d8d88433` failed the old telemetry expectation; the test-only correction above preserves exact denial counts and attribution. Earlier evidence at `dddf05d8be8195d567db0f989f70825b1246a6c1` (historical head): - `cargo nextest run -p buzz-auth -p buzz-db -p buzz-relay --lib --tests`: 2,139/2,139 passed. - `scripts/postgres-test-run.sh -p buzz-db -p buzz-relay --lib --tests --success-output final`: 1,236/1,236 passed with native PostgreSQL 16 and Redis. Docker's legacy `just test` wrapper was unavailable. - `just ci`: passed, including repository checks, tests and client builds. - **Pre-correction behavior (superseded):** Real ES256 assertions through TCP root/audio upgrades: valid evidence upgrades; missing evidence denies in Off/Shadow, including submitted/inventoried/approved deletion stages. Restoring the old query makes the regression fail on an assertion-less Off/submitted/root HTTP 101. - **Pre-correction behavior (superseded):** Deterministic unarchive race: Notify barriers pause an actual archived-host policy miss; the same required row becomes serving before binding. Both root/audio deny missing assertions and admit valid ES256 assertions in Off/Shadow. Ignoring bound-row policy makes this test fail on an assertion-less HTTP 101. - The full rerun exposed an existing cancellation fixture whose synthetic five-second replica fence expired between probes. Refreshing that test fence per operation preserves its timeout assertions; the full suite now passes. - Local probe: 100 serial uncached primary policy reads took 12.33 ms. Production latency is unmeasured. No packaged-app, human or deployed-fleet acceptance run. Live activation awaits BUZZ-512 and the staging verification tickets. Published with the three-layer stack in one atomic push through normal hooks. The published stack includes main `1055f79d048ef22c878d02a7afb04f49c753564f`, including access logging, storage reporting and mobile read-state changes. Access-slot integration and early HTTP refusal attribution belong to BUZZ-511; activation callback refusal close owners belong to BUZZ-512. Every changed layer requires review at its published head and new-head CI. Fresh trunk, remote membership, queue and immutable-lease guards passed before publication. All nine normal pre-push hook lanes passed; no hooks were bypassed. Remote heads, chained bases, membership, ready/draft states and native tracking were verified after publication. Trunk’s merged 0081 is unchanged. The parent upgrade expansion is 0082 and activation is 0083, with contiguous migrator versions, updated documentation and a populated-upgrade fixture starting at 81. Trunk’s storage-test repair supersedes our earlier local repair. Before publication, main advanced three commits from `1055f79d048ef22c878d02a7afb04f49c753564f` to `ba7d7825ea` (desktop deep-link settling and mobile pairing/sign-in). The normal branch-skew guard passed. Those later trunk commits are outside this validated published head; current-trunk CI and delta review remain required before merge. Generated with Codex --------- Signed-off-by: Brad Seiler <seiler@squareup.com> Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz>	2026-11-05 03:17:19 +00:00
Luke Tornquist	506a588f88	refactor(relay): remove database usage-metric queries (#9040) ... ## Summary The relay no longer runs database aggregate queries to emit usage gauges. Fleet and per-community counts of users, channels, relay members, workflows, git repos, communities, messages, and active users/channels are no longer relay metrics, and neither are the snapshot health gauges that came with them. ### Removed - **Gauges:** `buzz_total_users`, `buzz_total_channels`, `buzz_total_relay_members`, `buzz_total_workflows`, `buzz_total_git_repos`, `buzz_total_active_users`, `buzz_total_messages`, `buzz_total_active_channels`, `buzz_communities_estimated` / `buzz_communities_total`, and every DB-derived `buzz_community_*` series (users, channels, messages, relay members, workflows, git repos, active users, active channels). - **Per-community mode:** `BUZZ_USAGE_METRICS_PER_COMMUNITY=all` and everything it fed: the per-community in-memory gauges, `buzz_community_storage_bytes` / `buzz_community_storage_objects`, `buzz_storage_unmapped_community_bytes`, the community → host map query, and the emission-scope bookkeeping that cleared series for renamed or removed communities. No deployment set it. If one still does, the relay ignores it. - **Health signals:** `buzz_usage_snapshot_available`, `buzz_usage_snapshot_age_seconds`, `buzz_usage_query_skipped_total`, and the `usage_fleet_*` paths of `buzz_db_route_decision`. - **Code and config:** - `buzz-db/src/store/usage.rs`, including its aggregate queries. - The relay's `FleetUsageSchedule` and its cadence/deadline/retry machinery. - The replica-only telemetry route (`Db::route_usage_read`). - `BUZZ_USAGE_METRICS_REPLICA_MAX_AGE_MS` and `DbConfig::usage_metrics_replica_max_age_ms`. If a deployment still sets the env var, the relay now ignores it; startup doesn't fail. - **Reader maintenance pool operation:** `ReaderOperation::Maintenance` and its `close_on_drop` release path existed only for those telemetry reads, so they're removed too. The pool-acquire contract drops from twelve to eleven valid pairs, which lowers the per-pod series ceiling from 216 to 198. - **Dead code:** `Db::active_community_hosts` had no callers. ### Kept - **Leader:** the usage poller's leader lock, plus the leader's invite reaping and fleet storage-snapshot gauges. The session advisory lock (`SessionLeader`, `Db::try_lock_session_leader`) moves to `buzz-db/src/store/leader_lock.rs`, since storage-usage delivery uses it too and it has nothing to do with usage queries. - **In-memory gauges:** pod-local connection, subscription, and online-user gauges. They read process state, not the database. - **Writer maintenance pool operation:** `WriterOperation::Maintenance` stays. Dozens of writer-side maintenance paths across `buzz-db` use it (reapers, retention, the leader lock). The reader variant only ever served the telemetry reads. - **Storage-usage delivery:** unchanged. It reads the storage snapshot for billing, not for metrics. `docs/usage-metrics.md`, `docs/storage-accounting.md`, `.env.example`, and the Helm chart README are updated to match. ## What got simpler This is net deletion (~3.6k lines). One telemetry subsystem is gone: two collection families with independent cadences, availability and age semantics, relay-side deadlines, a dedicated replica routing policy, two config knobs, a per-community emission mode, and a pool-operation variant. Leader election now does only leader work, and it lives in its own module. ## Operational note Dashboards and monitors that read the removed series will go to no data once this rolls out. ## Testing - `cargo fmt --check`, workspace `cargo clippy --all-targets -D warnings`. - `buzz-db` lib, `observability_source`, and doc tests. - `buzz-relay` lib tests. - The Postgres lane for `buzz-db` (lib + tests) and `buzz-relay` (lib). All of these run on a remote workstation at the PR head. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 22:00:30 -05:00
Brad Seiler	a1f79e3aba	feat(relay): persist NIP-FI upgrades (BUZZ-510) (#8985) ... ## Summary Record that a community requires NIP-FI and provide an idempotent operator API to enable it and recover the original receipt. Acceptance checks the current owner, host, lifecycle and existing issuer trust in one transaction. It never creates trust. Receipts survive community deletion so an operation UUID cannot be reused. New acceptances default to disabled through `BUZZ_NIP_FI_UPGRADES_ENABLED`. Disabled calls return HTTP 503 `upgrades_disabled`; identical retries and authenticated readback still recover existing receipts. Policy is permanent: stopping new calls or rolling back code does not clear it. Last-issuer removal is refused for required communities, and bulk backfill leaves missing bindings for explicit operator repair. Apply additive migration 0082 before code. Existing communities remain optional. This layer records acceptance and reports only `pending` or `unavailable`; deploy and verify [per-request enforcement](https://github.com/block/buzz/pull/8988) and [session activation](https://github.com/block/buzz/pull/8991) before enabling acceptance. The global enforce floor remains. See [the operator contract](https://github.com/block/buzz/blob/seiler/BUZZ-510/relay-auth-upgrade/docs/community-auth-upgrades.md) for signing, recovery and containment. ### Related issue [BUZZ-510](https://linear.app/squareup/issue/BUZZ-510/relay-persist-community-nip-fi-requirements-and-expose-idempotent). Base of native stack `8989`; retains trunk's enforcement telemetry from [#8974](https://github.com/block/buzz/pull/8974). Includes a separate commit that repairs an observer timing flake in a desktop test required by the repository gate. ### Testing Controlled local regressions detect the old disabled error code and removal of the last issuer with `allow_last=true`; intended source is restored. Full PostgreSQL validation and the updated live relay/admin exercise remain pending. Human acceptance remains open. No deployment or production activation. Generated with Codex --------- Signed-off-by: Brad Seiler <seiler@squareup.com> Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz>	2026-11-04 19:56:07 +00:00
Luke Tornquist	4813b8cd15	relay: time every WebSocket read and every stored EVENT outcome (#9052) ... ## Summary Two gaps in relay telemetry: WebSocket reads (`REQ`, NIP-AR sync, `COUNT`) have no latency or outcome metric, and `buzz_event_processing_seconds` only measures stored EVENTs that succeed. **What this does** - **`buzz_ws_read_seconds{type, outcome, reason}`** (new histogram). It's recorded from `access::ReadLog::drop`. Every WebSocket read already finishes through that one seam, whether it ends `served`, `denied`, `error`, or `aborted` (future dropped mid-read), so this adds no new exit paths and changes no handler. Duration runs from the moment the handler takes the frame until `EOSE` is queued (`REQ`), the initial set ends (sync), or the total is sent (`COUNT`). `reason` is the existing fixed denial/error code (`query_timed_out`, `database_error`, `not_channel_member`, …), or `none`. The histogram's `_count` doubles as the outcome counter, so one metric gives both read latency and read failure rate. - It's recorded *before* the refusal budget check, because the budget only exists to suppress log lines. When `refused_read` is over budget, it records the sample without building the log line, so its cost under a refusal flood stays the same. - **`buzz_event_processing_seconds{outcome}`**. Before this change it was recorded only when `ingest_event` returned `Ok`. A store that was slow *and failing* (statement timeouts, pool exhaustion) produced no samples, so latency looked better while the store was degrading. It's now recorded on every persistent-ingest outcome: `accepted`, `not_accepted` (for example a duplicate), `invalid`, `auth`, `error`. Queries that sum across all outcomes still work. **Cardinality.** Every label value is a `&'static str` from a code-defined set: 3 types × 4 outcomes, with about 25 reasons that only appear alongside `denied`/`error`. On purpose, there's no `kind` label: `bounded_kind_label` still admits around 200 values, which is too many to cross with histogram buckets. **What got simpler:** `ReadOutcome::parts()` replaces the inline outcome→label match in `Drop`, so the log line and the metric share one mapping. There's no new instrumentation seam: the read metric reuses the access log's lifecycle. ### Related issue None found. This came out of a monitoring-gap review: without these signals, alerts can't catch a slow or failing WebSocket read path, or a failing EVENT store. ### Testing - `access::tests::every_read_is_measured_even_when_its_line_is_suppressed` (unit). It drives the real `ReadLog` constructors and `Drop`: 20 admission refusals against a 16-line budget, a handler denial past the budget, a `COUNT` error, a served sync and an aborted `REQ`. It asserts the exact `(type, outcome, reason, samples)` rows. Two mutants each fail it: removing the over-budget branch in `refused_read`, and moving the measurement after the budget check in `Drop`. - `handlers::event::tests::postgres_tests::event_processing_is_timed_for_accepted_and_refused_ingest` (Postgres lane). It sends one accepted text note, then a refused one from a cached-ban sender, through `handle_event`, and asserts one sample each under `accepted` and `auth` (a ban maps to the `auth` class). Removing the `Err`-arm measurement fails it. - Blox gate at `b314df43f2` (tree `4ca9ebc1`): `cargo fmt --check` clean. The new Postgres test passes, and removing the `Err`-arm measurement fails it. buzz-relay unit lane (`cargo nextest run -p buzz-relay --lib --bins`): 1708/1708 passed. - At `539796bd51` (the same code; only the second commit's test assertion differs): workspace clippy (`--all-targets -D warnings`) passed. The new unit test passed, and each of its two mutants failed it. buzz-relay Postgres lane: 604/605 passed. The one failure was this test's old `invalid` expectation, which `b314df43f2` corrects. - CI at `b314df43f2`: both new tests appear in the CI logs as passing (Rust / Unit Tests, PostgreSQL Domain / PostgreSQL Tests). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 13:15:51 -05:00
Luke Tornquist	25abfc4117	fix(loadgen): bound desktop reads at stop, keep timely EOSEs, model presence and probes (#9016) ... ## Summary Follow-up to #9005, addressing the review findings that merged unaddressed. None of them affect a relay. `buzz-ws-client` gains one method. Each one skews what the desktop client measures when the relay is slow, which is the case the load generator exists for. - **Reads stop at the stop.** A desktop HTTP read (including NIP-11) now races the run's stop and is abandoned there without being recorded, as EOSE waits already were. Before, a read that outlived the 15 s shutdown grace got its client cancelled and failed an otherwise healthy run. `Fetcher::finish` also keeps the read task's handle until the task ends, so a client cancelled during the join still aborts the task instead of detaching it. - **Root lookups are bounded.** The desktop's seeded-root lookups, for thread recovery and replies, now run under the desktop's 30 s read timeout and stop, not reqwest's 60 s. `roots()` takes an optional bound, and the basic human passes none. If the stop cuts a reply's lookup short, the reply isn't published. That applies to the basic human too: a reply whose root lookup finishes after the stop is dropped. - **An EOSE that arrives in time stays in time.** While a publish waits for its OK, `NostrWsConnection` buffers every other frame. The session loop used to judge EOSE deadlines before reading that buffer, and read one frame per pass. It now handles every frame already in hand (at most 256 per pass) before judging deadlines. Each frame is judged by when it arrived, not when it is read. `NostrWsConnection` keeps each buffered frame's arrival time, and the new `next_event_received` returns it. So an EOSE that arrived after its deadline while an OK wait held it counts as a timeout, and so does a frame that answers the probe late. Past that cap, a pass defers due deadlines, for at most one timeout from the first pass that defers them. While it is deferring, the pass only drains, so another OK wait can't expire an EOSE that's already in hand, and sustained traffic can't postpone a missing EOSE forever. - **Presence and the silence probe.** Both modes now open the app's live-only `presence` route (`{"kinds":[20001],"authors":[…],"since":…,"limit":0}`, up to 256 co-members) alongside the profile and membership routes. As in the app, presence takes no part in establishment: it isn't resumed or carried, and if it fails it doesn't cut a renewal's carry for the other routes. After a tick with no frame since the previous one, they send the app's probe, `REQ probe-N {"kinds":[44100],"#p":[viewer],"limit":0}`, which shows up as `desktop_probe` and is closed at its EOSE. Only frames actually received count, including a publish's OK when it arrives before the probe's deadline. A reaction with no target or a publish whose OK timed out doesn't count. Expiry is judged after the drain against the probe still outstanding. If no frame arrives within the EOSE deadline, the session ends as failed and the client reconnects with backoff. Before this, desktop runs left out presence fan-out and idle-client probes, so they understated load. - **Seed log.** "memberships added (N people)" now reads "memberships ensured", since a rerun adds nobody. - **Docs.** `perf/LOADGEN.md` notes that the six-read cap and flat 30 s timeout are simplifications (the app has neither), and that `legacy` is a lower bound. The pre-fix app also opened a per-channel route for each open or non-member channel and re-sent every route each tick, and the model leaves both out. Not done: an explicit count of REQs abandoned at a renewal (report polish; planned cancellations still aren't errors). ### What got simpler Reads abandoned at the stop now go through one helper, `run::bounded`, which the desktop reads and the root lookup share. Before, each read had its own timeout and its own stop check. `global_reqs` returns the routes the client actually opens, not a fixed pair. The probe and presence routes add one `Route` variant, one op and about 40 lines of session code. `Route::establishes` is now the one place that says which routes take part in establishment, replacing the two `Replay` special cases. "Heard" used to be inferred by comparing activity schedules before and after each step. It now comes from frames only, so `publish` reports whether the relay answered and that comparison is gone. ## Testing On a Blox workstation, at the tree of this PR's head: - `cargo fmt --all -- --check`, `cargo clippy -p buzz-test-client -p buzz-ws-client -p buzz-cli --all-targets -- -D warnings` - `cargo test -p buzz-test-client --lib` (parallel) and `cargo test -p buzz-ws-client`, plus the `loadgen::desktop` tests five more times New tests, each confirmed to fail when its fix is reverted: - `a_read_in_flight_at_the_stop_is_abandoned_not_waited_for`: an 8 s roster read at the stop. Without the stop race the client outlives `drive`'s 5 s stop limit. - `a_cancelled_finish_still_aborts_the_read_task`: aborting `finish()` mid-join ends the read task. With the old `take()`, the task is detached. - `an_eose_in_time_is_not_expired_behind_a_slow_ok`: EOSEs at 1.2 s, publish OKs at 3.5 s, deadline 2.5 s. Without the drain, all four routes time out. - `a_silent_tick_probes_and_an_unanswered_probe_reconnects`: the first connection never answers its probe, so the client reconnects and its next probe is answered. - `a_probe_answered_behind_a_late_ok_does_not_reconnect`: the probe's EOSE arrives in time, but it sits behind a heartbeat whose OK comes after the probe's deadline. With expiry judged against a deadline captured before the drain, the session fails and reconnects. - `a_publish_ok_in_time_answers_an_outstanding_probe`: no probe EOSE ever arrives, and heartbeat OKs answer each probe. - `a_reaction_with_no_target_is_not_heard_as_a_frame`: targetless reactions several times a second still leave the tick silent, so it probes. - `a_reply_whose_root_lookup_the_stop_cut_short_is_not_published`: the stop cuts an 8 s root search short, and OKs are withheld. No `kind 9` EVENT reaches the relay, and the client returns promptly. - `a_backlog_defers_deadlines_for_at_most_one_timeout`: the deferral bound and its reset. - `a_long_backlog_behind_slow_oks_keeps_its_timely_eoses`: 600 replayed events per REQ behind 3.5 s OKs, with a 1.5 s deadline and heartbeats every second. The test fails if the deferral is timed from the REQ instead of from the first pass that defers, and it fails if a second OK wait can run between the deferred passes. - `presence_is_a_live_only_global_only_while_someone_is_watched`: the route's order and its live-only filter in both modes, and that it's absent with nobody watched. - `a_late_eose_is_not_answered_behind_a_slow_ok`: the timely case above with EOSEs at 3 s instead, so all four routes time out, not succeed. - `a_probe_answered_late_behind_a_slow_ok_reconnects`: a probe's EOSE arrives 0.2 s after its deadline while a heartbeat waits for its OK, so the session fails and reconnects. The test also checks that a heartbeat followed that probe on the same connection, so the test can't pass on a probe that simply went unanswered. - `an_answered_probe_is_closed_at_its_eose`: `CLOSE probe-1` goes out before any next probe. - `routes_establish_once_and_a_renewal_holds_them_until_a_cut` gains a case: a renewal where presence times out keeps the other routes carried and establishes nothing. Each new test fails when its fix is reverted (seven mutants in the first round and six in the review round, run on Blox). The test harness now has a tick override that can go below the scenario's 1 s floor, and records EVENT frames by kind. The fake relay sends a REQ's replayed events after the REQ's delay, not before, and the fake API's slow reads sleep in `block_in_place`. No existing test changed behavior. Existing expectations moved for the presence route (one more live EOSE per client) and for reads now abandoned at the stop. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 12:57:38 -05:00
John Matthew Tennant	f9ef721e68	fix(media): keep transparency in image thumbnails with WebP (#9003) ... ## Summary This fixes transparent images showing a black background. Image thumbnails were always JPEG, which can't store transparency, so a transparent PNG or GIF upload, such as an agent avatar, rendered on black wherever a client showed its thumbnail. The relay now writes a lossy WebP thumbnail at `{sha256}.thumb.webp`, which keeps transparency, alongside the existing `{sha256}.thumb.jpg`: - **Encoding:** the WebP comes from libwebp through the `webp` crate, built from vendored C source, so the relay needs no system library. A GIF uses its first frame. - **Upload responses:** the upload descriptor's `thumb`, and its per-tenant rewrite, point to the WebP. - **JPEG kept:** installed clients derive `.thumb.jpg` themselves, so the relay keeps writing it until clients move to WebP with a JPEG fallback. Existing images still have only their JPEG thumbnail. - **Serving:** `/media/` serves both variants, each with its own content type. Other `.thumb.*` extensions still return 404. - **imeta:** a `thumb` may name either variant, and the relay checks that the named object exists. - **Storage sweep:** the bucket taxonomy counts both variants as the image's thumbnail bytes instead of classifying `.thumb.webp` as unknown. | 320px thumbnail | JPEG | PNG | lossy WebP (q80) | |---|---|---|---| | Transparent avatar (starter-team `fizz.png`) | 11.3 KB, black background | 65.1 KB | 13.8 KB | | Opaque avatar (`sprout-agent.png`) | 23.6 KB | 156.5 KB | 12.3 KB | | Screenshot (`channel-thread.png`) | 13.4 KB | 62.5 KB | 6.4 KB | Encoding a WebP thumbnail takes 3 to 20 ms. ### Related issue Supersedes #8772, which served a PNG at the `.thumb.jpg` path. ### Testing Red/green against a local relay on `:3000`, built from the branch under test, with Postgres, Redis and MinIO from `docker compose`: ```bash cargo test -p buzz-test-client --test e2e_media test_transparent_png_thumbnail_is_webp_with_alpha -- --ignored cargo test -p buzz-test-client --test e2e_media_extended test_ws_imeta_webp_thumb -- --ignored ``` - **On `main`:** both tests fail. The descriptor's `thumb` is `.thumb.jpg`, and the WebSocket send is rejected with `invalid: imeta thumb must be a local .thumb.jpg path`. - **On this branch:** both tests pass: - Uploading a half-transparent PNG returns a `thumb` ending `.thumb.webp`. - HEAD and GET on that URL return `Content-Type: image/webp`. - The decoded thumbnail keeps alpha 0 on the transparent half and alpha 255 on the opaque half. - `.thumb.jpg` still returns `image/jpeg`. - A kind 9 message whose imeta `thumb` names the `.thumb.webp` is accepted. ## Screenshots The starter-team `fizz.png` avatar uploaded to a relay built from this branch. Both thumbnails are shown as round avatars beside the original PNG. `.thumb.jpg` fills the transparent area with a solid disc, and `.thumb.webp` keeps the transparency.  🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: John Matthew Tennant <jtennant@squareup.com> Co-authored-by: Simone <0828be588d5c9c4d092c5b1cf01761d5aef63d6b104807fd8e8284d517dd1a99@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 10:11:00 -05:00
Luke Tornquist	d21f0c0fdb	perf(relay): keep 1 s read caches warm when the pool is saturated (#9012) ... ## What The relay's 1-second read caches (restriction state, tenant host, open channels) now date each entry from when the read checked out its pool connection, not from when the cache miss began. Concurrent misses on the restriction and tenant-host caches now share one database read. The open-channels cache already did this. ## Why Each of these caches serves an entry for 1 s after "the read that produced it began". Today that start time is taken before waiting for a pool connection. Once pool waits reach about a second (for example, many clients reconnecting at once), every entry is already expired by the time it lands. The cache hit rate falls toward zero exactly when the pool is saturated, so every request does its own read. Those reads queue for the same connections, which stretches the waits further. Requests that time out in the pool then fail closed (`internal error checking restriction state`, `community active check failed; refusing the socket`), and their clients reconnect and add more load. In a local synthetic reconnect load test with a 50-connection pool: - the restriction cache served fewer than one request in five from cache; - the tenant-host cache, which has a single key for the whole deployment, missed more often than it hit; - the open-channels cache missed hundreds of thousands of times on expired entries. ## How - **Stamp at checkout.** `buzz_db::ReadAt<T>` carries a value plus the instant its connection was checked out. The statement's snapshot is no older than that instant, so serving the value for 1 s after it keeps the documented staleness bound, however long the read queued. Three `*_read_at` methods return it: `moderation_restriction_state_read_at`, `lookup_community_by_host_read_at` and `get_listed_open_channel_ids_read_at`. The existing methods now call these and drop the value's timestamp. Each operation keeps one datastore span under its existing name, carried by the `*_read_at` method. - **Share concurrent misses.** `SharedReads<K>` is the per-key gate the open-channels cache already used, moved into its own module. The open-channels cache, the restriction cache and the tenant-host cache now all use it. The first miss reads; the others wait for it, then recheck the cache. A waiter waits for at most the writer pool's acquire timeout plus `SHARED_READ_STATEMENT_WAIT` (1 s). That covers the leader's own wait for a connection, which is what grows in a storm, so waiters keep sharing its read instead of queueing duplicate reads behind it. A failed or cancelled leader releases its waiters at once. A leader whose statement hangs stalls them for at most 1 s past the acquire timeout, never the statement timeout. - **Reject hosts no community can have.** `bind_community` rejects a host longer than `communities.host VARCHAR(255)` can hold (`buzz_core::tenant::MAX_HOST_LEN`, counted in characters) as unmapped, next to the empty-host check. An unauthenticated caller's arbitrarily long host therefore never reaches a resolver, cache, gate or the database. No staleness bound changes: every entry is still served for at most 1 s after its read's snapshot. Invalidation paths are unchanged. A read in flight at an invalidation can still refill the old row for at most that bound, as documented today. ## What got simpler The open-channels gate's hand-rolled `try_lock`/`timeout` block is replaced by `SharedReads::turn`. One gate implementation now serves three caches instead of one implementation that only one cache had. The gate's wait is set once per gate rather than passed at each call site. Provisioning and host binding share one `MAX_HOST_LEN` instead of a private constant. ## Tests - `shared_reads::tests`: one caller reads while the rest wait; a waiter gives up on a hung read after its bound. - `concurrent_restriction_and_host_misses_share_the_in_flight_read`: waiters on both caches use the leader's entry; with an unreachable database, any waiter that read on its own would fail. - `postgres_tests::cache_misses_behind_a_held_pool_share_one_fresh_read`: with a one-connection pool held for longer than every max age, eight concurrent misses on each of the three caches cost one database read, and each entry still lands fresh. - `tenant::tests::an_overlong_host_fails_closed_before_the_resolver`: a host over the limit fails as unmapped with no gate entry and no lookup; a host at the limit, including one in multibyte characters, is still looked up. - Mutation-checked: each of the following turns these tests red: stamping any of the three entries before the pool wait, dropping either new waiter path, letting waiters give up after the cache's max age, dropping the host length check, or counting the host length in bytes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 09:52:21 -05:00
Kalvin C	3d26ed7dfd	feat(relay): match verified names in profile search (#9019) ... ## Summary `POST /profiles/search` only matches kind:0 `name` and `display_name`. In communities that publish NIP-FI verified profiles, a person's surname or directory username often appears only in their relay-signed kind:30382, so typeahead cannot find them by it. - Where the community advertises verified profiles, the search also matches the `display_name`, `username` and `full_name` tags of each current kind:30382 head signed by the relay key. - Verified names need a trailing prefix of at least three characters, as in NIP-50, so a short prefix doesn't make the relay rank every verified head. kind:0 keeps one-character typeahead. Lookups from heads to `events` also match `created_at`, so Postgres prunes to one partition. - Results are paged by person. Each person returns their current kind:0, when they have one, then their current kind:30382, when they have one, whichever matched. A person is ranked by their best match across both sources, using the existing tiers. - NIP-11 `federated_identity.public_profile` gains `searchable: true`, so clients can rely on the response instead of a separate kind:30382 read. - Migration `0081` adds a GIN expression index over the head's name tags on `nip_fi_profile_heads` (built-in `jsonb_path_query_first` and `to_tsvector`, no new functions). The table holds one row per verified person. Like `0078`, the migration builds the index only while the table is empty; populated databases prebuild it with `CREATE INDEX CONCURRENTLY` (`docs/profile-search.md`), and the migration then checks it is valid and has the expected definition. Compatibility: the request schema is unchanged. Existing clients that keep only kind:0 events work as before and also find people who match only by verified name. Trust stays pinned to the relay `self` key; the event shape follows NIP-85 user assertions, but discovery is relay-advertised rather than kind:10040. ## Testing - `buzz-search` Postgres integration: new `profile_names_match_verified_names` covers forged-key heads, replaced heads, other communities, people without kind:0, per-person grouping, paging, and words split across sources; the existing profile-name tests pass unchanged. The test fails when the author filter is inverted. - `buzz-relay` end to end in enforce mode: `profile_search_matches_verified_names_where_advertised` publishes through the real writer and finds the person by surname only where verified profiles are advertised. - Desired-state and migration index definitions match. - `profile_names_use_verified_name_index` asserts the search scans the new index. Migration `0081` checked against an empty table, a rerun, a populated table without the index (raises), the documented concurrent prebuild, and an index with a different definition (raises). - Latency, 50k people each with a kind:0 and a verified head, release build, local Postgres, best of 5, 30 people per page: 11–22 ms with verified names versus 1–8 ms kind:0 only. `EXPLAIN` shows a bitmap scan on the new index. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Kalvin Chau <kalvin@block.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 21:25:15 -05:00
Luke Tornquist	865b150371	fix(db): use ScratchDb in the storage usage delivery test (#9032)	2026-11-04 00:30:32 +00:00
Will Pfleger	0a06c0b3cb	feat(relay): log reads, logins and connections on access target (#8376) ... 🤖 Adds an access log for relay reads, logins, connections and HTTP requests on the dedicated `buzz_relay::access` target. It writes to stdout, not `audit_log`, so reads never take the audit advisory lock. The log is shaped for volume: one line per HTTP request, one line per WebSocket read, and one line per WebSocket connection. ## Lines per operation | Operation | Lines | |---|---| | HTTP request on a matched route | 1 (`http`), except the skip list below | | HTTP request matching no route | 1 (`http`, `route="<unmatched>"`) at status 400 and above, otherwise none | | WebSocket connection | 1 (`close`, carrying the login) | | WebSocket `REQ` / `COUNT` / NIP-AR sync | 1 each (`req` / `count` / `sync`), plus the `buzz_ws_reads_total` counter | | WebSocket NIP-42 login | 1 (`auth`) per decided or cut-off attempt, capped after the decision as below | | Audio login | 1 (`auth`, at its final admission check) | | WebSocket `EVENT` | 1 (`Event ingested`, not on the access target) | ## HTTP requests One middleware, `http_access_log`, writes every HTTP line. It sits between `track_metrics` and `http_trace_layer`, so rebuilt errors, body-limit refusals and localhost refusals all reach it. It writes the line when the response is ready, which is not proof the body was delivered. A request dropped before then logs `request=aborted` at WARN with no `status`. - Every line carries `route` (the matched route template, never the raw path), `peer_addr` when the listener has one, `status`, `time_to_response_ms` and `http_method` (the HTTP verb). The verb stays on the line itself because the `http.request` span is absent under an access-only filter and when a pending request is dropped. - Handlers record what they learn on the request's `AccessSlot`: the login (`method`, `pubkey`, `pubkey_verified`, `community`), the decision, the filter summary, and the route's own fields. The decision is `outcome` (`success` / `denied` / `failed` / `challenged`) with a fixed `reason`, or `undecided` when the route recorded none. A refusal is final: a later admission doesn't replace it, so a replayed NIP-98 proof that verifies and is then refused logs `failed` / `nip98_replay`. - `outcome` describes the login. What the route did is in `status` and the route's fields: a rejected `/events` write logs `outcome=success` with `status=400`, `accepted=false` and its `reject_reason`. - Route fields: - `/query` and `/count`: the filter summary and `result_count`. An incremental `/query` response (`{changes:[...]}`) counts its changes, removals included. Artifact reads record their summary before the query runs, so a read dropped mid-query keeps it. - `/events`: `accepted`, `kind`, `event_id`, and on a refused write the parse `category` / `line` / `column` or a fixed `reject_reason` (such as `invalid` or `restricted`), never the ingest message. - Invite mint: `invite_id`, `expires_at` and `max_uses`. Invite claim: `joined`. - Admin feedback attachment reads: `feedback_id` and `attachment_sha256`. With `BUZZ_ADMIN_AUTH=disabled` they log `method=network_trusted` with no key, and Host or Origin refusals log their fixed reason. - Every other route logs only the login fields. - No field can hold an invite code, a token, or a Blossom read hash. Media routes log the `/media/{sha256_ext}` template. - Skip list: NIP-11 and other non-upgrade requests on `/`, `/info`, `/.well-known/nostr.json`, `/health`, `/_liveness`, `/_readiness`, and the SPA and static files the relay serves when enabled (web assets, invite landing pages, git web pages, the admin bundle) at any status. Successful WebSocket upgrades (101) log no `http` line. Refusals before the upgrade on `/` and the audio path log one, with the status actually sent. - Routes that aren't Nostr-key logins (the command-token disconnect endpoint, the workflow webhook, the git server's internal callback) log `outcome=undecided` with no key. - Dev-mode `X-Pubkey` requests log `method=x_pubkey`, `pubkey_verified=false`. - A successful `/query` or `/count` that returned nothing (`result_count=0`) logs at DEBUG: on staging, 93% of `/query` lines were empty polls. Denied, failed, aborted and error reads, reads with no `result_count`, writes and logins keep their levels. - Other successful lines and git's credential-less first request (`challenged`, with its 401) are INFO. Refusals, statuses of 400 and above, and aborted requests are WARN. In Enforce mode the line records the remapped status (for example 403). Denial bodies and headers are unchanged. ## WebSocket reads - `REQ` / `COUNT` / `sync` lines carry `pubkey`, `conn_id`, `sub_id`, the filter summary, the outcome (`served` / `denied` / `aborted` / `error` plus a fixed reason), and `events_queued` (REQ and sync) or `count` (a COUNT that counted). A `REQ` is `served` only once its EOSE is queued to the client. A read logs WARN only when it ends in `error`. `served`, `denied` and `aborted` are INFO, since `aborted` means the read ended before completing: the client disconnected or sent CLOSE, a same-ID `REQ` replaced it, or the session was cancelled. - A NIP-AR sync subscription writes its `sync` line when its initial set completes, not when live delivery ends. Its summary is the registered live filter (kinds and channels, never artifact predicates), and `events_queued` counts the initial set. - `REQ` lines stay at INFO even when empty at EOSE, because the subscription can still deliver live events later. - Filter summary, shared with HTTP `/query` and `/count`: `kinds`, channel ids, authors and `p` values each list the first 10 distinct values. The exact `kind_count` / `channel_count` / `author_count` / `p_count` (summed per filter) appears only when the list is `*` or shows fewer values than the filters held. A dimension that any filter leaves open logs `*`. `since` / `until` / `limit` take the widest bound and are absent when any filter leaves them open, so no field understates what the read covered. `ids` is a count only. NIP-50 search logs `search: true`, never the text. Only well-formed channel ids and pubkeys are listed. An artifact read's summary uses only the standard dimensions of each raw filter, so artifact predicates and client-defined tags (such as `#1`) are never logged. The summary is skipped when the access target is off. Under `=warn`, every WARN line, HTTP or WebSocket, still carries it. - Per connection, only the first 16 reads that are refused before they run (`auth_required`, quota, concurrency, overload) or denied are logged. The rest are counted as `reads_suppressed` on the close line. A refusal that arrives after the close line has been written, from a read still running when the client left, logs its own line, so every refusal is either logged or counted, never both. Served reads are never capped. ## WebSocket read counter `buzz_ws_reads_total{type="req"|"count"|"sync"}` counts every parsed `REQ`, `COUNT` and NIP-AR sync frame before admission, exported at zero from startup. It is a metric, so it counts at any `RUST_LOG`, including production's current `error`. ## WebSocket logins A NIP-42 login writes one `auth` line when it is decided. A successful login logs INFO `outcome=success` at its commit point, decided under the connection's close-reason lock, so a disconnect racing the login (session expiry, admin disconnect, revocation, ban or roster removal) either wins, and the login logs no success, or loses and closes the connection right after the success line. A refused or failed login writes one WARN `auth` line with `outcome`, a fixed `reason` from the shared `AuthReason` code set, `pubkey` and `pubkey_verified`. Root WebSocket `auth` lines carry `peer_addr`. AUTH frames after the login is decided are capped at 16 per connection, and the rest are counted as `auth_suppressed`. An attempt cut off by close or session expiry logs `aborted`. Audio logins log one `auth` line at their final admission check. ## Connections The close line is the connection's only line. It is owned by the upgraded socket, so it is written exactly once however the connection ends: a normal close, a refusal while checking the community is active (`refused=not_admitted`), the connection limit (`refused=connection_limit`), or the handler being dropped. It carries `conn_id`, `community`, `peer_addr`, `duration_ms`, the login, `events_queued` / `bytes_queued` (frames the writer accepted and flushed, not client delivery), and the two suppression counters. ## Error text - Database errors log as `DbError::kind()` (SQLSTATE or variant), including the command-executor event insert. Search errors, on both read paths and in the search-cancel task, log as `SearchError::kind()` / the same SQLx classifier. Workflow database and webhook errors log as their codes, since webhook URLs can be secrets. - Git NIP-98 failures log a fixed code. Git subprocess stderr moved to DEBUG because hook output can contain message text. ## Operator notes `.env.example` documents the target, its `event` values (`http`, `auth`, `req`, `count`, `sync`, `close`), the skip list, the levels, and how to turn it down: `buzz_relay::access=warn` keeps refusals, errors and aborted HTTP requests, and of WebSocket reads only those that end in `error`, and `=off` drops everything. It also lists the removed lines: `NIP-42 auth successful/failed`, `WebSocket connection established/closed`, `event=connect`, `HTTP bridge request`, `Event ingested via pipeline`, `relay invite minted`, `relay member added via invite`, `admin feedback attachment read`, and `addr` (now `peer_addr`). JSON lines no longer repeat the current span as `span`, since `spans` already ends with it. A line logged under an explicit `parent:` span loses that parent's fields. The default OTel export filter sets `buzz_relay::access=off`, so access lines aren't exported a second time as span events. `peer_addr` is the TCP peer. `BUZZ_CLIENT_IP_HEADER` optionally names a request header (for example `cf-connecting-ip` behind Cloudflare or `x-real-ip` behind nginx, matched case-insensitively). When set, `http`, `auth` (root and audio), `close` and connection-limit lines also log `client_ip`: the rightmost comma-separated entry of the header's last field, only if it parses as an IP address. It never replaces `peer_addr`, and it is only trustworthy when every request arrives through a proxy that overwrites the header. --------- Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>	2026-11-03 17:48:59 -05:00
Shu Hui	fa7a3bfbda	feat(relay): push per-community storage usage to kgoose (#8714) ... ### Why? To bill hosted communities for storage in Orb, kgoose needs per-community storage usage. The relay already computes it in a daily snapshot, but nothing delivered it anywhere. ## Summary - Adds opt-in `BUZZ_STORAGE_USAGE_URL`. When it is set, one leader-elected relay pod checks every 5 minutes for a new storage snapshot and POSTs it to kgoose's `POST /v1/buzz/storage-usage/record` (squareup/cash-server#144285) in consecutive batches of up to 50 communities until every community is sent. - Each batch is `{measured_at, communities: [{community_host, storage_bytes, storage_objects}]}`. Active communities with no stored bytes are reported as zero, and re-sends are safe because kgoose bills each community once per UTC day. - Retries on 408, 429, 5xx, and transport errors. No migration; documented in `docs/storage-accounting.md`. kgoose authenticates the relay as the `buzz` mesh service, so the relay needs mesh egress to kgoose-builderbot. ### Related issue None found. Complements #7899 (snapshot history in the DB). Receiving endpoint: squareup/cash-server#144285. ### Testing Unit tests cover batching, retry and reject handling, zero-filling, and a real HTTP delivery that checks the exact kgoose request body. A Postgres-gated DB test covers the new reads. End to end, a local relay delivered to a fake endpoint once per new snapshot. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Shu Hui <shu@squareup.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-11-04 09:40:06 +11:00
Salman Mohammed	6847882b14	fix(relay): stop logging unmapped-host NIP-FI denials as server failures (#8997) ... ## What this does In enforce mode, a request whose Host maps to no community gets NIP-FI's `authorization_unavailable` 503 (`docs/nips/NIP-FI.md`, Rejection and privacy table). tower-http's default classifier treats every 5xx as a failure, so each of these requests also logs a `tower_http::trace::on_failure` error. That line has no host or path, only `503` and `latency: 0 ms`. When a scanner probes a host this relay doesn't serve, it produces thousands of these lines in a minute, and they trip error-rate alerts even though nothing is down. This PR keeps the wire response the same and stops counting these denials as server failures: - `nip_fi_core::community_denial` builds the same `http_denial` response. When the Host is confirmed unmapped, it adds an internal `UnresolvedHost` response extension. Extensions never reach the wire, so the NIP-FI denial contract ([FI-TRACE-DENIAL-ORACLE]) still holds. - Each site that turns a community-resolution failure into a denial now uses it: - the router guard (`nip_fi_assertion_guard`) - the WebSocket upgrade (`check_nip_fi_at_upgrade`) - HTTP admission, for routes the guard exempts (`Stage::Community`) - The HTTP trace layer now uses `RelayResponseClassifier`. It treats marked responses as expected and keeps tower-http's 5xx-is-failure rule for everything else. Other 503s still log as failures. That includes an unavailable JWKS key source on a mapped host, `deny_protected`, a stale snapshot, and a failed database lookup. ## Why not return 4xx for unmapped hosts? NIP-FI.md maps "community/Host resolution unavailable" to 503, and `nip_fi_enforce_unmapped_host_is_503_before_verification` pins that ruling. Changing the status would be a spec change. This PR only changes how the relay logs these denials. ## Database community source With `BUZZ_NIP_FI_COMMUNITY_SOURCE=database`, a not-found host and a failed lookup (DB error, timeout, shed waiter) return the same denial class. The marker is set only when the host is confirmed unmapped: a missing Host, a non-bindable host, or a live negative-cache entry. Resolver outages stay unmarked and still log as failures. With the `env` source, every resolution failure means the host is unmapped, so it is always marked. ## Testing - `nip_fi_enforce_unmapped_host_is_503_before_verification` (guard, root upgrade, audio upgrade) and `enforce_unmapped_host_is_503_before_nip98_and_verifier` (HTTP admission) now also assert the marker. Mutation check: removing the guard marker turns the router test red. - New `nip_fi_mapped_host_dependency_503_is_not_marked_unresolved`: a mapped-host 503 from an unavailable key source is unmarked and still classified as a failure. - New `relay_response_classifier_skips_only_unresolved_host_denials`: covers marked 503, unmarked 503, 500, 200 and 401. - New `only_confirmed_unmapped_hosts_are_marked_as_expected_denials` (database source, all three sites): a negative-cached host and a non-bindable host are marked; a bound host whose lookup fails is not. Mutation check: marking every database failure turns it red. - `cargo clippy -p buzz-relay --all-targets -D warnings` is clean, and the pre-push `rust-tests` lane passed. - Local relay, before and after. Setup: enforce mode, `RUST_LOG=error`, one mapped host, JWKS unreachable on purpose. Both builds got the same requests: | Request | Status (both builds) | `on_failure` errors before | after | |---|---|---|---| | Unmapped host, 4 protected paths | 503 | 4 | 0 | | Unmapped host, WebSocket upgrade on `/` | 503 | 1 | 0 | | Mapped host, no assertion | 401 | 0 | 0 | | Mapped host, well-formed JWT, JWKS unreachable | 503 | 1 | 1 | | Unmapped host, `/assets/` path and `POST /api/invites/claim` | 404 | 0 | 0 | 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com> Co-authored-by: opus <f038cc09b32a776e7d95b65c267e9d851777ee2b6cdbd46a48edba01b1d10ebd@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 16:08:13 -05:00
Luke Tornquist	152441ae0a	feat(loadgen): model desktop clients and large-community shape (#9005) ... ## Summary The load generator models every community as many small open channels, and every human as a bare WebSocket client: four REQs, a little history, then nothing until the run ends. Large deployments don't look like that. They have thousands of people in a few communities, a handful of very wide channels, and some people in hundreds of channels. Their desktop clients also make most of their relay requests over HTTP and on timers. This PR adds opt-in scenario knobs for both, so a run can reproduce that shape and compare the desktop client before and after its recent request-volume fixes. Every new key defaults to the previous behavior. Existing scenarios parse, assign members and run exactly as before. ### Shape | Key | Default | Meaning | |---|---|---| | `shape.private_channels` | `false` | `seed` creates channels private and adds every member with kind:9000 before posting history. Members are the humans and agents assigned to each channel up to the largest ramp step, plus the hot thread's root and authors. Events carry up to 100 members each and run `seed_concurrency` at a time. A rerun adds nobody new, and any member the relay refuses fails the seed. | | `shape.wide_channels` | `0` | The first K channels of each community are wide channels. | | `population.wide_member_fraction` | `0.0` | Share of humans that join every wide channel on top of their own. | | `population.wide_activity_share` | unset | Share of its typing, messages and reactions a member of the wide channels sends there; the rest goes to its regular channels. Unset keeps the uniform pick, which sends half of a 3-wide/3-regular member's traffic to the wide channels and makes fan-out, not reads, the bottleneck. | | `population.tiers` | `[]` | `[{ fraction, channels }]`: that share of humans joins `channels` channels instead of `channels_per_member`. | Assignment is deterministic by global index, so `seed` and every `--shard` agree without coordination. With no wide channels and no tiers it is the old assignment, and a test pins that. ### Desktop clients (`[desktop]`) | Key | Default | Meaning | |---|---|---| | `fraction` | `0.0` | Share of humans that run the desktop model instead of the basic human. | | `mode` | `"current"` | `legacy` is the client before its volume fixes; `current` is after them. | | `windows`, `threads` | `3`, `1` | Open channel windows and threads, which are refreshed on connect. | | `tick_secs` | `60` | Recovery tick period. | | `backstop_secs` | `900` | `current` only: each tick item (subscription replacement, roster, unread repair) waits this long since it last ran. | | `renewal_secs` | `0` | Identity renewal period. `0` disables renewal, and the first renewal is jittered. | A desktop human keeps the basic human's presence, typing, messages and reactions. What it adds: - **Cold start.** It reads NIP-11 and opens its profile and membership subscriptions. Its channel subscriptions are batched per mode: - up to 10 channels: one replaying batch; - `legacy` with more than 10: one per-channel batch per 10 channels; - `current` with more than 10: a live-only subscription per 2048 channels, plus replays of 10 that are closed at EOSE. It also makes the app's HTTP reads: roster, channel names, sidebar, profiles, statuses, emoji, read state, unread repair, reminders, window catch-up plus notices for each window, and root plus thread page for each thread. A window catch-up follows the cursor in the relay's window bounds (kind 39006) for up to five older pages, until the bounds say history is exhausted. - **Route establishment.** `legacy` makes every read as it connects. `current` reads NIP-11, names and sidebar up front, and the rest as each subscription reaches EOSE, by its replay outcome. A replay of 500 events, a missed deadline, or the live-only batch over 10 channels is a cut, and so is a route the previous connection never caught up. Only an EOSE catches a route up: one recovered after a missed deadline cannot be resumed or carried by the next connection. - A continuous profile route skips its read (unless one is owed). - A continuous membership route only repairs unreads, without the read-state snapshot, unless a snapshot read failed since; then it runs the full repair. - A cut global reads in full, and channel routes always recover windows and threads. - **Ticks and renewals.** These follow each mode: - `legacy`: every tick replaces subscriptions and rereads roster and unreads, and a renewal is a full reconnect; - `current`: tick items wait for the backstop, measured from the last replacement, the last successful roster read, and the last full unread repair. A failed roster read, unread repair or read-state snapshot is owed and retries on the next tick; snapshot debt is tracked apart from repair debt, so a repair without the snapshot never settles it. A renewal carries its routes: nothing is read while every replay keeps continuity, and once one is cut every route establishes as on a reconnect. With more than 10 channels the live-only batch is always cut, so the channels recover and unreads are repaired, but globals are not reread. - **Reporting.** Each request type is its own op (`desktop_roster`, `desktop_unread`, `desktop_replay_eose`, …). - **Bounds.** At most 4 REQs set up at once, with a 10 s EOSE timeout. A tick replacement that misses it is closed, the batch it would have replaced stays, and the next tick retries. At most 6 HTTP reads in flight, with a 30 s timeout. A read that comes due while the same read is running or queued is not added again, and each read leaves the running set as soon as it settles, so a failed read can retry while a slow one is still under way. Connects and failed sessions back off. A stop abandons pending EOSE waits without counting them as failures. - **Not modeled.** The app also establishes channel routes when a tick replacement reaches EOSE; the generator does not. `perf/LOADGEN.md` documents the knobs and the model. `perf/scenarios/local-desktop.toml` is a short local example with every knob on. ## What got simpler - `human()`'s schedule loop (presence, typing, messages, reactions, session end) moved into one `Activity`. The basic and desktop humans share it, so neither client duplicates the schedule. The basic human's random draw order is unchanged. - Membership stays in one place. `member_channels` remains the single assignment function, and `channel_members` derives from it rather than from a second formula, so seeding and the clients can't drift apart. The desktop model is a new concept: about 1,420 lines of model in its own module, plus about 1,300 lines of tests. Everything else goes through existing seams: `Stats`/`Op`, `connect`, `Backoff`, `observe`, `roots` and `Net`. Identified debt, not fixed here: CI's `test-unit` runs `buzz-test-client --lib` with the filter `loadgen::`, so the five crate-root `tests::` in `lib.rs` (relay message parsing, auth and text event building, `collect_until_eose`) never run in CI. That gap predates this PR, which doesn't touch `lib.rs`. ## Testing - `cargo fmt --check`, `cargo clippy -p buzz-test-client --all-targets -- -D warnings`, and `cargo test -p buzz-test-client` (plain parallel): 84 lib tests pass. - Wide activity: a seeded `Rng` over 100k picks gives each share within 1%, and an unset share makes the identical draws to before. A run against a loopback relay checks that a human's typing lands only in its regular channels at share 0, and only in its wide channels at share 1. - Scenario: validation of every new key; deterministic wide-channel and tier assignment that honors the fractions; `channel_members` inverts the assignment up to the peak; the default assignment is unchanged; the shipped scenarios parse. - Seed: with private channels, a fake relay sees the following, in order: - every 9007 before any 9000, and every 9000 before any post; - owner signatures; - ≤100 members per event; - exactly the expected members per channel; - posts authored by channel members. A refused member fails the seed, and open channels add nobody. - Desktop: request planning, window bounds validation, route establishment, and tick and owed-read rules are unit-tested. End-to-end tests drive clients against loopback WS and HTTP fakes. The WS fake replays a planned number of events per subscription and can delay an EOSE or drop a connection. The HTTP fake answers window pages with relay-signed bounds, an older aux event and equal-timestamp rows, refuses unknown cursors, and can fail a read once. They check: - exact cold-start counts per op for `legacy` and `current` at 5 and 25 channels; - a window catch-up sends the head, then exactly the bounds' cursor (`until` + `before_id` of the last row, not the older aux event), and stops when the bounds say history is exhausted even though that page came back full; - after a `current` renewal, 499 replayed events everywhere read nothing over HTTP; 500 on the channel batch adds channel recovery plus a covered unread repair; 500 on membership adds roster, read state, unread and reminders plus channel recovery; 500 on profiles adds profiles, statuses and emoji plus channel recovery and a covered repair; - after a dropped connection, complete global replays read only NIP-11, channel recovery and a covered repair; truncated ones reread both globals in full; - with renewals shorter than the backstop, read-state snapshots still recur at the backstop; - a tick replacement past its EOSE deadline is closed, the old batch stays, the next tick replaces it, and one batch wire stays open; - a failed roster and unread read retry on the next tick, and successful reads do not run again; - a failed roster read retries on the next tick while an unrelated slow read is still running; - after a failed snapshot, a continuous membership establishment before any tick still reads the snapshot; - a route whose EOSE missed its deadline is recovered, and after a renewal with a complete replay it is recovered in full again rather than carried; - `legacy` ticks replace batches (new REQ before the CLOSE of the old one) and reread; - `current` ticks do nothing until the backstop, then replace live-only; - `legacy` renewals are full reconnects; - `current` renewals carry small sessions without rereading globals or channels, and recover channels in large ones. Each of the nine review-fix tests was checked against a mutant that restores the old behavior, and every mutant fails its test. - Smoke: a local relay with Postgres and Redis. I ran `seed` twice, then `run` of `local-desktop.toml` (60 s) once per mode. - Both seeds were accepted (780 of 780 posts each), including private channels and memberships for 340 people. Neither run aborted, no client was cancelled, and no op recorded an error. - Each run used 18 desktop clients: | op | `current` | `legacy` | |---|---|---| | `desktop_live_eose` | 266 | 483 | | `desktop_replay_eose` | 24 | 0 | | `desktop_roster` | 72 | 289 | | `desktop_read_state` / `desktop_unread` | 72 / 99 | 289 / 289 | | `desktop_profiles` | 18 | 73 | | `desktop_nip11` | 18 | 73 | | `desktop_window` | 372 | 596 | Window reads are below the first smoke (406 and 1086) because catch-ups stop where the relay's bounds say history is exhausted, rather than paging while pages came back full. They are above the previous revision (179 and 470) because each read now leaves the running set as soon as it settles, so a channel recovery is no longer dropped as a duplicate of a read that already finished inside a still-running batch. - The first smoke caught two relay contract mismatches that the fakes had missed: - top-level window pages need `before_id` with `until`; - the read-state snapshot answers with an object. Both are fixed, and the fakes now enforce both. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 15:12:39 -05:00
Luke Tornquist	88697f1c59	fix(relay): apply kind 5 / 9005 deletions atomically (#8944) ... ## Summary Kind 5 and 9005 deletions now store the event and soft-delete their targets in one transaction. Before this PR the deletion committed first. The target soft-delete ran afterwards as a best-effort side effect, and a failure there was only logged. Resending the deletion then returned `duplicate:` without retrying. A crash between the two commits had the same result. Either way the relay accepted a deletion while its target stayed live. The bridge `include_aux` closure still compensates for those rows. Its second hop returns deletions of the reactions and edits it found, so clients can cancel them. This PR keeps that hop, because rows already torn this way can't be repaired by a resend: - Ingest rejects events more than 900 seconds from server time. - Clients don't resend deletions that were accepted. - During a rolling deploy, old pods keep writing torn rows until they are replaced. Dropping the hop waits for a follow-up, once existing torn rows have had a verified repair and no old writers remain. ### Changes - **Atomic deletion.** Kind 9005, and kind 5 with `e` targets, go through `Db::insert_deletion_event`, driven by `side_effects::persist_deletion`. The write stores the event and its mentions, soft-deletes the targets and updates their thread counters. It follows the existing `persisted_atomically` path used for workflow deletion and kinds 9000/9002. - A failure stores nothing and rejects the event, so the client can resend. - A resend of a stored deletion inside the ingest window soft-deletes any target still live and is dispatched as inserted. It doesn't repair older rows. - Live 39005, reaction-table cleanup and the 9005 system message still run after the commit and stay best-effort. - Kind 5 deletions of `a` coordinates stay a side effect. - **Closure unchanged.** The channel window and legacy thread-reply paths keep both hops, and the docs (`docs/bridge-channel-window.md`, `docs/nips/NIP-CW.md`) are unchanged. ### What got simpler Deletion no longer has a window where the event is accepted but not applied. `handle_standard_deletion_event` is now only the `a`-coordinate path. Target planning and soft-delete for `e` targets live in one place, `persist_deletion`. ### Follow-ups (not in this PR) - Repair the existing torn rows: soft-delete the live targets of stored kind 5 / 9005 `e` deletions and fix their thread counters. Then drop the closure's deletion-of-aux hop. - Hop 1 still probes every partition once per row id. A partition bound or the indexed `reactions` table would cut that. ### Related issue N/A, found while investigating relay database CPU. ### Testing - `torn_deletions_still_cancel_live_targets_in_both_windows` (bridge Postgres) stores kind 5 and 9005 deletions without soft-deleting their reaction and edit targets. It queries the channel window and the legacy thread path with `include_aux`, without a resend, and asserts each live target comes back with its deletion. - `insert_deletion_event_target_failure_stores_nothing_and_resend_applies` (buzz-db Postgres) uses a trigger to make the target's soft-delete fail. It asserts nothing is stored and the target is unchanged, then that a resend applies both. - `insert_deletion_event_resend_repairs_a_live_target` (buzz-db Postgres): a resend repairs a deletion stored without its target's soft-delete, and a further resend changes nothing. - `failed_target_soft_delete_stores_nothing` and `resent_deletion_repairs_a_live_target` (relay Postgres) cover the same two cases through ingest, for kinds 5 and 9005. - `test_channel_window_rows_overlays_and_exact_multiple_exhaustion` (`e2e_nostr_interop`, Relay E2E lane) deletes its reaction and asserts the reaction leaves the closure. - Blox: fmt, clippy (`buzz-db`, `buzz-relay`, `buzz-test-client`), and the `buzz-db` and `buzz-relay` Postgres lanes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 14:38:26 -05:00
Luke Tornquist	b8b425e8b1	fix(db): drop a superseded head's mention rows with it (#8948) ... ## Summary Replacing a replaceable or addressable event soft-deletes the previous head, but its `event_mentions` rows stayed behind. Only the hard-delete kinds (read-state and mesh status) removed theirs, in a separate statement. No mention-index read (`#p` filters and feeds) returns a soft-deleted event, so those rows are pure cost: every `#p` read walks them and throws them away. A kind:39002 roster republish leaves one dead row per channel member, so a large channel piles them up with every join. Relay-emitted replaceable lists with p-tags do the same. The statement that retires the old head now deletes that head's mention rows as well, through a data-modifying CTE over `RETURNING id`, on all three supersede paths: - the relay's kind:39002 roster writer (`LockedMemberSnapshot::replace_member_event`) - `replace_addressable_event` (kind 0/3/10000–19999, including the relay-signed kind:13535 list) - the NIP-33 parameterized path Because the CTE covers every row the retire statement touches, a coordinate that somehow had more than one live row also gets cleaned up. The shared `superseded_with_mentions_sql!` macro sits at the crate root so all three mention-indexing replacement paths use the same statement. The kind:13534 membership-list and push-lease writers also retire heads, with a bare `UPDATE`, but neither indexes p-tags, so they leave no mention rows. ## What got simpler The hard-delete special case (`if hard_delete_superseded { DELETE FROM event_mentions … }`) is gone. Both retire statements go through one macro, so a head and its mention rows leave together on all three paths. ## Not in this PR - Mention rows already left behind by earlier supersedes. `docs/database.md` rules out data backfills in migrations, so these get cleaned up out of band, in batches. - Other soft-delete paths (NIP-09 deletion, moderation, channel deletion retiring its 39000–39002 events) also keep mention rows. They are low-volume, and are noted here as identified debt. ## Tests `superseded_heads_drop_their_mention_rows` (pg) has one table row per supersede path: the member-snapshot roster writer (lock, replace, release), the addressable path, the NIP-33 path, and the two NIP-33 kinds that are hard-deleted (NIP-RS read state and mesh status). In each case the old head keeps zero mention rows and the new head is indexed. Soft-delete rows check that the old head stays soft-deleted, and hard-delete rows check that it is gone. Reverting the macro, any single call site, or the hard-delete statement alone fails this test. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 14:30:52 -05:00
Luke Tornquist	298a555e34	perf(db): bind the channel scope as one array (#8946) ... ## Summary Event reads scoped to a set of channels pushed one bind parameter per channel: `channel_id IN ($5, $6, …)`. This applies to a viewer's access scope, which the relay adds to every REQ and `/query` without an `#h`, and to explicit multi-value `#h` filters. As a result, the SQL text depended on the viewer's channel count, and every count was a separate prepared statement. That caused two problems in production: - **Re-planning on every call.** A connection rarely reused a cached plan for these reads, so Postgres planned each one again. With no `since`, the plan covers every `events` partition. For `#p` mention reads (the `event_mentions` join), planning took far longer than executing the query. - **Hidden cost.** Monitoring that groups by statement split one query into one row per channel count, so its total cost never showed up in a single place. This PR binds the set as `channel_id = ANY($n)`, so one statement serves every channel count. On a production replica, the same query prepared with `ANY($5)` moved to Postgres's cached generic plan by its sixth run. Planning time dropped to near zero, and that plan was no slower than the custom one. ## What changed - `push_channel_scope` in `crates/buzz-db/src/store/event.rs` builds the channel-scope predicate as one array bind. - `build_query_events_sql` and `count_events_sql` each had their own copy of this block. Both now call the helper. - Semantics are unchanged: - Access scope still keeps channel-less global events. - Explicit `#h` still matches only the requested channels. - An empty set still means "no channels". ## What got simpler Two duplicated 20-line blocks became one helper. The statement count for channel-scoped reads no longer grows with channel count. ## Generic-plan exposure These statements were already persistent prepared statements, so a cached generic plan was always possible whenever the connection's statement cache held one. This change makes that reuse consistent; it doesn't introduce generic plans. Tag-value reads (`has_tag_values`) still run unnamed, as before. Plain (non-`#p`) access-scoped reads stay on custom plans. A production replica, with a viewer in hundreds of channels, ran the plain page read repeatedly under the default `plan_cache_mode`, and every run used the custom ordered index scan. Postgres estimated the generic plan as far more expensive, and when forced, it was orders of magnitude slower. Postgres switches only when the generic plan is no costlier than the average custom plan, so it never chose this one. The generic plan's cost comes from the `LIMIT` and `kind` values it can't see, not from the array length, and the per-arity `IN (…)` statements had the same blind spot. ## Tests - `channel_scope_sql_does_not_depend_on_channel_count` (unit): for the `#p` join and plain reads, for access scope and explicit `#h`, in both the page SELECT and the COUNT, the SQL for 1 channel equals the SQL for 40 channels. It also checks that the SQL uses `ANY(` and that global rows are retained only for access scope. - `channel_scoped_mention_reads_share_one_prepared_statement` (Postgres): on one connection, access-scoped `#p` reads with 1, 2 and 3 channels leave exactly one prepared `event_mentions` statement. With one mention per channel, one global mention, and one in an unscoped channel, every scope of 0–3 channels, with and without globals, returns and counts exactly its own rows. - The existing scope semantics tests still pass: - `access_scope_is_applied_before_historical_page_limit` - the explicit multi-channel `#h` page and count test - Mutation: reverting the helper to one bind per channel fails both new tests. ## Identified debt (not in this PR) - `get_last_message_at_bulk` still builds an `IN (…)` list per channel. It's a separate, lower-volume statement. - Superseded versions of replaceable roster events (kind 39002) keep their `event_mentions` rows. On large channels, a viewer's roster mention rows grow with every membership change. That's a follow-up. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 19:25:34 +00:00
Will Pfleger	f6198f6997	feat(relay): record NIP-FI enforce denials at parity with shadow (#8974) ... ## What this does In enforce mode, NIP-FI denials used to be returned without a reason counter or a log line. This PR records enforce denials with the same `WouldDeny` reasons, label sets and log fields as shadow. **Admission behavior does not change**: statuses, close codes and frame order are all the same. - **`buzz_nip_fi_enforce_total{route,stage,outcome,community}`**: the same shape as `buzz_nip_fi_shadow_total`, including an `admit`/`admit` stage, so `1 - admit/total` is the enforce denial rate. Each socket or HTTP request gets exactly one verdict. - **`buzz_nip_fi_enforce_session_end_total{route,reason=expired|revoked,community}`**: once per admitted session. - **`buzz_nip_fi_enforce_strict_proof_total{route,outcome,community}`**: the outcome of the proof check that actually decided the request. Nothing is verified twice. - **Log lines** `nip-fi enforce deny` and `nip-fi enforce session end` (info, `mode="enforce"`, from `buzz_relay::nip_fi_shadow`). Shadow's `nip-fi shadow would-deny` gains `mode="shadow"`. Neither mode logs admits. ### How each verdict is recorded once - **Sockets (root, audio).** `EnforceSession` moves Pending → Admitted(pubkey) → Ended under one std mutex, and records inside the lock with no awaits. - Root admits at its AUTH commit; audio admits at its join commit. - A refusal for a non-NIP-FI reason retires the session without a record. Audio retires it before rollback. - A deadline the effect gate sees before the expiry worker is claimed at that seam. - **Ends.** Revocation and expiry are recorded only by the terminal transition that wins (`publish_authorization_denied`), and only while the socket isn't already cancelled. A community close or cancellation that wins first records nothing. - **HTTP.** The router guard scopes each enforced request with one verdict. An agent token's verdict is deferred to the end of the request: - the first owner-decided refusal is the verdict, otherwise the request was admitted; - membership rechecks, batches, Git success and early returns each count once. - **Owner provenance** comes from the decision itself (`BanVerdict{outcome,source}` from one `restriction_state` read), with no rereads. - Reasons: `owner_banned`, `owner_ban_unavailable`, `not_stored_owner`, `owner_link_unavailable`. - When the agent's own ban read fails and there's a token owner, the result is `owner/unavailable`, as shadow records it. - A ban or failed restriction read that ingest catches on `POST /events` after membership passed replaces the request's admit with `owner/denied` when the owner's row decided the ban, or `owner/unavailable` on a failed read. An agent's own ban is not recorded as its owner's. ## Shadow-to-enforce map | Shadow signal | Enforce counterpart | |---|---| | `buzz_nip_fi_shadow_total` denials: upgrade, guard, HTTP admission, AUTH pairing, deny set, stale binding | `buzz_nip_fi_enforce_total` with the same stage/outcome, plus a `nip-fi enforce deny` line | | `buzz_nip_fi_shadow_total` owner stage: ws, audio, http | the same, from the original decision: root/audio ban seam and owner link; HTTP `enforce_membership`, smart Git, Git settings, buzz/v1 ban reread | | `buzz_nip_fi_shadow_total` `deadline/session_deadline` (pending) | `EnforceSession::expire`, from the worker's winning terminal or the effect gate that saw it first | | `buzz_nip_fi_shadow_total` admit | `buzz_nip_fi_enforce_total{stage="admit",outcome="admit"}`, one per socket or request | | `buzz_nip_fi_shadow_session_end_total{reason=expired\|revoked}` | `buzz_nip_fi_enforce_session_end_total`, same labels | | `buzz_nip_fi_shadow_strict_proof_total` | `buzz_nip_fi_enforce_strict_proof_total`, from the deciding check (a rejection is the route's 403, or 401 when the proof is missing) | | `buzz_nip_fi_shadow_disconnect_total{admin,accepted}` | `buzz_nip_fi_disconnect_total` (`api/nip_fi.rs:160`) | | `buzz_nip_fi_shadow_disconnect_total{admin,propagation_failure}` | `buzz_nip_fi_disconnect_propagation_failures_total` (`api/nip_fi.rs:180–185`) | | `buzz_nip_fi_shadow_disconnect_total{admin,capacity}` | `buzz_nip_fi_disconnect_capacity_rejections_total` (`api/nip_fi.rs:196–201`) | | `buzz_nip_fi_shadow_disconnect_total{cross_pod,capacity}` | `buzz_nip_fi_cross_pod_capacity_exceeded_total` (`api/nip_fi.rs:433–434`) | | `buzz_nip_fi_shadow_disconnect_total{cross_pod,poison}` | `buzz_nip_fi_cross_pod_shard_poison_total` (`api/nip_fi.rs:442–443`) | | `buzz_nip_fi_shadow_disconnect_total{cross_pod,lag}` | `buzz_nip_fi_disconnect_lag_total` (`main.rs:1596–1602`) | The enforce series was renamed from `buzz_nip_fi_enforce_denied_total` to `buzz_nip_fi_enforce_total` when it gained the `admit` stage. ## Tests Infra-free, in the `test-unit` lane: - `state::enforce_end_is_recorded_only_by_the_winning_terminal`: root scan, audio scan and deadline, each against a community close or cancellation first and FI first. - `nip_fi_http::enforce_strict_proof_records_the_deciding_check_once`: pass and reject, and no record when cardinality refuses. - `nip_fi_shadow_session::audio_owner_link_refusal_shares_roots_seam_and_reasons`: calls the shared owner-link helper with route `audio`, not a huddle socket. - The existing shadow, upgrade, router and HTTP enforce tests. Postgres lane: - `handlers::auth::enforce_root_auth_records_owner_refusals_from_one_decision`: clean admit then `expired` through the real expiry worker; banned token owner, banned stored owner, other stored owner and a failed ban read; authorization reads counted from the decision only. - `handlers::auth::enforce_root_deadline_seen_by_the_gate_first_is_recorded_once`: both orderings. - `audio::handler::enforce_join_commit_is_the_audio_admission`: drives the join-commit seam directly, not a huddle socket or the expiry worker: committed then expired; archived with an expiry held inside the rollback; gate-first deadline. - `api::bridge::token_owner_reaches_membership_on_every_nip_fi_http_route`: one verdict per request on every NIP-FI HTTP route, including sidebar, read-state, Blossom and Git, for an owner ban, a stored-owner conflict and a clear owner. - `api::bridge::token_owner_ban_reread_is_the_fixed_nip_fi_denial_on_http`: own ban, owner ban and unavailable at the reread, on sidebar and Git settings. - `api::bridge::token_owner_publish_ban_is_the_fixed_nip_fi_denial_on_http`: a late ban or failed read on `POST /events` at the write, typing, moderation and relay-admin checks records exactly one owner denial and no admit when only the owner is banned, and no owner denial when the agent alone or both are banned. - `audio::handler::tests::already_expired_audio_bootstrap_records_the_deadline`: an already-expired bootstrap records `deadline/session_deadline`, not a deny-set denial. Each new test names the mutation that turns it red. These were run red: - deleting root's `session.admit(pubkey)` - deleting audio's commit `session.admit(pubkey)` - deleting `effect_refused` - deleting enforce from `retire_shadow` - deleting the commit seam's `expire` - deleting the cancellation guard - deleting the own-read-failure provenance --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>	2026-11-03 14:21:32 -05:00
Daniel Rouleau	a9aa1f73b0	Enable incremental artifact reads on every relay (#8975) ... ## Why Compatible clients need incremental artifact reads for local collections on every Buzz relay. ## What Expose incremental artifact reads and capability discovery on every Buzz relay. ## How Serve incremental queries through the existing authentication and authorization checks. Keep the existing work budgets and the separate WebSocket sync capability. The HTTP test verifies capability discovery and a successful incremental query. ## Risk The relay starts no additional background work. Clients that use incremental reads consume database capacity. The net database load is not yet measured. Serving writers must maintain the modification timestamps. ## Testing Read-only SQL checks on November 3, 2026 verify the prerequisites below. Each database records successful migrations 76–78. All five required columns have the expected types, defaults, and nullability. All five indexes have the expected definitions and are valid, ready, and live. | Deployment | Schema and indexes | Running timestamp writers | | --- | --- | --- | | Production `bb-block` | Pass | `f6f6d9a` includes PR 8523 | | Production `bb-public` | Pass | `f3a7bab` includes PR 8523 | | Staging `bb-block` | Pass | `b4509de` includes PR 8523 | The checked columns are `artifact_heads.modified_at`, `artifact_revisions.modified_at`, `artifact_revisions.source_channel_id`, `channel_members.sync_modified_at`, and `communities.artifact_policy_modified_at`. The checked indexes are `idx_artifact_revisions_artifact`, `artifact_heads_modified`, `artifact_heads_channel`, `channel_members_sync`, and `artifact_revisions_moves`. At commit `62b26ad`, all 21 NIP-11 tests and the HTTP capability test pass on Linux with PostgreSQL 17 and Redis. The HTTP test checks capability discovery and a successful incremental query. All required push hooks also pass. No manual testing. Human confirmation remains pending. ## Bigger picture [Client PR 1507](https://github.com/block/buzz-app/pull/1507) uses the advertised capability for local artifact sync. Existing deployments receive this behavior when they deploy the relay version that includes this PR. Generated with Codex --------- Signed-off-by: Daniel Rouleau <drouleau@squareup.com>	2026-11-03 10:58:24 -08:00
Will Pfleger	ce47aefdc3	fix(relay): send membership notices on channel delete, moderation kick and moderation DM (#8977) ... Related: [buzz-app#2215](https://github.com/block/buzz-app/pull/2215) ## Problem A connected client learns about membership changes from kind:44100/44101 notices and then re-reads its channel list. Three relay paths changed a member's channel list without sending a notice, so clients only caught up on their periodic full read: - **Channel delete (9008):** no member was told. - **Moderation kick:** the kicked member wasn't told, and the channel's 39000/39002 discovery wasn't refreshed, so it still listed them. - **Moderation DM:** a new, existing or hidden moderation DM didn't surface live. ## Change - **Channel delete:** `soft_delete_channel_capturing_members` (buzz-db) reads the channel type and active members under deletion's existing membership lock, in the same transaction. A join either commits first and is included, or waits and is then refused. After the commit, discovery cleanup and cache invalidation, the relay sends a type-tagged 44101 to each captured member in a spawned task, eight at a time, outside any lock. The 9008 OK doesn't wait for them, and the task is started before the `channel_deleted` system message is emitted, so a system-message failure can't drop them. An already-deleted channel sends nothing. `soft_delete_channel` keeps its signature and delegates. - **Moderation kick:** `apply_kick_live_side_effects` refreshes 39000/39002 after the removal fence commits, then sends 44101 to the kicked member. If the refresh fails, it returns the error before the notice, so the moderation driver's existing mutation-marker recovery retries. If the channel was deleted since the kick, there's no roster to refresh: it removes the channel's discovery events and treats that as converged, so the action still finalizes. The removal/re-add fence and the re-added early return are unchanged. - **Moderation DM:** `send_moderation_notice` drops both participants' cached channel sets on every pod (`invalidate_membership`), as opening a DM does in `command_executor::handle_dm_open`, so the read the notice prompts includes the DM. After clearing `hidden_at`, it republishes the recipient's NIP-DV kind:30622 visibility snapshot (clients filter the sidebar on it), then emits discovery, then sends 44100. It sends the notice on every delivery, tagged `moderation_source` with the delivery's source id, so two deliveries in the same second each get their own 44100. The content JSON is unchanged. A failed snapshot or discovery returns an error, so the outbox retries. - **Ordering guarantee:** each notice is sent after its change is committed and readable on the relay instance that sends it. Ordering across relay instances, and other instances' cached channel sets, are best-effort, the same as for existing membership notices. - A failed notice delivery is logged and never undoes the change, like the existing 9001 path. There's no new retry system. ## Tests - `membership_notice_postgres_tests.rs`: one journey per path. A `#[cfg(test)]` barrier at the point a notice is signed parks the producer, keyed by recipient. While it's parked, the test reads the channel list (and, for the DM, the 30622 snapshot) as a client would on receipt. The barrier can also pin a notice's `created_at`. - The DM journey pins both deliveries to the same second, hides the DM through the real kind:41012 path between them, and checks what a client sees: a live 44100 for each delivery, with distinct ids, then a channel-list read that lists the DM and a visibility read that no longer hides it. - `moderation_dm_notice_follows_a_cached_channel_list` warms the recipient's channel set with a read before the DM exists, then checks the read on receipt of the notice lists the DM. - `soft_delete_captures_members_at_its_serialization_point` (buzz-db): a join and then a delete queue behind a held membership lock. The delete must return the joiner, and a later join is refused. - `crash_recovery_finalizes_kick_after_channel_deleted` (admin): commits a kick, deletes the channel, then runs the recovery worker. The action finalizes, and the member's `removed_at` is unchanged, so the removal doesn't run again. Each ordering test fails when its prerequisite is moved or removed: the kick notice ahead of the roster refresh, the delete notices ahead of discovery cleanup, the DM without the snapshot republish or the cache drop, the DM notice without its source tag (the second notice coalesces), and the member capture outside the lock. The recovery test fails if a deleted channel's `ChannelNotFound` propagates. No spec files change. This reads a channel delete as removing every member under the existing `NOSTR.md` notice promise. --------- Signed-off-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>	2026-11-03 13:57:34 -05:00
Salman Mohammed	d3a4909cca	perf(search): rank only the newest prefix matches and plan with the real tsquery (#8976) ... ## Summary Message search is one of the slowest relay queries. Short typeahead prefixes make it worst: `a:*` matches most of a community. **Why it was slow:** the tsquery was built inside a `CROSS JOIN LATERAL` subquery, so the planner saw an InitPlan and couldn't read the `search_tsv` statistics. It fell back to the default 0.5% selectivity for `@@`: for `a:*` on 1M rows it estimated 4,497 rows when 517,065 matched. Every search therefore took every GIN match, ran `ts_rank_cd` on each one, and top-N sorted them before `LIMIT 20`. This PR: 1. **Binds the real tsquery.** `push_tsquery` is unchanged. A separate `SELECT (…)::text` round trip resolves the tsquery, and the search binds it as `$q::tsquery`. An empty tsquery returns no hits without running the search. 2. **Plans each search for its own tsquery.** The search runs with `.persistent(false)`. A cached prepared statement can switch to a generic plan after five runs, and a generic plan can't see the tsquery. Forcing a generic plan lost the gain in testing. 3. **Ranks only the newest 1,000 prefix matches.** Prefix-mode relevance search ranks the newest `PREFIX_RANK_WINDOW` (1,000) matches. The inner `ORDER BY created_at DESC, id DESC LIMIT 1000` matches each partition's primary key read backwards, so for common prefixes Postgres streams the newest rows and stops at 1,000. For rarer terms it keeps the GIN bitmap plan, now with an accurate estimate. Full-text search and profile (kind:0) typeahead still rank every match. ### Behavior change: every human message search (product decision needed) Prefix mode isn't limited to keystroke typeahead. buzz-app (`useSearchMessages.ts`) and legacy desktop (`desktop/src-tauri/src/commands/messages.rs:184`) send `search_mode: "prefix"` for **all** message search. Full-text search runs only on the WS REQ path, which agents and the CLI use. So for people: - Results are the best-ranked of the newest 1,000 matches in the search's scope, not of all matches. If more than 1,000 newer messages also match, an older, more relevant message drops out of ⌘K results. The scope includes `#channel`, `from:` and date bounds, so a narrower search has the same limit within it. People have no full-text fallback. - The bridge honors `page`, and buzz-app loads later pages at 20 per page. Every page ranks the same window, so paging stays consistent, but results end after 1,000 matches (page 50). Full-text search (agents/CLI) and profile (kind:0) typeahead still rank every match. ### Perf risk: matches the backward scan has to walk far to find The planner picks the backward scan when it expects 1,000 matches near the top of the community, and it estimates that by treating each filter as independent. Two known shapes break that assumption. **Terms common in old messages but absent from recent ones.** The planner assumes matches are spread evenly over time. For a term that was common long ago but doesn't appear recently (an old project name, a former teammate), it can pick the backward primary-key scan and walk every newer row before reaching the first match. Reviewers measured this on 1.2M synthetic rows: about 2.05–2.1 s, against 0.7–1.8 s for the old query shape and 1.2 s with the GIN plan forced. The 20 s statement timeout and the #8954 cancel still bound it. The worst case is close to the old `a:*` worst case, but it now scales with community size, not match count. The planned EXPLAIN on production-shaped data will include such a term. **Terms common in channels the searcher can't read.** Human search always adds `channel_id = ANY(<readable channels>) OR channel_id IS NULL` (`build_search_channel_scope_filter`). A term that is common in private channels or other people's DMs can get a high match estimate, take the backward scan, and walk the community's rows to find a few readable matches, where the old shape used the GIN bitmap. This hasn't been reproduced yet; the production-shaped EXPLAIN will include such a term next to the old-only one. The same timeout and cancel bound it. ### Cost - One extra round trip per search to resolve the tsquery (about 0.03 ms of execution). It runs on the same pool checkout as the search (`cancel::fetch_all` takes the caller's connection), so a busy pool queues each search once. An IMMUTABLE SQL function could remove the round trip, but that needs a migration, so it's not done here. ### Related issue - Follows #8954 (cancel abandoned searches, bound the search pool) and #8964 (closed in favor of #8954). - Builds on #8947 (3-character server-side prefix minimum), merged from main. The resolve step passes its minimum. When the trailing prefix is too short, the resolved tsquery is NULL, and the search returns no hits without running the second query. - Client-side pairing: block/buzz-app#2210 (3-character last-word minimum for ⌘K message search). ## Testing **EXPLAIN ANALYZE on a synthetic 1M-row `events` table** (monthly partitions, GIN on `search_tsv`, kinds 9/40002/40008, custom plans, warm cache): | Prefix | Matches | Before | After | Plan after | |---|---|---|---|---| | `a` | 517k | 322 ms | **0.7 ms** | backward primary-key scan per partition, stops at 1,000 | | `th` | 137k | 127 ms | **1.2 ms** | backward primary-key scan, stops at 1,000 | | `a` (community with 10% of rows) | — | ~100 ms | **1–2 ms** | backward primary-key scan | | `deploy` | 8.2k | 15 ms | 5–10 ms | GIN bitmap, accurate estimate | | `d278` | 500 | 1.1 ms | 0.5 ms | GIN bitmap | | rare word | 0 | 0.07 ms | 0.03 ms | GIN bitmap | **Postgres integration tests:** - New test `prefix_relevance_ranks_only_the_newest_matches`. An old, highly relevant message drops out of prefix results behind 1,000 newer matches but still ranks first in full-text search. Prefix pages end at page 50. An old, best-ranked profile still tops profile typeahead behind 1,000 newer profile matches. Removing the window fails the test, and so does applying it to profiles. - `abandoned_search_cancels_its_statement`: the `pg_stat_activity` matcher now looks for `ts_rank_cd(`, because the old `search_query.query` text no longer exists. - Locally, each test ran against a fresh database because `cargo-nextest` isn't installed, so the hosted lane hasn't run yet. 26 of 27 pass. `artifact_search_uses_only_live_current_head` also fails on unmodified main in that local setup, so the failure isn't from this change. **Checks:** `cargo fmt --check` and `cargo clippy -p buzz-search --all-targets -D warnings` are clean. **Not yet done:** - hosted CI; - a manual check against a local relay with ⌘K typeahead; - EXPLAIN against production-shaped data, including a term common only in old messages and a term common only in channels the searcher can't read. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 13:25:27 -05:00