875 Commits

Author	SHA1	Message	Date
Luke Tornquist	50c10dc7b2	perf(relay): cache the community-active check on socket admission (#9109) ... ## What Socket admission (root and audio) and the ephemeral write fence now share one cache of "is this community active?". An entry is served for 1 s after its read checked out a pool connection, and concurrent misses for one community share one read. Archive and deletion drop the entry before they close the community's sockets. ## Why Every new socket runs `is_community_active` before it can authenticate. In a reconnect storm that means thousands of identical reads for one community. A read that times out in the pool refuses the socket (`community active check failed; refusing the socket`), and the client then reconnects and adds more load. On a local cold-start reconnect rig, this refusal count roughly tripled once #9012 freed pool capacity for other reads. It is now one of the top causes of connect failures. ## How - `buzz_db::Db::is_community_active_read_at` returns the answer together with the instant its connection was checked out, following the #9012 `ReadAt` pattern. `is_community_active` now calls it, and the datastore span stays under its existing name. - `community_active::CommunityActiveCache` holds an entry per community and a `SharedReads` gate. An entry is served for `COMMUNITY_ACTIVE_MAX_AGE` (1 s) after its read. Both answers are cached. A cached "inactive" fails safe and lasts at most 1 s after an unarchive. Errors are not cached. - **The archive sweep stays exact.** Admission registers a socket before it checks, so the sweep closes every socket registered before it. A socket that registers after the sweep must see the community as inactive. `apply_community_disconnect` invalidates the entry before the fence and sweep. Invalidation bumps an epoch, and a fill whose read began before the bump is not published. A post-sweep socket therefore reads afresh. It cannot take a pre-archive answer from the cache or from a shared in-flight read. Host invalidations (archive and unarchive writes, local and received) also drop the entry. - **Bound when the notification is missed.** A pod that misses the Redis disconnect admits sockets from a cached "active" for at most 1 s after the archive commits. Lifecycle revalidation then closes them. This matches the 1 s host-routing bound in `docs/community-auth-upgrades.md` ("archive and deletion … cached tenant binding can lag those changes by up to one second"). Deletion keeps its own write fence (`guard_community_write`). - Lifecycle revalidation and persistent ingest still read uncached. ## What got simpler The ephemeral fence's separate `serving_active_cache` is gone. It ran the same SQL predicate through `is_serving_active`, used a 10 s TTL and had no invalidation. The connect path, the WebSocket ephemeral path and HTTP typing now share one cache with one 1 s bound and one invalidation path. That leaves one fewer cache concept, and the ephemeral fence goes from 10 s stale to at most 1 s. ## Tests - `community_active::tests` (unit): - 16 concurrent misses cost one read, and an entry past its max age is re-read. - An invalidation during an in-flight read fences that fill. A waiter that joined the read reads afresh and gets "inactive". - Errors are not cached. - `connection::tests::bootstrap_admits_from_the_cached_community_answer`: with the database unreachable, a cached "active" admits a socket through the real `handle_connection`, and the socket gets its AUTH challenge. - `postgres_tests::cache_misses_behind_a_held_pool_share_one_fresh_read` now also covers this cache. With a one-connection pool held past every max age, eight concurrent admission checks cost one DB read, and the entry lands fresh. - `postgres_tests::archived_community_is_refused_after_invalidation_or_max_age`: after an applied archive, the next check refuses at once. An archive this pod never hears about refuses once the entry passes 1 s. - The ephemeral and typing fence tests now seed the new cache. - Mutation-checked on Blox. Each of these turns a test red: dropping the epoch fence, the max-age filter, the single-flight gate, the connect path's cached read, or the disconnect invalidation; stamping `read_at` before the pool wait; and calling the uncached read on the ephemeral path. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 14:09:14 -05:00
Luke Tornquist	ff15c347c2	fix(db): auto-migrate prebuild-guarded indexes on small databases (#8802) ... ## Problem Migrations `0076`, `0078` and `0081` raise an exception on any table with at least one row unless its index was prebuilt with `CREATE INDEX CONCURRENTLY`. A relay that runs migrations at startup (`BUZZ_AUTO_MIGRATE`, on by default in the Helm chart) and has data in those tables therefore crash-loops on upgrade: ``` migration error: while executing migration 81: error returned from database: prebuild idx_nip_fi_profile_heads_names concurrently; see docs/profile-search.md ``` `0081` (#9019) hits every auto-migrating relay that has published NIP-FI verified profiles. `0076`/`0078` (#8523) hit any relay with artifact data or channel members. Each of these tables is small enough on such relays that a plain index build takes milliseconds. The root cause is `docs/database.md` Rule 6. It told authors that an index on a populated table MUST be built concurrently by an operator ahead of time, so migrations were written to refuse populated tables. With auto-migration that refusal is a crash loop, not a safe stop. ## Fix The migrations are already applied on some databases, so they can't be edited: sqlx would reject the changed checksums. Instead, `run_migrations_locked` builds their missing indexes first. For each entry in `INLINE_INDEX_BUILDS` (`0076`, `0078`, `0081`), it runs migrations through the version before it, then builds any missing index. Each build is a plain `CREATE INDEX` in its own transaction, under the same 1s `lock_timeout` and 5s `statement_timeout` the migrations use. That is the bounded inline build `0049` already does: - **Small populated databases** upgrade the same way empty ones do. - **Large databases** still fail closed. Only a lock or statement timeout (`55P03` / `57014`) gets the "prebuild concurrently" hint, pointing at that migration's own doc. Other errors pass through unchanged. The blocked migration is not recorded. - **Prebuilt indexes** are found with `to_regclass` and skipped before `CREATE INDEX`, so deployments that prebuilt concurrently never request the writer-conflicting lock. - **Invalid prebuilt indexes** are left for the migration to reject, as before. `CONCURRENTLY` is not used in this path. The runner holds the migration advisory lock, and another replica blocked on that lock holds a snapshot that a concurrent build would wait on. ### Rules - `docs/database.md` Rule 6: with `BUZZ_AUTO_MIGRATE` on, every migration must apply its own change on a populated database and must never refuse one because it has rows. A new index is built inline under the migration's budgets. Only deployments that apply migrations out of band (auto-migration off) prebuild concurrently. - Root `AGENTS.md` (step 4) and `migrations/AGENTS.md` repeat the never-refuse rule. - `docs/artifact-sync-deployment.md` and `docs/profile-search.md` describe the startup behavior first and the concurrent prebuild as the out-of-band path. ## Tests - `populated_database_auto_migrates_through_every_migration` (Postgres) migrates to `0075`, seeds artifact rows, and checks four phases: 1. **Lock held:** while a writer lock is held, the `0076` build fails closed. 2. **Index prebuilt:** with the `0076` index prebuilt, the build is skipped. 3. **Partial `0078`:** a blocked `0078` build keeps the builds before it committed. 4. **Verified profile seeded:** with a verified profile head inserted before `0081`, a final startup succeeds. It asserts that the applied versions equal every embedded migration and that all six indexes are valid, ready and live. - `frozen_prebuild_guards_have_inline_builds` (no database) scans `migrations/`. It fails if any migration refuses a populated table (`RAISE EXCEPTION 'prebuild …`) without an inline build, and if a new guard is added. - `migration_execution_cannot_bypass_schema_destruction_lock` allows exactly one production `run_to`, inside `run_migrations_locked`. ## What got simpler - Auto-migrating relays no longer need a manual, operator-only prebuild step for small deployments. That restores the chart README's contract that `helm upgrade` is the whole upgrade procedure. - Rule 6 no longer steers authors toward refusing populated tables, and a test enforces that. - The change adds one table-driven step for three frozen migrations and no new configuration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 13:52:06 -05:00
Luke Tornquist	aa749a862e	perf(db): serve single-root thread reply reads from thread_metadata (#9075) ... ## Summary A plain thread-reply read, `{kinds:[9], #h:[channel], #e:[root], limit:100}`, goes through the generic `query_events` path. That path gathers every event tagging the root into a `MATERIALIZED` CTE, then sorts it to keep one page. On a root with 500k replies, each page costs about 514k buffer reads plus a temp-file spill. This PR serves single-root reply reads from `thread_metadata` instead. It walks `idx_thread_metadata_window` newest-first, fetches each reply by primary key, and applies every filter predicate before `LIMIT`. Page cost scales with the page size, not with the thread. ## What changed - **Routing (`crates/buzz-db/src/store/event.rs`).** A `query_events` read takes the new route when it has: - one channel, - one lowercase-hex `#e` root, - kinds limited to 9 and/or 40002, - no `#p` filter (any number of values; the relay marks such filters `exact_match`), - no `authors`, `pubkey` or `ids` selector. The route tests those per reply, so an author with no reply in a hot thread would walk the whole thread; the generic read narrows by author first. Every other `#e` read stays on the fenced path from #7854. - **Root check before routing.** `#e` must name a thread root. A cheap probe looks up the target in `thread_metadata` first. If the target is itself a reply, the read takes the generic path, which returns its direct children the NIP-01 way. A reply counts if its row has a root, or if it is a rootless legacy stub whose own NIP-10 markers resolve to a reply. - **COUNT unchanged.** COUNT post-filter fallbacks (`count.rs`, `bridge.rs`) set the new `EventQuery::exact_match` flag through `prepare_count_fallback`. Routing is skipped for them, so COUNT still counts every event that matches under NIP-01. - **Same predicates on both paths.** The predicate block moved out of `build_query_events_sql` into `push_event_filters`, which both paths call. Filters on the route therefore match the generic path, including the `#e` tag match, `since`/`until`/`before_id` and deletion. - **Unnamed statements.** Routed reads run as unnamed statements, so Postgres plans each one with its root and never caches a generic plan. The benchmark below shows the routed SQL is also fast under a forced generic plan, so this is a second safeguard. - **Narrowed semantics, documented.** Messages that name the root only in a bare `e` tag, a `mention` tag or a lone `root` marker are top-level under NIP-10. They have no `thread_metadata` row, so the routed read omits them. Live subscriptions still deliver them. `docs/nips/NIP-CW.md` gains a "Plain Thread-reply Filters" section describing this. A client that needs those messages can add another kind to the filter, which sends the read down the generic path. ## Benchmark (synthetic, Postgres 16, 500k-reply thread) Median of 5, newest page of 100. "Rare" is a root with 3 replies. "Generic" forces `plan_cache_mode = force_generic_plan`. | | main | this PR | |---|---|---| | hot root, custom plan | 2802 ms (513k buffers, 46,750 temp blocks written) | 1.6 ms (407 buffers) | | hot root, generic plan | 1779 ms | 0.6 ms | | rare root, custom plan | 1.2 ms | 0.8 ms | | rare root, generic plan | 61 ms | 0.08 ms | Author-filtered reads (`authors:[x], limit:1`) on the hot root: an author with no reply there takes 1.4 ms on the generic path but 1.7 s on the route, so author and id selectors stay generic. On both roots, the PR's SQL returns the same rows as main. ## Tests - `e_tag_filter_shape_tests::thread_reply_route_admits_only_single_root_message_reads` checks which reads take the route. Two roots, mixed kinds, no kinds, no channel, an uppercase root, a `#p` join, an `exact_match` read, and `authors`/`pubkey`/`ids` selectors all stay generic. The routed SQL is checked for the `thread_metadata` walk, the `#e` match and the window ordering, with no `MATERIALIZED`. - `postgres_tests::thread_reply_reads_use_thread_metadata` (Postgres lane) covers: - mixed 9/40002 replies, a nested reply and a soft-deleted reply, - a bare-`e` message that the route omits and the generic read (kinds `[9,40002,7]`) still returns, - a lone-`root` message, which the route omits too, - an `exact_match` read, which returns the same rows as the generic read, - a nested-parent target, which gets its direct child the generic way, - root stubs and a legacy parent with no row of its own, checked through the probe, - keyset paging, - no named prepared statement left behind after repeated runs. - `handlers::req::tests::postgres_tests::multi_p_thread_filter_keeps_nip01_results` (Postgres lane) runs a two-value `#p` thread filter through the real filter-to-query adapter and checks that a bare reference to the root still comes back. - `handlers::req::tests::count_fallback_fetches_one_extra_candidate` checks that COUNT fallbacks set `exact_match`. It is DB-free and was missing from the `buzz-relay --lib` allowlist, so this PR adds it to `just test-unit`. - Mutation-checked: the Postgres test fails if any of these is broken: - routing disabled, - persistent statements, - the nested-target probe dropped, - stubs treated as all replies or as no replies, - COUNT allowed to route, - the deleted filter dropped. - Blox gate: `cargo fmt`, clippy (`-p buzz-db -p buzz-relay --all-targets`), buzz-db lib (nextest and plain parallel `cargo test`), buzz-relay lib, and the buzz-db and buzz-relay Postgres lanes. One unrelated artifact-sync scale test timed out under load; it passes alone and in CI. ## What got simpler - One predicate builder replaces the inline block, so the two read shapes cannot drift apart. - One targeted route replaces the planned extra fence variant (a keys-only `MATERIALIZED` CTE). That variant was dropped because it reduced the spill but still scanned every reply. ## Identified debt (not in this PR) - The bridge `page` extension (`crates/buzz-relay/src/api/bridge.rs`, `extract_page_offset`, built for the kind:0 directory listing) still maps to SQL `OFFSET`. The routed read honours it too, for parity. Deep pages there should move to keyset (`until` + `before_id`). buzz-review-completed 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 11:14:12 -05:00
Logan Johnson	700e690193	feat(db): enforce unique listed channel names with an index (#8787) ... ## Summary Stacked on https://github.com/block/buzz/pull/8782 (merged), which makes the relay refuse a second open channel with a name that another channel in the community already has. That check lives in relay code, so an older relay, a direct database write or a missed code path can still create a duplicate. This PR adds the same rule to the database: a unique index on channel name, ignoring case, for the channels that show in the channel list. With the index in place, a duplicate fails at the write, whatever wrote it. The relay's own name check from https://github.com/block/buzz/pull/8782 is no longer needed, so this PR removes it. Clients still get the same refusal, `invalid: an open channel named "…" already exists`. The migration builds the index itself, also on a populated `channels` table, so an auto-migrating relay upgrades with no operator index step (the rule from https://github.com/block/buzz/pull/8802). It still refuses to run while duplicate names remain, because the index cannot exist over them. `docs/channel-name-uniqueness-deployment.md` gives the full rollout. ## Details - **The index** (migration `0083_channels_listed_name_unique.sql`): ```sql CREATE UNIQUE INDEX idx_channels_listed_name ON public.channels (community_id, lower(name)) WHERE visibility = 'open' AND NOT unlisted AND channel_type IN ('stream', 'forum') AND deleted_at IS NULL; ``` It covers open, not unlisted streams and forums that are not deleted, archived ones included. Private channels, DMs and unlisted channels may still repeat names. - **The migration builds the index inline, as `0049` does.** When `to_regclass` finds no index, it runs a plain `CREATE UNIQUE INDEX` under `lock_timeout` 1s and `statement_timeout` 5s. On a busy or very large table it fails on those budgets, records nothing, and the relay retries at its next start. When the index already exists (a concurrent prebuild), it skips `CREATE` so it never requests the write-blocking lock. In both cases it then checks that the index is valid, ready and has the exact expected definition. - **One refusal stays: duplicate names.** The migration refuses to run while listed duplicates remain, and its error names the rename script. The renames must go through the relay so clients get the new kind:39000, and a migration must not change rows (`docs/database.md` Rule 3). This is the "fails closed until its repair has run" case in `migrations/AGENTS.md`, so the writer release from https://github.com/block/buzz/pull/8782 and the rename are a required stop. - **Relay.** A write that takes the name of a concurrent, uncommitted write waits for it, and fails if that write commits. So the post-write check and the per-name lock from https://github.com/block/buzz/pull/8782 are removed. The store already maps the index violation to the same "name taken" error. - **Desired-state schema.** `schema/tables/public/channels.sql` has the same index, and the schema parity test compares its live shape between the desired-state schema and the migrations. - **Rollout** (`docs/channel-name-uniqueness-deployment.md`), in order: 1. Deploy https://github.com/block/buzz/pull/8782 and wait until no older relay pod serves. 2. Rename the existing duplicates in each community with that PR's script, while relays keep serving. The script needs `psql` and `python3`, and the operator key must be a member of each community. The doc covers archived communities and communities being deleted. 3. Confirm that no duplicate remains. 4. Only for a deployment that migrates out of band and is too large for the budgets: build the index with `CREATE UNIQUE INDEX CONCURRENTLY` and verify it. 5. Deploy this release. With auto-migration on, the relay applies 0083 at startup. With it off, run `buzz-admin migrate` first, never the file through `psql`. Rollback goes to the https://github.com/block/buzz/pull/8782 release only. Keep the index, and keep auto-migration off. - **Test fixtures.** Tests that inserted two open channels with one name now use distinct names. Tests that migrate a populated database through 0083 no longer prebuild the index. ### Related issue None found. Follows https://github.com/block/buzz/pull/8782. ### Testing - Migration tests (`listed_channel_name_postgres_tests.rs`) start from a database migrated through 0082: - With duplicates, 0083 fails and changes nothing. - After the rename plan is applied, a channel writer that holds its lock makes the inline build fail on the 1s lock budget, with nothing recorded. - With no writer, the populated table migrates with no prebuild, and a raw duplicate insert fails on the index. - An index with a different definition, and an invalid one, are refused. A valid prebuilt index is accepted, and not replaced, while another transaction is writing to `channels`. - The store and relay name tests from https://github.com/block/buzz/pull/8782 pass with the relay check removed: the index refuses the duplicate, and the client still gets `invalid:`. - The rename plan and script tests from https://github.com/block/buzz/pull/8782 run on a database at 0082, the state the script runs in. - Local Postgres runs on `108737dc0`, one fresh database per test as the nextest wrapper does: 38 of 40 buzz-db migration-lane tests pass (both listed-name tests, the migration parity test, the thread-window and invite-upgrade tests that migrate populated databases through 0083). The 2 `event_follow_up` contract tests fail locally the same way on main. All 49 buzz-db channel and channel-member store tests pass. ### Database review checklist - [x] Phase: contract. Before it: https://github.com/block/buzz/pull/8782 (writer release, required stop) and its rename script. Nothing after it. - [x] No trigger, function, procedure, rule or foreign key. - [x] No migration mutates rows. The renames happen before it, through the relay, with the script in https://github.com/block/buzz/pull/8782. - [x] Never refuses a table because it has rows. Fails closed only with duplicates (repair not run) or with an invalid or different index; then 0083 raises and the relay does not start. The release notes must name the release that contains https://github.com/block/buzz/pull/8782 as a required stop. - [x] Rule 4: `migrations/`, `schema/tables/public/channels.sql` and the parity test. A live catalog comparison covers the index. - [x] Deployed relay: the https://github.com/block/buzz/pull/8782 relay answers `invalid:` beside the index (its tests pass with the index present). A relay older than that still writes, but gets a database error, not `invalid:`, for a duplicate. That is why https://github.com/block/buzz/pull/8782 is a required stop. - [x] Timeouts and index build: `lock_timeout` 1s, `statement_timeout` 5s. The migration builds the index inline (plain `CREATE UNIQUE INDEX`, skipped when `to_regclass` finds a prebuilt one). Only out-of-band deployments too large for the budgets prebuild it with `CREATE UNIQUE INDEX CONCURRENTLY` (doc step 4). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 22:47:23 -05:00
Brad Seiler	64d12e2975	BUZZ-511: Honor relay mode for community NIP-FI enforcement (#8988) ... ## Summary Apply relay mode consistently across protected HTTP, root/audio WebSocket, Git, media and V1 admission. Off requires no NIP-FI, including for stored required communities. Shadow enforces only stored community requirements and observes other communities. Enforce requires NIP-FI for every community. Durable requirements survive all mode changes. Off retains ordinary tenant routing and authentication. It skips FI policy/dependency reads for admission, uses cached WebSocket tenant routing, and loads no FI binding snapshot at startup. Discovery reports `requires_nip_fi=false` and no FI capability in Off. Evaluating modes retain same-row WebSocket binding and immutable HTTP policy tenants, including host remaps, pending deletion and unarchive races. Global Enforce discovery retains its existing compatibility behavior. Host-specific root NIP-11 and `/info` expose the additive `requires_nip_fi` field with no-store. In evaluating modes, an unavailable policy returns 503 and an unknown or non-serving host returns 404. Off uses ordinary routing; a discovery lookup failure returns 404. Stored requirements in Shadow require usable FI dependencies. Admission does not establish activation. BUZZ-512 owns existing-session application, drain and mode-aware readback; its separate revision must report stored requirements as suspended in Off. Production baseline verification and human acceptance remain open. Community status now shares the bounded host cache: 10,000 entries, one-second leases measured before checkout, coalesced fills, and no cached misses or errors. Git reuses one request policy. Root/audio retain the final primary binding check: two reads on a cold successful upgrade, one on a cache hit. Off creates no verifier or JWKS source and reads no FI policy. Proof metrics count stored-required Shadow enforcement. Acceptance locks and validates the community row, takes the matching host advisory lock, and drains optional leases for 1.1 seconds before commit. Reads begun after commit see required across participating relay instances, even without Redis invalidation. The row lock closes single and double remap races. Admissions already in flight remain subject to BUZZ-512 activation fencing. ### Related issue [BUZZ-511](https://linear.app/squareup/issue/BUZZ-511/relay-enforce-per-community-nip-fi-policy-and-publish-compatible). [BUZZ-510 / #8985](https://github.com/block/buzz/pull/8985) is merged; this PR now targets main. The restack retains main’s #8974 denial telemetry and artifact capabilities. Related work includes #8526 tenant lookup errors and #8444 media authorization. ### Rollout and containment Keep `BUZZ_NIP_FI_UPGRADES_ENABLED=false`. [The rollout plan](docs/community-auth-upgrades.md#staging-and-containment) starts in staging with zero required rows and a 30-minute before/after traffic comparison. Initial abort limits include any policy timeout/503, policy-read p99 above 50 ms, writer acquisition p99 above 20 ms or twice baseline, and a five-minute 503 increase above 0.1 percentage point. These are initial limits to validate against measured staging results, not production capacity claims. Verify signed/unsigned free controls, real required root/audio upgrades, dependency failures, restart behavior and discovery before expansion. Also watch binding timeouts/errors and their current generic 404 responses on known serving hosts alongside writer-pool pressure; evaluating root/audio binding has a 500 ms budget; Off uses ordinary cached routing. In Shadow, a pre-511 rollback loses enforcement of accepted community requirements. Stop new upgrade calls, retain accepted policy and receipts, and repair with an enforcing binary or operator-approved containment. Never clear durable policy to restore access. Off deliberately suspends FI enforcement and must be an explicit operator mode choice. No production activation is claimed. All acceptance paths must implement the lease protocol before upgrades are enabled. Keep upgrades disabled through a mixed-version rollout. For rollback, disable new acceptance before removing the protocol while caching pods remain. A cold policy fill uses a six-statement transaction and a 400 ms server timeout within the 500 ms lookup budget. A contended policy fill returns 503. During the drain, archive, deletion, ownership, issuer-binding and competing acceptance writers for that community may wait; ordinary serving reads and writes retain their existing behavior. Host-binding changes retain the existing one-second routing bound. Local and Redis invalidation accelerate refresh; safety does not depend on delivery. Deploy the verified Shadow revocation behavior from #8991 before accepting required communities in Shadow. ### Testing Published review-fix head: `39dfe79b1fc04f10ee3eb94d6ea42cf686bafe1e`, tree `1e2e8d3abd39ee1d6821defe8c8d34de73726e08`. McKeon cleared this cache delta. Full affected auth (291), DB (192), pubsub (31) and relay (1,682) package tests passed, including integration binaries. Two previously documented hardcoded-Redis audit tests remained excluded. All-target clippy, all 1,297 native PostgreSQL tests and the full pubsub Redis suite passed. The full ACP package and `just ci` passed after rebuilding cached test executables that embedded a removed checkout path. No ACP source changed. Tauri's generated lockfile delta was archived and restored; it is excluded from this candidate. Both positive lease regressions passed. Four deliberately broken variants failed at the intended assertions: removing the drain left an optional lease alive after 202.935667 ms; skipping the fill lock failed contention refusal; moving the timestamp after the read failed the anchor check; taking the host lock before the row failed the remap-blocking check. Exact source was restored after each mutation. These are local validation results, not a deployed-fleet acceptance claim. All applicable normal push hooks passed. GitHub reports the exact tested head and tree; the published code has no delta from the reviewed candidate. Earlier mode-correction evidence follows for context. Current mode-correction head: `aa9abf07284b56a88eb78a2014e590f39d705288`, on main `af54258f19ac32667c027f889652bdeb479c8e13`. Its code equals reviewed `f1e7da5fc5`, plus the independently reviewed operator documentation follow-up; all 19 earlier remote patches are preserved. McKeon verified the correction and its mapping to the tested candidate. The mode-correction candidate passed full auth (291) and relay (1,679) package tests, clippy, and four explicit native PostgreSQL policy workflows. There were 623 ignored relay tests and two excluded Redis tests. Real PostgreSQL checks cover an ordinary unsigned caller on a stored-required host in Off with no FI token or dependencies, discovery false, durable policy retained, root/audio upgrades, host remaps and the unarchive race in evaluating modes. The `STORED_OFF` mutation fails the enum truth table only; `OFF_PRIMARY` fails cached upgrade routing. Restored source hashes match. Earlier failed compile/fixture attempts remain recorded. The first normal push hook attempt passed Tauri checks but failed the gateway startup verifier test (82/83 gateway tests passed). The remote remained unchanged. The retry rebuilt gateway artifacts with separate root and Tauri caches, then passed every normal push hook and published this head. No gateway source changes or hook bypasses were needed. Current-head hosted CI is tracked separately. Human acceptance and the staging binding-timeout/404 workflow remain open. The following retained validation and controls are historical and describe behavior before the mode correction; their Off-enforcement statements are superseded by the Summary above. Published head: `9a717fa89725f9430ff4a8bb78cd600affb8836d`, on main `1055f79d048ef22c878d02a7afb04f49c753564f`. CI, three schema checks, 2,136 ordinary tests, all 291 auth tests, 12 Redis tests, all 1,300 native PostgreSQL tests and discovery passed. The required wrapper passed units, then exited 1 at unavailable Docker bootstrap. BUZZ-510 is complete at `95c8d31c835d6f7ff6e432fb1ed248771d4f06dd`: CI, three schema checks, 2,127 ordinary tests, 12 Redis tests, all 1,295 native Postgres tests and discovery passed; its wrapper passed units and failed at unavailable Docker bootstrap. The first native run at historical BUZZ-510 `5380cb72` passed 1,291 tests and failed two event-follow-up queue assertions and the scale-sync case. All three passed serially, then the full 1,294-test run passed with normal concurrency. The original cause remains unresolved; failed runs and resource samples are retained. Standalone BUZZ-511 checks exposed access-slot callers, a Clippy warning and early HTTP 503 attribution that had remained in the child. All are now corrected in the parent. The unchanged real-router refusal regression passed before the latest commit. BUZZ-512's final tree is unchanged by moving these parent prerequisites. The remaining 46-file child delta was audited; no additional parent prerequisite was found. Earlier compile and native failures remain archived. The following earlier-head results and controls are historical; the current-head validation is reported above. At `48388834f99b8d045e243b619904cb3e27b47146`, based on BUZZ-510 `130f14a675edd7cbe6a69d1c9d7818bd1a51f46c` and reconciled main `152441ae0a56b204b42b6faa56a36ad2b98218b1`: - `just ci`: passed. - Four-package ordinary suite: 2,069 passed; full `buzz-auth` package: 291 passed. - Native Redis: 12 passed. Native PostgreSQL: 1,261 passed. PostgreSQL discovery checks passed. - Required `just test`: units passed, then exited 1 because Docker service bootstrap was unavailable. Native service results above are separate. - Real-router remap checks reject an A-only member under B policy, avoid A data after B admission, and advertise B in discovery despite a stale A cache. Removing each tenant pin separately fails its intended witness; restored source passes. An unmapped host cannot reuse a cached tenant. - Deadline witnesses advance their clock during real HTTP signature verification and retain socket primary-lookup barriers. Removing either final enforced or shadow HTTP deadline check separately fails its intended witness; restored source passes both deadlines and the single-resolution control. - Telemetry regression: unsigned dev proofs now expect the earlier NIP-98 `missing` denial; signed mismatched-key proofs retain pairing denial and signer attribution. Removing unsigned-proof rejection fails the exact two-counter assertion. Restored full relay package: 1,654 passed. - Preserved failed runs: two prior bottom-layer desktop gates failed an immediate inventory assertion, repaired with a bounded wait for the same observable arrays and verified with delayed-notification and frozen-provider controls. An earlier BUZZ-511 native run passed 1,242 tests and failed two deadline fixtures that relied on the tenant lookup removed by the policy pin; the test-only adaptation above replaces that seam. These failures remain archived. The first native run on bottom-layer `130f14a6` passed 1,255 tests and failed partition contention; its focused original test and fresh full 1,256-test run passed without source changes. The cause remains unresolved. The first CI run at `d8d88433` failed the old telemetry expectation; the test-only correction above preserves exact denial counts and attribution. Earlier evidence at `dddf05d8be8195d567db0f989f70825b1246a6c1` (historical head): - `cargo nextest run -p buzz-auth -p buzz-db -p buzz-relay --lib --tests`: 2,139/2,139 passed. - `scripts/postgres-test-run.sh -p buzz-db -p buzz-relay --lib --tests --success-output final`: 1,236/1,236 passed with native PostgreSQL 16 and Redis. Docker's legacy `just test` wrapper was unavailable. - `just ci`: passed, including repository checks, tests and client builds. - **Pre-correction behavior (superseded):** Real ES256 assertions through TCP root/audio upgrades: valid evidence upgrades; missing evidence denies in Off/Shadow, including submitted/inventoried/approved deletion stages. Restoring the old query makes the regression fail on an assertion-less Off/submitted/root HTTP 101. - **Pre-correction behavior (superseded):** Deterministic unarchive race: Notify barriers pause an actual archived-host policy miss; the same required row becomes serving before binding. Both root/audio deny missing assertions and admit valid ES256 assertions in Off/Shadow. Ignoring bound-row policy makes this test fail on an assertion-less HTTP 101. - The full rerun exposed an existing cancellation fixture whose synthetic five-second replica fence expired between probes. Refreshing that test fence per operation preserves its timeout assertions; the full suite now passes. - Local probe: 100 serial uncached primary policy reads took 12.33 ms. Production latency is unmeasured. No packaged-app, human or deployed-fleet acceptance run. Live activation awaits BUZZ-512 and the staging verification tickets. Published with the three-layer stack in one atomic push through normal hooks. The published stack includes main `1055f79d048ef22c878d02a7afb04f49c753564f`, including access logging, storage reporting and mobile read-state changes. Access-slot integration and early HTTP refusal attribution belong to BUZZ-511; activation callback refusal close owners belong to BUZZ-512. Every changed layer requires review at its published head and new-head CI. Fresh trunk, remote membership, queue and immutable-lease guards passed before publication. All nine normal pre-push hook lanes passed; no hooks were bypassed. Remote heads, chained bases, membership, ready/draft states and native tracking were verified after publication. Trunk’s merged 0081 is unchanged. The parent upgrade expansion is 0082 and activation is 0083, with contiguous migrator versions, updated documentation and a populated-upgrade fixture starting at 81. Trunk’s storage-test repair supersedes our earlier local repair. Before publication, main advanced three commits from `1055f79d048ef22c878d02a7afb04f49c753564f` to `ba7d7825ea` (desktop deep-link settling and mobile pairing/sign-in). The normal branch-skew guard passed. Those later trunk commits are outside this validated published head; current-trunk CI and delta review remain required before merge. Generated with Codex --------- Signed-off-by: Brad Seiler <seiler@squareup.com> Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz>	2026-11-05 03:17:19 +00:00
Luke Tornquist	506a588f88	refactor(relay): remove database usage-metric queries (#9040) ... ## Summary The relay no longer runs database aggregate queries to emit usage gauges. Fleet and per-community counts of users, channels, relay members, workflows, git repos, communities, messages, and active users/channels are no longer relay metrics, and neither are the snapshot health gauges that came with them. ### Removed - **Gauges:** `buzz_total_users`, `buzz_total_channels`, `buzz_total_relay_members`, `buzz_total_workflows`, `buzz_total_git_repos`, `buzz_total_active_users`, `buzz_total_messages`, `buzz_total_active_channels`, `buzz_communities_estimated` / `buzz_communities_total`, and every DB-derived `buzz_community_*` series (users, channels, messages, relay members, workflows, git repos, active users, active channels). - **Per-community mode:** `BUZZ_USAGE_METRICS_PER_COMMUNITY=all` and everything it fed: the per-community in-memory gauges, `buzz_community_storage_bytes` / `buzz_community_storage_objects`, `buzz_storage_unmapped_community_bytes`, the community → host map query, and the emission-scope bookkeeping that cleared series for renamed or removed communities. No deployment set it. If one still does, the relay ignores it. - **Health signals:** `buzz_usage_snapshot_available`, `buzz_usage_snapshot_age_seconds`, `buzz_usage_query_skipped_total`, and the `usage_fleet_*` paths of `buzz_db_route_decision`. - **Code and config:** - `buzz-db/src/store/usage.rs`, including its aggregate queries. - The relay's `FleetUsageSchedule` and its cadence/deadline/retry machinery. - The replica-only telemetry route (`Db::route_usage_read`). - `BUZZ_USAGE_METRICS_REPLICA_MAX_AGE_MS` and `DbConfig::usage_metrics_replica_max_age_ms`. If a deployment still sets the env var, the relay now ignores it; startup doesn't fail. - **Reader maintenance pool operation:** `ReaderOperation::Maintenance` and its `close_on_drop` release path existed only for those telemetry reads, so they're removed too. The pool-acquire contract drops from twelve to eleven valid pairs, which lowers the per-pod series ceiling from 216 to 198. - **Dead code:** `Db::active_community_hosts` had no callers. ### Kept - **Leader:** the usage poller's leader lock, plus the leader's invite reaping and fleet storage-snapshot gauges. The session advisory lock (`SessionLeader`, `Db::try_lock_session_leader`) moves to `buzz-db/src/store/leader_lock.rs`, since storage-usage delivery uses it too and it has nothing to do with usage queries. - **In-memory gauges:** pod-local connection, subscription, and online-user gauges. They read process state, not the database. - **Writer maintenance pool operation:** `WriterOperation::Maintenance` stays. Dozens of writer-side maintenance paths across `buzz-db` use it (reapers, retention, the leader lock). The reader variant only ever served the telemetry reads. - **Storage-usage delivery:** unchanged. It reads the storage snapshot for billing, not for metrics. `docs/usage-metrics.md`, `docs/storage-accounting.md`, `.env.example`, and the Helm chart README are updated to match. ## What got simpler This is net deletion (~3.6k lines). One telemetry subsystem is gone: two collection families with independent cadences, availability and age semantics, relay-side deadlines, a dedicated replica routing policy, two config knobs, a per-community emission mode, and a pool-operation variant. Leader election now does only leader work, and it lives in its own module. ## Operational note Dashboards and monitors that read the removed series will go to no data once this rolls out. ## Testing - `cargo fmt --check`, workspace `cargo clippy --all-targets -D warnings`. - `buzz-db` lib, `observability_source`, and doc tests. - `buzz-relay` lib tests. - The Postgres lane for `buzz-db` (lib + tests) and `buzz-relay` (lib). All of these run on a remote workstation at the PR head. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 22:00:30 -05:00
Brad Seiler	a1f79e3aba	feat(relay): persist NIP-FI upgrades (BUZZ-510) (#8985) ... ## Summary Record that a community requires NIP-FI and provide an idempotent operator API to enable it and recover the original receipt. Acceptance checks the current owner, host, lifecycle and existing issuer trust in one transaction. It never creates trust. Receipts survive community deletion so an operation UUID cannot be reused. New acceptances default to disabled through `BUZZ_NIP_FI_UPGRADES_ENABLED`. Disabled calls return HTTP 503 `upgrades_disabled`; identical retries and authenticated readback still recover existing receipts. Policy is permanent: stopping new calls or rolling back code does not clear it. Last-issuer removal is refused for required communities, and bulk backfill leaves missing bindings for explicit operator repair. Apply additive migration 0082 before code. Existing communities remain optional. This layer records acceptance and reports only `pending` or `unavailable`; deploy and verify [per-request enforcement](https://github.com/block/buzz/pull/8988) and [session activation](https://github.com/block/buzz/pull/8991) before enabling acceptance. The global enforce floor remains. See [the operator contract](https://github.com/block/buzz/blob/seiler/BUZZ-510/relay-auth-upgrade/docs/community-auth-upgrades.md) for signing, recovery and containment. ### Related issue [BUZZ-510](https://linear.app/squareup/issue/BUZZ-510/relay-persist-community-nip-fi-requirements-and-expose-idempotent). Base of native stack `8989`; retains trunk's enforcement telemetry from [#8974](https://github.com/block/buzz/pull/8974). Includes a separate commit that repairs an observer timing flake in a desktop test required by the repository gate. ### Testing Controlled local regressions detect the old disabled error code and removal of the last issuer with `allow_last=true`; intended source is restored. Full PostgreSQL validation and the updated live relay/admin exercise remain pending. Human acceptance remains open. No deployment or production activation. Generated with Codex --------- Signed-off-by: Brad Seiler <seiler@squareup.com> Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz>	2026-11-04 19:56:07 +00:00
Luke Tornquist	4813b8cd15	relay: time every WebSocket read and every stored EVENT outcome (#9052) ... ## Summary Two gaps in relay telemetry: WebSocket reads (`REQ`, NIP-AR sync, `COUNT`) have no latency or outcome metric, and `buzz_event_processing_seconds` only measures stored EVENTs that succeed. **What this does** - **`buzz_ws_read_seconds{type, outcome, reason}`** (new histogram). It's recorded from `access::ReadLog::drop`. Every WebSocket read already finishes through that one seam, whether it ends `served`, `denied`, `error`, or `aborted` (future dropped mid-read), so this adds no new exit paths and changes no handler. Duration runs from the moment the handler takes the frame until `EOSE` is queued (`REQ`), the initial set ends (sync), or the total is sent (`COUNT`). `reason` is the existing fixed denial/error code (`query_timed_out`, `database_error`, `not_channel_member`, …), or `none`. The histogram's `_count` doubles as the outcome counter, so one metric gives both read latency and read failure rate. - It's recorded *before* the refusal budget check, because the budget only exists to suppress log lines. When `refused_read` is over budget, it records the sample without building the log line, so its cost under a refusal flood stays the same. - **`buzz_event_processing_seconds{outcome}`**. Before this change it was recorded only when `ingest_event` returned `Ok`. A store that was slow *and failing* (statement timeouts, pool exhaustion) produced no samples, so latency looked better while the store was degrading. It's now recorded on every persistent-ingest outcome: `accepted`, `not_accepted` (for example a duplicate), `invalid`, `auth`, `error`. Queries that sum across all outcomes still work. **Cardinality.** Every label value is a `&'static str` from a code-defined set: 3 types × 4 outcomes, with about 25 reasons that only appear alongside `denied`/`error`. On purpose, there's no `kind` label: `bounded_kind_label` still admits around 200 values, which is too many to cross with histogram buckets. **What got simpler:** `ReadOutcome::parts()` replaces the inline outcome→label match in `Drop`, so the log line and the metric share one mapping. There's no new instrumentation seam: the read metric reuses the access log's lifecycle. ### Related issue None found. This came out of a monitoring-gap review: without these signals, alerts can't catch a slow or failing WebSocket read path, or a failing EVENT store. ### Testing - `access::tests::every_read_is_measured_even_when_its_line_is_suppressed` (unit). It drives the real `ReadLog` constructors and `Drop`: 20 admission refusals against a 16-line budget, a handler denial past the budget, a `COUNT` error, a served sync and an aborted `REQ`. It asserts the exact `(type, outcome, reason, samples)` rows. Two mutants each fail it: removing the over-budget branch in `refused_read`, and moving the measurement after the budget check in `Drop`. - `handlers::event::tests::postgres_tests::event_processing_is_timed_for_accepted_and_refused_ingest` (Postgres lane). It sends one accepted text note, then a refused one from a cached-ban sender, through `handle_event`, and asserts one sample each under `accepted` and `auth` (a ban maps to the `auth` class). Removing the `Err`-arm measurement fails it. - Blox gate at `b314df43f2` (tree `4ca9ebc1`): `cargo fmt --check` clean. The new Postgres test passes, and removing the `Err`-arm measurement fails it. buzz-relay unit lane (`cargo nextest run -p buzz-relay --lib --bins`): 1708/1708 passed. - At `539796bd51` (the same code; only the second commit's test assertion differs): workspace clippy (`--all-targets -D warnings`) passed. The new unit test passed, and each of its two mutants failed it. buzz-relay Postgres lane: 604/605 passed. The one failure was this test's old `invalid` expectation, which `b314df43f2` corrects. - CI at `b314df43f2`: both new tests appear in the CI logs as passing (Rust / Unit Tests, PostgreSQL Domain / PostgreSQL Tests). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 13:15:51 -05:00
Luke Tornquist	25abfc4117	fix(loadgen): bound desktop reads at stop, keep timely EOSEs, model presence and probes (#9016) ... ## Summary Follow-up to #9005, addressing the review findings that merged unaddressed. None of them affect a relay. `buzz-ws-client` gains one method. Each one skews what the desktop client measures when the relay is slow, which is the case the load generator exists for. - **Reads stop at the stop.** A desktop HTTP read (including NIP-11) now races the run's stop and is abandoned there without being recorded, as EOSE waits already were. Before, a read that outlived the 15 s shutdown grace got its client cancelled and failed an otherwise healthy run. `Fetcher::finish` also keeps the read task's handle until the task ends, so a client cancelled during the join still aborts the task instead of detaching it. - **Root lookups are bounded.** The desktop's seeded-root lookups, for thread recovery and replies, now run under the desktop's 30 s read timeout and stop, not reqwest's 60 s. `roots()` takes an optional bound, and the basic human passes none. If the stop cuts a reply's lookup short, the reply isn't published. That applies to the basic human too: a reply whose root lookup finishes after the stop is dropped. - **An EOSE that arrives in time stays in time.** While a publish waits for its OK, `NostrWsConnection` buffers every other frame. The session loop used to judge EOSE deadlines before reading that buffer, and read one frame per pass. It now handles every frame already in hand (at most 256 per pass) before judging deadlines. Each frame is judged by when it arrived, not when it is read. `NostrWsConnection` keeps each buffered frame's arrival time, and the new `next_event_received` returns it. So an EOSE that arrived after its deadline while an OK wait held it counts as a timeout, and so does a frame that answers the probe late. Past that cap, a pass defers due deadlines, for at most one timeout from the first pass that defers them. While it is deferring, the pass only drains, so another OK wait can't expire an EOSE that's already in hand, and sustained traffic can't postpone a missing EOSE forever. - **Presence and the silence probe.** Both modes now open the app's live-only `presence` route (`{"kinds":[20001],"authors":[…],"since":…,"limit":0}`, up to 256 co-members) alongside the profile and membership routes. As in the app, presence takes no part in establishment: it isn't resumed or carried, and if it fails it doesn't cut a renewal's carry for the other routes. After a tick with no frame since the previous one, they send the app's probe, `REQ probe-N {"kinds":[44100],"#p":[viewer],"limit":0}`, which shows up as `desktop_probe` and is closed at its EOSE. Only frames actually received count, including a publish's OK when it arrives before the probe's deadline. A reaction with no target or a publish whose OK timed out doesn't count. Expiry is judged after the drain against the probe still outstanding. If no frame arrives within the EOSE deadline, the session ends as failed and the client reconnects with backoff. Before this, desktop runs left out presence fan-out and idle-client probes, so they understated load. - **Seed log.** "memberships added (N people)" now reads "memberships ensured", since a rerun adds nobody. - **Docs.** `perf/LOADGEN.md` notes that the six-read cap and flat 30 s timeout are simplifications (the app has neither), and that `legacy` is a lower bound. The pre-fix app also opened a per-channel route for each open or non-member channel and re-sent every route each tick, and the model leaves both out. Not done: an explicit count of REQs abandoned at a renewal (report polish; planned cancellations still aren't errors). ### What got simpler Reads abandoned at the stop now go through one helper, `run::bounded`, which the desktop reads and the root lookup share. Before, each read had its own timeout and its own stop check. `global_reqs` returns the routes the client actually opens, not a fixed pair. The probe and presence routes add one `Route` variant, one op and about 40 lines of session code. `Route::establishes` is now the one place that says which routes take part in establishment, replacing the two `Replay` special cases. "Heard" used to be inferred by comparing activity schedules before and after each step. It now comes from frames only, so `publish` reports whether the relay answered and that comparison is gone. ## Testing On a Blox workstation, at the tree of this PR's head: - `cargo fmt --all -- --check`, `cargo clippy -p buzz-test-client -p buzz-ws-client -p buzz-cli --all-targets -- -D warnings` - `cargo test -p buzz-test-client --lib` (parallel) and `cargo test -p buzz-ws-client`, plus the `loadgen::desktop` tests five more times New tests, each confirmed to fail when its fix is reverted: - `a_read_in_flight_at_the_stop_is_abandoned_not_waited_for`: an 8 s roster read at the stop. Without the stop race the client outlives `drive`'s 5 s stop limit. - `a_cancelled_finish_still_aborts_the_read_task`: aborting `finish()` mid-join ends the read task. With the old `take()`, the task is detached. - `an_eose_in_time_is_not_expired_behind_a_slow_ok`: EOSEs at 1.2 s, publish OKs at 3.5 s, deadline 2.5 s. Without the drain, all four routes time out. - `a_silent_tick_probes_and_an_unanswered_probe_reconnects`: the first connection never answers its probe, so the client reconnects and its next probe is answered. - `a_probe_answered_behind_a_late_ok_does_not_reconnect`: the probe's EOSE arrives in time, but it sits behind a heartbeat whose OK comes after the probe's deadline. With expiry judged against a deadline captured before the drain, the session fails and reconnects. - `a_publish_ok_in_time_answers_an_outstanding_probe`: no probe EOSE ever arrives, and heartbeat OKs answer each probe. - `a_reaction_with_no_target_is_not_heard_as_a_frame`: targetless reactions several times a second still leave the tick silent, so it probes. - `a_reply_whose_root_lookup_the_stop_cut_short_is_not_published`: the stop cuts an 8 s root search short, and OKs are withheld. No `kind 9` EVENT reaches the relay, and the client returns promptly. - `a_backlog_defers_deadlines_for_at_most_one_timeout`: the deferral bound and its reset. - `a_long_backlog_behind_slow_oks_keeps_its_timely_eoses`: 600 replayed events per REQ behind 3.5 s OKs, with a 1.5 s deadline and heartbeats every second. The test fails if the deferral is timed from the REQ instead of from the first pass that defers, and it fails if a second OK wait can run between the deferred passes. - `presence_is_a_live_only_global_only_while_someone_is_watched`: the route's order and its live-only filter in both modes, and that it's absent with nobody watched. - `a_late_eose_is_not_answered_behind_a_slow_ok`: the timely case above with EOSEs at 3 s instead, so all four routes time out, not succeed. - `a_probe_answered_late_behind_a_slow_ok_reconnects`: a probe's EOSE arrives 0.2 s after its deadline while a heartbeat waits for its OK, so the session fails and reconnects. The test also checks that a heartbeat followed that probe on the same connection, so the test can't pass on a probe that simply went unanswered. - `an_answered_probe_is_closed_at_its_eose`: `CLOSE probe-1` goes out before any next probe. - `routes_establish_once_and_a_renewal_holds_them_until_a_cut` gains a case: a renewal where presence times out keeps the other routes carried and establishes nothing. Each new test fails when its fix is reverted (seven mutants in the first round and six in the review round, run on Blox). The test harness now has a tick override that can go below the scenario's 1 s floor, and records EVENT frames by kind. The fake relay sends a REQ's replayed events after the REQ's delay, not before, and the fake API's slow reads sleep in `block_in_place`. No existing test changed behavior. Existing expectations moved for the presence route (one more live EOSE per client) and for reads now abandoned at the stop. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 12:57:38 -05:00
John Matthew Tennant	f9ef721e68	fix(media): keep transparency in image thumbnails with WebP (#9003) ... ## Summary This fixes transparent images showing a black background. Image thumbnails were always JPEG, which can't store transparency, so a transparent PNG or GIF upload, such as an agent avatar, rendered on black wherever a client showed its thumbnail. The relay now writes a lossy WebP thumbnail at `{sha256}.thumb.webp`, which keeps transparency, alongside the existing `{sha256}.thumb.jpg`: - **Encoding:** the WebP comes from libwebp through the `webp` crate, built from vendored C source, so the relay needs no system library. A GIF uses its first frame. - **Upload responses:** the upload descriptor's `thumb`, and its per-tenant rewrite, point to the WebP. - **JPEG kept:** installed clients derive `.thumb.jpg` themselves, so the relay keeps writing it until clients move to WebP with a JPEG fallback. Existing images still have only their JPEG thumbnail. - **Serving:** `/media/` serves both variants, each with its own content type. Other `.thumb.*` extensions still return 404. - **imeta:** a `thumb` may name either variant, and the relay checks that the named object exists. - **Storage sweep:** the bucket taxonomy counts both variants as the image's thumbnail bytes instead of classifying `.thumb.webp` as unknown. | 320px thumbnail | JPEG | PNG | lossy WebP (q80) | |---|---|---|---| | Transparent avatar (starter-team `fizz.png`) | 11.3 KB, black background | 65.1 KB | 13.8 KB | | Opaque avatar (`sprout-agent.png`) | 23.6 KB | 156.5 KB | 12.3 KB | | Screenshot (`channel-thread.png`) | 13.4 KB | 62.5 KB | 6.4 KB | Encoding a WebP thumbnail takes 3 to 20 ms. ### Related issue Supersedes #8772, which served a PNG at the `.thumb.jpg` path. ### Testing Red/green against a local relay on `:3000`, built from the branch under test, with Postgres, Redis and MinIO from `docker compose`: ```bash cargo test -p buzz-test-client --test e2e_media test_transparent_png_thumbnail_is_webp_with_alpha -- --ignored cargo test -p buzz-test-client --test e2e_media_extended test_ws_imeta_webp_thumb -- --ignored ``` - **On `main`:** both tests fail. The descriptor's `thumb` is `.thumb.jpg`, and the WebSocket send is rejected with `invalid: imeta thumb must be a local .thumb.jpg path`. - **On this branch:** both tests pass: - Uploading a half-transparent PNG returns a `thumb` ending `.thumb.webp`. - HEAD and GET on that URL return `Content-Type: image/webp`. - The decoded thumbnail keeps alpha 0 on the transparent half and alpha 255 on the opaque half. - `.thumb.jpg` still returns `image/jpeg`. - A kind 9 message whose imeta `thumb` names the `.thumb.webp` is accepted. ## Screenshots The starter-team `fizz.png` avatar uploaded to a relay built from this branch. Both thumbnails are shown as round avatars beside the original PNG. `.thumb.jpg` fills the transparent area with a solid disc, and `.thumb.webp` keeps the transparency.  🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: John Matthew Tennant <jtennant@squareup.com> Co-authored-by: Simone <0828be588d5c9c4d092c5b1cf01761d5aef63d6b104807fd8e8284d517dd1a99@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 10:11:00 -05:00
Luke Tornquist	d21f0c0fdb	perf(relay): keep 1 s read caches warm when the pool is saturated (#9012) ... ## What The relay's 1-second read caches (restriction state, tenant host, open channels) now date each entry from when the read checked out its pool connection, not from when the cache miss began. Concurrent misses on the restriction and tenant-host caches now share one database read. The open-channels cache already did this. ## Why Each of these caches serves an entry for 1 s after "the read that produced it began". Today that start time is taken before waiting for a pool connection. Once pool waits reach about a second (for example, many clients reconnecting at once), every entry is already expired by the time it lands. The cache hit rate falls toward zero exactly when the pool is saturated, so every request does its own read. Those reads queue for the same connections, which stretches the waits further. Requests that time out in the pool then fail closed (`internal error checking restriction state`, `community active check failed; refusing the socket`), and their clients reconnect and add more load. In a local synthetic reconnect load test with a 50-connection pool: - the restriction cache served fewer than one request in five from cache; - the tenant-host cache, which has a single key for the whole deployment, missed more often than it hit; - the open-channels cache missed hundreds of thousands of times on expired entries. ## How - **Stamp at checkout.** `buzz_db::ReadAt<T>` carries a value plus the instant its connection was checked out. The statement's snapshot is no older than that instant, so serving the value for 1 s after it keeps the documented staleness bound, however long the read queued. Three `*_read_at` methods return it: `moderation_restriction_state_read_at`, `lookup_community_by_host_read_at` and `get_listed_open_channel_ids_read_at`. The existing methods now call these and drop the value's timestamp. Each operation keeps one datastore span under its existing name, carried by the `*_read_at` method. - **Share concurrent misses.** `SharedReads<K>` is the per-key gate the open-channels cache already used, moved into its own module. The open-channels cache, the restriction cache and the tenant-host cache now all use it. The first miss reads; the others wait for it, then recheck the cache. A waiter waits for at most the writer pool's acquire timeout plus `SHARED_READ_STATEMENT_WAIT` (1 s). That covers the leader's own wait for a connection, which is what grows in a storm, so waiters keep sharing its read instead of queueing duplicate reads behind it. A failed or cancelled leader releases its waiters at once. A leader whose statement hangs stalls them for at most 1 s past the acquire timeout, never the statement timeout. - **Reject hosts no community can have.** `bind_community` rejects a host longer than `communities.host VARCHAR(255)` can hold (`buzz_core::tenant::MAX_HOST_LEN`, counted in characters) as unmapped, next to the empty-host check. An unauthenticated caller's arbitrarily long host therefore never reaches a resolver, cache, gate or the database. No staleness bound changes: every entry is still served for at most 1 s after its read's snapshot. Invalidation paths are unchanged. A read in flight at an invalidation can still refill the old row for at most that bound, as documented today. ## What got simpler The open-channels gate's hand-rolled `try_lock`/`timeout` block is replaced by `SharedReads::turn`. One gate implementation now serves three caches instead of one implementation that only one cache had. The gate's wait is set once per gate rather than passed at each call site. Provisioning and host binding share one `MAX_HOST_LEN` instead of a private constant. ## Tests - `shared_reads::tests`: one caller reads while the rest wait; a waiter gives up on a hung read after its bound. - `concurrent_restriction_and_host_misses_share_the_in_flight_read`: waiters on both caches use the leader's entry; with an unreachable database, any waiter that read on its own would fail. - `postgres_tests::cache_misses_behind_a_held_pool_share_one_fresh_read`: with a one-connection pool held for longer than every max age, eight concurrent misses on each of the three caches cost one database read, and each entry still lands fresh. - `tenant::tests::an_overlong_host_fails_closed_before_the_resolver`: a host over the limit fails as unmapped with no gate entry and no lookup; a host at the limit, including one in multibyte characters, is still looked up. - Mutation-checked: each of the following turns these tests red: stamping any of the three entries before the pool wait, dropping either new waiter path, letting waiters give up after the cache's max age, dropping the host length check, or counting the host length in bytes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 09:52:21 -05:00
Kalvin C	3d26ed7dfd	feat(relay): match verified names in profile search (#9019) ... ## Summary `POST /profiles/search` only matches kind:0 `name` and `display_name`. In communities that publish NIP-FI verified profiles, a person's surname or directory username often appears only in their relay-signed kind:30382, so typeahead cannot find them by it. - Where the community advertises verified profiles, the search also matches the `display_name`, `username` and `full_name` tags of each current kind:30382 head signed by the relay key. - Verified names need a trailing prefix of at least three characters, as in NIP-50, so a short prefix doesn't make the relay rank every verified head. kind:0 keeps one-character typeahead. Lookups from heads to `events` also match `created_at`, so Postgres prunes to one partition. - Results are paged by person. Each person returns their current kind:0, when they have one, then their current kind:30382, when they have one, whichever matched. A person is ranked by their best match across both sources, using the existing tiers. - NIP-11 `federated_identity.public_profile` gains `searchable: true`, so clients can rely on the response instead of a separate kind:30382 read. - Migration `0081` adds a GIN expression index over the head's name tags on `nip_fi_profile_heads` (built-in `jsonb_path_query_first` and `to_tsvector`, no new functions). The table holds one row per verified person. Like `0078`, the migration builds the index only while the table is empty; populated databases prebuild it with `CREATE INDEX CONCURRENTLY` (`docs/profile-search.md`), and the migration then checks it is valid and has the expected definition. Compatibility: the request schema is unchanged. Existing clients that keep only kind:0 events work as before and also find people who match only by verified name. Trust stays pinned to the relay `self` key; the event shape follows NIP-85 user assertions, but discovery is relay-advertised rather than kind:10040. ## Testing - `buzz-search` Postgres integration: new `profile_names_match_verified_names` covers forged-key heads, replaced heads, other communities, people without kind:0, per-person grouping, paging, and words split across sources; the existing profile-name tests pass unchanged. The test fails when the author filter is inverted. - `buzz-relay` end to end in enforce mode: `profile_search_matches_verified_names_where_advertised` publishes through the real writer and finds the person by surname only where verified profiles are advertised. - Desired-state and migration index definitions match. - `profile_names_use_verified_name_index` asserts the search scans the new index. Migration `0081` checked against an empty table, a rerun, a populated table without the index (raises), the documented concurrent prebuild, and an index with a different definition (raises). - Latency, 50k people each with a kind:0 and a verified head, release build, local Postgres, best of 5, 30 people per page: 11–22 ms with verified names versus 1–8 ms kind:0 only. `EXPLAIN` shows a bitmap scan on the new index. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Kalvin Chau <kalvin@block.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 21:25:15 -05:00
Luke Tornquist	865b150371	fix(db): use ScratchDb in the storage usage delivery test (#9032)	2026-11-04 00:30:32 +00:00
Will Pfleger	0a06c0b3cb	feat(relay): log reads, logins and connections on access target (#8376) ... 🤖 Adds an access log for relay reads, logins, connections and HTTP requests on the dedicated `buzz_relay::access` target. It writes to stdout, not `audit_log`, so reads never take the audit advisory lock. The log is shaped for volume: one line per HTTP request, one line per WebSocket read, and one line per WebSocket connection. ## Lines per operation | Operation | Lines | |---|---| | HTTP request on a matched route | 1 (`http`), except the skip list below | | HTTP request matching no route | 1 (`http`, `route="<unmatched>"`) at status 400 and above, otherwise none | | WebSocket connection | 1 (`close`, carrying the login) | | WebSocket `REQ` / `COUNT` / NIP-AR sync | 1 each (`req` / `count` / `sync`), plus the `buzz_ws_reads_total` counter | | WebSocket NIP-42 login | 1 (`auth`) per decided or cut-off attempt, capped after the decision as below | | Audio login | 1 (`auth`, at its final admission check) | | WebSocket `EVENT` | 1 (`Event ingested`, not on the access target) | ## HTTP requests One middleware, `http_access_log`, writes every HTTP line. It sits between `track_metrics` and `http_trace_layer`, so rebuilt errors, body-limit refusals and localhost refusals all reach it. It writes the line when the response is ready, which is not proof the body was delivered. A request dropped before then logs `request=aborted` at WARN with no `status`. - Every line carries `route` (the matched route template, never the raw path), `peer_addr` when the listener has one, `status`, `time_to_response_ms` and `http_method` (the HTTP verb). The verb stays on the line itself because the `http.request` span is absent under an access-only filter and when a pending request is dropped. - Handlers record what they learn on the request's `AccessSlot`: the login (`method`, `pubkey`, `pubkey_verified`, `community`), the decision, the filter summary, and the route's own fields. The decision is `outcome` (`success` / `denied` / `failed` / `challenged`) with a fixed `reason`, or `undecided` when the route recorded none. A refusal is final: a later admission doesn't replace it, so a replayed NIP-98 proof that verifies and is then refused logs `failed` / `nip98_replay`. - `outcome` describes the login. What the route did is in `status` and the route's fields: a rejected `/events` write logs `outcome=success` with `status=400`, `accepted=false` and its `reject_reason`. - Route fields: - `/query` and `/count`: the filter summary and `result_count`. An incremental `/query` response (`{changes:[...]}`) counts its changes, removals included. Artifact reads record their summary before the query runs, so a read dropped mid-query keeps it. - `/events`: `accepted`, `kind`, `event_id`, and on a refused write the parse `category` / `line` / `column` or a fixed `reject_reason` (such as `invalid` or `restricted`), never the ingest message. - Invite mint: `invite_id`, `expires_at` and `max_uses`. Invite claim: `joined`. - Admin feedback attachment reads: `feedback_id` and `attachment_sha256`. With `BUZZ_ADMIN_AUTH=disabled` they log `method=network_trusted` with no key, and Host or Origin refusals log their fixed reason. - Every other route logs only the login fields. - No field can hold an invite code, a token, or a Blossom read hash. Media routes log the `/media/{sha256_ext}` template. - Skip list: NIP-11 and other non-upgrade requests on `/`, `/info`, `/.well-known/nostr.json`, `/health`, `/_liveness`, `/_readiness`, and the SPA and static files the relay serves when enabled (web assets, invite landing pages, git web pages, the admin bundle) at any status. Successful WebSocket upgrades (101) log no `http` line. Refusals before the upgrade on `/` and the audio path log one, with the status actually sent. - Routes that aren't Nostr-key logins (the command-token disconnect endpoint, the workflow webhook, the git server's internal callback) log `outcome=undecided` with no key. - Dev-mode `X-Pubkey` requests log `method=x_pubkey`, `pubkey_verified=false`. - A successful `/query` or `/count` that returned nothing (`result_count=0`) logs at DEBUG: on staging, 93% of `/query` lines were empty polls. Denied, failed, aborted and error reads, reads with no `result_count`, writes and logins keep their levels. - Other successful lines and git's credential-less first request (`challenged`, with its 401) are INFO. Refusals, statuses of 400 and above, and aborted requests are WARN. In Enforce mode the line records the remapped status (for example 403). Denial bodies and headers are unchanged. ## WebSocket reads - `REQ` / `COUNT` / `sync` lines carry `pubkey`, `conn_id`, `sub_id`, the filter summary, the outcome (`served` / `denied` / `aborted` / `error` plus a fixed reason), and `events_queued` (REQ and sync) or `count` (a COUNT that counted). A `REQ` is `served` only once its EOSE is queued to the client. A read logs WARN only when it ends in `error`. `served`, `denied` and `aborted` are INFO, since `aborted` means the read ended before completing: the client disconnected or sent CLOSE, a same-ID `REQ` replaced it, or the session was cancelled. - A NIP-AR sync subscription writes its `sync` line when its initial set completes, not when live delivery ends. Its summary is the registered live filter (kinds and channels, never artifact predicates), and `events_queued` counts the initial set. - `REQ` lines stay at INFO even when empty at EOSE, because the subscription can still deliver live events later. - Filter summary, shared with HTTP `/query` and `/count`: `kinds`, channel ids, authors and `p` values each list the first 10 distinct values. The exact `kind_count` / `channel_count` / `author_count` / `p_count` (summed per filter) appears only when the list is `*` or shows fewer values than the filters held. A dimension that any filter leaves open logs `*`. `since` / `until` / `limit` take the widest bound and are absent when any filter leaves them open, so no field understates what the read covered. `ids` is a count only. NIP-50 search logs `search: true`, never the text. Only well-formed channel ids and pubkeys are listed. An artifact read's summary uses only the standard dimensions of each raw filter, so artifact predicates and client-defined tags (such as `#1`) are never logged. The summary is skipped when the access target is off. Under `=warn`, every WARN line, HTTP or WebSocket, still carries it. - Per connection, only the first 16 reads that are refused before they run (`auth_required`, quota, concurrency, overload) or denied are logged. The rest are counted as `reads_suppressed` on the close line. A refusal that arrives after the close line has been written, from a read still running when the client left, logs its own line, so every refusal is either logged or counted, never both. Served reads are never capped. ## WebSocket read counter `buzz_ws_reads_total{type="req"|"count"|"sync"}` counts every parsed `REQ`, `COUNT` and NIP-AR sync frame before admission, exported at zero from startup. It is a metric, so it counts at any `RUST_LOG`, including production's current `error`. ## WebSocket logins A NIP-42 login writes one `auth` line when it is decided. A successful login logs INFO `outcome=success` at its commit point, decided under the connection's close-reason lock, so a disconnect racing the login (session expiry, admin disconnect, revocation, ban or roster removal) either wins, and the login logs no success, or loses and closes the connection right after the success line. A refused or failed login writes one WARN `auth` line with `outcome`, a fixed `reason` from the shared `AuthReason` code set, `pubkey` and `pubkey_verified`. Root WebSocket `auth` lines carry `peer_addr`. AUTH frames after the login is decided are capped at 16 per connection, and the rest are counted as `auth_suppressed`. An attempt cut off by close or session expiry logs `aborted`. Audio logins log one `auth` line at their final admission check. ## Connections The close line is the connection's only line. It is owned by the upgraded socket, so it is written exactly once however the connection ends: a normal close, a refusal while checking the community is active (`refused=not_admitted`), the connection limit (`refused=connection_limit`), or the handler being dropped. It carries `conn_id`, `community`, `peer_addr`, `duration_ms`, the login, `events_queued` / `bytes_queued` (frames the writer accepted and flushed, not client delivery), and the two suppression counters. ## Error text - Database errors log as `DbError::kind()` (SQLSTATE or variant), including the command-executor event insert. Search errors, on both read paths and in the search-cancel task, log as `SearchError::kind()` / the same SQLx classifier. Workflow database and webhook errors log as their codes, since webhook URLs can be secrets. - Git NIP-98 failures log a fixed code. Git subprocess stderr moved to DEBUG because hook output can contain message text. ## Operator notes `.env.example` documents the target, its `event` values (`http`, `auth`, `req`, `count`, `sync`, `close`), the skip list, the levels, and how to turn it down: `buzz_relay::access=warn` keeps refusals, errors and aborted HTTP requests, and of WebSocket reads only those that end in `error`, and `=off` drops everything. It also lists the removed lines: `NIP-42 auth successful/failed`, `WebSocket connection established/closed`, `event=connect`, `HTTP bridge request`, `Event ingested via pipeline`, `relay invite minted`, `relay member added via invite`, `admin feedback attachment read`, and `addr` (now `peer_addr`). JSON lines no longer repeat the current span as `span`, since `spans` already ends with it. A line logged under an explicit `parent:` span loses that parent's fields. The default OTel export filter sets `buzz_relay::access=off`, so access lines aren't exported a second time as span events. `peer_addr` is the TCP peer. `BUZZ_CLIENT_IP_HEADER` optionally names a request header (for example `cf-connecting-ip` behind Cloudflare or `x-real-ip` behind nginx, matched case-insensitively). When set, `http`, `auth` (root and audio), `close` and connection-limit lines also log `client_ip`: the rightmost comma-separated entry of the header's last field, only if it parses as an IP address. It never replaces `peer_addr`, and it is only trustworthy when every request arrives through a proxy that overwrites the header. --------- Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>	2026-11-03 17:48:59 -05:00
Shu Hui	fa7a3bfbda	feat(relay): push per-community storage usage to kgoose (#8714) ... ### Why? To bill hosted communities for storage in Orb, kgoose needs per-community storage usage. The relay already computes it in a daily snapshot, but nothing delivered it anywhere. ## Summary - Adds opt-in `BUZZ_STORAGE_USAGE_URL`. When it is set, one leader-elected relay pod checks every 5 minutes for a new storage snapshot and POSTs it to kgoose's `POST /v1/buzz/storage-usage/record` (squareup/cash-server#144285) in consecutive batches of up to 50 communities until every community is sent. - Each batch is `{measured_at, communities: [{community_host, storage_bytes, storage_objects}]}`. Active communities with no stored bytes are reported as zero, and re-sends are safe because kgoose bills each community once per UTC day. - Retries on 408, 429, 5xx, and transport errors. No migration; documented in `docs/storage-accounting.md`. kgoose authenticates the relay as the `buzz` mesh service, so the relay needs mesh egress to kgoose-builderbot. ### Related issue None found. Complements #7899 (snapshot history in the DB). Receiving endpoint: squareup/cash-server#144285. ### Testing Unit tests cover batching, retry and reject handling, zero-filling, and a real HTTP delivery that checks the exact kgoose request body. A Postgres-gated DB test covers the new reads. End to end, a local relay delivered to a fake endpoint once per new snapshot. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Shu Hui <shu@squareup.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-11-04 09:40:06 +11:00
Salman Mohammed	6847882b14	fix(relay): stop logging unmapped-host NIP-FI denials as server failures (#8997) ... ## What this does In enforce mode, a request whose Host maps to no community gets NIP-FI's `authorization_unavailable` 503 (`docs/nips/NIP-FI.md`, Rejection and privacy table). tower-http's default classifier treats every 5xx as a failure, so each of these requests also logs a `tower_http::trace::on_failure` error. That line has no host or path, only `503` and `latency: 0 ms`. When a scanner probes a host this relay doesn't serve, it produces thousands of these lines in a minute, and they trip error-rate alerts even though nothing is down. This PR keeps the wire response the same and stops counting these denials as server failures: - `nip_fi_core::community_denial` builds the same `http_denial` response. When the Host is confirmed unmapped, it adds an internal `UnresolvedHost` response extension. Extensions never reach the wire, so the NIP-FI denial contract ([FI-TRACE-DENIAL-ORACLE]) still holds. - Each site that turns a community-resolution failure into a denial now uses it: - the router guard (`nip_fi_assertion_guard`) - the WebSocket upgrade (`check_nip_fi_at_upgrade`) - HTTP admission, for routes the guard exempts (`Stage::Community`) - The HTTP trace layer now uses `RelayResponseClassifier`. It treats marked responses as expected and keeps tower-http's 5xx-is-failure rule for everything else. Other 503s still log as failures. That includes an unavailable JWKS key source on a mapped host, `deny_protected`, a stale snapshot, and a failed database lookup. ## Why not return 4xx for unmapped hosts? NIP-FI.md maps "community/Host resolution unavailable" to 503, and `nip_fi_enforce_unmapped_host_is_503_before_verification` pins that ruling. Changing the status would be a spec change. This PR only changes how the relay logs these denials. ## Database community source With `BUZZ_NIP_FI_COMMUNITY_SOURCE=database`, a not-found host and a failed lookup (DB error, timeout, shed waiter) return the same denial class. The marker is set only when the host is confirmed unmapped: a missing Host, a non-bindable host, or a live negative-cache entry. Resolver outages stay unmarked and still log as failures. With the `env` source, every resolution failure means the host is unmapped, so it is always marked. ## Testing - `nip_fi_enforce_unmapped_host_is_503_before_verification` (guard, root upgrade, audio upgrade) and `enforce_unmapped_host_is_503_before_nip98_and_verifier` (HTTP admission) now also assert the marker. Mutation check: removing the guard marker turns the router test red. - New `nip_fi_mapped_host_dependency_503_is_not_marked_unresolved`: a mapped-host 503 from an unavailable key source is unmarked and still classified as a failure. - New `relay_response_classifier_skips_only_unresolved_host_denials`: covers marked 503, unmarked 503, 500, 200 and 401. - New `only_confirmed_unmapped_hosts_are_marked_as_expected_denials` (database source, all three sites): a negative-cached host and a non-bindable host are marked; a bound host whose lookup fails is not. Mutation check: marking every database failure turns it red. - `cargo clippy -p buzz-relay --all-targets -D warnings` is clean, and the pre-push `rust-tests` lane passed. - Local relay, before and after. Setup: enforce mode, `RUST_LOG=error`, one mapped host, JWKS unreachable on purpose. Both builds got the same requests: | Request | Status (both builds) | `on_failure` errors before | after | |---|---|---|---| | Unmapped host, 4 protected paths | 503 | 4 | 0 | | Unmapped host, WebSocket upgrade on `/` | 503 | 1 | 0 | | Mapped host, no assertion | 401 | 0 | 0 | | Mapped host, well-formed JWT, JWKS unreachable | 503 | 1 | 1 | | Unmapped host, `/assets/` path and `POST /api/invites/claim` | 404 | 0 | 0 | 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com> Co-authored-by: opus <f038cc09b32a776e7d95b65c267e9d851777ee2b6cdbd46a48edba01b1d10ebd@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 16:08:13 -05:00
Luke Tornquist	152441ae0a	feat(loadgen): model desktop clients and large-community shape (#9005) ... ## Summary The load generator models every community as many small open channels, and every human as a bare WebSocket client: four REQs, a little history, then nothing until the run ends. Large deployments don't look like that. They have thousands of people in a few communities, a handful of very wide channels, and some people in hundreds of channels. Their desktop clients also make most of their relay requests over HTTP and on timers. This PR adds opt-in scenario knobs for both, so a run can reproduce that shape and compare the desktop client before and after its recent request-volume fixes. Every new key defaults to the previous behavior. Existing scenarios parse, assign members and run exactly as before. ### Shape | Key | Default | Meaning | |---|---|---| | `shape.private_channels` | `false` | `seed` creates channels private and adds every member with kind:9000 before posting history. Members are the humans and agents assigned to each channel up to the largest ramp step, plus the hot thread's root and authors. Events carry up to 100 members each and run `seed_concurrency` at a time. A rerun adds nobody new, and any member the relay refuses fails the seed. | | `shape.wide_channels` | `0` | The first K channels of each community are wide channels. | | `population.wide_member_fraction` | `0.0` | Share of humans that join every wide channel on top of their own. | | `population.wide_activity_share` | unset | Share of its typing, messages and reactions a member of the wide channels sends there; the rest goes to its regular channels. Unset keeps the uniform pick, which sends half of a 3-wide/3-regular member's traffic to the wide channels and makes fan-out, not reads, the bottleneck. | | `population.tiers` | `[]` | `[{ fraction, channels }]`: that share of humans joins `channels` channels instead of `channels_per_member`. | Assignment is deterministic by global index, so `seed` and every `--shard` agree without coordination. With no wide channels and no tiers it is the old assignment, and a test pins that. ### Desktop clients (`[desktop]`) | Key | Default | Meaning | |---|---|---| | `fraction` | `0.0` | Share of humans that run the desktop model instead of the basic human. | | `mode` | `"current"` | `legacy` is the client before its volume fixes; `current` is after them. | | `windows`, `threads` | `3`, `1` | Open channel windows and threads, which are refreshed on connect. | | `tick_secs` | `60` | Recovery tick period. | | `backstop_secs` | `900` | `current` only: each tick item (subscription replacement, roster, unread repair) waits this long since it last ran. | | `renewal_secs` | `0` | Identity renewal period. `0` disables renewal, and the first renewal is jittered. | A desktop human keeps the basic human's presence, typing, messages and reactions. What it adds: - **Cold start.** It reads NIP-11 and opens its profile and membership subscriptions. Its channel subscriptions are batched per mode: - up to 10 channels: one replaying batch; - `legacy` with more than 10: one per-channel batch per 10 channels; - `current` with more than 10: a live-only subscription per 2048 channels, plus replays of 10 that are closed at EOSE. It also makes the app's HTTP reads: roster, channel names, sidebar, profiles, statuses, emoji, read state, unread repair, reminders, window catch-up plus notices for each window, and root plus thread page for each thread. A window catch-up follows the cursor in the relay's window bounds (kind 39006) for up to five older pages, until the bounds say history is exhausted. - **Route establishment.** `legacy` makes every read as it connects. `current` reads NIP-11, names and sidebar up front, and the rest as each subscription reaches EOSE, by its replay outcome. A replay of 500 events, a missed deadline, or the live-only batch over 10 channels is a cut, and so is a route the previous connection never caught up. Only an EOSE catches a route up: one recovered after a missed deadline cannot be resumed or carried by the next connection. - A continuous profile route skips its read (unless one is owed). - A continuous membership route only repairs unreads, without the read-state snapshot, unless a snapshot read failed since; then it runs the full repair. - A cut global reads in full, and channel routes always recover windows and threads. - **Ticks and renewals.** These follow each mode: - `legacy`: every tick replaces subscriptions and rereads roster and unreads, and a renewal is a full reconnect; - `current`: tick items wait for the backstop, measured from the last replacement, the last successful roster read, and the last full unread repair. A failed roster read, unread repair or read-state snapshot is owed and retries on the next tick; snapshot debt is tracked apart from repair debt, so a repair without the snapshot never settles it. A renewal carries its routes: nothing is read while every replay keeps continuity, and once one is cut every route establishes as on a reconnect. With more than 10 channels the live-only batch is always cut, so the channels recover and unreads are repaired, but globals are not reread. - **Reporting.** Each request type is its own op (`desktop_roster`, `desktop_unread`, `desktop_replay_eose`, …). - **Bounds.** At most 4 REQs set up at once, with a 10 s EOSE timeout. A tick replacement that misses it is closed, the batch it would have replaced stays, and the next tick retries. At most 6 HTTP reads in flight, with a 30 s timeout. A read that comes due while the same read is running or queued is not added again, and each read leaves the running set as soon as it settles, so a failed read can retry while a slow one is still under way. Connects and failed sessions back off. A stop abandons pending EOSE waits without counting them as failures. - **Not modeled.** The app also establishes channel routes when a tick replacement reaches EOSE; the generator does not. `perf/LOADGEN.md` documents the knobs and the model. `perf/scenarios/local-desktop.toml` is a short local example with every knob on. ## What got simpler - `human()`'s schedule loop (presence, typing, messages, reactions, session end) moved into one `Activity`. The basic and desktop humans share it, so neither client duplicates the schedule. The basic human's random draw order is unchanged. - Membership stays in one place. `member_channels` remains the single assignment function, and `channel_members` derives from it rather than from a second formula, so seeding and the clients can't drift apart. The desktop model is a new concept: about 1,420 lines of model in its own module, plus about 1,300 lines of tests. Everything else goes through existing seams: `Stats`/`Op`, `connect`, `Backoff`, `observe`, `roots` and `Net`. Identified debt, not fixed here: CI's `test-unit` runs `buzz-test-client --lib` with the filter `loadgen::`, so the five crate-root `tests::` in `lib.rs` (relay message parsing, auth and text event building, `collect_until_eose`) never run in CI. That gap predates this PR, which doesn't touch `lib.rs`. ## Testing - `cargo fmt --check`, `cargo clippy -p buzz-test-client --all-targets -- -D warnings`, and `cargo test -p buzz-test-client` (plain parallel): 84 lib tests pass. - Wide activity: a seeded `Rng` over 100k picks gives each share within 1%, and an unset share makes the identical draws to before. A run against a loopback relay checks that a human's typing lands only in its regular channels at share 0, and only in its wide channels at share 1. - Scenario: validation of every new key; deterministic wide-channel and tier assignment that honors the fractions; `channel_members` inverts the assignment up to the peak; the default assignment is unchanged; the shipped scenarios parse. - Seed: with private channels, a fake relay sees the following, in order: - every 9007 before any 9000, and every 9000 before any post; - owner signatures; - ≤100 members per event; - exactly the expected members per channel; - posts authored by channel members. A refused member fails the seed, and open channels add nobody. - Desktop: request planning, window bounds validation, route establishment, and tick and owed-read rules are unit-tested. End-to-end tests drive clients against loopback WS and HTTP fakes. The WS fake replays a planned number of events per subscription and can delay an EOSE or drop a connection. The HTTP fake answers window pages with relay-signed bounds, an older aux event and equal-timestamp rows, refuses unknown cursors, and can fail a read once. They check: - exact cold-start counts per op for `legacy` and `current` at 5 and 25 channels; - a window catch-up sends the head, then exactly the bounds' cursor (`until` + `before_id` of the last row, not the older aux event), and stops when the bounds say history is exhausted even though that page came back full; - after a `current` renewal, 499 replayed events everywhere read nothing over HTTP; 500 on the channel batch adds channel recovery plus a covered unread repair; 500 on membership adds roster, read state, unread and reminders plus channel recovery; 500 on profiles adds profiles, statuses and emoji plus channel recovery and a covered repair; - after a dropped connection, complete global replays read only NIP-11, channel recovery and a covered repair; truncated ones reread both globals in full; - with renewals shorter than the backstop, read-state snapshots still recur at the backstop; - a tick replacement past its EOSE deadline is closed, the old batch stays, the next tick replaces it, and one batch wire stays open; - a failed roster and unread read retry on the next tick, and successful reads do not run again; - a failed roster read retries on the next tick while an unrelated slow read is still running; - after a failed snapshot, a continuous membership establishment before any tick still reads the snapshot; - a route whose EOSE missed its deadline is recovered, and after a renewal with a complete replay it is recovered in full again rather than carried; - `legacy` ticks replace batches (new REQ before the CLOSE of the old one) and reread; - `current` ticks do nothing until the backstop, then replace live-only; - `legacy` renewals are full reconnects; - `current` renewals carry small sessions without rereading globals or channels, and recover channels in large ones. Each of the nine review-fix tests was checked against a mutant that restores the old behavior, and every mutant fails its test. - Smoke: a local relay with Postgres and Redis. I ran `seed` twice, then `run` of `local-desktop.toml` (60 s) once per mode. - Both seeds were accepted (780 of 780 posts each), including private channels and memberships for 340 people. Neither run aborted, no client was cancelled, and no op recorded an error. - Each run used 18 desktop clients: | op | `current` | `legacy` | |---|---|---| | `desktop_live_eose` | 266 | 483 | | `desktop_replay_eose` | 24 | 0 | | `desktop_roster` | 72 | 289 | | `desktop_read_state` / `desktop_unread` | 72 / 99 | 289 / 289 | | `desktop_profiles` | 18 | 73 | | `desktop_nip11` | 18 | 73 | | `desktop_window` | 372 | 596 | Window reads are below the first smoke (406 and 1086) because catch-ups stop where the relay's bounds say history is exhausted, rather than paging while pages came back full. They are above the previous revision (179 and 470) because each read now leaves the running set as soon as it settles, so a channel recovery is no longer dropped as a duplicate of a read that already finished inside a still-running batch. - The first smoke caught two relay contract mismatches that the fakes had missed: - top-level window pages need `before_id` with `until`; - the read-state snapshot answers with an object. Both are fixed, and the fakes now enforce both. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 15:12:39 -05:00
Luke Tornquist	88697f1c59	fix(relay): apply kind 5 / 9005 deletions atomically (#8944) ... ## Summary Kind 5 and 9005 deletions now store the event and soft-delete their targets in one transaction. Before this PR the deletion committed first. The target soft-delete ran afterwards as a best-effort side effect, and a failure there was only logged. Resending the deletion then returned `duplicate:` without retrying. A crash between the two commits had the same result. Either way the relay accepted a deletion while its target stayed live. The bridge `include_aux` closure still compensates for those rows. Its second hop returns deletions of the reactions and edits it found, so clients can cancel them. This PR keeps that hop, because rows already torn this way can't be repaired by a resend: - Ingest rejects events more than 900 seconds from server time. - Clients don't resend deletions that were accepted. - During a rolling deploy, old pods keep writing torn rows until they are replaced. Dropping the hop waits for a follow-up, once existing torn rows have had a verified repair and no old writers remain. ### Changes - **Atomic deletion.** Kind 9005, and kind 5 with `e` targets, go through `Db::insert_deletion_event`, driven by `side_effects::persist_deletion`. The write stores the event and its mentions, soft-deletes the targets and updates their thread counters. It follows the existing `persisted_atomically` path used for workflow deletion and kinds 9000/9002. - A failure stores nothing and rejects the event, so the client can resend. - A resend of a stored deletion inside the ingest window soft-deletes any target still live and is dispatched as inserted. It doesn't repair older rows. - Live 39005, reaction-table cleanup and the 9005 system message still run after the commit and stay best-effort. - Kind 5 deletions of `a` coordinates stay a side effect. - **Closure unchanged.** The channel window and legacy thread-reply paths keep both hops, and the docs (`docs/bridge-channel-window.md`, `docs/nips/NIP-CW.md`) are unchanged. ### What got simpler Deletion no longer has a window where the event is accepted but not applied. `handle_standard_deletion_event` is now only the `a`-coordinate path. Target planning and soft-delete for `e` targets live in one place, `persist_deletion`. ### Follow-ups (not in this PR) - Repair the existing torn rows: soft-delete the live targets of stored kind 5 / 9005 `e` deletions and fix their thread counters. Then drop the closure's deletion-of-aux hop. - Hop 1 still probes every partition once per row id. A partition bound or the indexed `reactions` table would cut that. ### Related issue N/A, found while investigating relay database CPU. ### Testing - `torn_deletions_still_cancel_live_targets_in_both_windows` (bridge Postgres) stores kind 5 and 9005 deletions without soft-deleting their reaction and edit targets. It queries the channel window and the legacy thread path with `include_aux`, without a resend, and asserts each live target comes back with its deletion. - `insert_deletion_event_target_failure_stores_nothing_and_resend_applies` (buzz-db Postgres) uses a trigger to make the target's soft-delete fail. It asserts nothing is stored and the target is unchanged, then that a resend applies both. - `insert_deletion_event_resend_repairs_a_live_target` (buzz-db Postgres): a resend repairs a deletion stored without its target's soft-delete, and a further resend changes nothing. - `failed_target_soft_delete_stores_nothing` and `resent_deletion_repairs_a_live_target` (relay Postgres) cover the same two cases through ingest, for kinds 5 and 9005. - `test_channel_window_rows_overlays_and_exact_multiple_exhaustion` (`e2e_nostr_interop`, Relay E2E lane) deletes its reaction and asserts the reaction leaves the closure. - Blox: fmt, clippy (`buzz-db`, `buzz-relay`, `buzz-test-client`), and the `buzz-db` and `buzz-relay` Postgres lanes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 14:38:26 -05:00
Luke Tornquist	b8b425e8b1	fix(db): drop a superseded head's mention rows with it (#8948) ... ## Summary Replacing a replaceable or addressable event soft-deletes the previous head, but its `event_mentions` rows stayed behind. Only the hard-delete kinds (read-state and mesh status) removed theirs, in a separate statement. No mention-index read (`#p` filters and feeds) returns a soft-deleted event, so those rows are pure cost: every `#p` read walks them and throws them away. A kind:39002 roster republish leaves one dead row per channel member, so a large channel piles them up with every join. Relay-emitted replaceable lists with p-tags do the same. The statement that retires the old head now deletes that head's mention rows as well, through a data-modifying CTE over `RETURNING id`, on all three supersede paths: - the relay's kind:39002 roster writer (`LockedMemberSnapshot::replace_member_event`) - `replace_addressable_event` (kind 0/3/10000–19999, including the relay-signed kind:13535 list) - the NIP-33 parameterized path Because the CTE covers every row the retire statement touches, a coordinate that somehow had more than one live row also gets cleaned up. The shared `superseded_with_mentions_sql!` macro sits at the crate root so all three mention-indexing replacement paths use the same statement. The kind:13534 membership-list and push-lease writers also retire heads, with a bare `UPDATE`, but neither indexes p-tags, so they leave no mention rows. ## What got simpler The hard-delete special case (`if hard_delete_superseded { DELETE FROM event_mentions … }`) is gone. Both retire statements go through one macro, so a head and its mention rows leave together on all three paths. ## Not in this PR - Mention rows already left behind by earlier supersedes. `docs/database.md` rules out data backfills in migrations, so these get cleaned up out of band, in batches. - Other soft-delete paths (NIP-09 deletion, moderation, channel deletion retiring its 39000–39002 events) also keep mention rows. They are low-volume, and are noted here as identified debt. ## Tests `superseded_heads_drop_their_mention_rows` (pg) has one table row per supersede path: the member-snapshot roster writer (lock, replace, release), the addressable path, the NIP-33 path, and the two NIP-33 kinds that are hard-deleted (NIP-RS read state and mesh status). In each case the old head keeps zero mention rows and the new head is indexed. Soft-delete rows check that the old head stays soft-deleted, and hard-delete rows check that it is gone. Reverting the macro, any single call site, or the hard-delete statement alone fails this test. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 14:30:52 -05:00
Luke Tornquist	298a555e34	perf(db): bind the channel scope as one array (#8946) ... ## Summary Event reads scoped to a set of channels pushed one bind parameter per channel: `channel_id IN ($5, $6, …)`. This applies to a viewer's access scope, which the relay adds to every REQ and `/query` without an `#h`, and to explicit multi-value `#h` filters. As a result, the SQL text depended on the viewer's channel count, and every count was a separate prepared statement. That caused two problems in production: - **Re-planning on every call.** A connection rarely reused a cached plan for these reads, so Postgres planned each one again. With no `since`, the plan covers every `events` partition. For `#p` mention reads (the `event_mentions` join), planning took far longer than executing the query. - **Hidden cost.** Monitoring that groups by statement split one query into one row per channel count, so its total cost never showed up in a single place. This PR binds the set as `channel_id = ANY($n)`, so one statement serves every channel count. On a production replica, the same query prepared with `ANY($5)` moved to Postgres's cached generic plan by its sixth run. Planning time dropped to near zero, and that plan was no slower than the custom one. ## What changed - `push_channel_scope` in `crates/buzz-db/src/store/event.rs` builds the channel-scope predicate as one array bind. - `build_query_events_sql` and `count_events_sql` each had their own copy of this block. Both now call the helper. - Semantics are unchanged: - Access scope still keeps channel-less global events. - Explicit `#h` still matches only the requested channels. - An empty set still means "no channels". ## What got simpler Two duplicated 20-line blocks became one helper. The statement count for channel-scoped reads no longer grows with channel count. ## Generic-plan exposure These statements were already persistent prepared statements, so a cached generic plan was always possible whenever the connection's statement cache held one. This change makes that reuse consistent; it doesn't introduce generic plans. Tag-value reads (`has_tag_values`) still run unnamed, as before. Plain (non-`#p`) access-scoped reads stay on custom plans. A production replica, with a viewer in hundreds of channels, ran the plain page read repeatedly under the default `plan_cache_mode`, and every run used the custom ordered index scan. Postgres estimated the generic plan as far more expensive, and when forced, it was orders of magnitude slower. Postgres switches only when the generic plan is no costlier than the average custom plan, so it never chose this one. The generic plan's cost comes from the `LIMIT` and `kind` values it can't see, not from the array length, and the per-arity `IN (…)` statements had the same blind spot. ## Tests - `channel_scope_sql_does_not_depend_on_channel_count` (unit): for the `#p` join and plain reads, for access scope and explicit `#h`, in both the page SELECT and the COUNT, the SQL for 1 channel equals the SQL for 40 channels. It also checks that the SQL uses `ANY(` and that global rows are retained only for access scope. - `channel_scoped_mention_reads_share_one_prepared_statement` (Postgres): on one connection, access-scoped `#p` reads with 1, 2 and 3 channels leave exactly one prepared `event_mentions` statement. With one mention per channel, one global mention, and one in an unscoped channel, every scope of 0–3 channels, with and without globals, returns and counts exactly its own rows. - The existing scope semantics tests still pass: - `access_scope_is_applied_before_historical_page_limit` - the explicit multi-channel `#h` page and count test - Mutation: reverting the helper to one bind per channel fails both new tests. ## Identified debt (not in this PR) - `get_last_message_at_bulk` still builds an `IN (…)` list per channel. It's a separate, lower-volume statement. - Superseded versions of replaceable roster events (kind 39002) keep their `event_mentions` rows. On large channels, a viewer's roster mention rows grow with every membership change. That's a follow-up. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 19:25:34 +00:00
Will Pfleger	f6198f6997	feat(relay): record NIP-FI enforce denials at parity with shadow (#8974) ... ## What this does In enforce mode, NIP-FI denials used to be returned without a reason counter or a log line. This PR records enforce denials with the same `WouldDeny` reasons, label sets and log fields as shadow. **Admission behavior does not change**: statuses, close codes and frame order are all the same. - **`buzz_nip_fi_enforce_total{route,stage,outcome,community}`**: the same shape as `buzz_nip_fi_shadow_total`, including an `admit`/`admit` stage, so `1 - admit/total` is the enforce denial rate. Each socket or HTTP request gets exactly one verdict. - **`buzz_nip_fi_enforce_session_end_total{route,reason=expired|revoked,community}`**: once per admitted session. - **`buzz_nip_fi_enforce_strict_proof_total{route,outcome,community}`**: the outcome of the proof check that actually decided the request. Nothing is verified twice. - **Log lines** `nip-fi enforce deny` and `nip-fi enforce session end` (info, `mode="enforce"`, from `buzz_relay::nip_fi_shadow`). Shadow's `nip-fi shadow would-deny` gains `mode="shadow"`. Neither mode logs admits. ### How each verdict is recorded once - **Sockets (root, audio).** `EnforceSession` moves Pending → Admitted(pubkey) → Ended under one std mutex, and records inside the lock with no awaits. - Root admits at its AUTH commit; audio admits at its join commit. - A refusal for a non-NIP-FI reason retires the session without a record. Audio retires it before rollback. - A deadline the effect gate sees before the expiry worker is claimed at that seam. - **Ends.** Revocation and expiry are recorded only by the terminal transition that wins (`publish_authorization_denied`), and only while the socket isn't already cancelled. A community close or cancellation that wins first records nothing. - **HTTP.** The router guard scopes each enforced request with one verdict. An agent token's verdict is deferred to the end of the request: - the first owner-decided refusal is the verdict, otherwise the request was admitted; - membership rechecks, batches, Git success and early returns each count once. - **Owner provenance** comes from the decision itself (`BanVerdict{outcome,source}` from one `restriction_state` read), with no rereads. - Reasons: `owner_banned`, `owner_ban_unavailable`, `not_stored_owner`, `owner_link_unavailable`. - When the agent's own ban read fails and there's a token owner, the result is `owner/unavailable`, as shadow records it. - A ban or failed restriction read that ingest catches on `POST /events` after membership passed replaces the request's admit with `owner/denied` when the owner's row decided the ban, or `owner/unavailable` on a failed read. An agent's own ban is not recorded as its owner's. ## Shadow-to-enforce map | Shadow signal | Enforce counterpart | |---|---| | `buzz_nip_fi_shadow_total` denials: upgrade, guard, HTTP admission, AUTH pairing, deny set, stale binding | `buzz_nip_fi_enforce_total` with the same stage/outcome, plus a `nip-fi enforce deny` line | | `buzz_nip_fi_shadow_total` owner stage: ws, audio, http | the same, from the original decision: root/audio ban seam and owner link; HTTP `enforce_membership`, smart Git, Git settings, buzz/v1 ban reread | | `buzz_nip_fi_shadow_total` `deadline/session_deadline` (pending) | `EnforceSession::expire`, from the worker's winning terminal or the effect gate that saw it first | | `buzz_nip_fi_shadow_total` admit | `buzz_nip_fi_enforce_total{stage="admit",outcome="admit"}`, one per socket or request | | `buzz_nip_fi_shadow_session_end_total{reason=expired\|revoked}` | `buzz_nip_fi_enforce_session_end_total`, same labels | | `buzz_nip_fi_shadow_strict_proof_total` | `buzz_nip_fi_enforce_strict_proof_total`, from the deciding check (a rejection is the route's 403, or 401 when the proof is missing) | | `buzz_nip_fi_shadow_disconnect_total{admin,accepted}` | `buzz_nip_fi_disconnect_total` (`api/nip_fi.rs:160`) | | `buzz_nip_fi_shadow_disconnect_total{admin,propagation_failure}` | `buzz_nip_fi_disconnect_propagation_failures_total` (`api/nip_fi.rs:180–185`) | | `buzz_nip_fi_shadow_disconnect_total{admin,capacity}` | `buzz_nip_fi_disconnect_capacity_rejections_total` (`api/nip_fi.rs:196–201`) | | `buzz_nip_fi_shadow_disconnect_total{cross_pod,capacity}` | `buzz_nip_fi_cross_pod_capacity_exceeded_total` (`api/nip_fi.rs:433–434`) | | `buzz_nip_fi_shadow_disconnect_total{cross_pod,poison}` | `buzz_nip_fi_cross_pod_shard_poison_total` (`api/nip_fi.rs:442–443`) | | `buzz_nip_fi_shadow_disconnect_total{cross_pod,lag}` | `buzz_nip_fi_disconnect_lag_total` (`main.rs:1596–1602`) | The enforce series was renamed from `buzz_nip_fi_enforce_denied_total` to `buzz_nip_fi_enforce_total` when it gained the `admit` stage. ## Tests Infra-free, in the `test-unit` lane: - `state::enforce_end_is_recorded_only_by_the_winning_terminal`: root scan, audio scan and deadline, each against a community close or cancellation first and FI first. - `nip_fi_http::enforce_strict_proof_records_the_deciding_check_once`: pass and reject, and no record when cardinality refuses. - `nip_fi_shadow_session::audio_owner_link_refusal_shares_roots_seam_and_reasons`: calls the shared owner-link helper with route `audio`, not a huddle socket. - The existing shadow, upgrade, router and HTTP enforce tests. Postgres lane: - `handlers::auth::enforce_root_auth_records_owner_refusals_from_one_decision`: clean admit then `expired` through the real expiry worker; banned token owner, banned stored owner, other stored owner and a failed ban read; authorization reads counted from the decision only. - `handlers::auth::enforce_root_deadline_seen_by_the_gate_first_is_recorded_once`: both orderings. - `audio::handler::enforce_join_commit_is_the_audio_admission`: drives the join-commit seam directly, not a huddle socket or the expiry worker: committed then expired; archived with an expiry held inside the rollback; gate-first deadline. - `api::bridge::token_owner_reaches_membership_on_every_nip_fi_http_route`: one verdict per request on every NIP-FI HTTP route, including sidebar, read-state, Blossom and Git, for an owner ban, a stored-owner conflict and a clear owner. - `api::bridge::token_owner_ban_reread_is_the_fixed_nip_fi_denial_on_http`: own ban, owner ban and unavailable at the reread, on sidebar and Git settings. - `api::bridge::token_owner_publish_ban_is_the_fixed_nip_fi_denial_on_http`: a late ban or failed read on `POST /events` at the write, typing, moderation and relay-admin checks records exactly one owner denial and no admit when only the owner is banned, and no owner denial when the agent alone or both are banned. - `audio::handler::tests::already_expired_audio_bootstrap_records_the_deadline`: an already-expired bootstrap records `deadline/session_deadline`, not a deny-set denial. Each new test names the mutation that turns it red. These were run red: - deleting root's `session.admit(pubkey)` - deleting audio's commit `session.admit(pubkey)` - deleting `effect_refused` - deleting enforce from `retire_shadow` - deleting the commit seam's `expire` - deleting the cancellation guard - deleting the own-read-failure provenance --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>	2026-11-03 14:21:32 -05:00
Daniel Rouleau	a9aa1f73b0	Enable incremental artifact reads on every relay (#8975) ... ## Why Compatible clients need incremental artifact reads for local collections on every Buzz relay. ## What Expose incremental artifact reads and capability discovery on every Buzz relay. ## How Serve incremental queries through the existing authentication and authorization checks. Keep the existing work budgets and the separate WebSocket sync capability. The HTTP test verifies capability discovery and a successful incremental query. ## Risk The relay starts no additional background work. Clients that use incremental reads consume database capacity. The net database load is not yet measured. Serving writers must maintain the modification timestamps. ## Testing Read-only SQL checks on November 3, 2026 verify the prerequisites below. Each database records successful migrations 76–78. All five required columns have the expected types, defaults, and nullability. All five indexes have the expected definitions and are valid, ready, and live. | Deployment | Schema and indexes | Running timestamp writers | | --- | --- | --- | | Production `bb-block` | Pass | `f6f6d9a` includes PR 8523 | | Production `bb-public` | Pass | `f3a7bab` includes PR 8523 | | Staging `bb-block` | Pass | `b4509de` includes PR 8523 | The checked columns are `artifact_heads.modified_at`, `artifact_revisions.modified_at`, `artifact_revisions.source_channel_id`, `channel_members.sync_modified_at`, and `communities.artifact_policy_modified_at`. The checked indexes are `idx_artifact_revisions_artifact`, `artifact_heads_modified`, `artifact_heads_channel`, `channel_members_sync`, and `artifact_revisions_moves`. At commit `62b26ad`, all 21 NIP-11 tests and the HTTP capability test pass on Linux with PostgreSQL 17 and Redis. The HTTP test checks capability discovery and a successful incremental query. All required push hooks also pass. No manual testing. Human confirmation remains pending. ## Bigger picture [Client PR 1507](https://github.com/block/buzz-app/pull/1507) uses the advertised capability for local artifact sync. Existing deployments receive this behavior when they deploy the relay version that includes this PR. Generated with Codex --------- Signed-off-by: Daniel Rouleau <drouleau@squareup.com>	2026-11-03 10:58:24 -08:00
Will Pfleger	ce47aefdc3	fix(relay): send membership notices on channel delete, moderation kick and moderation DM (#8977) ... Related: [buzz-app#2215](https://github.com/block/buzz-app/pull/2215) ## Problem A connected client learns about membership changes from kind:44100/44101 notices and then re-reads its channel list. Three relay paths changed a member's channel list without sending a notice, so clients only caught up on their periodic full read: - **Channel delete (9008):** no member was told. - **Moderation kick:** the kicked member wasn't told, and the channel's 39000/39002 discovery wasn't refreshed, so it still listed them. - **Moderation DM:** a new, existing or hidden moderation DM didn't surface live. ## Change - **Channel delete:** `soft_delete_channel_capturing_members` (buzz-db) reads the channel type and active members under deletion's existing membership lock, in the same transaction. A join either commits first and is included, or waits and is then refused. After the commit, discovery cleanup and cache invalidation, the relay sends a type-tagged 44101 to each captured member in a spawned task, eight at a time, outside any lock. The 9008 OK doesn't wait for them, and the task is started before the `channel_deleted` system message is emitted, so a system-message failure can't drop them. An already-deleted channel sends nothing. `soft_delete_channel` keeps its signature and delegates. - **Moderation kick:** `apply_kick_live_side_effects` refreshes 39000/39002 after the removal fence commits, then sends 44101 to the kicked member. If the refresh fails, it returns the error before the notice, so the moderation driver's existing mutation-marker recovery retries. If the channel was deleted since the kick, there's no roster to refresh: it removes the channel's discovery events and treats that as converged, so the action still finalizes. The removal/re-add fence and the re-added early return are unchanged. - **Moderation DM:** `send_moderation_notice` drops both participants' cached channel sets on every pod (`invalidate_membership`), as opening a DM does in `command_executor::handle_dm_open`, so the read the notice prompts includes the DM. After clearing `hidden_at`, it republishes the recipient's NIP-DV kind:30622 visibility snapshot (clients filter the sidebar on it), then emits discovery, then sends 44100. It sends the notice on every delivery, tagged `moderation_source` with the delivery's source id, so two deliveries in the same second each get their own 44100. The content JSON is unchanged. A failed snapshot or discovery returns an error, so the outbox retries. - **Ordering guarantee:** each notice is sent after its change is committed and readable on the relay instance that sends it. Ordering across relay instances, and other instances' cached channel sets, are best-effort, the same as for existing membership notices. - A failed notice delivery is logged and never undoes the change, like the existing 9001 path. There's no new retry system. ## Tests - `membership_notice_postgres_tests.rs`: one journey per path. A `#[cfg(test)]` barrier at the point a notice is signed parks the producer, keyed by recipient. While it's parked, the test reads the channel list (and, for the DM, the 30622 snapshot) as a client would on receipt. The barrier can also pin a notice's `created_at`. - The DM journey pins both deliveries to the same second, hides the DM through the real kind:41012 path between them, and checks what a client sees: a live 44100 for each delivery, with distinct ids, then a channel-list read that lists the DM and a visibility read that no longer hides it. - `moderation_dm_notice_follows_a_cached_channel_list` warms the recipient's channel set with a read before the DM exists, then checks the read on receipt of the notice lists the DM. - `soft_delete_captures_members_at_its_serialization_point` (buzz-db): a join and then a delete queue behind a held membership lock. The delete must return the joiner, and a later join is refused. - `crash_recovery_finalizes_kick_after_channel_deleted` (admin): commits a kick, deletes the channel, then runs the recovery worker. The action finalizes, and the member's `removed_at` is unchanged, so the removal doesn't run again. Each ordering test fails when its prerequisite is moved or removed: the kick notice ahead of the roster refresh, the delete notices ahead of discovery cleanup, the DM without the snapshot republish or the cache drop, the DM notice without its source tag (the second notice coalesces), and the member capture outside the lock. The recovery test fails if a deleted channel's `ChannelNotFound` propagates. No spec files change. This reads a channel delete as removing every member under the existing `NOSTR.md` notice promise. --------- Signed-off-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>	2026-11-03 13:57:34 -05:00
Salman Mohammed	d3a4909cca	perf(search): rank only the newest prefix matches and plan with the real tsquery (#8976) ... ## Summary Message search is one of the slowest relay queries. Short typeahead prefixes make it worst: `a:*` matches most of a community. **Why it was slow:** the tsquery was built inside a `CROSS JOIN LATERAL` subquery, so the planner saw an InitPlan and couldn't read the `search_tsv` statistics. It fell back to the default 0.5% selectivity for `@@`: for `a:*` on 1M rows it estimated 4,497 rows when 517,065 matched. Every search therefore took every GIN match, ran `ts_rank_cd` on each one, and top-N sorted them before `LIMIT 20`. This PR: 1. **Binds the real tsquery.** `push_tsquery` is unchanged. A separate `SELECT (…)::text` round trip resolves the tsquery, and the search binds it as `$q::tsquery`. An empty tsquery returns no hits without running the search. 2. **Plans each search for its own tsquery.** The search runs with `.persistent(false)`. A cached prepared statement can switch to a generic plan after five runs, and a generic plan can't see the tsquery. Forcing a generic plan lost the gain in testing. 3. **Ranks only the newest 1,000 prefix matches.** Prefix-mode relevance search ranks the newest `PREFIX_RANK_WINDOW` (1,000) matches. The inner `ORDER BY created_at DESC, id DESC LIMIT 1000` matches each partition's primary key read backwards, so for common prefixes Postgres streams the newest rows and stops at 1,000. For rarer terms it keeps the GIN bitmap plan, now with an accurate estimate. Full-text search and profile (kind:0) typeahead still rank every match. ### Behavior change: every human message search (product decision needed) Prefix mode isn't limited to keystroke typeahead. buzz-app (`useSearchMessages.ts`) and legacy desktop (`desktop/src-tauri/src/commands/messages.rs:184`) send `search_mode: "prefix"` for **all** message search. Full-text search runs only on the WS REQ path, which agents and the CLI use. So for people: - Results are the best-ranked of the newest 1,000 matches in the search's scope, not of all matches. If more than 1,000 newer messages also match, an older, more relevant message drops out of ⌘K results. The scope includes `#channel`, `from:` and date bounds, so a narrower search has the same limit within it. People have no full-text fallback. - The bridge honors `page`, and buzz-app loads later pages at 20 per page. Every page ranks the same window, so paging stays consistent, but results end after 1,000 matches (page 50). Full-text search (agents/CLI) and profile (kind:0) typeahead still rank every match. ### Perf risk: matches the backward scan has to walk far to find The planner picks the backward scan when it expects 1,000 matches near the top of the community, and it estimates that by treating each filter as independent. Two known shapes break that assumption. **Terms common in old messages but absent from recent ones.** The planner assumes matches are spread evenly over time. For a term that was common long ago but doesn't appear recently (an old project name, a former teammate), it can pick the backward primary-key scan and walk every newer row before reaching the first match. Reviewers measured this on 1.2M synthetic rows: about 2.05–2.1 s, against 0.7–1.8 s for the old query shape and 1.2 s with the GIN plan forced. The 20 s statement timeout and the #8954 cancel still bound it. The worst case is close to the old `a:*` worst case, but it now scales with community size, not match count. The planned EXPLAIN on production-shaped data will include such a term. **Terms common in channels the searcher can't read.** Human search always adds `channel_id = ANY(<readable channels>) OR channel_id IS NULL` (`build_search_channel_scope_filter`). A term that is common in private channels or other people's DMs can get a high match estimate, take the backward scan, and walk the community's rows to find a few readable matches, where the old shape used the GIN bitmap. This hasn't been reproduced yet; the production-shaped EXPLAIN will include such a term next to the old-only one. The same timeout and cancel bound it. ### Cost - One extra round trip per search to resolve the tsquery (about 0.03 ms of execution). It runs on the same pool checkout as the search (`cancel::fetch_all` takes the caller's connection), so a busy pool queues each search once. An IMMUTABLE SQL function could remove the round trip, but that needs a migration, so it's not done here. ### Related issue - Follows #8954 (cancel abandoned searches, bound the search pool) and #8964 (closed in favor of #8954). - Builds on #8947 (3-character server-side prefix minimum), merged from main. The resolve step passes its minimum. When the trailing prefix is too short, the resolved tsquery is NULL, and the search returns no hits without running the second query. - Client-side pairing: block/buzz-app#2210 (3-character last-word minimum for ⌘K message search). ## Testing **EXPLAIN ANALYZE on a synthetic 1M-row `events` table** (monthly partitions, GIN on `search_tsv`, kinds 9/40002/40008, custom plans, warm cache): | Prefix | Matches | Before | After | Plan after | |---|---|---|---|---| | `a` | 517k | 322 ms | **0.7 ms** | backward primary-key scan per partition, stops at 1,000 | | `th` | 137k | 127 ms | **1.2 ms** | backward primary-key scan, stops at 1,000 | | `a` (community with 10% of rows) | — | ~100 ms | **1–2 ms** | backward primary-key scan | | `deploy` | 8.2k | 15 ms | 5–10 ms | GIN bitmap, accurate estimate | | `d278` | 500 | 1.1 ms | 0.5 ms | GIN bitmap | | rare word | 0 | 0.07 ms | 0.03 ms | GIN bitmap | **Postgres integration tests:** - New test `prefix_relevance_ranks_only_the_newest_matches`. An old, highly relevant message drops out of prefix results behind 1,000 newer matches but still ranks first in full-text search. Prefix pages end at page 50. An old, best-ranked profile still tops profile typeahead behind 1,000 newer profile matches. Removing the window fails the test, and so does applying it to profiles. - `abandoned_search_cancels_its_statement`: the `pg_stat_activity` matcher now looks for `ts_rank_cd(`, because the old `search_query.query` text no longer exists. - Locally, each test ran against a fresh database because `cargo-nextest` isn't installed, so the hosted lane hasn't run yet. 26 of 27 pass. `artifact_search_uses_only_live_current_head` also fails on unmodified main in that local setup, so the failure isn't from this change. **Checks:** `cargo fmt --check` and `cargo clippy -p buzz-search --all-targets -D warnings` are clean. **Not yet done:** - hosted CI; - a manual check against a local relay with ⌘K typeahead; - EXPLAIN against production-shaped data, including a term common only in old messages and a term common only in channels the searcher can't read. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 13:25:27 -05:00
Luke Tornquist	d44031f418	perf(relay): cut writer-pool round trips past the 48k relay ceiling (#8846) ... ## Summary Five changes that take load off the writer pool and the relay's connection bookkeeping. On a single-relay load rig, the clean ceiling moves from 40k to about 48k WebSocket clients. In the A/B, connect p99 at 48k fell from 4.1 s (abort) to 1.65 s. 1. **Ban/mute restriction cache with a 1 s max age.** HTTP API requests, publishes and ephemeral writes read `(community, pubkey)` restriction state through one cache. An entry is served for 1 s from when its read began, matching the open-channel cache's bound. Socket admission, moderation-command actor checks and the `/buzz/v1` ban recheck still read fresh. An admitted socket outlives the 1 s window, so a banned client that reconnects inside it would otherwise keep a socket open indefinitely. A ban drops the target's entry, and each owned agent's entry, on every pod as part of the live-disconnect fan-out, best-effort. Timeouts, untimeouts and unbans drop it on the pod that handles them. 2. **Fewer round trips per event read.** Replica reads skip the per-read `statement_timeout` tighten, because the reader pool installs the deadline at connect. They keep the savepoint, so a failed or cancelled read leaves the session usable and releases its locks. That takes them from 4 round trips to 3. Writer reads send `BEGIN` and the tighten as one message, which also takes them from 4 to 3. 3. **Concurrent open-channel cache misses share one read.** The first miss in a community takes a per-community gate and reads; the others wait for it and use its result. A waiter whose leader failed reads once on its own and does not queue again. A waiter also stops waiting after the cache's 1 s max age and reads on its own, because the leader's set would be stale by then anyway. So a leader whose read hangs stalls only its own request. 4. **`has_admitted_connection` is one map lookup.** A `(community, pubkey) → admitted count` index is maintained under each socket's pubkey lock in `mark_admitted`, `set_authenticated_identity` and `deregister`. It replaces a scan of every connection on each disconnect. 5. **Host → community cache with a 1 s max age.** Only successful lookups are cached; unmapped hosts and errors go to the database every time. ## What got simpler - The ban handler's separate pod-local cache drop is gone; the disconnect every ban already runs covers it on every pod. - One write-restriction gate instead of two. `enforce_cached_write_restriction`, `enforce_relay_membership_ephemeral` and the `cached_ban` parameter chain through `enforce_membership` / `admit_member` / `deny_banned` are deleted. - The ephemeral path's separate 30 s ban cache is gone, so the relay has one staleness rule for restriction state instead of two. ## Notes - Bans reach the restriction cache on every pod. Every ban path (signed command, report action, relay admin) closes the target's and its agents' sockets clusterwide, and each pod drops those entries in the same step. A dropped Redis publish, or a restriction read already in flight when the drop lands (it can refill the old row), falls back to the 1 s bound. Timeouts on other pods also rely on the 1 s bound, and so do unbans and untimeouts, which fail closed. - On the load rig the restriction-cache hit rate was 28%, because HTTP agents poll every 5 s. A longer TTL with Redis invalidation would raise it, but that is a separate change. ## Load test Single relay pinned to 8 cores, native Postgres 16, a seeded fixture with 2,000 open channels, and a 6-shard generator. Steady ramp with 120 s holds; a step aborts at p99 > 3 s. Each step adds 10% HTTP agents. | clients | base connect p99 | this PR connect p99 | base relay cores | this PR relay cores | |---|---|---|---|---| | 32k | 34 ms | 31 ms | 1.71 | 1.66 | | 40k | 45 ms | 26 ms | 2.33 | 2.15 | | 48k | 4,107 ms (abort) | 1,652 ms | 4.7 | 3.0 | | 56k | n/a | 4,535 ms (abort) | n/a | 4.0 | Open-channel re-reads over the run fell from 72k to 1.6k. The tenant cache hit rate was 94%. 48k is this branch's knee. A later run of the same build aborted at 48k: connect p99 2.2 s, but 29% subscribe errors from the concurrent-handler cap. With a 10 s restriction max age, it passed 48k, at connect p99 1.56 s, 60% hit rate and 10% fewer relay cores, and both arms stopped at 56k. ## Testing - `cargo fmt --check`, `cargo clippy -p buzz-relay -p buzz-db --all-targets -D warnings` - `cargo test -p buzz-db --lib` and `-p buzz-relay --lib`, run in parallel. The only failures were 8 DB-seeding tests that need a migrated default database; the Postgres lane covers them. Commits after that run changed the replica savepoint, the `/buzz/v1` recheck revert and some test edits. - Postgres lane at head: `scripts/postgres-test-run.sh -p buzz-relay -p buzz-db --lib --bins --tests`. All 1,183 tests passed. - Mutants. Each one must turn its new test red: - waiters skipping the gate - a no-op index removal - the tenant cache ignoring age - the restriction cache ignoring age - a waiter waiting on the gate with no limit - a disconnect that keeps the cached restriction row - replica reads dropping the savepoint buzz-review-completed 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 11:15:49 -05:00
Luke Tornquist	d005c7bb9c	fix(search): skip message prefix search below three characters (#8947) ... ## Summary Typeahead surfaces (the buzz-app and desktop topbars) send a NIP-50 search with `search_mode: "prefix"` whenever typing pauses. A one- or two-character trailing `:*` prefix matches a large share of every indexed message. The GIN index on `search_tsv` can't stop early or return rows in rank order, so Postgres fetches, rechecks and ranks every match before it applies `LIMIT`. In a large community, each of these queries costs seconds of database CPU. With this change, prefix-mode `search()` returns no hits when the trailing whitespace token yields a lexeme shorter than three characters (`PREFIX_MIN_CHARS`). The check runs inside the prefix tsquery SQL, on the output of the same `to_tsvector('simple', …)` call that decides which lexemes get `:*`. Punctuation therefore can't pad the count: `pr.`, `@pr`, `#pr`, `c++`, `don't` and `pre-p` are all held back, and `project ?` still runs, because `?` yields no `:*` lexeme. When the rule applies, the tsquery is NULL, so it matches nothing and the GIN index lookup returns no rows. Completed words in front of the trailing token don't matter: `"the p"` runs the same broad `p:*` lookup as `"p"`. Kind:0-only queries (profile typeahead) are exempt because there are few profiles, and the short-name exact-lexeme ordering relies on one- and two-character queries. `search_profile_names` is unchanged. Full-text mode is unchanged. Only the HTTP bridge (`POST /query` with `search_mode: "prefix"`) reaches prefix mode. WebSocket REQ search always uses full-text mode, so this change doesn't affect it. ## What got simpler The kind:0-only predicate is now computed once (`profile_only`) and shared by the new minimum and the existing exact-lexeme ordering. The minimum lives in the one SQL fragment that already tokenizes the query, so there is no second tokenizer in Rust to keep in sync with Postgres. No new concepts beyond one constant. ## Known limits - This bounds the worst case only. A common three-character prefix can still match enough rows to be expensive. Bounding that needs a product decision, such as a recency window for typeahead or a different index. - Two-character CJK words (for example `東京`) get no message typeahead until a third character. A trailing contraction or hyphenated word with a short part (`don't`, `e-m`) gets none until the next word is started. Full-text mode is unaffected. - Clients should not show "no results" for one- and two-character input. A client-side minimum is a companion change. That includes the legacy desktop topbar in this repo (`desktop/src/features/search/hooks.ts`, which searches at 2 characters globally and 1 in a channel); it is left out here to keep this PR relay-only. ## Testing - Extended `prefix_mode_matches_final_token_prefix_without_changing_full_text` with a table of prefix queries and the exact hit sets they must return. It pins no hits for a trailing token that yields a lexeme below three characters: alone, after a completed word (`project pr`), padded with punctuation (`pr.`, `@pr`, `#pr`, `...p`), as a contraction (`don't`), and hyphenated (`pre-p`). It covers `kinds = [9]`, no kinds, mixed `[0, 9]`, and `Newest` sort. Each of those rows has data it would match without the minimum. The same table has controls that must still match: `pro.`, `project ?` (punctuation-only trailing token), `pre-pro`, `t pan` (a short completed word stays exact and isn't limited), and a one-character kind:0 `p`. A full-text `don't` control shows that full-text mode is unaffected. `short_kind0_prefix_prioritizes_exact_lexeme_on_a_noisy_page` pins the kind:0 exemption. - Mutation-checked on Blox. Each of these fails a test: removing the minimum, removing the kind:0 exemption, and applying the minimum to every lexeme instead of only the trailing ones. - Synthetic plan check on Blox Postgres (300k rows, GIN on a generated `simple` tsvector, same `CROSS JOIN LATERAL` query shape). `'pr':*` took 3.6 s and read 10,411 buffers. The NULL tsquery took 0.01 ms: the bitmap scan returned 0 rows and read no buffers. - Two existing prefix tests used a two-character trailing token (`project pl`, `alpha be`). They now use three or more characters (`project plan`, `alpha bet`) and assert the same behavior. - Blox: `cargo fmt --check`, `cargo clippy -p buzz-search --all-targets -D warnings`, `buzz-search` unit tests, and the `buzz-search` Postgres lane (`scripts/postgres-test-run.sh`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 11:08:00 -05:00
Luke Tornquist	83bd8b1cae	fix(loadgen): sign public URLs when dialling connect_addr in plaintext (#8926)	2026-11-03 02:04:19 +00:00
Luke Tornquist	46da35810d	db: recycle pool connections after 5 minutes instead of 30 (#8951)	2026-11-02 19:35:27 -05:00
Luke Tornquist	02925ceb6e	fix(relay): cap bridge searches in flight per user (#8956) ... ## Summary Typeahead starts a bridge `/query` search on almost every keystroke. The client aborts the previous request. But if the abort never reaches the relay, for example because an intermediary keeps the upstream request open, every one of those searches keeps running. Each `(community, pubkey)` may now have **4** bridge searches in flight. The next search supersedes the oldest one. The superseded request returns **409** `{"error":"search superseded by a newer search"}`. It is deliberately not a 429: desktop arms its relay-wide admission gate on any 429, which would hold every relay request for 10 s, and the CLI retries a `rate-limited:` 429. Its future is dropped, so #8954's cancel-on-drop cancels its statement on the server. - **Newest wins.** The newcomer is never rejected, because the newest keystroke is the one the user is waiting for. - **The cap is above 1.** Sessions that share a pubkey keep their parallel searches: several agent sessions, or people search and message search running side by side. - **Bookkeeping lives in process.** `SearchSlots` holds a `Mutex<HashMap<key, VecDeque<(id, CancellationToken)>>>`. A slot removes itself by id when its search finishes, fails, is dropped or is superseded. So a superseded search can never remove the newer search that replaced it, and a user's entry is removed when their last search leaves. - **A search that finishes as it is superseded still returns its result.** The `select!` is biased toward the search. The limit is per relay process, not global. That is enough to bound how much one user can pile onto one process's search pool. ## What got simpler Nothing was removed. The change adds one small, self-contained module and one call site. It also wires the new tests into the unit lane (`Justfile test-unit` and `scripts/run-tests.sh`). The relay's unit lane is an allowlist, so without that wiring the tests would run nowhere in CI. ## Tests `api::bridge::search_slots::tests`: - `the_search_past_the_cap_supersedes_the_oldest` - `a_finished_search_frees_its_slot` - `a_superseded_search_leaves_its_successor_in_place` - `no_entry_outlives_its_searches` - `a_superseded_search_is_a_conflict_not_back_pressure` Each of these mutants made at least one test fail: - Removing the cap. - Replacing the by-id removal with `pop_front`. - Leaving empty user entries in the map. - Answering supersession with 429. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 00:33:09 +00:00
Luke Tornquist	949f61dfab	fix(search): cancel abandoned searches and bound the search pool (#8954)	2026-11-02 23:19:23 +00:00
Phil Azar	8f83c63209	fix(db): preserve partition reader grants (#8485) ... ## Summary PostgreSQL does not copy parent table grants to new partitions. A reader can query the `events` parent but fail direct partition reads after the relay creates another month or replaces its catch-all. Copy each managed parent's explicit table-level `SELECT` grants to its new children inside the existing maintenance transaction. The parent ACL supplies the policy; there is no new setting or deployment-specific role name. Writes, column-only grants, owner entries, and grant options are excluded. A grant failure rolls back the table's partition changes. Existing children and manually created or attached partitions still need grants from database provisioning. Apply the [companion initializer fix](https://github.com/squareup/bb-utility/pull/2253) for existing partitions before relying on automatic grants for future ones. ### Related issue No duplicate found. The closest related PR is #8464, which enables catch-all advancement by default; this change independently preserves reader access during that maintenance. ### Testing No manual testing. Production rollout and the Fivetran setup check remain pending. Generated with Codex Signed-off-by: coder2 <4acc49cb93576448ba82e631688c51a40bf03d6a889d78fddf57917d198de10f@buzz.block.builderlab.xyz> Co-authored-by: coder2 <4acc49cb93576448ba82e631688c51a40bf03d6a889d78fddf57917d198de10f@buzz.block.builderlab.xyz>	2026-11-02 15:27:05 -05:00
Logan Johnson	5f8d0aab98	fix(relay): refuse a second listed open channel with the same name (#8782) ... ## Summary Two open channels in one community could have the same name, so the channel list showed two `#square-ios` rows and people could not tell them apart. Now the relay refuses to create, rename or open up a channel when that would give it the same name as another channel in the channel list, ignoring case. The client gets `invalid: an open channel named "…" already exists`. The rule follows Slack: an archived channel keeps its name until it is deleted. Private channels, DMs and unlisted channels may still repeat names. So a refusal never reveals that a private channel exists. The PR also adds a script that renames the duplicates that exist today. It sends each rename through the relay, so relays keep serving while it runs. The stacked PR https://github.com/block/buzz/pull/8787 then adds a database index that enforces the rule. This PR is the release that must be fully deployed before that index. ## Details **Which channels the rule covers.** A *listed* channel: open, not unlisted, a stream or forum, not deleted. Archived channels are included. The check runs on create, rename, private → open and unlisted → listed. **Relay check** (`crates/buzz-db/src/store/channel.rs`, `crates/buzz-relay/src/handlers/ingest.rs`): - The check runs after the write, in the same transaction, under a per-name lock. So two concurrent writers cannot both commit the same name. - A refused create or edit (kind:9007 or kind:9002) gets `OK false "invalid: an open channel named … already exists"`. Nothing from it is stored. - The relay gives the same answer when the database index from https://github.com/block/buzz/pull/8787 refuses the write. That covers the rollout, and a rollback from that release. - A channel create with no `h` tag (the new channel's id) is now refused before it is stored. Before, such a create ran after the event was stored, so a refusal (for example, a taken name) still answered "accepted". All first-party clients send `h`. **Operator rename** (`crates/buzz-relay/src/handlers/side_effects.rs`). The rename script needs to rename channels it does not own, archived ones included. So a relay operator may now send a channel edit that changes only the name of a listed channel. Everything else still needs the channel's owner or admin. - "Operator" means `RELAY_OPERATOR_PUBKEYS`, the owner fallback, or an `operator` row in `relay_operators`. A moderator is not an operator. - The rename also applies to an archived listed channel, whatever the operator's role in that channel, and the channel stays archived. Other edits of an archived channel are still refused. - Private channels, unlisted channels and DMs still need the channel's owner or admin. The relay decides "listed" under the channel row lock, so a channel made private during the rename is not renamed. - The rename takes the normal edit path: the name check, the stored event, the channel metadata event and fan-out. **Rename script** (`scripts/maintenance/rename_duplicate_listed_channel_names.sh`, with the read-only plan `plan_duplicate_listed_channel_renames.sql`): - In each group of listed channels that share a name, the oldest unarchived channel keeps the name. Each other channel becomes `<name>-<first 8 chars of id>`, or `<name>-<id>` when that is also taken. When all the channels in a group are archived, the oldest keeps the name. - **Starter channels.** Released desktop and mobile apps create `#general` and `#welcome-everyone` when a community has no open, unarchived stream with that name. Under the new rule, the relay refuses that create while an archived channel has the name, and the apps' setup fails. So the script also renames `general` and `welcome-everyone` when every listed channel with that name is archived. That frees the name for the apps. - The apps create each starter at a fixed id, derived from the relay's address. If a channel already has that id, the relay refuses the create there whatever its name, so a rename cannot help. For that name the script renames nothing, and an archived starter at its id keeps the name. The script computes the ids from `BUZZ_RELAY_URL` (or `BUZZ_STARTER_ORIGIN` when the apps use a different address) and prints them, so the operator can check them against a client. A relay-side fix for those clients is planned separately. - The operator key goes in `BUZZ_PRIVATE_KEY`. It must be a relay operator and a member of the community. The script stops at the first auth refusal and says so. - The script stops before the plan for an unknown host, an archived community and a community being deleted, and says what to do. - A failed rename (the new name is now taken, or the relay can't be reached) is reported, and the script goes on. At the end it reads the plan again and exits 1 if any rename remains. `--dry-run` prints the plan. - The script replaces the earlier direct-SQL repair and `buzz-admin reconcile-channels --metadata`, which needed every relay pod stopped. **Duplicates that exist today** (read on 2026-10-30): `ml-platform-public` (`d41a31f8…`, `7be55f5a…`) and `square-ios` (`79f8e9d4…` archived, `fe246d3e…` live). ### Rollout The full procedure, with exact SQL and recovery steps, is in `docs/channel-name-uniqueness-deployment.md` in https://github.com/block/buzz/pull/8787. 1. **Deploy this PR.** Wait until no older relay pod serves; check the image of every pod. An older relay can still create a duplicate, and it refuses the operator rename. 2. **Rename the existing duplicates in each community**, with relays serving. The key must be an `operator`, not a moderator, and a member of the community (`RELAY_URL=wss://<host> buzz-admin add-member --pubkey <npub>`). ```sh DATABASE_URL=postgres://... BUZZ_PRIVATE_KEY=<operator key> \ scripts/maintenance/rename_duplicate_listed_channel_names.sh --dry-run <community host> # review the plan, then run it again without --dry-run ``` For the duplicates known today, the newer `ml-platform-public` and the archived `square-ios` (`79f8e9d4…`) get the `-<id8>` suffix. 3. **Confirm that no duplicate remains** in any community (query in the doc). 4. **Build the index, then deploy https://github.com/block/buzz/pull/8787.** Rollback from https://github.com/block/buzz/pull/8787 goes to this release only, keeping the index, with auto-migration off. ### Related issue None found. ### Testing - **Live local relay.** The script ran against a local relay built at 9ee09a619, with relay membership required and six seeded channels (`ops`, `OPS`, an archived `Ops`, a private `ops`, `dev`, `DEV`): - `--dry-run` listed the three renames. - With the operator key not yet a member, the script stopped at the first refusal, named the membership precondition and renamed nothing. - After `buzz-admin add-member`, it renamed all three and exited 0. The archived channel stayed archived, the private `ops` kept its name, and a second run renamed nothing. - With the community archived, the script stopped with the unarchive message. - Rerun at 0df245e3c as part of the full rollout in https://github.com/block/buzz/pull/8787: the relay refused a new duplicate over the wire, and the script renamed the seeded duplicates with the relay serving. - **Relay tests** (`group_state_postgres_tests.rs`): a taken name is refused as `invalid:` and nothing is stored; a create with no `h` tag is refused before it is stored; an operator can rename only a listed channel, only the name, archived or not, and a moderator cannot; an operator rename is refused if the channel stops being listed before it is saved. - **Store tests** (`channel_name_postgres_tests.rs`): create, rename, private → open and unlisted → listed are refused for a taken name; private and unlisted names may repeat; an archived channel keeps its name and a deleted one frees it; of two concurrent creates with one name, the second is refused once the first commits. - **Plan and script tests** (`listed_channel_name_postgres_tests.rs`) run the shipped SQL and shell script: which channel keeps the name, the starter cases, names at the 255-character limit, names with control characters or a leading `-`, and the auth and community stops. The starter-id computation matches the mobile app's test vectors. - The new guards were each removed once to confirm that a test fails. - The full local Postgres test runs (buzz-db, buzz-relay, buzz-admin) pass, except 6 tests that also fail locally on main and pass in CI. ### Database review checklist - [x] Phase: writer release (deploy), plus a maintenance script. The stacked PR adds the contract index; this release is its required stop. - [x] No trigger, function, procedure, rule or foreign key. - [x] No migration. The script writes nothing to the database itself. It reads a plan, and the relay applies each rename as a normal channel edit. `docs/database.md` Rule 3 asks for maintenance scripts in their own PR. The script is here because it depends on the operator rename in this PR. - [x] Contract: not in this PR. - [x] Rule 4 locations: no schema change. - [x] Deployed relay: the schema is unchanged, so the deployed relay keeps working. An older relay refuses the operator rename, so run the script only after this release is fully deployed. This release also works beside the contract index (tests above). - [x] Timeouts: no index build or bulk write here. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 13:23:40 -05:00
Matt Toohey	322fdbf086	test(relay): hold the replica's second aux page and move media's DB tests (#8740) ... ## Summary Two buzz-relay test fixes from the 23–30 Oct flake audit. Both are test-only. This is one of six PRs split from the reviewed `flakey-tests` branch (siblings below). ### `thread_window_bridge_restarts_both_aux_hops_after_replica_failure` (3 runs) - **Runs:** 40799617319, 40924179886 and 40930100994 (#8697, docs only), all on 10-29 in PostgreSQL Tests, attempt 1 (jobs 123208043033, 123632125866, 123651641392). None of the branches touches thread_window, and each run passed on a re-run. Signature: `failure_postgres_tests.rs:46`, "barrier must precede closure completion", left: 2, right: 1. - **Cause:** `pause_after_aux_page` polls the request until a `thread_window_aux` span closes, then expects exactly one. The fixture's 1,001 reactions make two replica pages at `AUX_LIMIT` 1000, and each page is one round trip on the transaction already held. A loaded runner can deschedule the test thread after page 2's query is written. Postgres then answers first, and the same poll reads page 2 too. - **Change:** `test_support` gains a `PausingProxy`, a trimmed copy of buzz-db's private one with only the hold-everything switch. The replica reader pool connects through it. `AuxPages` pauses it when the first aux span closes. That runs inside the request's poll, after page 1's response is consumed and before page 2's query is written, so page 2 stays pending however fast Postgres answers. The test resumes the proxy after it terminates the reader's backend. The held page then sees the termination, and the closure restarts on the writer, as before. The other three `pause_after_aux_page` callers read from the writer and pass `None`. ### `api::media` tests that need Postgres (fail every time Flake Check selects them) - **Run:** 40954304982 (#8658), 10-29, Flake check (Rust), job 123730434146. All six `test_state()` tests panic at the community-seed `expect` after the 30 s sqlx acquire timeout. - **Cause:** when a hunk lands outside a test body, Flake Check selects every test in the file. It skips only `postgres_tests::`, `postgres_*` binaries and `external_infra`, and its job has no database. These six tests seed `relay.example` through Postgres but sat in plain `mod tests` without `#[ignore]`. So any PR that changes non-test code in `media.rs` fails that lane, and at least 8 open PRs touch the file. No other lane ran these tests either. - **Change:** move `test_state`, the `media_get_*` helpers, `media_request` and the six tests into `tests::postgres_tests`. Mark each test `#[ignore = "requires Postgres"]`, and point `test_state` at `test_support::database_url()`, as the module's other states do. The PostgreSQL lane now runs them, and Flake Check skips them. The change also drops imports `mod tests` no longer uses, and updates the Justfile comment that gave `api::media` as the reason the infra-free lane is scoped to `api::admin`. - When `media.rs` is staged, the pre-commit secret scan reads the whole file and flags the module's existing public ES256 test key (`TEST_EC_PKCS8_PEM`). The key is now written on one line with a sadscan marker, as `invites.rs` already does. Its bytes are unchanged. ### Related issue None found. **Conflicts with #8451 and #8444**, which both already conflict with `main`. The one-line `TEST_EC_PKCS8_PEM` rewrite is the only new conflict hunk with each. The test move merges cleanly with both. ### Testing Verified locally: - Each of the four `pause_after_aux_page` tests passes when run alone against local Postgres through `scripts/postgres-test-run.sh`. - A forced check, not committed: after the barrier, the test gave page 2 another 500 ms over ten re-polls and asserted the page count stayed at 1. It passed with the pause and failed with `None`, with the CI signature (left: 2, right: 1). A true same-poll burst needs the OS to preempt between sqlx's write and read, so it was not forced directly. - `check-postgres-test-discovery.py` accepts the touched files and all 621 files under `crates/`, and `scripts/test-postgres-test-discovery.sh` passes. Each of the six media tests passes when run alone against local Postgres. - `cargo fmt --check` and `cargo clippy -p buzz-relay --all-targets -- -D warnings` are clean under 1.98.1. Only hosted CI can show the full lanes, which were not run locally. The six media tests have never run in CI, so this PR's PostgreSQL lane is their first hosted run. Pushed with `git push --no-verify`, because the pre-push suites fail when the laptop is under load. ### Readiness An agent review of the combined `flakey-tests` branch (b432453af) found no blockers. `buzz-review-completed` is not added, because the human test step (AGENTS.md "Before opening a PR", step 3) hasn't happened yet. ### Siblings - #8739 ci: Hermit fetch deadline and sccache fallback - #8741 test(mobile): RunnerTests title fade and alert wait - #8742 fix(desktop): deep-linked message hold (product change) - #8743 test(desktop): four specs racing scroll settles and focus - #8744 test(desktop): five settle classes and remote-owned-mentions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 08:28:32 -07:00
Matt Toohey	cde6c1bf5b	feat(relay): auto-join new community members to default channels (#8594) ... ## Summary Adds **default channels**: relay owners and admins can flag channels that every new community member is automatically joined into. **Setting (contract)** - New `channels.auto_join` column (migration `0080`, constant default `false`, so no table rewrite). - kind:9002 accepts `["auto_join", "true"|"false"]`. Authorization comes from the actor's **relay** roster role (owner/admin), not a channel role, so the actor doesn't need to be a channel member. API tokens need `AdminChannels`. - Only open, unarchived `stream` channels qualify. This is checked under the channel row lock, so one event can open and flag a channel in one step, and a concurrent archive can't slip past the check. - kind:39000 carries a bare `auto_join` tag only while the setting is effective (on, open and unarchived). Every change posts one `auto_join_changed` kind:40099 notice. - SDK `build_update_channel(.., auto_join)`, `buzz channels update --auto-join <true|false>`, a NOSTR.md "Default channels" section, and a NIP-CL tag-table row. **Behaviour** - kind:9021's body is pulled out into `join_channel_as_self`, so auto-join uses the same path as a self-join: the channel's `join_role` applies, and it emits `member_joined` 40099, 39000/39002 and 44100. - `auto_join_default_channels` runs inline right after a member is added at three sites: the v2 invite claim, the v1 invite claim, and kind:9030 add. It does not run on login or reconnect, so a later kind:9022 leave stands. - Best effort: each failure is logged and counted (`buzz_auto_join_failures_total`) but never retried, because the roster membership has already committed. This deliberately departs from Review-Proven Rule 1 and is documented in NOSTR.md. `buzz-admin add-member` writes the roster row only. **Communities without a membership roster** (`BUZZ_REQUIRE_RELAY_MEMBERSHIP=false`) - These have no invite claim or kind:9030 to hook, so the three sites above join nobody there. Instead, after the AUTH `OK`, the relay joins a person to every effective default channel they have **never had a membership row in** (`auto_join_never_joined_channels`, spawned so admission never waits on it). - The channel's own membership history is the once-only record, so this adds no new table, column or migration. A kind:9022 leave or a kind:9001 removal keeps the row with `removed_at`, so the person stays out. A new device or another relay pod finds the row and does nothing. `put_first_member` checks the row again under the `put_member` locks, so two admissions racing each other join once. - A channel flagged later is joined on each person's next admission. A failed membership write leaves no row, so the next admission retries it. - Each channel's join holds the session's NIP-FI effect permit, so once the socket closes or its admission expires, no further channels are joined. - Agents are skipped: any socket with an admitted owner (a NIP-OA tag, a NIP-FI agent token, or a stored `agent_owner_pubkey`). - The kind:9021 side effects moved into `announce_self_join`, so both join paths emit the same `member_joined`, 39000/39002 and 44100. **Review follow-ups** - `get_accessible_channels` now selects `auto_join`. Without it, every call failed with `ColumnNotFound`. - The final authorization branch of the 9002 validator now refuses by default. `list_auto_join_channels` filters to `stream` channels itself. ## Testing - Postgres-lane journeys: - `edit_metadata_postgres_tests`: who may set `auto_join`, the 40099 notice, the effective-state tag, and the refusals. - `claim_auto_joins_default_channels_once`: v2 and v1 claims, and leave then re-claim. - `adding_a_member_auto_joins_default_channels`: kind:9030 add. - Removing the auto-join call at any one of the three sites fails the matching test (checked by mutation). - `root_auth_joins_people_to_default_channels_without_a_roster`: through `handle_auth` on a roster-less community. A person is joined, a NIP-OA agent is not, and a leave holds when the person is admitted again. Mutations checked: removing the spawn, and dropping the agent check. Each check waits for that admission's own auto-join outcome (a skip reports through the same test hook as a finished task) and fails if none arrives. A third mutation drops the agent check and delays the task 3s; it still fails. - `admission_auto_join_stops_when_the_session_ends`: closing the socket before the first or second channel stops the remaining joins; a later admission joins what was skipped. Mutation checked: dropping the effect permit. - `first_member_put_joins_once_and_never_undoes_a_leave` (buzz-db): the person is joined once with the channel's `join_role` (`guest` on an announce channel), a second put does nothing, after a leave the channel is neither listed nor joined, and two racing puts write exactly once. Mutations checked: removing the row check, removing the channel lock, and removing the `NOT EXISTS`. - e2e `test_invite_claim_auto_joins_default_channel` against a local relay. - Unit coverage in the SDK and CLI builders and `group_metadata_tags`. ## What got simpler The roster-less path adds no schema and no new concept: it reuses the channel's membership history as its record. It also shares the kind:9021 join code. It replaces the per-device client workarounds, [buzz-app#2029](https://github.com/block/buzz-app/pull/2029) and [buzz#8795](https://github.com/block/buzz/pull/8795), which are now closed. ## Related - Client half (buzz-app): https://github.com/block/buzz-app/pull/1624 ## Demo Recorded against a local relay built from this branch with throwaway identities (Alice = owner, Bob = new member); no real accounts or hosted relays involved. Relay at 141aa01d4, client at buzz-app bb0d97d86 (browser dev build). **Owner turns on "Starter channel for new members" on two channels.** Alice creates `#general` and `#welcome`, then flips the switch on each (~0:25 and ~0:41). The switch greys out while the kind 9002 is in flight and only turns on once the republished 39000 carries the `auto_join` tag. https://github.com/user-attachments/assets/e590d343-11d3-4385-af43-2cfaf19f8468 **A brand-new member joins and lands in a starter channel.** Bob claims Alice's invite with empty storage; from ~0:18 he is reading `#welcome` with both `#general` and `#welcome` in the sidebar, and at 0:27 opens `#general` as a member with no Join prompt. The relay logged one `auto-joined default channel` line per flagged channel. https://github.com/user-attachments/assets/037a0aaa-b25d-474d-a316-8d25e5e54f89 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Matt Toohey <contact@matttoohey.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com> Signed-off-by: tornquist <tornquist@squareup.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-11-01 14:17:41 -05:00
fsola-sq	7d00c70805	feat(relay): publish NIP-FI verified profiles (BUZZ-433) (#8774) ... ## Summary Publish historical verified names from valid human NIP-FI assertions as relay-signed kind 30382 events. Keep profile publication optional and separate from access decisions. Disconnects and access changes initiate asynchronous cleanup without retracting profiles verified after the original trigger time. Publish `display_name` as the chosen short display handle and `username` as the account handle. Carry optional `full_name` in the signed assertion into a matching kind 30382 tag. Adding, changing, or removing the full name updates the profile immediately; retractions remove every name field. Older profiles without `full_name` remain valid. Adds opt-in issuer labels, discovery, and schema migration 0079. Deploy and drain all access writers before enabling publication; use a stable relay key and a shared publication-policy revision. Update clients before enabling assertions with `full_name`. ### Related issue Implements the Relay portion of [the verified-profile specification](https://github.com/block/buzz/pull/8745) at `500829a9` (BUZZ-433). This is the closest existing PR found. Companion client: https://github.com/block/buzz-app/pull/2005. Companion issuer: [KGoose/bb-identity #144773](https://github.com/squareup/cash-server/pull/144773), which includes `nostr_public_profile` in valid human assertions. ### Testing Passed the lifecycle test against a disposable local PostgreSQL database, including full-name addition, change, and removal: `cargo test -p buzz-db store::verified_profiles::postgres_tests::profile_lifecycle_retains_assertion_ordering --lib -- --ignored --exact` with `BUZZ_TEST_DATABASE_URL` set to that database. Independent agent review found no remaining blockers. No live Relay test or human verification yet. This remains a draft and does not attest the repository's human-testing checklist. For end-to-end testing, enable publication for a test issuer, admit a member with a signed profile claim, and query kind 30382. Verify short display handles, optional full names, changed names, missing-claim retraction, disconnect cleanup, and a later admission surviving delayed cleanup. Check the companion app against the same relay. Generated with Codex --------- Signed-off-by: Franco Sola <fsola@squareup.com> Signed-off-by: Bradley Axen <baxen@squareup.com> Co-authored-by: Bradley Axen <baxen@squareup.com> Co-authored-by: mbot <3754f8729004d95654c46dbab3129e4ab9ef05cc2534e2a3fbfc155983bd637b@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Codex <noreply@openai.com>	2026-10-31 15:42:34 +00:00
Matt Toohey	b7b545f090	test(db): drop scratch databases and roles even when a test fails (#8747) ... ## Summary `buzz-db` tests that create their own databases or roles dropped them as the last line of the test body. A failed assertion or a timeout skipped that line and left the database or role on the shared Postgres server. Seven files carried their own copy of `create_scratch_db_through` and its drop helper. - Adds `ScratchDb` and `ScratchRole` guards to `crates/buzz-db/src/test_support.rs`. Their `Drop` runs `DROP DATABASE IF EXISTS ... WITH (FORCE)` or `DROP ROLE IF EXISTS` on a fresh admin connection, on a separate thread with its own runtime. `ScratchSchema` already works this way, so the guards work under both runtime flavours and while a panicking test unwinds. `WITH (FORCE)` ends any sessions still open on the database, so pools need not be closed first. - A failed drop prints a warning instead of panicking, because a panic during unwinding would abort the test binary. - The seven per-file helpers and every end-of-test `DROP DATABASE` in `buzz-db` now use the guard. The replica-fence masked-activity test's least-privilege role uses `ScratchRole`. Admin pools and `close()` calls that only served cleanup are removed. Test behaviour is otherwise unchanged. - `crates/buzz-db/TESTING.md` now points new tests at the guards. - The second commit fixes a comment in `armed_pool_rejects_old_channel_inserts_through_public_api` that still described the old end-of-test drop. **Existing leftovers need dropping by hand.** This change stops new leftovers but does not remove the ones earlier failed runs already left in dev `buzz-postgres` containers. There were 24 on the author's machine, named `stall_r_*`, `buzz_admin_*`, `floor_probe_*`, `fence_*` and `cxl_*`. List them, check the list, then drop each one: ```sql SELECT datname FROM pg_database WHERE datname ~ '^(stall_r|buzz_admin|floor_probe|fence|cxl)_'; DROP DATABASE "<name>" WITH (FORCE); ``` The guards are private to `buzz-db`. `buzz-push-gateway`, `buzz-admin` and `buzz-relay` tests still create and drop scratch databases by hand, which could get a shared copy of the guards in a follow-up. ### Related issue None found. This continues the test-cleanup work in #8727. ### Testing - Ran the 70 affected `buzz-db` tests against the local `buzz-postgres` container: 68 passed. Neither failure comes from this change. `cluster_global_probe_commits_tokens_and_sessions_prove_coverage` hit the known `MaskedActivity { masked: 1 }` flake under load and passed when run alone. Because it panicked while holding a `ScratchDb`, it also showed a panicking test leaves nothing behind. `serving_lease_statement_timeout_cancels_slow_statement` failed with `VersionMismatch(56)`, because the shared dev database already has another branch's migration 56 applied. - After the run, the server's database and role lists matched the snapshot taken before it. - A temporary test panicked while holding an open transaction, an exclusive table lock and a live session as the scratch role. It left no database or role behind under either runtime flavour and printed no drop warnings. That test is not part of this PR. - `cargo fmt --check` and `cargo clippy -p buzz-db --all-targets -- -D warnings` pass. The tests ran before the commits were cherry-picked onto current main. Only clippy was re-run afterwards. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-31 10:35:59 +11:00
Matt Toohey	d14e95f90e	test: stop desktop, buzz-agent and mobile tests leaking temp files (#8748) ... ## Summary Six test files wrote into the shared temp dir on every normal run and never removed what they wrote. **Desktop and buzz-agent (first commit)** - `desktop/src/protectedFeatures/tauriCommand.test.mjs` wrote its fake Tauri CLI and three result files straight into `/tmp` under pid-suffixed names. They now live under one `mkdtempSync` root that an `after()` hook removes. - `crates/buzz-agent/tests/permission_boundary.rs` wrote a `buzz_perm_calllog_*` file for every test whose tool call reached `fake_mcp`, five per run. `call_log_path` now returns a `TempDir` guard with the log path inside it, so the log is deleted when the test ends, including when an assertion panics. - `cancel_kills_inflight_tool_via_mcp_notification` in `crates/buzz-agent/tests/regressions.rs` spawns `buzz-dev-mcp`, which the agent stops with SIGKILL, so the server's `buzz-dev-mcp-*` and `buzz-dev-mcp-session-*` TempDirs were never deleted. The test now passes the server a `TMPDIR` it owns and removes. **Mobile (second commit)** The three mobile test files left 12 `buzz-video-*` files and 4 `transcode_*` dirs per run. - `video_viewer_test.dart` and `video_controls_test.dart` pointed their fake `path_provider` at `Directory.systemTemp`. The viewer downloads each video there as `buzz-video-*`, and its async delete on close never completes under FakeAsync. Each test now gets its own temp dir for the fake, and an `addTearDown` removes it with `deleteSync`. In `video_controls_test.dart` that teardown is registered first, so it runs after the viewer is disposed. - Four `media_upload_test.dart` tests faked `transcodeVideoToMp4` by writing `out.mp4` into a new `systemTemp` `transcode_*` dir. The upload service deletes `out.mp4` but not its dir. The fakes now create that dir under the test's own temp root, which the test already deletes in `finally`. Production code is unchanged. **`buzz-dev-mcp`'s SIGKILL shutdown is left for a follow-up:** the agent still stops the server with SIGKILL, so outside tests the server's temp dirs are still left behind. Sending SIGTERM first is the real fix. ### Related issue None found. This continues the test-cleanup work in #8727. ### Testing Each run below was bracketed by listings of `/tmp` and the per-user temp dir (`/var/folders/.../T`), so new leftovers would show up. - `tauriCommand.test.mjs`: 3/3 pass, with no new `buzz-tauri-*` or `buzz-fake-tauri-*` entries. - `cargo test -p buzz-agent --test permission_boundary`: 9/9 pass, with no new `buzz_perm_calllog_*` files. The tests still check that the call log is written, so the log really is going into the guarded dir. - `cancel_kills_inflight_tool_via_mcp_notification` passes with `buzz-dev-mcp` built, so it ran rather than skipped, and left no new `buzz-dev-mcp-*` dirs. - Mobile: `video_viewer_test` (7 tests), `video_controls_test` (20) and `media_upload_test` (38) pass, each run on its own. With `TMPDIR` pointed at a scratch dir, the old versions left 12 `buzz-video-*` files and 4 `transcode_*` dirs, and the new versions left it empty. - After cherry-picking onto current main, these pass: - `cargo clippy -p buzz-agent --all-targets -- -D warnings` - biome and `node --check` on `tauriCommand.test.mjs` - `dart format` and `flutter analyze` on the three mobile files 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-31 10:35:26 +11:00
Luke Tornquist	4cf2a3bd66	feat(loadgen): add a hot-thread stressor and teardown (#8746) ... ## Summary `buzz-loadgen` spreads load evenly: every channel gets the same history and the same short thread chains. That never puts heavy load on a single thread. A busy conversation, such as agents talking to each other, can give one root a huge number of replies, and thread-summary work grows with it (see #8635). This PR adds an optional `[hot_thread]` scenario section so a load test can measure that case. - **Seed:** `seed` posts a hot root in community 0, channel 0, then `seed_replies` replies directly under it. The root comes from a dedicated identity that posts nothing else, so `run` finds it however many other roots the channel has. The replies go out concurrently, like the rest of the seed, so the run starts with the thread already large. - **Run:** `authors` hot-thread clients are scaled by each ramp step's `users`, the same way humans and agents are. Each one replies to the hot root over HTTP `/events` and reads the thread's newest 100 replies over HTTP `/query`, at `replies_per_hour` and `fetches_per_hour`. - **Metrics:** two new report ops, `hot_reply` and `thread_fetch`. The replies are timed, so humans subscribed to that channel also record delivery latency for them. - **No root, failed run:** an author that can't find the seeded root counts a `thread_fetch` error on every attempt, so a run without the hot thread fails instead of quietly measuring nothing. - **Off by default:** the section is disabled while `seed_replies` and `authors` are both 0, so existing scenarios are unchanged. `perf/scenarios/local-smoke.toml` gets a small hot thread, and `perf/LOADGEN.md` documents the knobs. ### Teardown The PR also adds `buzz-loadgen teardown <scenario>`, so a staging fixture can be removed after the load test. - For each community from `target.community_host_template`, it first calls `POST /operator/communities/archive` as the fixture owner, using the operator key `seed` uses. It then calls `POST /operator/communities/delete`. - The deletion `request_id` is derived from `run_id` and the host, so a re-run replays the same requests. That retries failures and reports progress without a `deletion_request_conflict`. - The relay's deletion drain does the actual deletion asynchronously. - If the fixture owner does not own a host, the host is reported `absent` and left untouched. - A scenario without a host template is refused. - Deleted hosts stay as tombstones, so `LOADGEN.md` says each fixture needs a fresh `run_id`. The signing has to happen as each derived owner, which is why this lives in the generator rather than in deployment config. ## What got simpler Nothing got simpler; this adds a scale vector. To keep it small, the PR reuses what's already there: - the seeder's post plan, plus `find_roots` with a content prefix - the existing HTTP helper, whose auth tag is now optional - humans' delivery measurement A separate chain-depth knob and dedicated viewer clients were left out. `shape.thread_depth` already seeds chains up to the relay's depth cap, and humans in the hot channel already receive the replies. ## Testing - `loadgen::seed::tests::the_hot_thread_seeds_direct_replies_under_its_own_root` checks that the hot root and its replies are seeded, that every reply is direct (depth 1), and that a rejected root fails the seed. - `loadgen::run::tests::hot_thread_authors_reply_to_the_hot_root_and_fail_without_it` checks several things: - authors post replies tagged to the hot root; - root lookups query only the hot-root identity, and a newer reply-shaped decoy is never taken as the root; - scheduled reads all target the hot root's channel and id; - the run fails when the root isn't seeded. - `loadgen::run::tests::hot_thread_authors_start_nothing_once_the_run_stops` runs 8 hot authors directly. It holds every in-flight `/events` reply at a gate until the stop is signalled, so each author's next reply is already overdue when the gate opens. It then checks that exactly 8 replies were posted, none started after the stop. Removing the post-`select!` stop check fails the test, and the gate is released before any assertion, so a failure can't hang the job. - The run now signals the stop before the last step's snapshot instead of after building that step's report. Under a loaded parallel test run, work started in that gap leaked into `shutdown`. - `loadgen::teardown::tests::teardown_deletes_only_the_fixture_owners_communities_and_replays_on_rerun` runs teardown twice against a fake operator that checks the owner, requires archive before delete, and conflicts on a different request id. It checks that: - the owned host is archived and deletion-requested; - another owner's host and an unseeded host are reported absent and left untouched; - a 503 on archive is reported as a failure; - the second pass converges to the same result; - a deletion already past membership purge, where archive returns 404, is replayed and reported at its stage under `requested`; - the same run under another host template gets its own request ids: another owner's host is reported absent rather than conflicting with a request id already used. If the request id is replaced with a fresh one per run, the test fails. - `loadgen::teardown::tests::teardown_refuses_a_scenario_without_templated_communities` - All of these run in the existing `loadgen::` unit lane. - Teardown was run end to end at `28192004` against a local relay. The relay had operator keys configured, local S3 behind it, and `buzz-admin deletions drain`. The run covered: - seeding three templated communities; - teardown, with every fixture archived and a deletion request at its derived id; - a re-run that converged; - the drain, which tombstoned the fixtures; - teardown again after the purge, which reported each fixture at `retention_pending` and exited 0; - a re-seed of the torn-down run, which failed at provisioning. A community outside the fixture stayed active, with no deletion request. - At `28192004`, `cargo fmt --check`, `cargo clippy -p buzz-test-client --all-targets -D warnings` and six parallel runs of `cargo test -p buzz-test-client` pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-30 13:52:30 -04:00
Logan Johnson	a56966b777	feat(relay): NIP-AR sync subscriptions (#8522) ... 🤖 ## Summary Clients that show a list of artifacts (tasks, config entries and similar records) had no way to get the complete list and then stay current. They could only page through query results, and a page is not proof that the list is complete. This PR adds **sync**. A client sends one subscription and gets every matching artifact as one consistent snapshot, then a signal that the set is complete. After that it gets the current version of each artifact that changes. When the relay cannot guarantee an exact copy, it closes the subscription with a `resync` reason, so the client never misses a change without knowing it. It also adds a **head lookup**. It returns the current version of named artifacts, deleted ones included, so that "create or restore" flows can retry safely. This implements the Sync section of NIP-AR (the artifact protocol spec), which is the first commit of this PR. ## Details - **Request.** A normal `REQ` whose only filter has `"artifact": "current"` and at least one channel in `#h`. This is the same current-state filter as HTTP `/query`; on a `REQ` it becomes a sync, because a `REQ` already means "stored set, `EOSE`, then live". The client does not page: the relay streams the whole set until `EOSE`, so `limit` and `offset` are refused. Other artifact views (`history`, `head`) stay HTTP-only, as on main. Other clients and other `REQ` shapes do not change. - **Initial set.** The relay first registers for live changes and waits until Redis confirms that registration. Then it reads every matching current version in pages ordered by artifact ID, all inside one read-only `REPEATABLE READ` transaction. So all pages come from the same database snapshot. `EOSE` comes only after a complete read. A failed read, an unreadable channel, or a read that takes too long ends with `CLOSED` and no `EOSE`. - **Live changes.** Fan-out never sends the published event to a sync. It only records which artifact changed. The sync then reads that artifact's current version and sends it, one artifact at a time. So a client never gets an older version after a newer one. Changes that arrive during the initial read are held and sent after `EOSE`. Changes are signalled per channel, so a sync skips an artifact whose `type` or `d` does not match its filter: those never change, so it was never in the set. A board that syncs three record types in the same channels no longer gets each change three times. - **When the relay sends `resync`.** The Redis link drops, reconnects, or skips a message it cannot decode. Local fan-out lags. The connection's send queue is full. A moderator redacts an event in the channel. The reader's channel access changes (remove, leave, metadata edit, channel delete). An access change ends the whole sync, also when the sync names other channels. A sync cannot drop one channel and still claim an exact set. - **Publish retry.** When a Redis publish fails for an event kind that syncs use, the relay retries it in the background until it succeeds. - **Relay info (NIP-11).** The relay now lists `sync` (`max_events: 4096`), adds the `head` mode, and sets `websocket_query_extensions: true`. - **No migration.** Sync reads the existing `artifact_heads` table. - **Next to incremental reads (#8523).** #8523 added a separate HTTP protocol, `artifacts.incremental` / `"artifact":"incremental"`. It keeps a saved store up to date after a disconnect, but it does not prove a complete set. The two protocols use different NIP-11 keys and filter values, so they do not collide. They share one copy of the current-state access rule: `artifact_query::predicates`, which #8523 reads through, now applies the view join and calls `push_scope`, and the sync snapshot and head lookup call `push_scope` too. - **A failed access lookup in fan-out resyncs.** The access gate fails closed when a visibility or membership lookup fails. For a sync, that would drop a change with no signal, so the gate also reports those recipients and each one gets `resync: access check failed`. ### Known cost To find out which redactions remove an artifact that a sync covers, the relay would need an extra database read. So every redaction in a synced channel resyncs that channel's syncs, and so does every channel metadata edit. Each resync reads the set again, up to 4,096 artifacts. This is correct, but it adds load to channels where people often delete chat messages. We can narrow it later if it becomes a problem. ### Related issue None found. The design was reviewed as NIP-AR Sync r28. ### Testing - New Postgres tests: keyset pages see one snapshot, and an artifact created after the snapshot opens is in no page; the head view includes deletions; a private or unknown channel is refused. - New relay end-to-end tests on real Postgres and Redis, using the real `REQ` handler, ingest and fan-out: initial set in order, then `EOSE`, then current versions for an update and a delete; a chat message causes no frame; `CLOSE` stops the sync. Other tests: removing the reader from one channel resyncs the whole sync; an unreadable or unknown channel gets `CLOSED restricted` with no `EOSE`; a node-wide loss resyncs. - Unit tests for the parser, request routing, fan-out diversion, the held-change limit, and access-change signals that must arrive before the access filter. - Lifecycle tests from an adversarial review: every frame is checked against the subscription owner, also when a `CLOSE` or a replacing `REQ` arrives while the frame waits for queue room or while a live head read is in progress. A resync recorded while `EOSE` waits ends the sync with no `EOSE`. The held list refuses a change before it stores it (4096 entries, 4 MiB of held removals). - End to end: a relay container built from this branch, with the buzz-app #1523 browser build and the Work plugin. An edit from another client showed in the open board 4 to 25 ms after the write returned, with no refresh. - `cargo clippy --all-targets -D warnings` is clean for the four changed crates. All `buzz-relay` lib tests pass except `api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo`. That test returns 504 on this machine. It is a mesh networking test, and this PR does not touch that code. --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-30 13:31:30 -04:00
Daniel Rouleau	a57349b237	feat(relay): add incremental artifact reads (#8523) ... ## Summary Buzz clients need to recover task changes after a disconnect. This adds incremental reads of changed artifacts with database modification timestamps and a 120-second overlap. Each result contains the current revision or an explicit removal. A separate read of known IDs confirms accepted edits and restores access state. Pages use a timestamp and a stable ID to handle equal timestamps. Access grants include older records. Access policy changes require discovery and revalidation of saved IDs. The reader does not load a list of all readable channels. Ordinary artifact writes update their own rows. ### Incremental reads and WebSocket Sync This HTTP protocol maintains an eventually consistent local store. It uses `artifacts.incremental` and `"artifact":"incremental"`. It does not prove a complete set at one database snapshot. Absence never proves deletion or loss of access. [WebSocket Sync](https://github.com/block/buzz/pull/8522) serves a separate use case: a complete set at `EOSE`, followed by live updates. It owns `artifacts.sync`. Both protocols can coexist. The second PR to merge must reuse the shared access helper in `artifact_query.rs` when it resolves the overlapping edits. The gate is `BUZZ_ARTIFACT_INCREMENTAL_ENABLED`. Errors use the `incremental_` prefix. Unlisted open channels require membership or an explicit `#h` scope. Listing changes reset incremental progress. Incremental reads exclude TTL artifacts. Known-ID reads return `remove` for them. The `#h` scope limits candidates before pagination. Initial reads omit deleted heads. Later reads retain deletion removals. ### Why the new timestamps are needed `created_at` comes from the event signer. `received_at` records event arrival. Neither advances when access changes or an event is redacted. Sync needs database timestamps for those changes: | Column | Why sync needs it | | --- | --- | | `artifact_heads.modified_at` | Finds current records that change through an edit, deletion, or redaction. | | `artifact_revisions.modified_at` | Dates retained move records. This lets sync remove a task from its former channel when the reader cannot access its new channel. | | `channel_members.sync_modified_at` | Finds grants, rejoins, and removals. A grant must include older tasks whose own timestamps do not change. | | `communities.artifact_policy_modified_at` | Requests recovery after a relevant channel visibility, TTL, or deletion change. Clients revalidate saved IDs. Ordinary task writes do not update it. | The database sets these timestamps with `clock_timestamp()`. The 120-second overlap covers the accepted delay between timestamp assignment and commit visibility. `BUZZ_ARTIFACT_INCREMENTAL_ENABLED` defaults to off. Apply migrations `0076`–`0078` and upgrade all writers before enablement. The 120-second delay bound is an operating assumption. [Deployment steps](https://github.com/block/buzz/blob/drouleau/artifact-snapshots/docs/artifact-sync-deployment.md) cover index prebuilds and rollback. ### Related issue No duplicate found. Related work: [read permissions](https://github.com/block/buzz/pull/8338), [statement limits](https://github.com/block/buzz/pull/8461), and [atomic channel settings](https://github.com/block/buzz/pull/8516). ### Testing Signed HTTP E2E passes on `f3d6ac852e7ce4e2cc53bd891cef87832a868d72` after CI passes. It includes main through `ef4130da4`. A standalone local relay uses a fresh PostgreSQL 17 database, Redis, and signed NIP-98 HTTP requests. All 78 migrations apply through relay startup. All behavioral writes and reads use HTTP. The tests cover create, edit, delta reads, known-ID recovery, private moves, access grants, removals, rejoins, visibility resets, redaction, and deletion. Unlisted tests cover hidden broad discovery, named scope, known-ID reads, member access, and policy resets. They also cover bootstrap across two pages, overlap replay, and 101 move removals across pages. Decoded cursors contain no private revision IDs. With the incremental gate off, `artifacts.incremental` is absent and a signed incremental request returns `503 incremental_unavailable`. Unsigned requests are rejected. Signed tests also cover TTL exclusion, current-ID removals for TTL artifacts, moves into TTL channels, deleted initial heads, and scoped pages with exactly 100 and 3 upserts. A separate local scale test on `babcdb766` reads 10,050 tasks across 60,002 memberships. It uses seeded fixtures and a second relay state with the same signing key. The initial read takes 8.31 seconds over 101 pages with 4,923-byte events. A read with no changes takes 5.70 milliseconds. The candidate keeps the same database query. These are local measurements, not production limits. Cold pages repeat candidate scans and access checks. Writer load needs validation before fleet enablement. ```sh cargo test -p buzz-relay sync_downloads_ten_thousand_tasks_with_sixty_thousand_memberships -- --ignored --nocapture ``` The behavioral run uses `cargo build -p buzz-relay --bin buzz-relay`, then `cargo run -p buzz-relay --example artifact_incremental_final_oct30_f3`. The temporary harness drives the built relay and is removed after the run. The worktree is clean at the tested SHA. It does not exercise a native client. Git and S3 behavior are outside this validation. The [protocol](https://github.com/block/buzz/blob/drouleau/artifact-snapshots/docs/nips/NIP-AR.md#incremental-reads) defines progress, removals, limits, and recovery. Relay verification covers the signed HTTP path. Native client verification belongs to [the client PR](https://github.com/block/buzz-app/pull/1507). Generated with Codex --------- Signed-off-by: Daniel Rouleau <drouleau@squareup.com>	2026-10-30 08:30:46 -07:00
Luke Tornquist	cc04cdca1a	perf(relay): route author- and #p-scoped global subscriptions by selector (#8666)	2026-10-30 08:05:40 -04:00
Luke Tornquist	0e8fb49bd0	test(relay): fix main's build after the test-state helper rename (#8718)	2026-10-30 11:31:46 +11:00
Matt Toohey	ef4130da4b	fix(db): keep lease releases durable and deflake PostgreSQL lane tests (#8676) ... ## Summary When the user-deletion worker dropped a lease heartbeat, its pooled connection could be left inside an open transaction. The worker's final lease `release` then reported success but never committed, and a rerun saw the request as "leased by another worker" for up to 60 s. The heartbeat is now one autocommit `UPDATE`. The PR also fixes six PostgreSQL-lane test flakes from the 21–28 Oct 2026 CI audit. It moves one relay test off the 100 ms-acquire test helper before that test starts flaking the same way. This is one of seven PRs split out of the `flakey-tests` branch (siblings below). It touches only `buzz-db`, `buzz-deletion` and `buzz-relay`. Every relay file except `connection.rs` and `boot_lifecycle.rs` changes only for the helper rename in the last row. ### Related issue None found. Nothing on `main` (eda26b942) or in an open PR covers these. G, a single hermit `pgschema` 504 that needs a workflow change, is out of scope. ### Flaky tests fixed | Test | Failing runs | Cause | Change | |---|---|---|---| | `buzz-deletion` `user::worker::postgres_tests::dummy_executor_failure_retries_same_community_stage` (:47) and `next_step_advances_one_handler_at_a_time_through_completion` (:131) | 40439041234 (main), 40399635380#3, 40413732817#1 | **Product bug.** `run_stage_with_heartbeat` drops the renewal future when the stage handler wins `select!`. The heartbeat opened a transaction. sqlx 0.9 bumps `transaction_depth` only after `BEGIN` returns, so a `begin()` dropped mid-`BEGIN` queues no `ROLLBACK`, and the connection goes back to the pool inside an open transaction. The worker's autocommit `release` then ran inside it and never committed. | `EntityDeletionStore::heartbeat` is one autocommit `UPDATE`, guarded by `lock_owned`'s ownership/status/expiry predicate. A superseded lease is never renewed. A new regression test, `dropped_heartbeat_leaves_no_open_transaction_for_the_release`, drops the real heartbeat mid-`BEGIN` through a pausing proxy. | | `buzz-db` `session_statement_timeouts_bound_writer_and_reader_sessions` (runtime/tests.rs:3138, `PoolTimedOut`) | 40128463987, 40314950864#1, 40374286094 (main), 40394821009#1, 40413732817#1 | Each config's lazy reader pool dialled cold inside the 150 ms `READER_ACQUIRE_TIMEOUT`. `Db::new` never runs the boot ping. | Warm the reader the way the relay boots it (`read_pool_boot_ping_for_tests` + `await_idle_floor`). | | `buzz-relay` `startup_owns_the_dependency_sampler` (boot_lifecycle.rs) | Torn scrape: 39865872891 (main), 39624074244#2, 40385799750#1, 40413293801#1. Early exit: 39704588623 | metrics-exporter-prometheus 0.18.3 snapshots counters, then gauges. The scrape that first shows the completion gauge can therefore predate the `checks_total` counters. | Assert on a second scrape, taken after the gauge is seen. On an early relay exit, the helper now panics with the exit status, stdout and stderr. The early exit itself was not reproduced. | | `buzz-db` `sample_writer_waits_for_shared_floor_lock_before_sampling_time` (replica_fence.rs:1086) and `replica_floor_probe_waits_for_shared_writer_and_records_after_release` (runtime/tests.rs:3452) | 39601763602 (main), 40391445284#1, 39825846222, 40301667418 | Both scan cluster-wide `pg_stat_activity` but ran outside the serialized `postgres-cluster-global` group. Connection churn from other tests masked activity (`MaskedActivity { masked: 1 }`). | Rename both with a `cluster_global_` prefix. `nextest show-config test-groups --profile postgres-ci` lists both in the group. | | `buzz-db` `stalled_reader_sessions_release_their_slots_for_new_connections` (reader_pool_postgres_tests.rs:354) | 40399635380#2 | The floor warms one connection, so the second warm-up acquire dials cold inside 150 ms. | Retry only that warm-up acquire on `PoolTimedOut`, for up to 5 s (#8472's precedent). The stall under test keeps the production budget. | | `buzz-db` `readiness_check_cancellation_balances_waiter_and_inflight_connection` (runtime/tests.rs:391, `Elapsed`) | 39785370353 | The test aborted `pg_sleep(5)` after a fixed 25 ms. Recovery lands at server start + 5 s, but the bound is abort + 5 s, a ~22 ms margin locally. | Poll `pg_stat_activity` until the query is active, then abort. Use `pg_sleep(1)` within the unchanged 5 s bound. | | `buzz-relay` `upgrade_would_deny_line_is_written_at_auth_while_open` (connection.rs) | None yet (preventive, added by #8634) | It used the 100 ms-acquire `test_state_with_database_url` against the real test database, the same shape as the `PoolTimedOut` flakes above. | Use `test_state_with_database_url_and_pool_options` with sqlx's default timeout. The 100 ms helper is renamed `test_state_with_unreachable_database_url`; all its remaining callers use closed ports. | Local before/after counts are from the original `flakey-tests` commit (1f0451915), on macOS with OrbStack Postgres: - Regression test: 3/3 FAIL before, 3/3 PASS after. - Worker stress: 3 failures in 288 iterations before; 2,100 passing runs after. - Statement timeouts: 44/50 FAIL before, 50/50 PASS after. - Sampler: 2/200 FAIL before, 200/200 PASS after. - `MaskedActivity` under `pgbench -C`: 2/50 iterations FAIL before. After, the lane passed 3× (613/613) but was not re-run under pgbench. - Stalled reader: 10/30 FAIL before, 30/30 PASS after. - Forced late-`Execute` cancellation: 10/10 FAIL before, 10/10 PASS after. ### Verification This branch was pushed without local re-verification, by the user's choice: local test runs were overloading the machine and failing for that reason. **Hosted CI is the check.** It was pushed with `git push --no-verify`, because the pre-push hook runs clippy and unit tests. - Every file on this branch is byte-identical to its final version on `flakey-tests` (67185df6a), where the counts above were measured. - Not re-run for this PR: clippy, nextest and the Postgres lanes. - Checked for this PR: `git diff --check`, `rustfmt --check` on the changed files, and a conflict-free merge with all six siblings onto `main`. ### Readiness This PR has a product fix (the `buzz-db` lease heartbeat). Under AGENTS.md "Before opening a PR", it needs an agent review and a **human check against a local relay** before `buzz-review-completed` is added. Suggested check: 1. With local Postgres and a relay, submit and approve a user deletion with `buzz-admin user-deletions submit` / `approve`. 2. Run `buzz-admin user-deletions run` until it completes, then run it again at once. 3. Working looks like this: both runs complete, neither reports the request as leased by another worker, and `buzz-admin user-deletions inspect <id>` shows no leftover lease. The original race is rare and is not expected to reproduce by hand; this check confirms releases still commit. ### Merge order Independent of the other six. It merges cleanly onto `main`, and with every sibling, in any order. ### Sibling PRs Split from `flakey-tests` (one PR per area): #8677 (relay E2E helper + Windows shell), #8678 (mobile), #8679 (Tauri apt cache + Windows Rust cap), #8680 (desktop product races), #8681 (desktop E2E state gates), #8682 (desktop E2E composer + row actions). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-30 10:00:41 +11:00
Matt Toohey	1893c3ab5a	test: dedupe collect_until_eose by event id and uncap two shell tests (#8677) ... ## Summary Fixes two CI flakes from the 21–28 Oct 2026 audit, both in test code with no product change: - The relay E2E helper `collect_until_eose` now drops repeat copies of an event id. - Two Windows shell tests drop a 5 s cap that a cold Git Bash start can exceed. This is one of seven PRs split out of the `flakey-tests` branch (siblings below). ### Related issue None found for these two. The buzz-terminal `passwd_shell()` `getpwuid` race (`env_fence_tests::resolve_shell_falls_through_to_passwd_not_the_default`, run 38590095289) is left to open PR #7687, which already replaces `getpwuid`. #8636 and #7976 edit `buzz-test-client/src/lib.rs`, and #7607 and #7915 edit `buzz-dev-mcp/src/shell.rs`, but none of them changes `collect_until_eose` or these timeouts. ### Flaky tests fixed | Test | Failing runs | Cause | Change | |---|---|---|---| | Relay E2E `e2e_project::test_project_tombstone_deletes_coordinate_and_spares_members` (e2e_project.rs:338, "left: 2 right: 1") | 40408678412#1, job 121879630394 (PR #8620, schema-only) | The EVENT handler sends OK, then fans out in a spawned task. `handle_req` registers the live subscription before its history query, and `req.rs`'s `seen_ids` dedups history rows only. If fan-out runs after the REQ is registered, the event arrives once live and once from history, both before EOSE, and `collect_until_eose` kept every frame. | `collect_until_eose` keeps the first copy of each event id, as production clients do. Its doc comment says a test of the relay's own dedup must read frames with `recv_event`. A new unit test, `collect_until_eose_keeps_one_copy_of_each_event_id`, uses a fake relay that sends the duplicate. | | Windows `buzz-dev-mcp` `shell::tests::basic_echo` (shell.rs:1026, exit 124) | 39675698081#1, job 119485363399 (green on re-run) | 124 is the shell tool's own timeout exit code. `basic_echo` capped the shell at 5 s, and it is one of the first tests in the process to start Git Bash, which can take longer than 5 s on a cold runner. | `timeout_ms: None` (the production 120 s default) in `basic_echo` and in `workdir_is_honored`, which has the same exposure. `timeout_fires` still covers timeouts. | Pre-check: no e2e test relies on the relay's own history dedup. Every exact-count assertion uses a single filter. The NIP-33 "only newest remains" assertions would see a failed replacement as two different ids, and this dedup keeps both. Local evidence is from the original `flakey-tests` commit (fc4518d39): - The new unit test: FAIL before, PASS after. - 13 of the 14 e2e files that call the helper, plus `e2e_thread_roots`, against an isolated local relay. All pass, except `e2e_relay` `test_unarchive_emits_member_added_notification`. That test fails by itself 3/5 with the original helper too, and CI's Relay E2E lane doesn't select it. - The Windows failure can't be reproduced on macOS. The Windows Rust lane is the check. Follow-up (out of scope): make the relay deliver each event id at most once per subscription, keeping the live/history overlap. ### Verification This branch was pushed without local re-verification, by the user's choice: local test runs were overloading the machine and failing for that reason. **Hosted CI is the check.** It was pushed with `git push --no-verify`, because the pre-push hook runs clippy and unit tests. - Both files are byte-identical to their final versions on `flakey-tests` (67185df6a). - Checked for this PR: `git diff --check`, `rustfmt --check`, and a conflict-free merge with all six siblings onto `main`. ### Readiness Test-only change. AGENTS.md's readiness steps (agent review, then human confirmation) are still outstanding, so `buzz-review-completed` is not set. ### Merge order Independent of the other six. It merges cleanly onto `main`, and with every sibling, in any order. ### Sibling PRs Split from `flakey-tests` (one PR per area): #8676 (PostgreSQL lease + lane), #8678 (mobile), #8679 (Tauri apt cache + Windows Rust cap), #8680 (desktop product races), #8681 (desktop E2E state gates), #8682 (desktop E2E composer + row actions). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-30 10:00:21 +11:00
Luke Tornquist	efcdf00101	relay: open the search pool lazily so a dead reader can't block boot (#8662)	2026-10-29 22:48:55 +00:00
Tom Brow	4b21a10c4b	fix: don't notify workflow owners through their attribution tag (#8694) ... A workflow message names its owner in a marked p tag, ["p", owner, "", "owner"], for attribution. #p live, history and push matching, operator listeners, desktop, mobile and the iOS NSE treat that tag as attribution rather than a mention. A workflow that mentions its owner adds a plain p, so the owner is notified only when actually mentioned. NOSTR.md records the #p exception. Co-authored-by: studio <dc2fc481343672df31a689840c09299972e704dac592e0da6f18486646a7ee3e@buzz.block.builderlab.xyz> Signed-off-by: Tom Brow <tomb@squareup.com>	2026-10-29 15:21:56 -07:00
Luke Tornquist	1f16f621d8	perf(relay): cut connect-burst CPU in log filtering and open-channel lookup (#8670) ... ## Summary A cold relay taking a burst of new connections spent most of its CPU on two things that don't scale with useful work. This PR removes both. **Log filtering held a contended lock.** `EnvFilter` (tracing-subscriber 0.3.23) takes a shared `RwLock` read on every span enter and exit, even when no directive names a span. Every poll of an instrumented future enters its span, and the relay installs two per-layer `EnvFilter`s (logs and OTEL), so every worker contends on those locks. Plain `target=level` directive lists now filter through `tracing_subscriber::filter::Targets`, which keeps no per-span state. Directives that name spans or fields still use `EnvFilter`, which is the only filter that understands them. **Each reader re-read the community's open channels.** A reader's accessible channels are its member channels plus the community's listed open channels, and the open half is the same for every reader. The per-reader cache stored the union, so every miss fetched every open channel row again. The lookup is now two cached halves: - member channels, per reader, cached as before; - open channels, once per community. The shared half is used for a bounded time. Cross-pod invalidation is best-effort, so a pod can miss the drop that follows a channel becoming private, unlisted or deleted elsewhere. Each cached open set records when its database read began, and readers use it only for `OPEN_CHANNELS_MAX_AGE` (1 s) after that. After that the next reader re-reads it. **Accepted trade-off:** if an invalidation is lost, any reader on that pod can be granted a just-revoked open channel, history included, for at most 1 s. That includes a reader with no cache entry, which on `main` always read the database. This is a new exposure for those readers. With no bound, the shared set could have served that channel for its 10 s TTL. Readers with a cached entry already had a 10 s window on `main`, so for them the bound is tighter than before. An exact per-community version check would remove the window but needs a schema change, so it's tracked as a follow-up. Invalidations still drop the halves. A community-wide accessible-channel or channel-deleted invalidation drops both halves, and a membership invalidation drops only that reader's member half. The age bound counts from the start of the read, so open-set fills need no fence: a fill that raced a drop is still at most 1 s stale. Uncached callers of `get_accessible_channel_ids` get the same set as before, now built from the two halves. Open-set reads are counted by `buzz_open_channels_cache_hits_total` and `buzz_open_channels_cache_misses_total{reason="absent"|"expired"}`. **Metric meaning change:** `buzz_accessible_channels_cache_hits_total` and `buzz_accessible_channels_cache_misses_total` keep their names but now count only the per-reader member half. Dashboards that read them as whole-lookup hit rates should add the open-set counters above. Empty levels (`target=`) stay on `EnvFilter`. `Targets` reads an empty level as ERROR while `EnvFilter` reads it as TRACE, so only `target=level` directives with a non-empty level take the `Targets` path. ## Results Measured with the relay load generator (#8636) on one Linux machine. The relay ran as a release build on 8 pinned cores, with local Postgres, Redis and MinIO; the generator ran on separate cores. The fixture is one community with 2,000 open channels. Clients connect in a burst against a cold relay: 48k WebSocket clients and 4.8k HTTP agents. Each arm restored the same database snapshot and ran the same generator binary and scenario. Arms were rotated (base/before/after, then before/after/base, then after/base/before), and the table shows medians of three runs per arm. Both columns after base also carry #8666, so the comparison isolates this PR's design. - **before** is this PR's first revision, which had an unbounded shared open set. - **after** is this head, where the open set is bounded to 1 s. | | base | before | after | |---|---|---|---| | relay cores (mean) | 6.37 | 2.39 | 2.45 | | connect p99 | 3,870 ms | 40 ms | 42 ms | | subscribe EOSE p99 | 12,952 ms | 110 ms | 112 ms | | history EOSE p99 | 1,252 ms | 68 ms | 72 ms | | live delivery p99 | 26,421 ms | 54 ms | 55 ms | | HTTP query p99 | 16,426 ms | 44 ms | 43 ms | | DB tuples returned/s | 1.04M | 43k | 49k | | DB transactions/s | 4,397 | 4,065 | 4,062 | | relay RSS peak | 12.6 GB | 10.7 GB | 10.7 GB | | errors | 16, plus 241k throttled | 0 | 0 | The 1 s bound costs about 1,400 extra open-set reads per run, within run-to-run noise of the unbounded version. One run in each of the two right-hand arms had a slow tail (connect p99 0.8–1.3 s), which the medians absorb. An earlier revision bound the shared set to a per-reader digest instead. That was exact, but in the same kind of A/B its per-miss digest query returned about 28 times the tuples per second of the unbounded version. It gave back the database savings, so this revision uses the time bound. The open-channel gain scales with how many listed open channels a community has. The filter gain doesn't depend on the fixture. ## Rollout There's no flag, so the change takes effect on deploy. There's no schema or data change, so reverting the commit is the rollback. 1. Deploy to staging and compare relay CPU and connect and EOSE p99 under a connection burst. 2. Watch `buzz_open_channels_cache_misses_total{reason="expired"}`. Expect at most about one per second per active community per pod. 3. Confirm the deployed `RUST_LOG` and `BUZZ_OTEL_FILTER` are plain `target=level` lists, so they take the `Targets` path. Other values fall back to `EnvFilter` and keep today's cost. ## What got simpler - `invalidate_channel_deleted_local` now reuses the community-scoped accessible-channel drop instead of repeating it. - Tests seed accessible channels through one `seed_accessible_channels` helper instead of writing cache internals. ## Tests - `telemetry::tests::only_plain_target_level_directives_use_targets` checks which directive strings are plain. It covers span directives, field directives, whitespace, empty segments, empty targets (`=debug`) and bad levels. - The rewritten `env_filter_tests` assert on captured log output rather than `Display` strings or cached callsite interest, so they are stable when run in parallel. `span_directives_keep_env_filter_semantics` proves that `[conn]=debug` still enables events only inside the span. - `state::tests::postgres_tests::a_lost_open_set_invalidation_grants_a_revoked_channel_only_until_max_age` runs two relay states on one database, with no pub/sub between them. For each revocation (private, deleted, unlisted), it: 1. primes the reader pod; 2. revokes the channel in SQL and invalidates only the writer pod; 3. shows that a new reader is still granted the channel within the window; 4. ages the cached set past the limit; 5. asserts that a new reader and an earlier cached reader are both denied. Dropping the age check fails it. Never caching the open set fails its within-window control. - `env_filter_tests::directive_filters_log_what_env_filter_logs` emits events at every level for six targets, including prefix lookalikes. It asserts that the filter built for each of 17 directive lists logs exactly what `EnvFilter` logs for that list. Routing `buzz_relay=` to `Targets` fails it. - `state::tests::accessible_channels_join_member_and_open_halves` checks the merge. The two existing scoped-invalidation tests now also assert the open half. - These state tests, and the two existing ones, were not in the `just test-unit` allowlist. They are infra-free, so they are now selected by exact name, and `scripts/run-tests.sh` mirrors that. Identified debt, not fixed here: the per-reader member half keeps the pre-existing unfenced read-then-insert (a raced fill only reaches that reader, as before). `handlers::req::tests::expiry_during_stalled_{history,search}_read_releases_permit_and_claim` are not selected by any CI lane. buzz-review-completed 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 21:59:17 +00:00
Brad Seiler	1fe120165f	BUZZ-272: Deny workflow runs for banned owners (#8242) ... ## Summary A workflow whose saved owner is banned from the community can no longer create runs through any trigger: webhook, channel-event, cron, or signed command trigger. Every one of those doors already calls `WorkflowEngine::check_owner_authority`, which re-checks the owner's channel role. That gate now also reads the owner's authoritative restriction state. A ban denies the run, and a failed lookup fails closed. Unbanning restores admission. On the webhook door the denial is the same generic 404 as before, and no run is queued. On the event and cron paths, a ban denial is logged at WARN at most once per workflow every ten minutes, and at DEBUG in between. A failed lookup and every other denial still warn every time. The handler-local ban lookup in `api/bridge.rs` is removed because the shared gate covers it. This is a point-in-time check: a ban committed after the lookup can race with run creation. It does not cancel existing runs. Community timeouts do not block runs in this change. This PR also carries the shared mesh receive fix needed for independent validation (see Shared carry). ### Related BUZZ-272. - Invitations: [#8237](https://github.com/block/buzz/pull/8237). - Route coverage: [#8239](https://github.com/block/buzz/pull/8239). - Live-session revalidation: [#8241](https://github.com/block/buzz/pull/8241). - Staging runbook: [#8235](https://github.com/block/buzz/pull/8235). ### Testing Postgres lane: - `on_event_denies_run_while_owner_banned`: a banned owner's channel-event trigger creates no run, and after unban it creates one. - `cron_tick_skips_banned_owners_without_consuming_the_claim`: one tick skips a banned owner and an agent whose human owner is banned, while an unbanned workflow on the same tick still fires. After unban, a tick for the same scheduled instant fires both, so the denial did not consume their scheduled-fire claim. - `on_event_denies_run_when_restriction_lookup_fails`: the event path fails closed when the restriction read errors, and fires once the read succeeds. - `webhook_admission_enforces_owner_ban_tenant_scope_and_lookup_failure`: the HTTP door returns the same generic 404 for a wrong secret, a banned owner, a forbidden role and a failed restriction lookup, with no run created. It also covers another tenant's allowed control and unban. Unit lane: - `ban_denial_warns_once_per_interval_while_other_denials_always_warn`. Results: - At `81c367f48`, CI passed: Unit Tests 572/572, PostgreSQL Tests 1169/1169 and Relay E2E. A local-relay E2E also passed: - unbanned webhook: 202 and one run; - banned webhook: generic 404 and no run; - banned event trigger: no run; - after unban: one run. - At `72c8d8bbc` (merged with main), CI passed: Unit Tests (including all 171 buzz-workflow unit tests), PostgreSQL Tests 1227/1227 (all four ban tests listed by name), and both `Build (linux/amd64|arm64)` image jobs, which build with `--locked`. A Blox run at the same head also passed `cargo metadata --locked`, fmt and clippy, and the buzz-workflow unit tests passed there with no database reachable. - Not covered: the signed trigger has no dedicated test. It calls the same gate, which was confirmed by reading the code. ### Shared carry #8237, #8239, #8241 and #8242 each carry the same mesh-receive fix, boot and scraper regressions, and push-lease fixture correction. Whichever lands first owns them, and the others drop them on their next main merge. ### What got simpler The ban check now lives in the one gate every run-creation door already shares, so there is one fewer handler-local branch. Cron, event and signed triggers get the same protection without per-door code. The cron loop body is now `cron_tick(now)`, so a test can drive one tick without the 60-second sleep. buzz-workflow's infra-free unit tests ran in no CI lane. `just test-unit` and its cargo fallback in `scripts/run-tests.sh` now run them. buzz-review-completed 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: seiler <seiler@squareup.com> Signed-off-by: Brad Seiler <seiler@squareup.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Codex <codex@openai.com> Co-authored-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 17:27:24 -04:00
Luke Tornquist	f539ef4a87	docs(buzz-acp): fix stale prepare_reconnect doc comment (#8697) ... ## Summary Follow-up to #8653. The `prepare_reconnect` doc comment in `crates/buzz-acp/src/relay.rs` still said it runs `begin_reconnect` "with the real clock and RNG". `begin_reconnect` stopped taking a clock when `live_until` moved to #8688, so the comment now says "with the real RNG". Comment-only change. **What got simpler:** nothing in the code changes; the doc now matches it. ### Related issue N/A. This was a review note on #8653. ### Testing Comment-only change, so I ran nothing locally. CI covers fmt and clippy. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 17:24:57 -04:00
Luke Tornquist	76cd4294f6	fix(relay): close a REQ whose history read failed instead of sending EOSE (#8692) ... ## Summary **Merge order:** this lands and deploys **before** [#8688](https://github.com/block/buzz/pull/8688). That PR's buzz-acp resume cursor counts on a failed history read never ending in EOSE. Until #8688 ships, buzz-acp re-sends a REQ as soon as it gets an error CLOSED. The relay's REQ rate limit bounds that loop, and buzz-acp already parks on a rate-limited CLOSED. #8688 replaces this with a per-subscription backoff. When a historical query fails for a reason other than a statement timeout, the relay logs the error and still sends `EOSE`. EOSE means "you have every stored event" (`ARCHITECTURE.md` §5, NIP-01). A client that resumes from its last-seen point therefore treats the failed read as complete, advances past it, and never replays that history. This is RPR #1: a failure must not become an authoritative empty result. The history task now returns the CLOSED reason to send: - **Statement timeout:** `["CLOSED", sub_id, "error: query timed out"]`, unchanged. - **Any other DB error:** `["CLOSED", sub_id, "error: database error"]`. - **Cleanup:** CLOSED goes through the existing `close_if_owner` path, so the subscription, its sub-registry filters and its topic refcount are released, the same as on a timeout. - **Docs:** `ARCHITECTURE.md` §5 now states this contract. **Clients don't hot-loop on it.** Checked at `main` 7c165c37: - **Desktop:** `relayClosedPolicy.ts` / `relayClosedRecovery.ts` classify `error: database error` as retryable. Live subscriptions back off from 1s to 30s and keep their `since`. History-mode requests reject. - **Mobile:** `relay_closed_policy.dart` / `relay_session.dart` do the same, with their own capped backoff. - **buzz-acp:** today it re-REQs at once, bounded by the relay's REQ rate limit. [#8688](https://github.com/block/buzz/pull/8688) adds a per-subscription backoff that also restores the loss record. ### What got simpler - **One constant for the CLOSED reason.** `"error: database error"` was a string literal repeated across `req.rs` (5×) and `count.rs` (4×). It is now one constant, `DATABASE_ERROR_CLOSED`. - **One mapping function.** `history_read_closed_reason` maps a DB error to its CLOSED reason in one place. ### Identified debt (not changed here) - **NIP-50 search path.** After a non-timeout full-text-search or hydrate error, the search path still sends `EOSE` with partial results. It needs the same treatment, but it has a separate result-assembly path, so it's left out of this PR. ### Related issue Adds the regression wpfleger96 asked for in [this review on #8653](https://github.com/block/buzz/pull/8653#pullrequestreview-5692776411): a history error followed by a reconnect still replays the history. ### Testing New Postgres test `api::bridge::postgres_tests::failed_history_read_closes_and_reconnect_replays_the_history`: 1. Lock the replica's `events` table past its 300ms `statement_timeout` and send a history REQ. It must get exactly `["CLOSED","t","error: query timed out"]`. 2. Break the `events` table on both the writer and replica pools and send a REQ. It must get exactly `["CLOSED","h","error: database error"]`. 3. After each CLOSED, the subscription map, the sub-registry filters and the topic refcount must all be cleared. 4. Restore the table and send the same REQ on a new connection. It must get the stored event, then `EOSE`. Run on Blox at 95a8255a: - `cargo fmt --check` - `cargo clippy -p buzz-relay --all-targets -D warnings` - Postgres lane via `scripts/postgres-test-run.sh -p buzz-relay` (499/499) - The new test and `routed_read_cancel_maps_to_timeout_contract` 5× - **Mutation checks:** each of these makes the new test fail at its frame assertion. - Restoring `EOSE` for a non-timeout error. - Mapping every failure to `error: database error`. - **`just test` integration half:** run by hand, because `_ensure-services` can't start its Redis container on that workstation while a native redis-server holds the port. - Workspace `cargo test --test '*'`: every binary passes except two `buzz-acp` `git_bootstrap` tests. - Those two fail the same way on that workstation without this change, so they come from the environment. - `buzz-db` and `buzz-auth` integration tests pass. End-to-end against a locally built relay at e5621f94, with buzz-acp from #8688: - **While `events` was missing:** every channel REQ got `CLOSED error: database error`, and no channel EOSE was sent. - **After the table was restored:** the next retry replayed the missed mention, followed by EOSE. The later commit only changes tests and docs. buzz-review-completed 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 17:03:05 -04:00