796 Commits

Author	SHA1	Message	Date
ravarora2	2d289f720d	BUZZ-301: Add user deletion request intake (#8496) ... ## Summary Add the first testable piece of user deletion: submit a synthetic request, list requests, and inspect one by UUID. Operators can verify that intake reaches PostgreSQL before we add inventory, approval, and execution. Add one operator-global `entity_deletion_requests` table with an additive migration and matching desired-state schema. Identical submissions return the same row. Conflicting submissions fail without changing it. Only `demo:` targets are accepted, and requests stay `submitted`. Keep the commands in `buzz-deletion`, reached through a thin `buzz-admin user-deletions` adapter. Reuse the existing database connection setup. This change adds no worker, stage history, community gates, or deletion operations. ### Rollout plan This is an expand-only schema change plus an operator CLI. Existing relay code does not use the new request table. No backfill or real deletion is part of this release. 1. **Order the PRs.** Land #8496 first, using 0068; main now owns 0067 for channel join roles. Rework #8489 to build on this table with the next available migration number. Renumbering its existing CREATE TABLE migration alone is insufficient; both PRs currently create the same request table. 2. **Apply on staging.** For deployments that manage schema separately, apply the reviewed additive migration before using the new CLI. Where auto-migration is enabled, the new relay binary applies it at startup. Keep existing tables and records unchanged. 3. **Verify on staging.** Use the new `buzz-admin` binary with `DATABASE_URL` pointing to staging. Submit a unique synthetic label, list requests, and inspect the returned UUID: ```bash buzz-admin user-deletions submit --target demo:staging-intake-1 --requested-by <operator> buzz-admin user-deletions list --limit 5 buzz-admin user-deletions inspect <returned-uuid> ``` Verify one `entity_deletion_requests` row with status `submitted`. Repeat the identical submission and verify the same UUID and timestamps. Use a new label if this label was used by an earlier staging test. 4. **Apply beyond staging.** After verification, use the deployment's reviewed schema/release workflow to add the table in the remaining environments. Requests run only through the operator CLI; this release starts no deletion worker. No feature flag is needed for this synthetic intake: the database accepts only `demo:` targets, and real deletion is absent. 5. **Rollback.** Keep the additive table and saved requests; stop using the new commands and roll back the binary if needed. After 0068 is applied, an older binary with auto-migration enabled can fail at startup with sqlx `VersionMissing`. A binary rollback therefore requires migration execution to be disabled on the older relay (`BUZZ_AUTO_MIGRATE=false`, including the corresponding deployment configuration). ### Related work This is a smaller first slice of the full-workflow draft [#8489](https://github.com/block/buzz/pull/8489), which is preserved for later extraction. The two PRs overlap and should not both be merged as written. ### Testing The author submitted `demo:rarora-manual-1` to a separate local PostgreSQL 17 database. Repeating the same submission returned its original UUID and timestamps. List, inspect, and a direct SQL query all returned the same saved request. <details> <summary>Actual command inputs and outputs</summary> Commands ran from the new worktree with `DATABASE_URL` set to the local `buzz_user_deletion_submit` database. Outputs below are stdout; connection diagnostics on stderr are omitted. **Submit again — returns the existing request without creating a duplicate** ```bash ./target/debug/buzz-admin user-deletions submit \ --target demo:rarora-manual-1 --requested-by rarora ``` ```json { "id": "2e5bcdec-ab68-44d2-9743-1dd9fa7d2f89", "entity_type": "user", "target": "demo:rarora-manual-1", "status": "submitted", "requested_by": "rarora", "reason": null, "created_at": "2026-10-20T19:57:44.516698Z", "updated_at": "2026-10-20T19:57:44.516698Z" } ``` **List saved requests** ```bash ./target/debug/buzz-admin user-deletions list --limit 5 ``` ```json [ { "id": "2e5bcdec-ab68-44d2-9743-1dd9fa7d2f89", "entity_type": "user", "target": "demo:rarora-manual-1", "status": "submitted", "requested_by": "rarora", "reason": null, "created_at": "2026-10-20T19:57:44.516698Z", "updated_at": "2026-10-20T19:57:44.516698Z" } ] ``` **Inspect one request** ```bash ./target/debug/buzz-admin user-deletions inspect 2e5bcdec-ab68-44d2-9743-1dd9fa7d2f89 ``` ```json { "id": "2e5bcdec-ab68-44d2-9743-1dd9fa7d2f89", "entity_type": "user", "target": "demo:rarora-manual-1", "status": "submitted", "requested_by": "rarora", "reason": null, "created_at": "2026-10-20T19:57:44.516698Z", "updated_at": "2026-10-20T19:57:44.516698Z" } ``` **Read the table directly** ```bash docker exec buzz-user-deletion-dev-20261020 \ psql -X -U buzz -d buzz_user_deletion_submit -c \ "SELECT id, entity_type, target, status, requested_by FROM entity_deletion_requests ORDER BY created_at DESC;" ``` ```text id | entity_type | target | status | requested_by --------------------------------------+-------------+----------------------+-----------+-------------- 2e5bcdec-ab68-44d2-9743-1dd9fa7d2f89 | user | demo:rarora-manual-1 | submitted | rarora (1 row) ``` </details> A separate CLI smoke test also verified that a conflicting operator and a non-demo target fail without changing the saved request. The existing community-deletion `list --limit 1` command passed against the smoke database. See [the intake walkthrough](729233c726/docs/operator-user-deletion-intake.md) for setup, commands, expected fields, and the SQL query. Generated with Codex --------- Signed-off-by: Ravneet Arora <rarora@squareup.com>	2026-10-22 10:49:27 -04:00
Logan Johnson	a2d28eb6c4	fix(relay): apply channel setting changes atomically and lock every membership writer (#8516) ... 🤖 ## Summary This PR fixes two ways the relay could report a channel change that did not fully happen. **1. Channel settings could be saved only in part, while the app was told that they all worked.** When an owner or admin changed a channel's settings (name, description, topic, public or private, message lifetime, archived or not, who can join), the relay saved the request first and then applied each setting with its own database write. If one write failed, the relay only logged the error. It skipped the remaining settings and still reported success. Sending the request again did nothing, because the relay saw it as already saved. Example: a switch from public to private could fail while the app showed success, and the channel stayed public. Now saving the request, applying every setting and saving the "settings changed" messages in the channel happen in one database transaction. Either all of it is saved or none of it is: - If anything fails, nothing is saved and the client gets an error. It can send the same request again. - A repeat of a request that already succeeded is reported as a duplicate and changes nothing. This is also true when the repeat arrives while the first copy is still being saved. - A request to restore a channel that is not archived, or to change an archived channel without restoring it, is refused and nothing is saved. Before, the relay saved the request and a "settings changed" message, and then the restore failed. - Live updates go out only after the save commits. These are: removing access for people who lose it, and pushing the new messages and channel info to clients. **2. A published member list could include someone who had just been removed.** For some channels the relay publishes a list of the channel's members. While it builds that list, it holds a per-channel membership lock. Kicks by a relay admin and the automatic kick after a deploy did not take that lock. A kick could commit after the list was read, so the published list still showed the removed member. Channel deletion also did not wait for the lock. Kicks, channel deletion and the automatic add when someone joins a Huddle (voice call) now all take the lock first. The published list then includes the change, or the change waits until the list is saved. ## Details - **One lock order for all channel settings writes.** Before, each writer took its own locks. Review kept finding writers that took them in a different order or forgot one. `lock_channel_for_settings_write` (`crates/buzz-db/src/store/channel.rs`) is now the only place that takes them. The order is: 1. the membership lock, only when the public/private setting or the "who can join" setting (`join_role`) changes. Without it, a member add could read the old rule while the change commits. 2. the message-lifetime (TTL) lock: exclusive for a lifetime change, shared otherwise. 3. the channel row (`FOR NO KEY UPDATE`). This does not block the key lock that a member insert takes. The row lock also orders two copies of the same request, so the second copy sees the first as already stored. - The function returns a `ChannelSettingsWriter`. The writer borrows the transaction that took the locks, so the compiler stops it from being used with another transaction or after commit. The settings writes (`update_channel_on`, `set_topic_on`, `set_purpose_on`, `set_archived_on`) require the writer. They refuse a change its locks do not cover. - `Db::apply_channel_metadata_event` stores the channel-edit event (kind:9002) and its mentions, applies the settings and stores the "settings changed" messages (kind:40099) in one transaction. The older single-purpose writes (`update_channel`, `set_topic`, `set_purpose`, `archive_channel`, `unarchive_channel`) use the same helpers and behave as before. - `ingest.rs` and `validate_admin_event`: channel edits no longer check archived/active state before the write. The transaction checks it after its duplicate check, under the row lock. A refusal there goes back to the client as `invalid:`. Channel edits have their own save path and skip the general post-save step. `handle_side_effects` refuses them, so no caller can go around the transaction. - `side_effects.rs`: `emit_system_message` is split into signing and sending. The messages are signed before the transaction and sent after it. All "settings changed" messages from one edit share one timestamp and come in a fixed order (topic, purpose, visibility, lifetime, archive). - A change to "who can join" (`join_role`) is applied in the same transaction and announced with its own `join_role_changed` message. - **Membership lock outside settings** (`relay_admin_actions.rs`, `channel_members.rs`, `audio/handler.rs`): both kick paths (`execute_kick_with_marker`, `deploy_kick_member`) take the membership lock before they remove the member. The marker kick still undoes the removal if its lease check fails after waiting. The Huddle join takes its locks through a new `ChannelMembershipLocks` token, and the auto-add is a method on that token, so it cannot run without the lock. `soft_delete_channel` takes the full settings lock order. - **Limit of the membership lock:** it covers member changes, public/private and "who can join" changes, and channel deletion. It does not cover archiving: archive writes and the cleanup of expired temporary channels do not take it. The cleanup keeps its single UPDATE. A comment in `channel.rs` explains why that is safe and why it must not be split into a read and a write. ### Related issue Follow-up to the review of #8143 (which added the "who can join" setting). That review found that channel edits could be applied in part. No separate issue. ### Testing New PostgreSQL tests in `crates/buzz-relay/src/handlers/edit_metadata_postgres_tests.rs`: - `failed_edit_stores_nothing`: a test trigger makes one write fail partway through the transaction. Nothing is stored or changed, on the first send or a resend. Without the trigger, the same signed request applies in full. - `resent_edit_is_a_duplicate`, `resent_archive_and_unarchive_are_duplicates`: a resend after success is a duplicate and adds nothing. - `resend_validated_after_the_first_commit_is_a_duplicate`: a resend validated after its first copy committed is a duplicate, with no second change and no second message. - `concurrent_copies_of_a_restore_apply_once`: two copies of one restore, held at the channel lock until both are waiting, apply once. The other copy is a duplicate. - `unarchive_of_an_active_channel_is_refused`, `join_role_edit_is_applied_and_announced`. Lock tests in buzz-db: each one holds a member list in progress and checks that the writer waits for it. They cover a "who can join" change (`channel_members.rs`), the deploy kick and the marker kick (`relay_admin_actions.rs`), and channel deletion (`channel.rs`). The kick and delete tests fail when the new locks are removed. These tests and the existing channel-edit tests in `group_state_postgres_tests` pass locally against a migrated PostgreSQL database. --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-22 10:32:43 -04:00
Ian Oberst	347ce88d5d	fix(db): keep pgschema re-apply from dropping runtime partitions (#8487) ... ## Summary pgschema doesn't compare partition bounds. Re-applying `schema/schema.sql` to a database a relay has used therefore plans `DROP TABLE IF EXISTS ... CASCADE` for every `events` / `delivery_log` monthly the partition manager created at runtime, and those tables' rows are deleted with them. Since #8464 the manager advances the catch-all by default, so any local dev or test database a relay has started against now hits this. One example is the persistent `buzz` database that `scripts/start-relay-for-tests.sh` re-applies. This adds a root `.pgschemaignore` that hides dated monthlies (`events_p2*`, `delivery_log_p2*`) from pgschema. The monthlies declared in `schema/partitions/` are exempted with `!` patterns, so pgschema still manages them. pgschema reads the file from its working directory, and every caller already runs it from the repo root: the test-relay launchers, `postgres-test-setup.sh`, the desktop release smoke and `_ci-relay.yml`. Both partition files now carry a note pointing at it, and AGENTS.md gotcha #7 adds "run pgschema from the repo root" next to the reconcile rule. New-install layout is unchanged. **Why the drops haven't deleted data yet:** on a reconciled database a re-apply also plans to drop and re-add the `events` primary key on the parent and on each declared child. The apply then fails with `cannot drop inherited constraint "events_p2026_01_pkey"` (42P16). That happens with or without this PR. The planned drops run in the same single transaction, so they roll back, and the runtime monthlies have survived so far. Once that primary-key drift is fixed, re-applies will succeed and this ignore file is what keeps them from dropping the runtime monthlies. The primary-key fix is a separate follow-up. ### Related issue Follow-up to #8464. Salvaged from an earlier branch; no existing issue found. ### Testing - `pgschema_reapply_keeps_runtime_partitions` (Postgres) follows the real bootstrap path in a dedicated database: - `bin/pgschema apply` from the repo root, then `scripts/reconcile-schema-after-pgschema.sql`; - the relay's default maintenance pass (create plus advance); - asserts that both catch-alls were attached before the pass and that their bounds moved after it; - asserts that each parent's `p2026_07` monthly exists. The catch-all advanced past the declared bounds but kept its name. - asserts that `pgschema plan` contains no `DROP TABLE` and no line mentioning any runtime monthly. - Falsified both ways: - With `.pgschemaignore` removed, the test fails and prints the planned drops. - Without the reconcile step, it fails with `reconcile did not attach both catch-alls`. - `pgschemaignore_exempts_exactly_the_declared_monthlies` checks that the `!` exemptions exactly match the dated monthlies in `schema/partitions/`. - The admin parity test now shares the `pgschema` helper, so it also runs from the repo root like the real callers. - All `runtime::migration` and `store::partition` tests pass against local Postgres after merging current main, including the ignored ones: 104 passed, run with `--test-threads=1` because the full nextest Postgres lane needs local `dropdb`. `clippy -D warnings` and the pre-push lanes pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Ian Oberst <ioberst@block.xyz> Co-authored-by: Ian Oberst <ioberst@block.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-21 17:14:22 -07:00
Luke Tornquist	2f2f6e2774	fix(db): bound every statement and the #p mention scan (#8461) ... ## Problem An agent `#p` + `#h` + `since` REQ (the `require_mention` subscription in `buzz-acp`) ran for over 45 minutes on a production writer and held `AccessShare` on every `events` partition. There were two causes: 1. **The mention side of the `#p` join was unbounded.** `build_query_events_sql` joined `event_mentions` on `(community_id, event_id)` only. The planner picked a nested loop whose outer side was every mention of the pubkey, ever, and whose inner side range-scanned the channel index from `since` on each partition. Cost ≈ total mentions × channel rows since `since`. 2. **Nothing cancelled it.** The writer `statement_timeout` defaulted to `0`. Only `#e` reads had a deadline (the 20s cap from #7854), and read-replica sessions had none at all. `pg_stat_statements` shows this shape has a multi-thousand-second max going back months, so it is chronic and was not introduced by a recent change. ## Change **`#p` mention bound** (`store/event.rs`, `push_mention_join`, shared by the page SELECT and COUNT) - Join on the full event key, `AND e.created_at = m.event_created_at`. Mention rows copy the event's `created_at` at insert (migration 0007 already relies on this equality), so the join can prune `events` partitions by time. - Repeat `since`/`until` on `m.event_created_at`, so the outer scan is a range on the existing `idx_event_mentions_pubkey_created`. `until` is a superset bound for the keyset cursor; the exact `(created_at, id)` cursor test stays on `e`. - Repeat `kinds` on `m.event_kind` when the filter has kinds. Every production mention writer binds the event kind (`insert_mentions` in `store/runtime/mod.rs`, and the replaceable-event path in `store/replaceable.rs`, back to the first Postgres schema), and no migration inserts mention rows, so this drops no rows written by the relay. Rows imported out of band with a NULL `event_kind` would no longer match a kinds-filtered `#p` read; a read-only production count on 2026-10-20 found 0 such rows on both production reader databases. The pushdown makes `idx_event_mentions_pubkey_kind_created` available to the planner for the `require_mention` kind-9 shape; its use has not been observed (see below). An E2E fixture with a hand-set NULL `event_kind` row confirmed the caveat: base returns 8 events, head 7. - The feed's mentions and needs-action queries (`store/feed.rs`, `build_mentions_query` / `build_needs_action_query`) join `event_mentions` the same way, so each joined row probes one `events` partition instead of all of them. Their `ORDER BY m.event_created_at ... LIMIT` already kept them from running away; this removes the per-row cross-partition probe. - Observed plan (E2E at `9cb6638b1`, before the `kinds` pushdown; 60,000 historical mentions + 2,500 recent decoys, stock planner, `EXPLAIN (ANALYZE, BUFFERS)` of the captured production SQL): the mention side is a bounded range on `idx_event_mentions_pubkey_created` (13 rows), then a hash join with a scan of the recent `events` partition. Base scanned 2,509 recent events and probed `event_mentions_pkey` 2,509 times. Head: **1.1 ms / 319 shared hits**; base: **8.5 ms / 10,049 shared hits**. Not measured against the production replica. A fresh `EXPLAIN (ANALYZE, BUFFERS)` replay at `a7065c367`, after the pushdown, ran on a smaller fixture where the planner chose a sequential mention scan, so it neither confirms nor replaces these numbers. **Reads: 20s deadline on every generic event read** - `query_events_on` and `count_events_on` always run under the transaction-local 20s deadline that `#e` reads used (`fetch_with_read_deadline`, formerly `fetch_with_e_tag_deadline`). It only tightens: shorter operator or caller caps are kept. - Read-replica sessions get a 20s session default through `after_connect`, using the same tighten-only SQL. - Cost: every generic REQ history read and COUNT now pays BEGIN + `set_config` + ROLLBACK (three extra round trips; before, only `#e` reads did). In the local E2E the end-to-end history read was still faster at head (median 4.1 ms vs 10.2 ms at base, 9 samples each, warm cache), so the round trips are dominated by the bounded plan; this does not isolate them or establish production p99. The round trips also lengthen how long each such read holds its writer connection; that hold time has not been measured against production pool sizing. - A replica read cancelled at its deadline (57014) is returned, not re-run on the writer. Usually the cancel means expensive work the writer would only repeat. It can also mean the replica waited on a lock only it holds (a replayed `ACCESS EXCLUSIVE` from primary DDL), when the writer would have answered quickly; that miss is the accepted price of keeping runaway reads off the primary. The rationale now lives once on `DbError::is_statement_cancelled` instead of in a dozen copied comments. The generic routed reads and `ReadSession` already did this; the eight routed reads with their own SQL (feed mentions, needs-action and activity; thread replies; channel window; relay membership; thread window and its auxiliary scan) now do too. A cancelled auxiliary scan also keeps its replica session instead of degrading it to the writer for the rest of the request. Other replica failures (e.g. hot-standby recovery conflicts) still fall back to the writer. - Paths with their own tighter `SET LOCAL` (artifact 2–5s, thread window 4s, personal reads, usage 5–15s, serving lease 1.5s) are unchanged. **Writes: every writer session is bounded** - The default `statement_timeout` changes from `0` to 120s (`DEFAULT_STATEMENT_TIMEOUT_MS`). - `BUZZ_DB_STATEMENT_TIMEOUT_MS` still overrides it. `0` or any value above 300s installs 300s (`MAX_STATEMENT_TIMEOUT_MS`), so a session can no longer disable it, and a clamped value logs a `warn!` when the pool is built. Only the operator-approved, fenced destruction paths below lift it, per transaction. **Long-running writer paths, audited** | Path | Behavior | |---|---| | Migrations / schema destruction (`with_exclusive_schema_destruction_lock`) | Unchanged: these lift lock and statement timeouts on their own detached connection | | Partition maintenance | Unchanged: own 5s `statement_timeout` | | Partition audit (`buzz-admin`) | Unchanged: 30s | | Community deletion inventory, purge and logical verification (`inventory_schema` whole-tenant counts, `purge_postgres` whole-community `DELETE` per table in one tx, `verify_postgres_logically_deleted` whole-tenant `EXISTS` per table right after the purge) | Lifted (`statement_timeout = 0`) transaction-locally by `lift_statement_timeout`, like migrations; lock and idle timeouts still bound them. Inventory counts each table in its own lifted transaction, so each table's lock and snapshot are released as soon as its count finishes, as the autocommit counts did. A cancelled purge would retry from scratch and eventually block, so a cap here would turn large-tenant deletion into an operator override | | Audit appends (`buzz-audit`, audit pool inherits the writer cap) | Waits on the audit-chain advisory lock. With `BUZZ_DB_LOCK_TIMEOUT_MS=0` (or at least the statement cap), `statement_timeout` (57014) cancels the wait instead of `lock_timeout` (55P03). The relay's audit worker now retries both with the same backoff instead of dropping the entry on 57014; admission refusal stays terminal | On HTTP routes a statement timeout already maps to the distinct retryable 503 (`api/mod.rs`, `statement_timeout_maps_to_distinct_503`). A cancelled REQ fails its history read instead of hanging. ## Tests - `p_tag_query_bounds_mention_side_by_time` (unit): page and COUNT SQL carry the `created_at` join and the `since`/`until`/`kinds` mention bounds, and the bounds are absent when the filter has none. - `p_tag_read_with_mention_bounds_keeps_results` (Postgres): through the production mention insert, unbounded, `since`, window and keyset-cursor reads return exactly the mentioned rows, and COUNT matches. Every bound sits exactly on a mentioned row's second and the cursor page includes a same-second sibling after the cursor id, so an off-by-one on either mention-side bound fails it. - `plain_read_is_cancelled_at_deadline_when_timeout_disabled` / `count_events_cancels_at_read_deadline_when_timeout_disabled` (Postgres): the production `#p` + `#h` + `since` shape and a plain COUNT are cancelled at 20s with the session timeout disabled. Previously these covered `#e` only. - `writer_statement_timeout_is_always_bounded` (unit) and `session_statement_timeouts_bound_writer_and_reader_sessions` (Postgres): through `Db::new`, writer sessions install `2min` / `5min` (for `0` and for 600s) / `45s`, and replica sessions install `20s`. - `deletion_whole_tenant_stages_outlive_the_writer_statement_timeout` (Postgres): on a scratch desired-state database with a 500ms writer cap, the real `inventory_schema`, `purge_postgres` and `verify_postgres_logically_deleted` each wait behind a held `ACCESS EXCLUSIVE` lock on `events` for over 1s and still complete, a plain statement is cancelled (57014), and pooled sessions still read `500ms` afterwards. A lifted transaction reads `statement_timeout = 0`, so a bounded lift (e.g. `'300000'`) fails it, as does removing any of the three `lift_statement_timeout` calls. - `mentions_query_is_tenant_scoped_and_joins_mentions_by_composite_key` / `needs_action_query_is_tenant_scoped_and_joins_mentions_by_composite_key` (unit): both feed queries carry the `e.created_at = m.event_created_at` join. - `replica_statement_cancel_propagates_without_writer_rerun` (Postgres): with the replica locked past its cap, each of the six routed reads above returns 57014 instead of the writer's copy. Removing the cancel arm in any of them fails it. - `thread_window_statement_cancel_propagates_without_writer_rerun` (Postgres): a thread-window read parked behind a replica lock and cancelled (57014; the window's own 1s `lock_timeout` would otherwise fire first) returns the cancel instead of the unlocked writer's window, and an auxiliary scan timed out on a replica session returns 57014 and leaves the session on the replica. Removing either guard fails it. - `audit_worker_retries_statement_timeout_until_original_entry_is_appended_once` (Postgres, `buzz-relay`): the production audit worker, with `lock_timeout` disabled and a 100ms statement cap, keeps the entry through repeated 57014 cancels while the chain lock is held and appends it exactly once after release. Treating only 55P03 as retryable fails it. - The existing writer and audit-pool GUC tests now expect `5min` for a disabled statement timeout. ## After deploy, watch - `buzz_db_route_decision{decision="replica",reason="statement_cancelled"}`: replica reads that now fail instead of falling back to the writer, including the replica-only lock case above. - `buzz_db_pool_acquire_wait_seconds` and `buzz_db_pool_waiters` on the writer pool: every generic REQ history read and COUNT now holds its connection for three extra round trips. ## Database checklist (`docs/database.md` § Review checklist) This PR changes schema-dependent `buzz-db` queries (the `#p` join and its `m.event_kind` pushdown). It does not change the schema. - **Phase:** none. No schema change, so this is not an expand, backfill or contract step. Nothing before or after it is required. - **Triggers, functions, procedures, rules, foreign keys:** none added. - **Row mutations / backfill:** no migration, and no row is mutated or backfilled. The one data-shape dependency is mention rows with a NULL `event_kind`: a kinds-filtered `#p` read no longer matches them (see the `kinds` bullet under Change). Relay writers always bind `event_kind`, so only rows imported out of band are affected. A read-only production count on 2026-10-20 found 0 such rows on both production reader databases. - **Contract migration:** N/A. - **Rule 4 locations:** none touched. No table, index or desired-state file changes, so no parity test applies. The new bounds use the existing `idx_event_mentions_pubkey_created` and `idx_event_mentions_pubkey_kind_created`. - **Deployed relay vs. new schema:** there is no new schema. The currently deployed relay and this relay run against the same tables. - **Lock and statement timeouts:** this PR is about them. Generic reads get a 20s deadline, replica sessions a 20s default, and writer sessions a 120s default capped at 300s. Community-deletion transactions lift `statement_timeout` per transaction; migrations and schema destruction are unchanged (see "Long-running writer paths, audited"). No index builds and no operator prebuild step. ## What got simpler - One read-deadline path instead of an `#e`-only branch in `query_events_on` and `count_events_on`. - The `#p` join SQL that was duplicated between SELECT and COUNT now lives in one `push_mention_join`. - The deletion tests share one scratch desired-state database helper instead of an inline copy. - "Is this query bounded?" now has one answer: yes, unless it runs in a migration, schema destruction, or community-deletion transaction. ## Identified debt (not in this PR) - `buzz-acp` re-sends the same filter after a timeout `CLOSED` (no backoff or narrowing). With the `#p` bound the `require_mention` read should finish well inside 20s, so this is a follow-up, not a blocker. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-21 19:35:03 -04:00
Logan Johnson	d97f6b4c6e	feat(relay): add announce channels with a join_role setting (#8143) ... 🤖 ## Summary Channels are Buzz's main way to control access, but today they cannot separate reading from writing. In an open channel, anyone who can read can also write. So there is no way to say "many people read, a few people write". Announcements, release feeds, plugin release records and shared configuration all need that. This PR adds **announce channels**: anyone can follow and read them, but only members can post. It does this with one new channel setting and without changing what any role means. - **Roles mean the same thing in every channel.** A *guest* can read and leave, but cannot write. A *member* (or *bot*) can write. *Admins* and *owners* also manage the channel. The relay already had the guest role, but no client used it until now. - **A new setting, `join_role` (`member` | `guest`), picks the role a new person gets** when nobody chooses one: when they join by themselves, or when someone adds them without a role. The default, `member`, keeps today's behavior for every existing channel. - **An open channel with `join_role = guest` is an announce channel.** Anyone can find, read and join it. People who join are guests, so they cannot write until someone makes them a member. The relay enforces these rules, so a client cannot get around them. What each role can do (the same in every channel): | Action | Guest | Member or bot | Admin / owner | | --- | --- | --- | --- | | Read, leave | ✅ | ✅ | ✅ | | Post, reply, edit, delete own, react, artifacts, canvas, forum, pins, topic, typing, workflow messages, save or run workflows, huddles | ❌ | ✅ | ✅ | | Add people | ❌ | ✅ as member, guest or bot | ✅ any role | | Make a guest a member (person or agent) | ❌ | ✅ | ✅ | | Any other role change | ❌ | ❌ | ✅ | | Change name, about, visibility, TTL, `join_role` | ❌ | ❌ | ✅ | Other rules: - **Joining by yourself.** A person who joins (kind:9021), or adds themselves (kind:9000 with their own key), gets the channel's `join_role`. They may ask for `join_role` or a lower role, never a higher one. So in an announce channel nobody can join as a member by themselves. - **Non-members.** A non-member may write only where joining would let them write: an open channel with `join_role = member`. That is today's behavior for open channels. - **Agents.** Agents follow the same rules as people. Any member who can write may make a guest agent a member. Limits that only an agent's owner may lift belong to the later IFC work (its "restricted" agent level), not to this PR. That work only refuses more actions, so it can be added later without breaking anything here. - **Leaving always works.** A member may leave an archived channel too (kind:9022, or kind:9001 naming themselves). The last owner still cannot leave. - **Agent owners keep edit and delete.** Today the human owner of an agent can edit (kind:40003) and delete (kind:9005) that agent's messages without being a member. This PR keeps that existing behavior in announce channels too, and documents and tests it. It does not let the owner post. - **Changing the setting changes no roles.** Existing members can still write after a change to `guest`. Existing guests stay read-only after a change back to `member`. A refused write gets `restricted: only members can post in this channel`. ## Details - **One write rule.** The relay's channel write check decides from the actor's role and the channel's `join_role`. Some event types skip that check and run their own: message edit, deleting your own message (kind:9005), topic and purpose (kind:9002), add-member (kind:9000), workflow messages and joining a huddle. Each of these now uses the same rule. Membership changes (add, remove, leave) do not need write access, so a guest can always leave. If the relay finds a stored `join_role` value it does not know, it treats it as `guest`, so an unknown value can never give more access. - **Admission is decided in the database transaction.** The membership write (`put_member`) makes the whole admission decision inside its transaction, after it takes the channel's membership lock: `join_role` for a missing role tag, the self-join ceiling, guests cannot add people, who may grant admin or owner, and role changes (`MemberRole::may_change_role`). The relay's request check uses the same rules, but the database is the authority, so a write that skips the request check cannot get around them. A change to `visibility` or `join_role` takes the same lock, so an add and a setting change cannot interleave. - **No creator exception.** `put_member` used to give owner to a channel's creator who added themselves to a private channel without an active membership. A removed creator could use that: their join (kind:9021) passes the open-channel check, the channel becomes private before the join takes the membership lock, and the join restored them as owner. Channel creation already makes the creator owner in the same transaction, so the exception is removed. A removed creator is now admitted like anyone else. - **Huddles.** Joining a huddle needs write access in the huddle channel **and** in its parent channel. The relay checks both before the join. When it commits the join, it applies the same rule again while it holds both channels' membership locks: a member needs a role that can post, and a non-member needs an open channel whose `join_role` is `member`. A guest in either channel cannot join. A join is also refused when the joiner loses membership or the channel's `join_role` changes before the commit. - **One lock order.** Each writer that takes more than one of a channel's locks takes them in this order: membership lock, TTL lock, channel row. Settings changes, roster publication, TTL refresh and huddle joins all use this order. Before this change, a `join_role` change on an ephemeral channel at the same time as a roster publication could deadlock. - **Workflows.** Saving a workflow (kind:30620) skips the generic write check, so it now checks the sender's role itself and needs a writing role before anything is stored. Every path that starts a workflow run, including the manual trigger (kind:46020), needs the workflow's owner to hold a writing role now (and an elevated role for an elevated workflow). So a workflow saved while its owner was a member stops running after the owner is made a guest. - **Channel names are checked before storage.** kind:9002 and kind:9007 refuse a name longer than the `channels.name` column (255 characters) before the event is stored. Before this, the relay accepted the event, then the name update failed, and the rest of the update (for example `join_role`) was silently skipped. This PR only adds this check. Storing the event and applying the update in one transaction is a larger change, and it is not in this PR. - **Changing the setting.** Owners and admins send a channel-metadata update (kind:9002) with `["join_role", "guest"]` or `["join_role", "member"]`. Buzz applies these updates as partial updates; a missing tag means "no change". The relay posts a `join_role_changed` system message in the channel. - **What clients see.** The relay signs a channel description event (kind:39000) that clients read. One shared builder now writes it for both the relay and the `buzz-admin` backfill, which had a copy that had drifted from the relay's. It always publishes an explicit `["join_role", …]` tag, plus the standard NIP-29 group access flags: | `visibility` | `join_role` | `public` / `private` | `restricted` | `closed` | | --- | --- | --- | --- | --- | | open | `member` | public | | | | open | `guest` | public | ✓ | | | private | either | private | ✓ | ✓ | This also fixes an old bug: `closed` ("join requests are ignored") was published for every channel, even open channels that accept joins. - **CLI and SDK.** `buzz channels update --join-role guest|member`. `buzz_sdk::build_update_channel` takes a new `join_role` argument and always writes it as an explicit tag. - **Docs.** `NOSTR.md` has an "Announce channels" section with the role table and the flag table. - **Tests.** Relay end-to-end tests, run in the CI relay E2E step: - `test_nip29_announce_channel_wire`: who can set `join_role`, the kind:39000 flags, joining as a guest, refused guest and non-member posts, reactions and adds, member posts, a member making a guest a member who then posts, changing the setting back without changing roles, and leaving. - `test_nip29_member_raises_guest_agent`: a member who is not the agent's owner can make a guest agent a member. - `test_nip29_announce_channel_wire_workflow_send_needs_write_role`: a guest's workflow message is refused; a member's is accepted. - `test_nip29_departure_wire_archived_channel_allows_self_leave`: guests and members can leave an archived channel. - `test_private_channel_any_member_can_invite` now expects a private-channel guest to be refused when it adds someone (it used to expect success). - `test_nip29_announce_channel_wire_workflow_commands_need_write_role`: saving and triggering a workflow, by a member and by a guest, in announce and normal channels; a guest cannot run a workflow saved while it was a member. - `test_nip29_announce_channel_wire_overlong_name_is_refused_before_storage`: a kind:9002 with an overlong name and `join_role` is refused, and kind:39000 still shows the old settings. - `test_nip29_announce_channel_wire_agent_owner_exception`: an agent's owner, as a guest and as a non-member, can edit and delete the agent's messages, while ordinary guests are still refused. - `test_nip29_guest_is_read_only_in_normal_channel`: a guest in an ordinary channel cannot write, and joining an ordinary channel still gives `member`. Unit tables cover the self-join rule for each `join_role`, and every actor, current role and requested role for the role-change rule. Database tests check the admission rules in `put_member` (`put_member_enforces_announce_admission`), that an add which waits on an uncommitted change to `guest` is decided on the new setting (`put_member_waits_for_a_concurrent_join_role_change`), and that a member can make a guest a member but not an admin. Unit tests cover the name limit and the workflow owner rule (`owner_authority_denies_roles_that_cannot_write`). A relay test covers huddle joins with different roles in the huddle and its parent, and a commit after a demotion, a `join_role` change or a lost membership (`huddle_join_needs_a_write_role_in_the_huddle_and_its_parent`). Lock-order tests: `join_role_change_waits_for_a_roster_publication_on_an_ephemeral_channel` reproduced a real deadlock (`40P01`) against the old order. `f2d_join_lock_order_serializes_membership_and_settings_writers` checks that membership and settings writers wait for a join and then succeed. `removed_creator_cannot_rejoin_a_private_channel` checks that `put_member` refuses a removed creator on a private channel, and `removed_creator_join_racing_a_private_switch_is_refused` holds the membership lock so a removed creator's kind:9021 passes the open-channel check, then makes the channel private before the join continues. Each new test was run against the old code and failed. ## Rollout 1. **Before the deploy, count guest rows** (`SELECT count(*) FROM channel_members WHERE role = 'guest' AND removed_at IS NULL`). We expect none. Any guest that exists loses the ability to post, change the topic and add people when this ships. If some exist, decide first whether to make them members. 2. Deploy. Every channel stays `join_role = member`, so nothing else changes. 3. **Do not set any channel to `join_role = guest` until no old relay is running.** Old relays ignore the column, so they would let guests and non-members write in that channel. 4. Rerun the `buzz-admin` channel reconcile so open channels drop the wrong `closed` flag. ## Migration and rollback - Migration `0067` adds `channels.join_role TEXT NOT NULL DEFAULT 'member'` with a CHECK constraint. Postgres stores a constant default in table metadata, so existing rows are not rewritten and there is no backfill. The relay runs the migration at startup. - Existing channels keep the old `closed` flag until their next change, or until the reconcile in Rollout step 4. - **Before rolling back the relay**, set every announce channel back to `join_role = member`. An older relay ignores the column and does not stop guests from writing, so those channels would accept posts from their guests and give `member` to new joiners again. ## Not in this PR Client UI: a Follow button, hiding the composer for guests, and an admin toggle. Clients that ignore the new tag keep working. A guest who tries to post sees the `restricted` error. ## Stack 1. #8143 (this PR): announce channels 2. #8494: store the bot role only for keys with a recorded agent owner --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-21 16:01:11 -04:00
Logan Johnson	c37d6b8eb6	test(relay): cover system channels in the workflows API and push matcher (#8509) ... 🤖 ## Summary Test-only. This adds end-to-end coverage for the last two system channel read paths that had none. It changes no behavior. A system channel is a channel type the relay leaves out of ordinary reads. A client sees one only when it names the channel or opts in. The system channels spec lists every read path that must follow this rule. The Postgres end-to-end tests already covered subscriptions, counts, search, the HTTP bridge and membership notices, but not these two: - **Workflows API.** A workflow names its channel, so reading a workflow in a system channel follows ordinary channel access. Members can read it. Outsiders can read it only when the channel is open. For a private one, an outsider gets 404, the same answer as for a missing workflow. - **Push notifications.** This is database-backed matcher coverage: the test stores real channel rows, memberships and push leases, loads the matcher's context from the database, and runs the production matcher. It builds the match batch directly, so it does not cover the outbox or gateway delivery. A member's push filter that names no channel gets a wake for a message in an ordinary channel, but not for a message in a system channel. A member's filter that names the system channel by `#h` gets a wake for that channel only (the rule since #8517). A non-member's filter gets neither. ### Related issue Part of the system channels spec (risks section: end-to-end cases for the workflows API and push). Rebased on #8507 and #8517, which made push follow the read rule: a system channel pushes only to a member's filter that names it by `#h`. No duplicate PR found. ### Testing - Both tests pass against local Postgres and Redis on this branch. - Each test fails when the behavior it covers is removed: without the named-channel check in the workflows API, a member gets 404 for its own system channel's workflow; if the push matcher lets a filter that does not name a system channel match it, the member's broad lease also wakes; if the matcher never pushes system channels, the named lease does not wake. - clippy (`--tests -D warnings`) and fmt pass. --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-21 13:09:45 -04:00
Logan Johnson	cd8a968179	fix(relay): push system channel events to a lease that names them (#8517) ... 🤖 ## Summary Mobile clients can now get push notifications for a system channel, but only when they ask for that channel by its ID. Before this change, the relay never sent push for system channels, even when a client asked for one directly. System channels are channels the app uses for its own purposes (for example, agent attention). They are hidden from clients that do not know about them. #8507 made push use the same visibility rule as normal reads: a push subscription sees a system channel, or any channel type added later, only if its filter names that channel ID in `#h`. With that rule, an older client can never get system channel pushes by accident. The extra "never push system channels" rule in NIP-CL now only blocks clients that ask on purpose, such as a future Live Activities subscription. This PR removes that special case. ### Details - `push_runtime.rs`: system channels now follow the same rule as other named-only channel types. The relay still checks membership before it sends a wake. - `NamedOnlyChannels::is_system` had no other caller, so it is removed. - NIP-CL and NOSTR.md now describe the shared rule instead of the ban. - No behavior change for current clients. They do not name system channels in their push subscriptions. ### Related issue Follow-up to #8507 (see https://github.com/block/buzz/pull/8507#issuecomment-6316180106). ### Testing - `named_only_channel_pushes_only_to_a_filter_that_names_it` now expects a wake for a system channel named by `#h`. It also checks that a broad filter still gets no wake, and that a non-member still gets no wake. - `cargo test -p buzz-relay --lib push_runtime`, `cargo clippy -p buzz-relay --all-targets -D warnings`, and `cargo fmt --check` pass. Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-21 10:38:57 -04:00
Logan Johnson	4dea51f644	fix(relay): scope push matches with the read path's channel rule (#8507) ... 🤖 ## Summary This is a consistency and forward-compatibility fix. It does not change what users of current clients see. Mobile push notifications now decide which channels can notify a device with the same rule that reads use, with one exception that the protocol requires: system channels never push. Before this change, push and reads used two separate rules: - **Reads** (subscriptions, counts, search and the bridge) show only the original channel types (stream, forum, DM, workflow) unless the filter names the channel. - **Push** skipped only `system` channels. Today `system` is the only non-original type, so the two rules give the same result. But a channel type added later would be hidden from reads and would still send push notifications to every member. This PR removes that gap: push now calls the same function that reads use, so a future type gets the correct push behavior with no extra work. ## What changes - A channel of an original type pushes as before. Membership is still required. - A system channel never pushes, even when a push filter names it. NIP-CL says: "The relay MUST NOT send push notifications for system channels." This is the same as before. - A channel of a type added later pushes only to a push filter that names that channel with `#h` (the channel tag), as reads do. - A push filter that names one channel does not get pushes from other named-only channels. - Events that are not in a channel (for example gift wraps) match as before. Push filters cannot carry the `#t` or `#P` opt-in that reads accept, so for push, naming the channel means `#h` only. ## Impact on current clients: none - System channels still never push, and no current channel type is affected by the later-type rule. - The mobile app names channels by `#h` only for joined, unarchived DM channels (`mobile/lib/shared/push/push_subscription_provider.dart`). DMs are an original type. - Naming a channel cannot get around membership. A push filter that names a channel still gets nothing unless its owner is a member of that channel. - Database cost is unchanged: one per-batch query replaces another, on the same channel list, with no extra round trip. - The push docs (NIP-PL and the push suppression doc) do not mention channel types, so they need no update. ## Details - The push matcher loads the named-only channels in each batch and checks each lease filter with `filter_channel_scope`, the function REQ, COUNT, search and the bridge already use. Before that check, it rejects system channels outright. - This removes `system_channel_ids_among`, which push was the last caller of. ### Related issue Part of the system channels spec (§2, rule 5). Builds on #8181 and #8469. No duplicate PR found. ### Testing - A `match_job` unit test covers: unnamed filters on an original-type channel, on a channel-less event, on a system channel and on a later-type channel; a later-type channel named by `#h` (1 wake); a system channel named by `#h` (0 wakes); an unrelated `#h` (no widening); and non-members. Removing either the shared scope check or the system exception fails it. - Before this PR, no test covered the push channel filter, not even for `system`. --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-21 08:42:30 -04:00
Jason Holmes	ce3fc78cbd	feat(ws-client): send custom headers on the WebSocket upgrade (#8159) ... ## Summary NIP-FI relays verify the `Nostr-Federated-Identity` assertion on the WebSocket upgrade request (#7224) and pair it with the NIP-42 AUTH key, but `NostrWsConnection` can only connect from a bare URL, so no `buzz-ws-client` consumer can present an assertion today. This adds two additive constructors: - `NostrWsConnection::connect_with_headers(url, headers)` - `NostrWsConnection::connect_authenticated_with_headers(url, keys, auth_tag, headers)` The existing `connect` and `connect_authenticated` delegate with an empty `HeaderMap`, so their behavior is unchanged (`connect_async(&str)` already went through `into_client_request`). `HeaderMap`, `HeaderName`, and `HeaderValue` are re-exported so callers don't need a matching `tungstenite`/`http` dependency. Caller headers are added after the handshake headers; the doc comment notes that supplying a handshake header such as `Sec-WebSocket-Key` replaces it. tungstenite doesn't follow redirects, so upgrade credentials can't be forwarded to another origin. Follow-up (not in this PR): renewing assertions on long-lived connections. The relay ends the session at the assertion's deadline, so clients reconnect with a fresh assertion; that's caller policy rather than a transport concern. ### Related issue NIP-FI spec: #1485. Relay enforcement: #7224, #7264. No existing client-side issue or PR found. ### Testing - New `connect_with_headers_sends_headers_on_upgrade` test runs a local `accept_hdr_async` server and asserts the header arrives on the upgrade. Confirmed it fails when the headers are dropped. - `cargo test -p buzz-ws-client` (4 passed) - `cargo clippy -p buzz-ws-client -p buzz-test-client -p buzz-cli --all-targets -- -D warnings` - `cargo fmt --all -- --check` - Not run: full `just ci` (desktop/mobile toolchains unavailable in my environment) and an end-to-end connect against a local relay with NIP-FI enforcement on. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Jason Holmes <holmes@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-20 20:24:35 -07:00
ngoodger-block	76d792f288	BUZZ-357: feat(cli): token-lean agent output format for event reads (#8234) ... ### Why? Agents read channel history with `buzz messages get/thread`. Every returned event includes its 128-hex `sig`. Every agent-authored message also carries a NIP-OA `auth` tag, which is another owner pubkey and a 128-hex signature. An agent can't use either of these, but pays for them on every history read. Measured with Claude's tokenizer, that is about 88 tokens per human message and about 180 per agent message, so a history read shrinks 33% for short human messages and 40% once agent messages are in the mix (details in the comment below). The existing `compact` format is fixed at `{id, content, created_at}` (#6884), which drops the author, tags and reply target, so agents can't use it. ### What? - **New `--format agent` for event reads.** Applies to `messages get/thread/search` and `feed get`. - Drops `sig`, and trims the NIP-OA `auth` tag to `["auth", <owner>]` so agents can still tell an author is an agent and who owns it. - Folds the NIP-10 `root`/`reply` tags into a `reply_to` field, using the same resolver as relay ingest, plus `thread_root` on nested replies. That's the value `messages send --reply-to` takes. - Keeps every other tag exactly as `json` output shows it: channel, mentions, edit targets, diff metadata. - Non-event reads return the same output as `json`. - **Default for agents.** Without `--format`, the CLI falls back to `BUZZ_OUTPUT_FORMAT`; an empty or unknown value means `json` with a warning, so a harness/CLI version mismatch never breaks commands. buzz-acp sets it to `agent` for every agent runtime through `default_agent_env` and forwards the effective value in the explicit MCP server environment, so operator or persona env still wins, and `--format json` still returns full signed events. buzz-agent adds the variable to its MCP env allowlist. - **Docs and cleanup.** The managed-agent skill doc (version bumped to 7) and `AGENTS.md` describe the new default. The two duplicate `format_events` copies in `messages.rs` and `feed.rs` are now one `commands/event_output` module, which also does the NIP-10 tag parsing. The `json` and `compact` output contracts are unchanged. ### Example output https://github.com/block/buzz/pull/8234#issuecomment-6259679347 ### Testing Started `buzz-acp run` with the deterministic task-agent fixture, then passed its `session/new` MCP environment to Codex 0.163.0 (`codex app-server`). With no format override the MCP child received `agent`; operator overrides of `json` and `compact` reached the MCP child unchanged. The probes used temporary local sessions and made no model requests. ### Related Same change on the other agent read surfaces: each drops `sig`, trims `auth` to `["auth", <owner>]` and keeps thread context. - block/buzz-app#1388: buzz-app MCP read lines (text, `claimed owner`/`reply_to`) - squareup/cash-server#139132: hosted agents' read tools (JSON events, raw NIP-10 tags) Generated with Codex --------- Signed-off-by: Nikolaj Goodger <ngoodger@squareup.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-21 12:54:05 +11:00
Will Pfleger	7e6461242a	feat(audio-client): accept a client request in connect (#8497) ... ## Summary Related: [block/buzz-app#1424](https://github.com/block/buzz-app/pull/1424) (native federated sessions; attaches the assertion on Huddle audio once this lands) `buzz_audio_client::connection::connect` took the audio URL as `&str`, so a host had no way to add headers to the WebSocket upgrade. buzz-app needs to send its NIP-FI user assertion (`Nostr-Federated-Identity: Bearer <jwt>`) on the Huddle audio socket. The relay checks NIP-FI when the audio connection upgrades (`audio/handler.rs` → `check_nip_fi_at_upgrade`). This PR widens the parameter to `impl IntoClientRequest + Unpin`, the same bound `connect_async_tls_with_config` already takes. Nothing else changes: TLS, timeouts and the NIP-42 handshake work as before. Existing `&str`/`&String` callers, including `desktop/src-tauri/src/huddle/relay_api.rs`, compile unchanged. --------- Signed-off-by: Will Pfleger <wpfleger@block.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>	2026-10-20 22:58:04 +00:00
Jason Holmes	a5767c67de	fix(sdk): reject malformed mention pubkeys before signing p tags (#6372) ... ## Summary `buzz_sdk::builders::mention_tags` lowercased whatever string it was handed and pushed it straight into a `p` tag, so malformed mention pubkeys were signed onto real events. A composer that mentioned someone this way looked successful while notifying nobody, and the bad tag propagated to every relay and client that read it. Reproduced before fixing: `""`, `"not-a-pubkey"`, `"zzzz"`, `"../../etc/passwd"`, and 63/65-character strings were all accepted and emitted verbatim as `p` tags by `build_message` and `build_forum_post`. Fixes #6291 ## Acceptance criteria → evidence Following @kiranmagic7's [recommendation on the issue](https://github.com/block/buzz/issues/6291): | # | Criterion | Evidence | |---|---|---| | 1 | Validate in the **private `mention_tags`**, not the public `normalize_mention_pubkeys` | `crates/buzz-sdk/src/builders.rs` — the single changed line inside `fn mention_tags`. `normalize_mention_pubkeys` is untouched, so no public signature changes. | | 2 | Reuse the existing **`check_pubkey_hex`** helper | `let lower = check_pubkey_hex(hex, "mention pubkey")?;` replaces `hex.to_ascii_lowercase()`. No new helper added. | | 3 | Reuse the existing **`SdkError::InvalidInput`** | `check_pubkey_hex` already returns `InvalidInput`; no new error variant is introduced. | | 4 | Cover all **3 builders** | `malformed_mention_pubkeys_are_refused_by_every_builder` asserts against `build_message`, `build_forum_post`, **and** `build_forum_comment` — all three share `mention_tags`. | | 5 | Reject **empty / non-hex / 63-char / 65-char** | Same test's table: `""`, `"not-a-pubkey"`, `"zzzz"`, `"../../etc/passwd"`, `"a"*63`, `"a"*65`, plus `"z"*64` and a 64-char path-like string so a length-only check cannot pass. | | 6 | Preserve valid **uppercase canonicalization** | `valid_mentions_are_canonicalized_and_deduped_across_case` — uppercase hex is accepted and the emitted `p` tag is lowercase, across all three builders. | | 7 | Preserve valid **dedupe** | Same test: the same pubkey in both spellings collapses to exactly one `p` tag. Pre-existing `message_mentions_deduped` still passes. | | 8 | Preserve **`TooManyMentions` precedence** for >50 | `too_many_mentions_takes_precedence_over_validation` — a 51-entry list that also contains a malformed entry still returns `TooManyMentions`. The cap check stays ahead of validation. `mentions_at_the_cap_still_sign` pins that the boundary itself did not move. | | 9 | Inspect **#3036 overlap**; keep the patch narrow | See the compatibility note below. | | 10 | Make these tests actually **gate** | `Justfile` + `scripts/run-tests.sh` — see "Why the runner change is here". | Out of scope by design: switching to `nostr::PublicKey::from_hex`. Its stricter on-curve rule would change every one of `check_pubkey_hex`'s ~26 call sites, which is a repo-wide consistency decision rather than part of this fix. The doc comment records that reasoning. ## Why validation lives in `mention_tags` `mention_tags` is the single choke point every mention reaches, shared by all three builders, and callers can hand those builders an explicit list without ever going through `normalize_mention_pubkeys`. Validating here closes that bypass without breaking the normalizer's public API. This also aligns the SDK with behavior already shipping elsewhere: desktop's own `events.rs::mention_tags` already validates with `check_pubkey` on the identical structural rule (64 ASCII hex). The SDK was the inconsistent one. One malformed entry refuses the whole list rather than being filtered out — a silently dropped mention is precisely the failure being fixed, and filtering would preserve it. ## Why the runner change is here Adding `buzz-sdk` to `just test-unit` and the mirrored `scripts/run-tests.sh` fallback is a CI-behavior change, so it deserves its own justification: **without it the regression tests above would never run in CI.** `buzz-sdk` is a workspace member, so it was getting `clippy` and `check`, but no CI job executed a single one of its tests. This is the exact hazard the `Justfile` already warns about for other crates — *"nothing in CI runs `cargo test --workspace` — workspace membership alone buys clippy/check, not a single executed test."* Verified with a positive/negative control rather than by reading the config: - **Negative control** — with the runner files at pristine `main` and a deliberately panicking `buzz-sdk` test in the tree, `just test-unit` exited **0** and printed "All tests passed!". The crate name never appeared in the log. - **Positive control** — same sabotaged test, with only these two runner edits applied: `just test-unit` exited **1** with `buzz-sdk tests FAILED`. The sabotage test was removed afterward; it is not part of this PR. Both entries are needed because the two lists (nextest path and the no-nextest fallback) must stay in step. This is scoped strictly to making the crate's existing tests execute — no other CI behavior is touched. ## Compatibility with #3036 #3036 (`fix/nip10-reply-p-tags`) also edits `mention_tags`: it adds `parent_author: Option<PublicKey>` to `ThreadRef`, emits a parent-author `p` tag in `thread_tags`, and seeds `mention_tags`'s dedup set from `p` tags already on `tags`. This PR replaces only the `let lower = ...` line inside the loop, so the two changes are compatible in substance. Mechanically, whichever lands second needs the trivial merge fixup: #3036 adds `parent_author: None` to every `ThreadRef` literal, including the ones in the tests added here. One correction to the paragraph above, from an independent merge trial: the fixup is not *only* the `parent_author: None` additions. Merging #3036 into this head also produces one **textual conflict in `crates/buzz-sdk/src/builders.rs`**, confined to the `mention_tags` doc comment — both branches rewrote it, this PR to explain the validation and #3036 to explain the seeded dedup set. The resolution is to keep both blocks; there is no logic conflict, since this PR changes the `let lower = ...` line and #3036 changes the `seen` initializer. For reference, merging #3036 into plain `main` leaves `builders.rs` auto-merging cleanly, so this conflict is specific to the combination. Once the doc comment is resolved and `parent_author: None` is added to the three new `ThreadRef` literals here, `buzz-sdk` compiles and the only failing test is #3036's own `self_reply_drops_the_parent_p_tag` — which fails identically with #3036 merged into `main` **without** this PR, because `build_message` gained `.allow_self_tagging()` in #4975 after #3036's merge base. ## Test plan - [x] `cargo test -p buzz-sdk` — **267 passed, 0 failed** (5 new tests; pre-existing `message_mentions_deduped`, `message_too_many_mentions`, and the three `*_preserves_self_mention_p_tag` tests still green) - [x] `cargo fmt --all -- --check` — clean - [x] `just clippy` (workspace, `-D warnings`) — clean - [x] `just test-unit` — all 10 steps pass, now including `buzz-sdk` - [x] `just file-size-check` — pass - [x] `just desktop-tauri-fmt-check` / `desktop-tauri-check` / `desktop-tauri-test` — pass (2684 tests; `desktop/src-tauri` consumes `buzz-sdk`) - [x] Full `just ci` — the remaining steps are JS/Dart (`desktop-test`, `desktop-build`, `web-build`, `mobile-test`); this change touches only Rust + two runner scripts, so CI covers them here All gate results above were confirmed at commit `5fe5929de39388cada354593ac875d3c42c3d830`. ## Open questions for maintainers Out of draft and ready for review. Two calls I'd like from a maintainer: 1. **Do the runner entries belong here?** Adding `buzz-sdk` to `just test-unit` and `scripts/run-tests.sh` is a CI-behavior change riding along with a bug fix. I kept it because without it the regression tests above never execute in CI (controls in the section above), but I'll split it into its own PR on request. 2. **Is leaving `check_pubkey_hex` alone right for now?** Switching to `nostr::PublicKey::from_hex` would add an on-curve check but changes the contract at all ~26 call sites, which reads as a separate repo-wide decision rather than part of this fix. ### A note on CI The only check reporting here is **DCO Check** (pass). `CI`, `Desktop Release Candidate`, and `Docker image` are all sitting in `action_required` because this PR comes from a fork — a maintainer has to approve the workflow run before they execute. That's the standard first-time-contributor gate, not a failure, and I can't clear it from my side. Every gate result in the test plan above is from local runs at `5fe5929de39388cada354593ac875d3c42c3d830`, so CI has not independently confirmed them yet. Signed-off-by: rust worker <cb8459a6e4c936835834cef03b8331fe806108dd03da97d39c7036a7d710e860@buzz.block.builderlab.xyz> Signed-off-by: Jason Holmes <holmes@squareup.com> Co-authored-by: rust worker <cb8459a6e4c936835834cef03b8331fe806108dd03da97d39c7036a7d710e860@buzz.block.builderlab.xyz>	2026-10-20 13:44:19 -07:00
Luke Tornquist	fcdabcf774	docs(db): database target architecture and schema-change rules (#8273) ... ## Summary Adds `docs/database.md`, which covers two things: - **The target architecture.** `schema/tables/public/` is the declarative source of truth, written within the constraints that out-of-band, online schema tools can apply. Deployments at scale apply schema changes out of band; auto-migration stays supported for smaller and self-hosted deployments, and `migrations/` stays. The database has no triggers, functions, or foreign keys; that behavior lives in the owning application transaction or in durable application work. Data repair runs through the backfill framework in `docs/data-backfills.md`, not through migrations. - **Rules for schema changes made now:** 1. Add no triggers, functions, procedures, rules, or foreign keys, with no exceptions. New tenant-owned tables are fenced by community admission in the application. Startup and deletion validation already tolerate missing fences, but desired-state bootstraps still attach a fence trigger to every `community_id` table, so a new tenant-owned table is blocked until that loop stops (part of the fence-retirement work). The doc includes checklists for adding a tenant-owned table and a provenance-only table. 2. Migrations change schema only. They never mutate existing rows. Until the backfill framework ships, a repair may be a reviewed, idempotent, batched one-off script in `scripts/maintenance/` that takes the community deletion lock and touches only active communities. A repair that writes historical channel events must keep replica routing closed until its WAL has replayed. 3. Update every schema location: `migrations/`, `schema/tables/public/`, the manifest and sibling directories, the pgschema reconcile script where it applies, and both homes of the existing `push_gateway_*` tables (the target is for the gateway to own its tables alone). Never edit a merged migration. Add catalog-level parity tests for what you touch. 4. Every migration must be safe for the relay that is currently deployed, under both supported modes: auto-migration at startup and out-of-band application. Either way the old relay runs against the new schema for a while. Because auto-migration can apply expand and contract in one boot, a contract migration must fail closed when its backfill has not run, and a contract that depends on a writer release (such as dropping a trigger whose work moved into the app) must name that release as a required stop in the release notes, because no row check can prove it. The order is expand, deploy, backfill, then contract. The doc includes a table of changes that break the old relay and the safe form of each. 5. Bound every lock, build `migrations/` indexes concurrently (the declarative schema uses plain `CREATE INDEX`), and keep changes within what out-of-band apply can make online. AGENTS.md (Product Contract), ARCHITECTURE.md (buzz-db), and new path-local `migrations/AGENTS.md`, `schema/AGENTS.md` (a pointer), and `crates/buzz-push-gateway/migrations/AGENTS.md` point to the doc. The path-local files mean agents editing those directories get the rules automatically. AGENTS.md also had a stale claim that migrations are "auto-applied on relay startup". This PR corrects it: `BUZZ_AUTO_MIGRATE` defaults off in the binary and on in the Helm chart, and both modes are supported. ## What got simpler The rules were scattered across `schema/schema.sql` comments, AGENTS.md Common Gotcha #7, the backfill spec, `docs/thread-window-deployment.md`, and test names. They now live in one document, and the rest link to it. No code changes. ## Testing This PR changes documentation only. I checked links and diffs, and every claim in the doc points to code at the base commit: the layout and parity tests in `crates/buzz-db/src/runtime/migration.rs`, `EXPECTED_SCOPED_TABLES` and the serving-catalog check in `crates/buzz-db/src/store/deletion.rs`, and `BUZZ_AUTO_MIGRATE` in `crates/buzz-relay/src/main.rs`. The new `AGENTS.md` files aren't `.sql`, so neither the sqlx migrator nor the schema layout test picks them up; CI's layout, migration-count, and deletion-parity tests pass. At an earlier head, a real relay against isolated PostgreSQL 17 showed that a new unfenced `community_id` table serves normally and makes community deletion report `unknown=<table>`. Since then, the fence requirement at startup was removed on main; the doc now cites `migration_schema_fence_objects_are_optional_for_serving_and_deletion` for that. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-20 15:47:05 -04:00
Ian Oberst	ca3be55018	fix(relay): advance the partition catch-all by default (#8464) ... ## Summary The migrations leave `events` and `delivery_log` with a catch-all partition that already covers every month from 2026-07. Only catch-all advancement replaces it with monthly partitions. Advancement was off by default, so any deployment that left out `BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED` routed every new row into the catch-all. An operator then had to repair it by hand: drain the catch-all, swap it for monthly partitions, and re-attach. This has already happened in one staging environment. - **Advancement on by default.** `BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED` now defaults to `true`, and an explicit `false` still disables it. `BUZZ_PARTITION_MANAGER_CREATE_ENABLED=false` remains the kill switch for all automatic partition DDL. - **Prompt retry while the current month routes to the catch-all.** If a startup or periodic pass leaves the current month routed to an empty catch-all that the maintenance planner would replace, the audit loop retries on its existing backoff. That is 5 s, doubling to a 60 s cap, for at most 8 retries (about 5 minutes). After that it returns to `BUZZ_PARTITION_AUDIT_INTERVAL_SECS`. Before this change, a failed first pass, such as a lock timeout during a rollout, waited the full 15 minutes. - The routine monthly advance, where only the last lookahead month falls into the catch-all, stays on the normal interval. Each attempt queues an `ACCESS EXCLUSIVE` request on the parent, so it shouldn't be retried without urgency. - The budget resets once the catch-all no longer awaits advancement. - A failed attempt keeps the last known state. - Layouts the planner refuses, such as a catch-all that holds rows, need an operator, so they are never retried early. Neither is a policy with create or advance disabled. - The check is `PartitionTableAudit::awaits_catch_all_advance(policy)`, plus a `PartitionAudit` helper that applies it to every table. It is derived from `plan_maintenance`, so it can't drift from the planner's rules. - **Fresh-install test.** A new `#[ignore = "requires Postgres"]` test runs every migration in a dedicated database. It checks that both tables start out waiting on the catch-all and that create-only maintenance does not fix them. It then checks that the first pass with advancement produces canonical monthlies, an empty `{table}_p_future`, trigger parity and a non-degraded audit, and that a second pass is a no-op. ### Related issue None found. ### Testing **Automated:** - `cargo test -p buzz-db --lib partition`, including `--ignored` against local Postgres: 21 unit tests and 55 Postgres tests pass. - `cargo test -p buzz-relay` for the config tests and the partition schedule tests passes. - `cargo clippy -p buzz-db -p buzz-relay --all-targets -- -D warnings` and `cargo fmt --check` are clean. - Mutation checks on `awaits_catch_all_advance`. Each fails the new unit test: - matching any window month instead of the current month; - replacing the planner check with the policy flags alone; - short-circuiting the current-month gate. **Local relay** (`BUZZ_AUTO_MIGRATE=true`, no advance env var): - **Fresh database:** startup committed `advanced` for both tables, creating `p2026_07` through `p2027_04` plus `_p_future`. - **Failed first pass:** I held `LOCK TABLE ONLY communities IN SHARE ROW EXCLUSIVE MODE` for 4 s across startup. The startup pass failed with `lock_timeout` at 17:07:20.8, and the first periodic retry committed `advanced` for both tables at 17:07:29. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Ian Oberst <ioberst@block.xyz> Co-authored-by: Ian Oberst <ioberst@block.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-20 16:20:18 +00:00
Luke Tornquist	039e1c8d59	feat(relay): write the NIP-43 member list in the membership transaction (#8482) ... ## Summary The relay-signed NIP-43 member list (kind:13534) was published after each membership change had already committed. If that publish failed, or if a change never republished at all (startup `bootstrap_owner`), the list stayed wrong until the background reconcile sweep found and fixed it. #8467 moved that sweep off the startup path. This PR removes the reason the sweep exists. Now every `buzz-db` function that changes `relay_members` writes the new list **in the same transaction**, as its last statement before commit: - It takes the per-community list lock last, reads the members, signs the list, retires the old one and inserts the new one. - If the list write fails, the membership change rolls back and the caller gets an error. - A call that changes nothing (already a member, role mismatch, host exists, limit reached) writes no list. The functions return `MembershipWrite<T> { outcome, snapshot }`. Callers push `snapshot` to live subscribers after commit. That push is the only thing left outside the transaction; if it fails, live clients see the change on their next fetch, and the stored list is already correct. Write paths covered (RPR #2, every write path): `add_relay_member`, `claim_relay_membership`, `claim_relay_invite`, `remove_relay_member`, `remove_relay_member_if_role`, `update_relay_member_role`, `bootstrap_owner` / `provision_owner`, `transfer_ownership`, `backfill_from_allowlist`, and the owner insert in `create_community_with_owner[_and_nip_fi_issuers]`. Community deletion removes the members together with the community, so it writes no list. The relay keypair is passed in explicitly instead of being stored on `Db`, so the compiler finds every caller. ### Behavior changes - **Open relays get an accurate list too.** Provisioning and ownership transfer used to publish only when `require_relay_membership` was on. The write is now unconditional. - **Same-second changes can't reuse an id.** The list's `created_at` is one second past the newest earlier list for that community, including retired ones, or the current time if that is later. Before, adding and then removing a member in the same second could produce an event with the same id as an earlier list. The insert then failed, and the stale list stayed live. A burst of changes can push `created_at` ahead of the clock by up to one second per change; it returns to the clock once the burst ends. - **`buzz-admin add-member` / `remove-member`** no longer build their own list (`publish_membership_list_with_bump` is deleted). They push the list the database wrote. Concurrent CLI runs now serialize on the database lock rather than depending on `run.sh` running one at a time. - **Owner mutations lock the community row `FOR NO KEY UPDATE`.** `transfer_ownership` and `bootstrap_owner` used `FOR UPDATE`. The list insert takes `FOR KEY SHARE` on that row through `events.community_id`'s foreign key, so a repair holding the list lock, or a removal or role update holding a member row, deadlocked with a concurrent owner change. `FOR NO KEY UPDATE` still conflicts with archive, unarchive and deletion admission, so the lifecycle ordering is unchanged. - **Startup delivers its list to other replicas.** When backfill or `bootstrap_owner` changes members at startup, the relay fans the newest list out through pub/sub once its state is ready, so clients connected to other replicas during a rolling deploy see an owner rotation without a refetch. - **New `buzz-admin republish-members`.** Rewrites the community's list from the stored members in one community-write transaction and pushes it live. This replaces the sweep as the repair for a list that is missing or stale: a `relay_members` change made directly in SQL, a restore, or a deployment that turns on `BUZZ_REQUIRE_RELAY_MEMBERSHIP` after running without it. ## Rollout Nothing rewrites a list automatically once the sweep is gone. A community whose list drifted before this deploys stays wrong until its next membership change or until an operator runs `buzz-admin republish-members`. Either deploy #8467 and let its sweep complete one pass first, or run `republish-members` for each community after this deploys. The command repairs only the community that `RELAY_URL` maps to, so on a relay hosting many communities, run it once per community host. ## What got simpler Deleted: - `reconcile_nip43_membership_snapshots`, `spawn_nip43_membership_reconciler`, its leader lock key and #8467's standby retry - the three `nip43_membership_snapshot_needs_reconciliation*` drift reads (one already deprecated) - `publish_nip43_membership_list` / `publish_nip43_membership_locked`, the separate after-commit republish at every call site, and `buzz-admin`'s duplicate publisher - the reconcile metrics `buzz_nip43_membership_reconciliations_total` and `buzz_nip43_membership_reconciliation_failures_total` - the after-commit publish metrics `buzz_nip43_membership_publications_total` and `buzz_nip43_membership_publication_seconds`, and the `publish_nip43_membership_locked` transaction metric label. A failed list write is now a failed membership write and shows up in the caller's existing error handling and the transaction metrics of its write path. - the `BUZZ_NIP43_RECONCILE_INTERVAL_SECS` setting There is now one way the list gets written, and it can't disagree with `relay_members` after commit. **Unused deploy config (outside this repo):** hosted deployments that set `BUZZ_NIP43_RECONCILE_INTERVAL_SECS` can drop it. It is now ignored. Dashboards or alerts on the four deleted metrics will stop receiving data. ## Testing New Postgres tests (`buzz-db`): - `relay_members::every_membership_write_commits_its_list`: for each write path, the returned list is the single live kind:13534 and names exactly the committed members. No-op calls return no list and leave the live list alone. - `relay_members::a_failed_list_write_rolls_back_the_membership_change`: holds the list lock, waits for the join to block on it, cancels that statement, and checks that the member was not added and the previous list is still live. - `relay_members::concurrent_joins_all_land_in_the_list`: 8 concurrent joins; the final list names all 8. - `relay_members::same_second_return_to_an_earlier_membership_writes_a_new_list`: add, remove, add, remove within a second; every list is newer than the last and no id repeats. - `relay_members::owner_transfer_and_list_repair_both_commit` and `relay_members::owner_transfer_and_member_removal_both_commit`: hold a repair (list lock) or removal (member row) open, wait until a transfer blocks behind it, then finish it. Both transactions commit; with `FOR UPDATE` back in place, one of them fails. - `community::create_community_with_owner_is_atomic_and_create_only` and `relay_invite::bounded_claim_exhausts_and_existing_member_retry_does_not_consume` now also check the list. Each new test was checked to fail against a mutation of the new code that brings back the old behavior (details in the thread). Stacked on #8467. Merge order: #8467, then #8468, then this. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-20 11:39:13 -04:00
Luke Tornquist	c99872f749	fix(relay): delete the startup NIP-33 d_tag backfill (#8468) ... ## Summary Delete the NIP-33 `d_tag` backfill that ran on every relay boot. On every boot, before the listener opened, the relay ran: ```sql SELECT DISTINCT community_id FROM events WHERE kind BETWEEN 30000 AND 39999 AND d_tag IS NULL ``` No index covers `d_tag IS NULL`. The only `d_tag` index is the partial `idx_events_parameterized`, which is `WHERE d_tag IS NOT NULL`. So this query scanned every partition of `events`. In production it averaged about 11s per boot, max 19s (`buzz_startup_phase_seconds{phase="d_tag_backfill"}`), and found nothing. These rows shouldn't exist anymore: - The backfill was the one-time migration for #246, the change that added `d_tag`. - In `crates/buzz-db/src`, every production write path sets the column for kinds 30000–39999 through `extract_d_tag`: the main insert, `replaceable.rs`, `channel_members.rs`, `relay_members.rs`, and `push.rs`. A new source test, `every_production_event_insert_names_d_tag`, now enforces that. - The `events` inserts that omit `d_tag` are all test fixtures. `scripts/backfill-d-tag.sql` is still there as the manual fallback for any database that predates #246 and somehow never booted a relay since. ### What got simpler - Removed `Db::backfill_d_tags` and its admission test. - Removed the source-shape assertion in `observability_source.rs`. - Removed the `d_tag_backfill` startup phase. - `StartupStep::ALL` is now a slice, so removing a step no longer means editing a hand-kept array length. ### Notes - The `buzz_startup_phase_seconds{phase="d_tag_backfill"}` series stops reporting. - **Rebased onto main after #8467 merged.** Both remove a startup phase. With the fixed-length `ALL` array the two merged cleanly in git into a 17-entry `[Self; 18]`, which doesn't compile. This PR makes `ALL` a slice. - **Before merging**, run this read-only query against production and confirm it returns `0`: `SELECT count(*) FROM events WHERE kind BETWEEN 30000 AND 39999 AND d_tag IS NULL`. It scans every partition, so use a read replica. If it isn't `0`, run `scripts/backfill-d-tag.sql` first. The cutover script copies legacy `d_tag` values as-is, so only this query proves historical data is clean. - **Self-hosted installs** should run the same check before upgrading. ### Related issue None found. Companion to #8467, which also targets relay startup time. ### Testing - New `every_production_event_insert_names_d_tag` (`crates/buzz-db/tests/observability_source.rs`): every production `INSERT INTO events` must name `d_tag` in its column list. Dropping `d_tag` from `push.rs` makes it fail and name the function. - Runtime E2E against a real relay: boot has no `d_tag_backfill` phase; kind:30023 replacement via WebSocket and HTTP keeps only the newer event, an unrelated `d` survives, and an older replay can't win (confirmed in SQL). - Stacked on #8467 at `501dfd83`: a fresh two-replica boot runs all enabled startup phases, with neither `nip43_reconcile` nor `d_tag_backfill` present. Both replicas pass readiness and return the same signed message over authenticated HTTP and WebSocket, and outsiders are denied. - Blox, at `501dfd83`: `just fmt-check clippy security-review-check file-size-check` passes. Every `test-unit` step and `just test` pass except the `buzz-acp` `git_bootstrap` native-git tests, which fail the same way at base `b6aa5539` on Blox, and one `buzz-agent` catalog timing test that fails only under full-suite load and passes when run alone. - `e27139de` adds only a doc comment on `every_production_event_insert_names_d_tag`, saying which SQL shapes the text match covers. No code changed, so the results above still apply. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-20 11:22:09 -04:00
klopez4212	c9e35f8ac7	fix(mobile): restore Solo chats from channel labels (#8466) ... Restore Solo chats and project placement on mobile using `buzz.solo` labels and creator metadata. Invalidate stale channel details when identity changes so opening and sending preserve refreshed Solo identity; shared chats retain ordinary sending. New mobile Solo chats publish the discovery label, and the personal send guard enforces ownership even when route hints or legacy placement suggest Solo. The current desktop producer lives in `block/buzz-app`. [Contract documentation](0201ca0572/docs/solo-mobile-contract.md) links its production creation call to relay identity generation and mobile regression coverage. Public metadata fields are documented. Validation: clean analysis; 3,542 mobile tests (5 skipped), 4 push checks, and 280 core unit/doc tests passed. Removing the cache fix reproduces the send failure. Local push hooks were skipped with owner approval after an unrelated terminal timeout and disk exhaustion; full local CI is not green. GitHub CI must validate the pushed commit. The initial build was installed/launched on iPhone; follow-up native and live cross-client confirmation remain pending. The ownership regression supplies a valid owned member recipient and checks the exact rejection. A New chat UI test captures the real signed kind-9007 command. Both regressions were mutation-verified: removing the ownership guard or the production creation label causes the corresponding test to fail. --------- Signed-off-by: kenny lopez <klopez4212@gmail.com> Co-authored-by: Terp <5856f6900e8e0a4e0ea1f4f8d2ee33e4753d2119bc122f94ecafc660b46c0511@buzz.block.builderlab.xyz>	2026-10-20 11:10:24 -04:00
Logan Johnson	25bcb791b4	fix(relay): show only legacy channel types to reads that name no channel (#8469) ... 🤖 ## Summary Every Buzz channel has a **type**, such as an ordinary chat channel (`stream`), a forum, a DM or a workflow. A client app can only show the types it was built to understand. This PR makes the relay (the Buzz server) follow one rule everywhere: **a client sees a channel only if it knows that channel's type.** #8406 applied this rule to membership notices. This PR applies it to the reads that list channels without naming one: the channel list, "channels I belong to", the sidebar's unread list and artifact lists. Before, these reads left out one type by name: system channels. Now they return only the four types that every client has always known: `stream`, `forum`, `dm` and `workflow`. Today the result is the same for every user, because `system` is the only other type. ### Backward compatibility: old clients stay safe - A client that was released before a new channel type exists will never receive channels of that type in its lists. It cannot show such a channel correctly, so it does not get one. - Requests that old clients send today return exactly what they returned before. Nothing changes for them now or when a type is added later. ### Forward compatibility: new types can be added safely - Before, the rule was a deny list: "everything except `system`". Each new type would appear in every deployed client by default, unless someone remembered to update five separate checks in the relay. A missed check would show the new type to clients that cannot handle it. - Now the rule is an allow list that is frozen: the four original types. A new type is hidden by default, with no relay change needed. Clients that understand it can request it on purpose, the same way current clients request system channels today. Each new type gets its own opt-in, so adding a type cannot break clients that are already in use. ## Details - One frozen list, `buzz_core::channel::LEGACY_CHANNEL_TYPES`. Membership notices (#8406) now use the same constant, so channel reads and notices cannot drift apart. The duplicate `LEGACY_NOTICE_CHANNEL_TYPES` is removed. - One SQL helper, `legacy_channel_type_sql`, builds `channel_type IN ('stream', 'forum', 'dm', 'workflow')` from that list. It replaces the five `<> 'system'` checks: - `get_accessible_channel_ids` (both halves): the implicit channel scope of any filter without `#h`, for example kind 39000 channel metadata, REQ, COUNT and search; - `get_accessible_channels`: the channel list; - the personal-read sidebar roster; - artifact queries. - A filter that names a channel (`#h`, or `#d` on kinds 39000–39003) reads it under the ordinary rules, whatever its type: the reader is a member, or the channel is open. The uncached named-channel lookup, `get_readable_named_only_channels`, now returns every readable non-legacy channel with its type, not only system channels. Before, a non-member could not read an open later-type channel, and `#d` read nothing. Workflow run reads and thread windows use the same lookup. - The relay keeps each named channel in the scope of the filter that names it only. This also holds when it re-checks membership for a named channel that the cache lacks. - The opt-in to system channels (`#t:["system"]` or `#P` on kinds 39000–39003) adds readable **system** channels only. It never adds a channel of another type, even one that a sibling filter in the same request names. The relay scope type `NamedOnlyChannels` keeps the type with each channel, so the opt-in can tell them apart. - NIP-CL and NOSTR.md state the rule. ### Related issue Follows #8406 (the same rule for membership notices). ### Testing New Postgres tests use a scratch database with a `future_type` value added to `channel_type`, as a later migration would add one: - `reads_that_name_no_channel_leave_out_a_type_added_later`: the channel scope, the channel list and the sidebar roster leave out the later type and keep the legacy types. - `artifacts_of_a_later_channel_type_are_read_only_by_name`: artifact current and history queries and counts without `#h` leave out the later type. With `#h` naming the channel, they return it. - `channels_of_a_later_type_are_read_only_by_name`: production read paths (WS REQ, WS COUNT, HTTP `/query` and `/count`). A member reads a private later-type channel and anyone reads an open one, by `#h` and by `#d`; an outsider cannot read the private one; a read that names no channel returns neither. In one request, a filter that names the channel plus a `#P` or `#t:["system", <label>]` sibling: the sibling does not get it. - `a_named_channel_of_a_later_type_stays_in_its_named_filter`: the membership re-check for a channel joined mid-request, with an unnamed sibling and a `#P` sibling. Only the naming filter gets the channel. Each test fails when its predicate goes back to `<> 'system'`, when the named lookup loads only `system`, or when the opt-in adds every named-only channel. The #8406 notice tests and the system channel Postgres tests pass on this branch. --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-20 11:04:29 -04:00
Luke Tornquist	e6c56ee5ee	fix(relay): take NIP-43 snapshot reconcile off the startup path (#8467) ... ## Summary Before the relay opened its listener, it reconciled the NIP-43 membership snapshot for every active community. Each community costs two writer round trips, one after the other. On a production deployment with about 80k communities, this one step averaged 174s of a roughly 180s boot (max 324s). Pods in an availability zone farther from the writer took 3–4× longer than pods in the writer's zone. The step doesn't need to block startup. The snapshots live in the shared database, and the replicas that are already running serve the same snapshots. A new replica has nothing of its own to repair before it serves. The sweep before bind did the same work as the background maintenance sweep; only the metric labels differed. This PR: - Removes the NIP-43 sweep that ran before bind. - Moves the background loop into `side_effects::spawn_nip43_membership_reconciler`. Its first sweep now runs as soon as the task starts. Before, the loop skipped its first tick, so the first maintenance sweep waited a full `BUZZ_NIP43_RECONCILE_INTERVAL_SECS`. - Runs that loop on one replica only. The replica holding the `NIP43_RECONCILE_LOCK_KEY` session advisory lock sweeps; the others retry the lock every `NIP43_STANDBY_RETRY` (5s, capped at the interval). This reuses the usage-metrics leader lease (a detached Postgres session), renamed from `UsageMetricsLeader`/`try_lock_usage_metrics` to the generic `SessionLeader`/`try_lock_session_leader`. - Waits a full interval after each sweep (sleep after sweep), so a sweep longer than the interval never runs straight into the next one. - `reconcile_nip43_membership_snapshots` is now the only entry point and keeps its original signature (`-> Result<usize>`). It's no longer deprecated. ### What got simpler Deleted: - `Nip43ReconciliationPurpose` (the `Bootstrap`/`Maintenance` purposes) and `reconcile_nip43_membership_snapshots_with_purpose` - `Db::bootstrap_community_hosts` and `Db::nip43_membership_snapshot_needs_reconciliation_for_bootstrap` - The startup progress log (`NIP-43 startup reconciliation progress`, `nip43_progress_due`) and its test - The `nip43_reconcile` startup phase: `StartupStep::ALL` goes from 19 to 18 steps The leader lock reuses the existing usage-metrics lease rather than adding a second leader primitive. `StorageAccountingLeader` is still a near-duplicate of it; folding that in is left as identified debt. ### Trade-offs - **Repair before serving.** A newly started replica no longer guarantees it has repaired stale snapshots before serving. The running fleet doesn't guarantee that today either: the background sweep is what repairs drift, and the leader starts it as soon as it takes the lock. - **Rollout load.** Before, every replica swept before binding, and every replica also ran the background loop. Now there is one sweeper in steady state, so writer load from this job drops from one sweep per replica to one per deployment, and rollouts no longer stack boot sweeps. Standbys add one lock attempt every 5s each. - **Leader handoff.** When the leader's process exits (including SIGKILL), its connection closes, Postgres releases the session lock, and a standby retries the lock at least every `NIP43_STANDBY_RETRY` (5s) and sweeps as soon as it gets it. Each attempt also waits for a writer connection, so under database pressure takeover can take longer than 5s. Before this, a standby waited a full interval (3600s in production), so after a rollout replaced the leader nothing swept for up to an hour. If the leader is lost with its host or network instead, the lock is held until the database server's TCP keepalive drops the session, and nobody sweeps meanwhile. The usage-metrics leader already has the same exposure. The lease is checked only between sweeps, so a leader that loses its session mid-sweep finishes that sweep while the new leader starts one; the per-community publication lock keeps that overlap safe. - **Snapshot window.** A missing or stale kind:13534 snapshot is served until the leader's sweep reaches that community; before, a booting replica blocked until it was repaired. Admission reads canonical `relay_members`, so this window doesn't change who's admitted. It does affect clients that read the snapshot: desktop onboarding shows the membership-denied page to a member missing from it, the desktop/mobile member lists and join alerts lag, and desktop mesh discovery keeps its existing allowlist when the snapshot is missing. In production the inline publications that keep it current have not failed since 2026-10-07, and the sweep repaired no snapshots in the last 30 days. - **Dashboards and alerts.** The `buzz_startup_phase_seconds{phase="nip43_reconcile"}` series stops reporting. Anything charting or alerting on it should move to the sweep's existing counters (`buzz_nip43_membership_reconciliations_total`, `buzz_nip43_membership_reconciliation_failures_total`). ### Related issue None found. ### Merge order #8468 is stacked on this PR. Both remove a startup phase, and with the old fixed-length `StartupStep::ALL` array, merging both would have built a 17-entry `[Self; 18]` that doesn't compile. Merge this PR first; #8468 makes `ALL` a slice. ### Testing - `background_reconciler_sweeps_immediately` (private scratch database): spawns the production reconciler with a 24h interval and 24h standby retry, and requires the first sweep to publish a missing snapshot. Fails if the skipped first tick is restored. - `background_reconciler_standby_takes_over_on_short_retry` (private scratch database): with a 24h interval and 100ms standby retry, nothing publishes while another session holds the leader lock; once it's released the standby must publish within 10s. Fails if the standby sleeps the full interval again (checked by reverting that line). - `test_usage_metrics_lock_has_single_owner_and_releases_on_drop` covers the renamed lease. - `maintenance_sweep_skips_archived_and_deleted_communities` now calls the single entry point. - Runtime E2E at `574b28b3`: two real relays sharing one database, 3600s sweep interval, production 5s standby retry. While the leader was alive, the standby made only lock attempts (no sweep or snapshot-comparison reads). SIGKILL of the actual lock holder, 3/3 trials: the survivor took the lock in 4.81–4.86s and repaired the snapshot within 10ms after that. Both replicas served authenticated reads while the initial publication was stalled. - Blox, at `574b28b3`: `just fmt-check clippy security-review-check file-size-check` passes. Every `test-unit` step and `just test` (118 suites) passes except the `buzz-acp` `git_bootstrap` native-git tests, which fail the same way at base `b6aa5539` on Blox (environment, not this change). --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-20 10:19:46 -04:00
Wes	6e6173acd5	test(buzz-db): fix two PostgreSQL-lane timing flakes (#8472) ... ## Summary Two timing flakes in the PostgreSQL lane, one fix each. ### Deletion lock order `abort_serializes_before_fence_without_deadlock` failed on main ([run 39330176879](https://github.com/block/buzz/actions/runs/39330176879)): `post-lock lease verification rejects aborted request: 1`. The test slept 50ms and assumed `abort` had queued on the community advisory lock by then. On a slow runner, `fence` queued first, won the lock, and succeeded. The test now waits until `pg_locks` shows each contender waiting on that community's lock, `abort` first and then `fence`. Postgres grants queued advisory locks in arrival order, so `abort` always wins. The sleeps and their timing guess are gone. ### Saturated-reader fixture `routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold` failed this PR's first run ([run 39335038111](https://github.com/block/buzz/actions/runs/39335038111)): `establish the reader's sole connection: PoolTimedOut`. Setup opens the reader's only connection through the production lazy pool, which has a 150ms acquire timeout. On a loaded runner that first connect took longer, so the test failed before reaching the budget it measures. Setup now retries on `PoolTimedOut` within 5s. The pool and its 150ms budget are unchanged for the routed read under test. Test-only. Production code is unchanged. ## Validation - Deletion test against an isolated local Postgres 17 database: 40/40 passes. Saturated-reader test: 15/15. - `cargo fmt --check`, `cargo clippy -p buzz-db --tests -D warnings`, pre-push Rust unit tests: pass. - Three other `store::deletion::postgres_tests` fail locally on unmodified `main` as well (local environment). CI's PostgreSQL lane is the authoritative check for them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Wes <wesbillman@users.noreply.github.com> Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-19 23:40:43 +00:00
Wes	b6aa5539e3	feat(mobile): account onboarding, key recovery and hosted reconnect (#8375) ... ## Behavior Adds optional mobile Builderlab sign-in, identity backup/recovery, community discovery/creation, and account-aware reconnect. **Builderlab features are disabled by default:** when `BUZZ_AUTH_DOMAINS` is unset or empty, account login, creation, backup/recovery, reconnect prompts, and background account restoration are hidden/disabled. QR/code pairing, links, and invites retain Nostr-only entry. Saved account state is retained for later opt-in. `--dart-define=BUZZ_AUTH_DOMAINS=communities.buzz.xyz,communities.public.builderlabstaging.xyz` opts those DNS domains and their subdomains into required account setup. The same policy gates pairing, deep links, durable community admission, and reconnect. A nonempty list enables Builderlab features; `BUZZ_BUILDERLAB_URL` alone does not. Enabled flows still require their backend services. Existing Nostr relay authentication remains mandatory. `BUZZ_HOSTED_RELAY_AUTH=true` enables scoped relay assertions and reconnect prompts only for configured authentication domains. Configure assertion issuance and each community's issuer binding before enabling assertions; enable relay enforcement last. Account credentials stay at Builderlab. Wrong-account recovery preserves keys; failed writes are not replayed; authentication is bounded by request deadlines. Hosted flows preserve keys across cancellation/restart, reconcile interrupted creation/invitation acceptance, and support manual nsec file import/export. Account identities are scoped to backend and authenticated subject; reusing an existing local key for an unbound account requires an explicit selection. Recovery reconciles the advertised and recovered inventories before automatic selection. Successful self-hosted admission retires its pending key, and iOS startup removes interrupted export files. Adds authenticated relay `GET /operator/communities/joined` for the companion Builderlab membership endpoint. ## Validation - Follow-up review repair at `66a12ac92`: self-hosted entry presents readable choices and import despite incomplete enumeration, while generation still requires a complete empty inventory; explicit selection uses exact-key lookup. Consuming a value-equal duplicate now advances dispatch. Reconnect requires an existing account binding or a subject-scoped backup receipt before signing an assertion, and pins the subject inside the serialized account request. An empty different account is refused without sending key proof. Focused checks: configured domains 58 passed; default 49 passed, 9 hosted-only skips. Separate isolated mutations fail the incomplete-inventory sheet, exact-key reuse, duplicate dispatch and reconnect journeys (including an empty-account-only bypass); restored controls pass. Evidence uses synthetic storage/network boundaries, not physical Android or live backend association. All mandatory commit/push hooks passed; fresh GitHub CI is pending. - Bounded review repair at `d3cc0003f`: incomplete secure-storage enumeration now blocks automatic identity generation while explicit known-key reads retain their fallback; declined/failed onboarding links release the queue; HTTP 429/5xx keeps Retry regardless of response encoding. Default app-gate coverage exercises a bind link followed by ordinary community entry. Focused checks: configured domains 47 passed; default 38 passed, 9 hosted-only skips. Removing the three production fixes in an isolated source/Git/build copy fails the identity-resolution, declined/failed queue, and Retry journey assertions; clean and restored controls pass. Storage and HTTP failures are synthetic; physical Android and live outage behavior were not exercised. Published after all mandatory commit/push hooks passed (mobile analysis/tests, branch-skew and file-size gates); fresh GitHub CI remains pending. - Existing whole-PR source review cleared the reconnect defects at `fb15d5b82`. - Domain-policy increment: focused environment, entry, app-gate, pairing and onboarding journeys pass with empty and configured domains. Existing pairing provider/page tests pass. Changed Dart files analyze cleanly. - Isolated source/Git/build mutation: forcing paired credentials through hosted login fails the default-build journey; clean and restored controls pass. CI now runs configured-domain journeys in addition to the default-build suite. - Integrated main through `5b433d929` at `0067c069c`, including its pinned Flutter upgrade. Preserved the PR welcome UI with the updated transition API and added Material surfaces around onboarding selection/consent rows. Focused post-merge runs pass in both configurations; all mandatory local push gates passed (mobile analysis/tests, Rust tests, desktop-native checks, branch-skew and file-size checks). - Merge-review repair at `a315bce11`: focused identity/storage/admission and widget journeys pass, including account switching with an empty or already-bound second account, explicit reuse, and self-hosted removal/rejoin with another key. Default-domain journeys pass too (hosted-only cases intentionally skip there). - Isolated source/Git/build mutation checks: bypassing account scoping, reuse consent, pending-key retirement, or recovery inventory reconciliation fails the relevant journey/invariant assertions; clean and restored controls pass. - Native `RunnerTests/IdentityFilePluginTests`: both tests pass on the iPhone 18 Pro iOS 27 simulator with an isolated app identifier. At `24a0dcaac`, the picker journey seeds residue in Library/Caches, matching path_provider_foundation, and verifies plugin startup removes it while preserving an unrelated file. The earlier tmp-only fixture did not exercise the actual export location. This is synthetic native cleanup/picker evidence, not a real terminated-share or live account round trip. - Follow-up: Caches and tmp cleanup; fallback to individually known identity slots when Android bulk enumeration fails; explicit replacement of a refused self-hosted key preserves its original before swapping the retry slot. Configured-domain journeys/invariants: 34 passed; default: 31 passed, 3 hosted-only skips. Isolated enumeration/retry mutations fail with passing controls. Bulk-read fallback preserves but cannot enumerate arbitrary other-account slots until bulk reads recover. - Configured-domain CI exposed a stale pairing-status assertion after account scoping. The repaired test checks the admission mismatch, preserved account/transfer keys, and no admitted community. The configured-domain CI selection passes locally (56 tests); removing the production mismatch guard fails the updated test, with clean/restored controls passing. This is synthetic coordinator evidence, not physical QR acceptance. - Final review repair (`40e1e6101`): native iOS export creation uses atomic complete file protection; normal and restart cleanup remain. Account/policy/claim HTTP uses one 30-second total deadline with request abort and body cancellation. Removed the unrelated NIP-FA spec edit. - Focused account/admission/export tests: 25 pass; configured-domain account/admission/onboarding journeys: 33 pass. Isolated deadline and native-dispatch mutations fail, with passing clean/restored controls. Native export/picker tests: 2 pass on iOS 27 simulator. The real file-protection attribute assertion is physical-device-only because simulator provides no Data Protection attribute; locked-device protection and interrupted real sharing have not been observed. - Opt-in UI: configured-domain CI selection passes (57 tests); default selection passes (7 tests, 10 hosted-only skips); URL-only plus whitespace/empty domains passes (6 tests, 9 hosted-only skips). Changed Dart files analyze cleanly. Independent mutations that unconditionally initialize the coordinator in the app gate, welcome entry, or link form fail the default journey; clean/restored controls pass. - Native welcome screens exercised on iPhone 18 Pro / iOS 27 simulator in default and configured builds: default exposes community link / QR / pairing code only; configured build additionally exposes Builderlab connection and community creation. No live hosted login was exercised in this check. - Delivery at `e48fe73a9`: all mandatory push hooks passed (full mobile analysis/tests, Rust tests, desktop-native checks, branch and file-size checks). GitHub checks had not yet populated at the immediate post-push snapshot. ## Review disposition | Feedback | Disposition | | --- | --- | | Explicit self-hosted entry under unreadable inventory, duplicate invite dispatch, reconnect cross-account binding ([entry](https://github.com/block/buzz/pull/8375#discussion_r4318421622), [queue](https://github.com/block/buzz/pull/8375#discussion_r4318421640), [reconnect](https://github.com/block/buzz/pull/8375#discussion_r4318455512)) | Repaired at `66a12ac92`; evidence above. Reconnect refuses unproven ownership rather than silently establishing a binding. | | Incomplete Android identity inventory, stranded deep-link queue, transient HTTP retry classification ([review](https://github.com/block/buzz/pull/8375#pullrequestreview-5576864143)) | All three repaired in `d3cc0003f`; focused journeys and isolated mutation evidence above. | | Account cross-link, incomplete recovery inventory, committed invite checkpoint, pending local key retirement/replacement | Repaired in the preceding commits; journey and mutation evidence above. | | iOS interrupted key export ([review](https://github.com/block/buzz/pull/8375#pullrequestreview-5575253252)) | Native cleanup plus complete protection at creation implemented. Physical-device lock and real interrupted sharing acceptance remains pending. | | Trickle responses bypass idle timeout; unrelated NIP-FA edit ([review](https://github.com/block/buzz/pull/8375#pullrequestreview-5575764556)) | Total request deadline/abort implemented; spec edit removed. | | Production domains rejected | Ordinary entry now needs no Builderlab configuration. Opt-in account-required domains must still use supported community mappings. | | Lost Android Keystore recovery | Backup exclusions prevent the reviewed stale-restore scenario; genuine key loss still has no in-app recovery/reset and remains deferred. | | Native secp256k1 dependency question | Obsolete against the current base: upstream C integration landed in #8398; no package diff remains here. | | Operator membership endpoint contract | Documentation now states the operator chooses the pubkey; no end-user session binding is implied. Additional handler-level coverage is optional follow-up. | | Optional Android partial document cleanup/activity retention; local work after backup cancellation | Follow-up hardening, not claimed implemented. Cancellation prevents remote backup upload. | | Semgrep crypto inventory notices | Informational inventory, not demonstrated defects. | ## Remaining draft gates Native QR/browser-return acceptance remains unverified for this revision. Optional Builderlab onboarding still needs backend backup-status and joined-community routes, recovery policy/configuration, and real backup/recovery/reconnect acceptance. The empty-domain build removes these services as prerequisites for ordinary pairing; it does not make the optional hosted flows backend-ready. Human acceptance and current CI are still required before merge. Review dispositions retained: production auth-domain configuration must use supported community mappings; enabling `builderlab.xyz` does not add support for `buzz.block.builderlab.xyz`. Recovery after genuinely losing an Android Keystore key remains a follow-up, not a newly implemented capability. Human acceptance steps: build once without `BUZZ_AUTH_DOMAINS` (including with a previously saved account) and confirm no Builderlab controls/restoration while ordinary link and QR/code entry work. Build with supported domains and confirm connection/creation appear; exercise browser return and live backup/recovery/reconnect. On physical iOS, exercise export and interrupted sharing/locked-device protection. Record results before marking review complete. --------- Signed-off-by: Wes <wesbillman@users.noreply.github.com> Co-authored-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz> Co-authored-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@wes.communities.public.builderlabstaging.xyz>	2026-10-19 15:46:58 -06:00
Logan Johnson	cc6336d985	feat(relay): tag every membership notice with its channel type (#8406) ... 🤖 ## Summary - When someone is added to or removed from a system channel, the relay now sends them the usual "you were added / removed" notice. Before, it sent nothing, so a client could learn about a new system channel only by rereading every channel. - **A client sees a notice only if it knows the channel's type.** Every membership notice now carries its channel type. Clients that do not send `#t` (every client today) see only the types that existed before system channels, so they see no change. A type that Buzz adds later is hidden from them by default. - The notices are stored like every other membership notice, so a client that was offline can catch up with `since` instead of rereading all its channels. ## Details - Every kind:44100 (added) and kind:44101 (removed) notice now carries `["t", <channel_type>]`, for example `["t","stream"]` or `["t","system"]`. - **One place reads the channel type:** `emit_membership_notification` in `side_effects.rs`, where the relay signs the notice. A channel's type is set when it is created and no code path changes it, so the tag stays correct. If the channel row is missing, the relay sends no notice rather than an untagged one. - **The rule (NIP-CL):** - a filter with `#t` matches a notice only when the notice's type is one of its `#t` values; - a filter without `#t` matches only the **legacy types**: `stream`, `forum`, `dm`, `workflow`. This list is frozen (`LEGACY_NOTICE_CHANNEL_TYPES`), so a new type never joins it; - a notice without a `t` tag was stored before this change. It is always a legacy type, so the relay treats it as one, and replay across the upgrade stays complete. - **Fails closed.** The rule lives in two shared places, and both default to the legacy types. A read path that is not updated behaves as it does today: - `filters_match` / `notice_visible` (buzz-core). Live fan-out, the REQ history post-filter, COUNT, the HTTP bridge and push all match through it. - `EventQuery::notice_channel_types` (buzz-db). Its SQL keeps only visible notices before `LIMIT`, so page limits and COUNT stay exact. `None` means legacy types. Only `filter_to_query_params` sets it, from the filter's `#t`. - A client that knows every current type sends one filter: `{"kinds":[44100,44101],"#p":[<me>],"#t":["stream","forum","dm","workflow","system"]}`. NIP-CL and NOSTR.md describe this. - **Only the target receives a notice.** A notice names the target, the channel and who made the change, and the relay signs it, so knowing its id is not permission to read it. 44100/44101 are now in `RESULT_GATED_KINDS`, the one list used by the `ids` exemption check, the per-result reader check (WS and HTTP history, push), live fan-out (same pod and Redis) and COUNT. This also closes the same id lookup for ordinary channels' notices. - No migration: 44100/44101 are already excluded from search indexing. ## Testing - `buzz-core`: a table test covers untagged, `stream`, `system` and a future type against filters without `#t`, with `#t` for one type, and with `#t` for every type. Other kinds keep the plain `#t` match. - Postgres tests (`system_channel_postgres_tests.rs`): live fan-out, REQ history, COUNT and the HTTP bridge, for a filter without `#t`, a `#t:["system"]` filter and an all-types filter; an untagged legacy notice; a non-target with an `ids` filter on each path; and a raw `EventQuery` that does not set the field. - I broke each part on purpose and the tests failed: no type tag on the notice, no rule in `filters_match`, no SQL rule, and no untagged clause in the SQL. - End to end: I built a relay container from f24f8e7ba and ran buzz-app's `container-relay.e2e.test.ts` (block/buzz-app#1290) against it. The live system channel test passed 5 of 5 runs. Paired client change: block/buzz-app#1290. It works unchanged against this relay. --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-19 16:29:32 -04:00
Will Pfleger	91e7b469b8	fix(relay): wait for the peer's Close before dropping any WebSocket (#8426) ... 🤖 A NIP-42 login refused for a NIP-FI key mismatch could reach the client as an abnormal close (`1006`, no reason) instead of the `NOTICE "restricted: authorization denied"` followed by close `1008` that every other authorization denial produces. The relay did write the NOTICE and the `1008` Close, but the connection task dropped TCP as soon as the writer finished, before the peer finished the close handshake. The client could then lose frames it hadn't read yet. It depends on timing, so it showed up intermittently and is not specific to key pairing. ## Relay route - `recv_loop` now hands back the read half even while a frame handler is still running. Frames are handled one at a time while the connection is live. Once it is cancelled, an in-flight handler (for example a Nostr `CLOSE` waiting on the subscription lock behind a REQ stuck in Redis) is spawned to finish instead of holding the reader. It is never dropped partway, so subscription retirement stays atomic, and cleanup joins it before the expiry barrier and application cleanup. - As soon as the writer finishes, `await_client_close` keeps the transport open until the client's Close arrives or the stream ends. It runs concurrently with joining the heartbeat, auth-timeout, expiry and auth-cancel tasks and the connection's application cleanup (subscriptions released, registry entry removed, presence cleared). Cleanup never waits on the peer, so a nonresponsive client cannot delay registry counts, presence or `has_admitted_connection`. The socket never waits on cleanup, so a stalled Redis (the pool has no wait timeout) or an expiry join blocked on its quiescence barrier cannot hold the socket and its file descriptor open past the bounded wait. The active-connection gauge and the connection permit are released after both finish. - The connection permit (`BUZZ_MAX_CONNECTIONS`) is now taken at the start of `handle_connection`, before the NIP-FI gate and the community-active bootstrap check, and held through the active handler or the refusal drain and its bounded wait. Previously a socket in bootstrap, or refused there, held no permit, so a burst of refusals could open sockets beyond the limit. Over the limit, the socket is still dropped with no frames. `buzz_ws_connections_active` still counts only connections that reached the active handler. ## Huddle-audio route - The relay and audio bootstrap drains share one bounded helper. A socket refused before admission waits after its community registry entry is released. - Every exit of the audio handler gets the bounded peer wait. Exits whose cleanup follows their terminal frames (`room_full`, `room_ended`, version mismatch, archived channel, owner rejection or unreachable, failed commit confirmation) and the normal teardown hand the socket to a background task that waits for the client's Close, so lease releases, `48102` writes and room archiving on Postgres and Redis never hold the socket open. The remaining exits send their frames last, and `handle_active_audio_connection` waits once they return. The wrapper joins the background task before returning, so the audio connection permit is held until the socket ends, as on the relay route. - Audio error exits now follow their existing JSON error with a Close frame: | Exit | Close code | | --- | --- | | `huddle_relay_draining` | `1012` service restart | | `join_rejected` (owner resolution failed), `huddle_owner_unreachable`, failed channel lookup | `1011` internal error | | `huddle_audio_unavailable`, `unsupported_version`, `upgrade_required`, `room_full`, `room_ended`, archived channel, owner rejection, NIP-FI authorization denial | `1008` policy | Audio login failures and authorization refusals outside NIP-FI `authorization_denied` still send only their existing error frame, then get the bounded wait. - The audio read half is an enum, so the hand-off has no `expect()` that can panic. ## Timing - The peer wait is bounded by `WS_TERMINAL_FLUSH_TIMEOUT` (1s), and drains to end-of-stream so a client-initiated Close gets its reply. The terminal writer flush has its own 1s budget, so the worst case for terminal transport work is about 2s per refused or closed socket. - Terminal frame writes (`flush_terminal_frames`, `send_exit_frames_bounded`) still give up on the first failure, but a failed or timed-out send is now logged at `debug` instead of being discarded. - The graceful-shutdown comment in `main.rs` now describes per-stage budgets (restart jitter, restart-Close acknowledgement, terminal flush, peer wait) rather than a per-socket total. The 30s hard drain bounds every socket, and the 35s process shutdown budget fits within the chart's 60s `terminationGracePeriodSeconds`. ## Tests Real-socket tests in the infra-free relay lane of `just test-unit` and `scripts/run-tests.sh` unless marked PostgreSQL: - The key-mismatch login and a session-expiry denial (as a control) both assert NOTICE then `1008` / `authorization denied` on the wire. - A raw-TCP client that never answers Close: the relay holds the socket for at least the bounded wait, has deregistered the connection by the time the socket ends, and ends it on its own under a 10s watchdog. - A raw-TCP client that withholds its Close until the connection has deregistered: the relay ends the socket in under the 1s cap, which fails if cleanup waits on the peer. - A raw-TCP client that never answers while final cleanup is stalled at a test pause point: the socket still ends after the bounded wait. - A raw-TCP client that never answers Close on the relay bootstrap drain, the audio bootstrap drain and an early audio-handler exit: the relay holds each socket for the bounded wait. - On the relay route with a one-slot budget through the real `handle_connection`: a refused bootstrap holds the slot until its socket ends, and a bootstrap stalled on the community check already holds the slot before the AUTH challenge is sent. - An established relay session whose reader is blocked in a Nostr `CLOSE` handler on a held subscription lock when the session expires: the socket ends within the bounded wait while the lock is held, and cleanup completes with the subscription retired once it is released. - An established relay session whose expiry task is stalled behind a held session effect permit: the socket still ends within the bounded wait. - PostgreSQL: an established audio session whose teardown is stalled after its Close, and a stalled archived-channel exit, both end within the bounded wait from the hand-off. - PostgreSQL: a fast refusal (unsupported protocol version) through the real `handle_audio_connection` with a one-slot permit budget keeps the permit taken until that socket ends. - PostgreSQL: `huddle_audio_unavailable` sends its error and then Close `1008` as the very next frame. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>	2026-10-19 16:13:27 -04:00
Will Pfleger	af7979e1ca	test(auth): read the clock once in NIP-FI until test (#8457) ... 🤖 The exact NIP-FI assertion-lifetime boundary test could fail when its JWT helper read the clock, then ES256 signing crossed a whole-second boundary. The token's `until` claim could land one second beyond the verifier's `now + skew + maximum_age` ceiling even though both were intended to use the same test instant. This changes the test helper to accept a captured instant and uses that instant for both the JWT time claims and `verify_at`. The boundary remains strict: `now + 3630s` is accepted and `now + 3631s` is rejected. No production code changes. Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Alia <d32955ad69077062930cc46cfe2df30ca9aaf6f8e76422681265e9e9af704d78@buzz.block.builderlab.xyz> Co-authored-by: Alia <d32955ad69077062930cc46cfe2df30ca9aaf6f8e76422681265e9e9af704d78@buzz.block.builderlab.xyz>	2026-10-19 17:37:12 +00:00
Will Pfleger	5b433d9292	feat(relay): resolve NIP-FI issuer bindings from the database (#8377) ... Stack: #8372 (merged) → this PR → #8399 → #8401 Makes relay login checks read the per-community NIP-FI issuer bindings that #8372 stores. Without this, the relay can't keep up with bb-public, where new communities are created at every signup and a static `BUZZ_NIP_FI_COMMUNITIES` list won't do. **Source switch.** `BUZZ_NIP_FI_COMMUNITY_SOURCE=database` resolves each community's trusted issuers from `community_nip_fi_issuers`, filtered by the issuer registry, instead of the static list. Each community gets its own policy ID: the base ID no longer hashes every community audience, and the final ID hashes the base plus this community's audience, so policy IDs change once (see `docs/operator-nip-fi-bindings.md`). `Config::from_env` refuses to start, in every mode, on an unknown source, on `database` together with `BUZZ_NIP_FI_COMMUNITIES`, and on `database` without `BUZZ_NIP_FI_PROVISION_BINDING`. `BUZZ_NIP_FI_RESOLVER_*` and `BUZZ_NIP_FI_SNAPSHOT_*` are validated in every mode, so a bad value fails the deploy that sets it. In `off` mode no resolver is built and no snapshot is loaded. **Resolver (`nip_fi_resolver.rs`).** - **Snapshot:** each pod loads every binding into memory, capped at `BUZZ_NIP_FI_SNAPSHOT_MAX_ROWS` (going over is a failed load). It refreshes every 30 s, start to start, with jitter that only shortens the delay and backoff after failures. Each attempt is limited by `BUZZ_NIP_FI_SNAPSHOT_REFRESH_TIMEOUT_MS`, which must be shorter than the interval. A refresh whose deadline passes before it publishes publishes nothing. The old snapshot is dropped off the runtime. - **Freshness:** a host in a snapshot loaded under two minutes ago is answered from memory. Any other host, and every host once the snapshot is stale, goes to a live lookup. Every answer carries a deadline two minutes after the read behind it started, and WS, HTTP and audio recheck it just before admitting. - **Live lookups:** bounded and single-flight. In-flight lookups are capped by `BUZZ_NIP_FI_RESOLVER_MAX_INFLIGHT` (default 8, clamped below the database pool size), and the pending hosts and their waiters have fixed caps. Positive answers are used for that admission only. A "not found" is cached for 5 s, up to `BUZZ_NIP_FI_RESOLVER_NEGATIVE_CACHE_MAX` hosts, and only when no snapshot was published and no binding was created while it ran. A lookup that panics still leaves the pending map. - **Removal:** a removed binding stops admitting new connections within 30 s plus the refresh timeout (40 s by default) while refreshes succeed, and within two minutes of the last good load while they fail. Open WebSocket sessions are not rechecked when a binding is removed; they end at their own session deadline. **Shadow mode** resolves through the same path as enforce, including the database lookup and the freshness rules. It keeps the resolved binding's deadline and checks it at the same point as enforce; a binding that goes stale during the tenant lookup is recorded as `stage=deadline, outcome=unavailable`, without changing the legacy response or closing the socket. With `source=database` its `community` metric label is only `bound` or `unmapped`; with `source=env` it keeps the existing per-community audience label, so current shadow dashboards are unchanged. Each request resolves its Host once: the router guard's resolution, failures included, is reused by shadow recording and admission. **Provisioning.** Both operator provisioning paths drop this pod's cached "not found" for the new host after a commit that bound issuers, so the creator's first login looks it up. **Startup and readiness.** With `source=database` and a mode other than `off`, the first snapshot loads before any listener binds, under its own startup phase `nip_fi_snapshot_load`. A failed load exits non-zero, so a pod is never ready without a snapshot. **Operational consequence:** boot already fails when the database is unreachable; what is new is that a snapshot over `BUZZ_NIP_FI_SNAPSHOT_MAX_ROWS` or a load past its timeout also crash-loops new pods. After two minutes the old pods' snapshots go stale and most logins fall through to the 8-slot live lookup, which sheds the rest with 503. The way out is raising the cap or timeout, or `mode=off`. **Metrics.** `buzz_nip_fi_binding_snapshot_*` covers loads, failures, rows and freshness. `buzz_nip_fi_binding_snapshot_loaded_timestamp_seconds` is written on every refresh attempt, so alert on `time() - x > 120` together with `absent(x)`. `buzz_nip_fi_binding_snapshot_rows` against `buzz_nip_fi_binding_snapshot_max_rows` gives the headroom to the row cap. `buzz_nip_fi_resolver_*` covers live lookups and the negative cache. The env vars and alerts are documented in `.env.example`, the chart README and `docs/operator-nip-fi-bindings.md`, which also has the staged off → shadow → enforce rollout for `source=database` with go/no-go signals at each step, and the over-cap and load-timeout failure modes with their ways out. **Regression coverage.** Router-level tests run the database resolver through the real `build_router` for root WS, audio and protected HTTP, in enforce and shadow mode. Over PostgreSQL and Redis, two pods using the env community source (two exact bindings) replay assertions retained from before a disconnect through a real root NIP-42 session, the audio deny-set check and signed NIP-98 requests: they are denied on both pods, and a root session upgraded before the disconnect is refused by the post-registration deny-set check. Postgres tests in `buzz-db` check that archived and quiescing communities, and issuers missing from the registry, never reach the snapshot or the live lookup. Process-level `boot_lifecycle` tests spawn the real relay with `source=database`: an over-cap or timed-out startup load exits non-zero with no health listener bound, and a loadable snapshot lets readiness turn ready. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>	2026-10-19 10:32:18 -04:00
Tom Brow	305fb7b5cd	fix(push-gateway): request default notification sound (#8431) ... Push banners arrive silently because the gateway's fixed APNs body never requests a sound, even though the iOS app requests sound permission. Add `aps.sound: "default"` to that constant so ordinary alerts request the system sound even if the notification extension is not invoked. Preserve the existing alert text, mutable-content flag, and class-blind, content-free delivery contract; synchronize its normative specification and executable models. Companion client workaround: [#8429](https://github.com/block/buzz/pull/8429). This gateway change also covers the original-payload fallback and requires no mobile source change. Sound remains subject to iOS settings and authorization. It does not bypass Focus or Silent Mode, select critical alerts, or guarantee vibration. Every public-profile wake requests the same sound, including generic/restricted fallback banners. ### Validation - Extended the real HTTP transport test: reproduced missing sound (`null` versus `default`) before the fix, then passed with the fix. All 79 runnable gateway library tests pass; 17 infrastructure/manual tests remain ignored. - Added a guard against drift between the emitted bytes, NIP-PL, and both formal models. Fixed-payload and stateful-delivery models and mutation checks pass. - Inspected the 0.24.0 (7) iOS source: sound permission is requested; ordinary enrichment and restricted fallback copy incoming content. A macOS UserNotifications probe confirms copy retention, but does not test iOS Intents or audible device delivery. - Focused existing native resolver/handoff tests pass (35 tests on macOS). These cover resolution and fallback behavior, not iOS sound playback. - Deployed exact head [b8a0eb6](b8a0eb6b78) to the personal dev push gateway (Railway deployment `b3e52fc8-d1d3-48d7-ae21-1a83c5db2e99`). Verified the dev relay already routes to that gateway. On an iPhone 12 mini running the existing Buzz Dev 0.22 (10140), without the client workaround, automation captured the test banner and Tom confirmed audible sound on October 18. - Broad local validation reached an unrelated ACP startup-deadline failure in `signals_cancel_startup_turn_and_pending_stdin_without_hanging`. No ACP source or dependency changed, and ACP does not depend on the gateway. The full local gate is not reported as passing. - Local desktop pre-push tests passed 3,420 cases but failed two unchanged agent-discovery checks (a 300 ms subprocess deadline and a login-shell PATH comparison). Desktop does not depend on the gateway. Retried publication with only the Rust-suite and desktop-Tauri lanes excluded under the standing unrelated-failure exception; other hooks remain enabled. ### Before merge - [x] Verify a real push banner and audible sound through the dev relay and candidate dev gateway on an iOS device without the client workaround. Signed-off-by: Tom Brow <tomb@block.xyz>	2026-10-18 15:59:19 -07:00
Bradley Axen	3ffbfa693c	feat(relay): start a channel with its members in one create (#8421) ... ## What A kind:9007 can now name the channel's first members with `p` tags. Those members are added in the same database transaction that creates the channel. Resending a create whose OK was lost is now accepted instead of refused. ## Why Starting a new Solo chat today takes about 6–8 sequential relay round trips before the first message goes out: create, read, add each agent (kind:9000), read the roster, then send. With members in the create, a client can start a chat with its agents as two writes: the create, then the message as soon as the create is OK. The resend fix makes the create safe for an outbox to retry. Today an identical resend after a lost OK is refused with `duplicate: channel already exists`, so the client can't tell success from a real UUID collision. ## How - **buzz-db:** `Db::create_channel_with_event` writes the channel, its owner, its members (role `member`, `invited_by` = creator) and the 9007 event itself in one transaction. A channel row therefore never exists without its create event. `create_channel_with_id` is unchanged for its other callers. - **Validation** (`side_effects::initial_members` and the `validate_admin_event` 9007 branch): - Each `p` must be a 32-byte hex pubkey. - Duplicates and the creator are dropped. - At most 32 members. - An `h` tag is required when there are members. - Each member goes through the same `decide_channel_add_policy` check a kind:9000 uses, so an agent that refuses third-party adds keeps the whole create out. - **Side effects:** each initial member gets the same `member_joined` system message and member-added notification a kind:9000 would post. The fallback in `handle_create_group` creates a bare channel when the pre-created row is missing; it announces no members, because it added none. - **Resend:** when the UUID is already taken, the same transaction checks whether this exact event is the stored create. - If it is, the create is OK'd as `duplicate:` (it is the same signed create). - Anything else, including a changed create from the same author, is refused and writes nothing, so it can never re-run side effects or announce members. - A resend that arrives while the first create is still uncommitted waits on its channel insert. It is confirmed once the first commits, or creates the channel itself if the first rolled back. Before this, a resend in that window was refused, and the app then withdrew a chat that existed. - **No compensation:** ingest no longer soft-deletes a pre-created channel when the event insert fails. The rollback covers it. - **Announcements:** a failed `member_joined` notice for one initial member is logged at `error` and the loop continues, so the members after it still get theirs. A resend writes nothing, so nothing would retry them. ## Compatibility A 9007 with malformed `p` tags, or more than 32 of them, used to be accepted with the tags ignored. It is now rejected with `invalid:`. The relay's own SDK and CLI builders send no `p` tags on 9007. ## Out of scope - The kind:9000 OK still doesn't prove the membership change committed: side-effect failures are logged and OK'd, and a duplicate replay skips side effects. The new path doesn't depend on kind:9000. - Events in separate HTTP requests are still unordered. Clients send the first message after the create's OK. ## Tests `store::channel::postgres_tests` (buzz-db): - `a_resend_beside_an_open_create_confirms_it_once_committed`: holds a create open, checks that the resend waits, commits, and expects `Exists { resent: true }` with one owner and one member. - `a_resend_beside_a_failed_create_creates_the_channel`: the same, but the first create rolls back. - `a_create_whose_event_fails_leaves_no_channel`: an event the insert refuses leaves no channel row, and a retry creates the channel. - `another_create_for_a_taken_channel_writes_nothing`. Two mutations, each caught: committing the channel before its event fails the "leaves no channel" test, and looking up the event before the channel insert waits fails the "confirmed once committed" test. `handlers/create_with_members_postgres_tests.rs`, all through the real ingest path: - `create_adds_its_p_tags_as_members` checks: - the roles and the 39002 roster - exactly one `member_joined` notice - that the agent can post with no separate add - `create_respects_the_members_add_policy`: an agent with policy `nobody` gets an `invalid:` rejection, and no channel is created. - `resending_a_create_confirms_it`: - an identical resend is accepted - a changed create (from the creator or from someone else) is refused - the roster is unchanged and nobody new is announced - Unit tests for `initial_members`: dedupe, creator excluded, bad hex, the 32-member cap, and the missing `h` tag. Each new test was mutation-checked against the change it covers: no members, no policy check, refuse every resend, and let a changed creator create through. Each mutation turns its test red. Agent exercise on a local relay, over `POST /events` the way the app sends: - The 9007 with `p` = agent was accepted in 54–97ms (debug build). - The kind 9 was accepted 12–15ms after the create's OK. - The agent was in the 39002 roster, and the operator listener delivered the mention. - An identical resend came back `duplicate:` (accepted), and a changed create was refused. Local suite runs at 10adcf6dd (the transaction fix), on PG14: - `postgres-test-run.sh -p buzz-relay -p buzz-db`: 28 failures. That is the same 27 as below, plus `personal_read_frontier_is_monotonic…`, which hit a statement timeout under load and passes when run alone. - `cargo test -p buzz-db -p buzz-relay`: the same 6 `api::media` failures and the flaky `mesh_demo` test. Earlier local suite runs: - New tests pass and clippy is clean. - `postgres-test-run.sh -p buzz-relay -p buzz-db`: 27 failures (partition maintenance, a huddle links test, a redis boot test), and the same 27 fail on a96ace4e5. - `cargo test -p buzz-relay`: 6 `api::media` failures (they need a DB at the default URL) also fail on base. - One `api::mesh_demo` test is flaky: it passed 2 of 3 branch runs and failed once on a timeout. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Bradley Axen <baxen@squareup.com> Co-authored-by: mbot <3754f8729004d95654c46dbab3129e4ab9ef05cc2534e2a3fbfc155983bd637b@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: claude00 <fb8b76a01257932b8ca35060862c29bd09b2137638573731c2d5074fc02c5385@buzz.block.builderlab.xyz> Co-authored-by: ART <ea39c1c8acf57180161ef4ba4ed68abf8589f5291956b909a5aa991fc66664ca@buzz.block.builderlab.xyz>	2026-10-18 11:32:09 -07:00
Will Pfleger	6286a54321	feat(relay): store per-community NIP-FI issuer bindings (#8372) ... Stack: this PR → #8377 → #8399 → #8401 bb-public creates communities all the time, so a hand-edited `BUZZ_NIP_FI_COMMUNITIES` map can't list them. This PR stores an explicit NIP-FI issuer allowlist for each community in the database and writes it when the community is created. Login checks don't read the table yet; that comes in #8377 together with the database-backed resolver. Operator steps are in [`docs/operator-nip-fi-bindings.md`](docs/operator-nip-fi-bindings.md). **Storage.** Migration `0066_community_nip_fi_issuers` adds `community_nip_fi_issuers (community_id, issuer, created_at)`, with primary key `(community_id, issuer)` and the community write fence attached. The assertion audience is `https://<communities.host>`, so the table has no `aud` column. Zero rows means no binding. The table is included in whole-community deletion and purge. **One bindable rule.** `BUZZ_NIP_FI_PROVISION_BINDING` (`{"host_suffix": ..., "authorized_issuers": [...]}`) is parsed in `buzz-auth` with `BUZZ_NIP_FI_ISSUERS`, whatever `BUZZ_NIP_FI_MODE` is set to, and both the relay and `buzz-admin` use the same parser. Each authorized issuer must be a configured issuer (1 to 2048 bytes), and the suffix must not cover any host declared in `BUZZ_NIP_FI_COMMUNITIES`; an unreadable declaration refuses startup even in mode `off`. `ProvisionBindingPolicy::is_bindable(host)` is true for a host exactly one label under the suffix that is not reserved. It decides creation-time binding, explicit `bind`, and what the lookup route reports. **Binding on creation.** Rows are written in the same transaction as the community insert on all three creation paths (`POST /operator/communities` `create_only`, its legacy convergence branch, and startup bootstrap), and only when the call actually created the community, so existing rows are never rewritten. **Reserved host.** A host whose `https://<host>` equals a configured `command_audiences` value never holds a binding. Operator provisioning of it is rejected with a 400 and the `reserved_host` error code, bootstrap creates it unbound, and the availability route reports it unavailable with an error code. `main.rs` calls `verify_reserved_hosts_unbound` at startup; if a reserved host has rows the relay refuses to start and names the fix (`buzz-admin nip-fi unbind --host <h> --issuer <i>`, or revert the command audience). A database failure during that check is a separate `Check` error. **Operator commands.** - `buzz-admin nip-fi backfill` pages unbound communities by host (1000 per page) and binds each page with one `INSERT ... SELECT` limited to active communities. It skips communities being deleted, always prints `{rows_bound, unbound, unbound_hosts, truncated}`, and exits non-zero while any bindable community is still unbound. Archived communities are bound on purpose. Both bind paths take the shared deletion lock and then lock the community row (`FOR NO KEY UPDATE`, in id order for a batch), the same order deletion uses, and the backfill insert also requires the community to still have no binding, so an operator `bind` that lands after the backfill listed a community wins and the default issuer is not added beside it. - `buzz-admin nip-fi bind --host --issuer` refuses a reserved host, a host outside the suffix, or an unconfigured issuer. - `buzz-admin nip-fi unbind --host --issuer` refuses to remove a community's last issuer (archive or delete the community instead), with one exception: when a valid provisioning policy shows that backfill can't bind the host (it's reserved, or outside the suffix), the last issuer can be removed. That is the reserved-host recovery path in the runbook. A missing or malformed policy doesn't block `unbind`; it just means the exception never applies. **Lookup route.** `GET /operator/communities/nip-fi-binding?host=` (NIP-98, `RELAY_OPERATOR_PUBKEYS`) reads the database and returns `{host, active, issuers[]}`. `host` is the stored host. It returns 404 for an unknown host, 200 with `active:false` for an archived or quiescing community, and `issuers:[]` when no provisioning policy is set. It lists only issuers that are configured and on a bindable host. kgoose depends on this contract; it is also documented in the runbook. **NIP-FA.** The `relay_url` bullet now also says: "An adapter MAY obtain its exact relay list from a configured, authenticated source; it MUST still compare `relay_url` exactly." **Deployment dependency.** The kgoose config side is already handled by [cash-server#137671](https://github.com/squareup/cash-server/pull/137671). In public-community mode it requires the command audience not to be a community audience. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>	2026-10-17 13:13:49 -04:00
Luke Tornquist	97e1f6a082	fix(db): validate channel roster snapshots in the app writer (#8316) ... ## Summary Phase 1 of retiring the channel roster snapshot trigger (`trg_events_guard_channel_roster_snapshot`, migration 0032). Every admitted writer that can store a kind 39002 roster snapshot now runs the same check the trigger runs, on the same transaction and under the same lock: - `LockedMemberSnapshot::replace_member_event` (relay's live roster publisher) validates the snapshot it already holds the membership lock for, before soft-deleting the old head. - `insert_event_on` and `insert_event_with_thread_metadata_tx` (generic event writes). - `replace_parameterized_event_in_transaction_impl` (inside its existing savepoint) and `Db::replace_addressable_event` (admin reconciliation path), after dominance checks and before the old head is deleted. The validator (`validate_channel_roster_snapshot`) is a direct port of `guard_channel_roster_snapshot()`: four-field `p` tags, 64-char hex pubkey, empty relay field, valid role, and an exact match against active `channel_members` rows. It raises the same `23514` messages, so callers still see `DbError::Sqlx(sqlx::Error::Database(..))` with the same code and text. **Lock order:** replacement key, then channel-membership key, then the canonical read, then the old-head `UPDATE` and the write. Legacy writers took the membership key later, at the trigger's INSERT, after the old-head row locks. Taking it before the `UPDATE` is not the same order, but it is strictly safer: it removes the row-lock/membership-key cycle. The membership writers in `channel_members.rs` take the same membership key before they change rows. Not every membership writer does yet: `execute_kick_with_marker` (reached from report enforcement) and `deploy_kick_member` in `relay_admin_actions.rs` set `removed_at` without the key, so a kick can commit after a roster passes validation and before that roster commits, leaving the removed member listed. The retained trigger has the same gap, so this PR neither causes nor closes it. When the trigger also runs it re-takes a key this transaction already holds; transaction advisory locks are re-entrant, so this adds no wait. The validator is one verdict query on the happy path. On failure it raises the trigger's exact error via a `DO` block, so mixed-version behavior is identical; phase 2 can switch to a typed `DbError`. **Unchanged on purpose:** the trigger, its function, migration 0032, and the startup probe `verify_channel_roster_fence_behavior`. Pods still on the old version depend on them. Phase 2 drops them once this version is running everywhere. ## Tests New PG tests in `channel_members.rs` `postgres_tests`, each with only the roster trigger disabled (the helper asserts the catalog check then fails): - `app_roster_guard_matches_trigger_for_roster_input_table`: 19 shape and membership cases (3/5-field tags, short and non-hex pubkeys, relay field, unknown role, duplicate, role mismatch, missing/extra member, uppercase hex, non-`p` tags, empty rosters, all roles). For each one, the app outcome through `Db::replace_addressable_event` must equal the trigger's outcome on a raw INSERT, including SQLSTATE and message. - `app_roster_guard_locked_writer_rejects_stale_and_accepts_current_without_trigger` - `app_roster_guard_covers_generic_and_parameterized_writers_without_trigger` - `app_roster_guard_rechecks_after_concurrent_membership_commit_without_trigger`: the 0032 race. A newer membership commits while an older snapshot waits on the membership key. The barrier is `pg_locks` with a bound, not sleeps. `stale_legacy_roster_cannot_replace_new_locked_snapshot` (`replaceable.rs`) and `desired_schema_rejects_stale_legacy_roster_role` (`channel_members.rs`) now simulate the old binary inline (replacement key, old-head `UPDATE`, raw INSERT) instead of calling the app API, so they test the retained trigger again. The replaceable.rs one keeps its one-connection pool, so it still catches self-deadlock. `observability_source.rs` gains `every_production_event_insert_runs_the_roster_guard`, which requires a real guard call in every production `INSERT INTO events` function, with explicit exceptions (NIP-43 membership and push lease rows, both `channel_id` NULL, and the two rolled-back startup probes), plus a self-test. Falsified: removing one guard call fails the source policy; dropping the role clause or `lower()` fails the parity table; disabling the trigger fails both legacy tests. The PG tests are picked up by the `postgres-ci` nextest filter (`test(/postgres_tests::/)`); the source policy runs in `just test-unit`. ### Coverage limits (review follow-up) - `every_production_event_insert_runs_the_roster_guard` is a presence check. It rejects missing, commented, and quoted calls, but it does not prove a call is awaited, reachable, or passed the event's real kind and channel. Behavior is covered by the PG tests above. The test now says this in its doc comment. - Exemptions match by function header. A kind 39002 insert added inside one of the four exempt functions would pass the policy. This is documented next to the exemption list. - The 19-case parity table and the race test drive `Db::replace_addressable_event` only. The generic, parameterized, and locked writers get the mismatch case only, and the parameterized path's membership-key-before-delete ordering is not exercised concurrently. All five writers call the same validator, so this PR doesn't widen these tests. - `disable_roster_trigger` commits `DISABLE TRIGGER` on the test database. That is safe under the per-test database harness the `postgres-ci` profile uses, but not against a shared database. Its doc comment now says so. ## Validation Run on a Blox workstation (private PG 16 and Redis) against a tree byte-identical to this commit: - `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo nextest run -p buzz-db --lib`: pass. - `cargo nextest run -p buzz-db --test observability_source`: 19 passed. - `scripts/postgres-test-run.sh -p <pkg>` for buzz-admin, buzz-audit, buzz-deletion, buzz-push-gateway, buzz-relay, buzz-search, buzz-workflow: pass. - buzz-db: the full run passed 513 and was stopped at a 240 s deadline on `writer_pool_rejects_non_read_committed_database_default`, which hangs the same way on the unmodified base. A second run excluding the three host-flaky runtime tests already reproduced on the base passed all 511 others. ## Simpler? Not yet. While the trigger is still in place, the check runs twice. Phase 2 deletes the trigger, its function, the startup catalog and behavior probes, the migration.rs equivalence asserts, and the pgschema reconcile entry, which leaves one check in Rust. ## Phase 2 (separate PR, after fleet rollout) **Prerequisite before claiming complete roster serialization (phase 2 or anywhere else):** make both kick paths take the channel membership key before they change rows, and add a concurrent kick-vs-roster-publication test. The retained trigger does not prove this; it has the same gap. A new migration drops the trigger from `events` and its partitions, then drops `guard_channel_roster_snapshot()`. It also removes the schema files and includes, `verify_channel_roster_fence*` and the relay startup call, the migration.rs asserts, and the probe tests, and updates `scripts/reconcile-schema-after-pgschema.sql`. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-16 17:07:25 -04:00
Luke Tornquist	60de1db0b8	fix(db): admit fleet sweeps per tenant instead of in-SQL fence predicate (#8318) ... ## What Removes every production call to the SQL fence predicate `community_write_allowed()` from application SQL, so no application mutation depends on the boolean predicate anymore. This is one prerequisite for BUZZ-315 part B; #8322 (tolerating absent fence objects) is a separate one. The triggers and SQL functions stay in this PR because mixed-version pods still depend on them. The predicate isn't inlined. It's a VOLATILE plpgsql function that takes the deletion lock and *then* reads lifecycle state on a fresh snapshot, and an inline expression would read state from the statement's start snapshot. A separate earlier check would race deletion. So each sweep now: 1. reads its candidate community IDs with a plain read, bounded the same way the original statement was, 2. opens one `AdmittedTx` per community (`begin_community_sweep_transaction`, a thin wrapper over `begin_community_write_transaction` that turns an admission refusal into "skip this tenant"; the four periodic sweeps go through `sweep_admitted_communities`, which also owns their error handling), 3. runs the original mutation scoped to `community_id = $1` and commits. | Site | Change | | --- | --- | | `runtime/push_enqueue.rs` | Deleted the redundant predicate. This transaction already calls `guard_transaction`. | | `store/relay_invite.rs` `reap_expired_relay_invites` | Keeps the global expiry order and the 1,000-row ceiling. Candidates are pre-filtered to active tenants so fenced invites can't fill the batch. Admission is still what authorizes the write. | | `store/workflow.rs` `prune_scheduled_workflow_fires_before` | Prunes per admitted community and sums the row counts. | | `store/channel.rs` `reap_expired_ephemeral_channels` | Archives per admitted community and returns the committed rows, so the relay still runs their system and discovery follow-ups. The candidate read has no lifecycle pre-filter (it has no `LIMIT`, so admission alone decides). | | `store/event.rs` `backfill_d_tags` | Backfills per admitted community at startup. | | `store/push.rs` `claim_due_match_batch_with_loader` | Picks the earliest due active community, admits it, and claims under the same `FOR UPDATE OF q SKIP LOCKED` and batch limit. It skips at most 16 tenants that fence between the read and admission. | | `store/push.rs` `reap_exhausted_matches` | Reaps per admitted community. | **Tenant failures.** The four periodic sweeps (invite reap, workflow fire prune, channel reap, exhausted match reap) log a warning and continue when one tenant fails. That tenant's rows are left unmodified and stay eligible, so the next tick retries them. A tenant error doesn't abort processing of the other tenants already in the selected candidate page. Invite cleanup keeps its oldest-first 1,000-row selection, so if one tenant owns that entire page and its delete keeps failing, tenants outside the page wait. No persistent tenant-local failure for that delete is known in the current schema, and base aborted the whole page on any error. The returned count or rows cover the tenants that committed. Claim, enqueue and the startup backfill still return the error to the caller. **Database outages.** A connection-level error (pool acquire timeout, closed pool, socket I/O; `DbError::is_connection_unavailable`) isn't tenant-local, so it stops the periodic sweep at that tenant instead of moving on. Without this, an outage that began after the candidate read cost one writer acquire timeout and one warning per remaining tenant, and invite cleanup runs inside the usage-metrics leader tick. Rows already committed are still returned, so reaped channels keep their follow-ups. Remaining tenants retry next tick. **Known limit (pre-existing, tracked separately):** - *Claim at a locked head.* `claim_due_match_batch_with_loader` returns `None` once an admitted tenant claims zero rows. If every due row in the oldest active tenant is row-locked, younger tenants aren't tried on that call. Base behaved the same way. This PR doesn't change it. ## Tests All tests use a one-session fixture (`test_support::admission_fixture`) with `session_replication_role = replica`, so the trigger backstop is off and only application admission protects the fenced tenants. The fixture shadows `community_write_allowed` with a function that raises, in a per-call `admission_test_<uuid>` schema. - One test per site: a quiescing tenant's rows are untouched while an active tenant in the same sweep still progresses. - Race tests for invite reap, channel reap and claim. These are the sites where the candidate read already filters out fenced tenants. Each test holds the tenant's deletion lock, waits until the sweep is blocked in admission, fences the tenant in that transaction, and asserts its rows are untouched. The claim test also asserts that a second active tenant is claimed in the same call. - `channel_reap_returns_committed_rows_after_earlier_tenant_error` and `periodic_reaps_continue_after_tenant_error_and_retry_retained_rows` inject a per-tenant failure. They check that later tenants commit, that the failed tenant's rows are retained, and that the next call processes them. - `fleet_sweep_stops_at_first_unavailable_connection` runs the sweep helper against a closed pool with three candidates and asserts only the first is attempted. `fleet_sweep_returns_committed_results_when_database_drops_mid_page` (PostgreSQL) commits the first tenant, closes the pool, and asserts the first tenant's result is returned and the sweep stops at the second of three candidates. `connection_unavailable_covers_only_pool_and_io_failures` pins the classification; the per-tenant failure tests above confirm a database-raised error still continues. Falsification, each with the tests failing as expected: | Mutation | Result | | --- | --- | | Original SQL restored at invite sweep, push claim, d-tag backfill | 0/3 pass (`legacy SQL fence predicate called`) | | `guard_community_write` returns `Ok(())` after taking the lock | race tests + channel reap test 0/4 pass | | Per-tenant errors abort the sweep again | 0/2 pass | | Sweep helper `continue`s instead of stopping on a connection-level error | `fleet_sweep_stops_at_first_unavailable_connection` and `fleet_sweep_returns_committed_results_when_database_drops_mid_page` fail | | Sweep helper drops already-committed results when it stops (`return Vec::new()`) | `fleet_sweep_returns_committed_results_when_database_drops_mid_page` fails | | Sweep helper removed from the source-policy chokepoint check | `serving_table_writes_expose_syntactic_chokepoint_or_guarded_tx_routes` fails | The PostgreSQL tests are under `postgres_tests` modules with `#[ignore = "requires Postgres"]`, which puts them in the `.config/nextest.toml` PostgreSQL filter. CI evidence is under Verification. ## Verification CI at head `5d37fbda` ([run 38925996275](https://github.com/block/buzz/actions/runs/38925996275)) is green. The PostgreSQL job ([job 117062895683](https://github.com/block/buzz/actions/runs/38925996275/job/117062895683)) passed 1017/1017. That includes named PASS lines for all 30 `admission_postgres_tests`, `fleet_sweep_returns_committed_results_when_database_drops_mid_page`, and both writer-pool tests listed below. Unit Tests ([job 117058136482](https://github.com/block/buzz/actions/runs/38925996275/job/117058136482)) has PASS lines for `connection_unavailable_covers_only_pool_and_io_failures` and `fleet_sweep_stops_at_first_unavailable_connection`. Local gates ran on a Linux cloud workstation against private PostgreSQL 16 and Redis. The tree was `735170df`, which is this head rebased on `main` `c0ab789`, so the gates built the merge result. | Gate | Result | | --- | --- | | `cargo fmt --all -- --check` | pass | | `cargo clippy --workspace --all-targets -- -D warnings` | pass | | `cargo nextest run -p buzz-db --lib --tests` | 185 passed | | `scripts/test-postgres-test-discovery.sh` | pass | | PostgreSQL lane for buzz-db (excluding the two writer-pool tests below) | 530 passed | | PostgreSQL lanes for buzz-admin, buzz-audit, buzz-deletion, buzz-push-gateway, buzz-search, buzz-workflow | all pass | | PostgreSQL lane for buzz-relay | 413/426. The 13 `group_state_postgres_tests` fail with `PoolTimedOut` on unmodified `main` `c0ab789` too (see below) | | The three mutations in the falsification table above (`continue` ×2, `return Vec::new()`) | each fails its target test | These failures happen only on the workstation, and each one passes in CI's PostgreSQL job at this head: - `writer_pool_metrics_stop_after_session_timeout_setup_failure` stalls. It's an unchanged writer-pool configuration test, and it stalled the same way at earlier heads. - The 13 relay `group_state_postgres_tests` fail at the same point, `ensure community: Sqlx(PoolTimedOut)`, on unmodified `main` `c0ab789`. This PR doesn't touch `buzz-relay`. ## What got simpler - No application mutation calls the boolean predicate `community_write_allowed()` anymore. Dropping the fence objects still has other dependencies: serving-write lease acquisition calls `assert_community_write_allowed()` (`store/deletion.rs:3068`), catalog validation requires both functions (`store/deletion.rs:1195-1198`), and the triggers and compatibility SQL still use them. Those belong to #8322 and part B, not this PR. - The claim's `target` CTE is gone, and every sweep uses the same admission-and-skip helper. - The four periodic sweeps share one loop (`sweep_admitted_communities`) for admission, skip, continue-on-tenant-error and stop-on-outage, instead of four copies of it. - The source policy treats that helper as a tenant-local chokepoint, so `backfill_d_tags` no longer needs a reviewed exception. - The per-tenant loops add code, because each community now gets its own admitted transaction. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-16 15:46:37 -04:00
Luke Tornquist	2745a419a8	buzz-db: tolerate absent universal write-fence objects (expand) (#8322) ... ## What This is the expand step for retiring the 32 universal `community_write_fence_*` triggers and their SQL functions (BUZZ-315). Relay pods auto-migrate at boot (`BUZZ_AUTO_MIGRATE`). The drop migration therefore runs while the previous release is still serving, and that release must not depend on the objects it drops. Today the app depends on them in four places, and this PR removes all four from its own code paths. Seven in-SQL `community_write_allowed` predicates in fleet sweeps and push (`push_enqueue.rs`, `channel.rs`, `event.rs`, `push.rs`, `relay_invite.rs`, `workflow.rs`) are #8318's scope. The release is independent of the fence functions only with both this PR and #8318. **It changes no schema.** The objects stay and keep working until the follow-up drop PR. | Old dependency | Now | |---|---| | Startup and readiness (`validate_serving_catalog_on`) require `community_write_allowed`, `assert_community_write_allowed`, `enforce_community_write_fence` and all 32 triggers | No longer required. Still required: the tenant tables, `community_deletion_lock_key` and the enabled community tombstone trigger. | | Serving-lease admission runs `SELECT assert_community_write_allowed($1)` | Uses `guard_community_write`, the same shared lock and state check as ordinary serving writes, in the same transaction as the lease `INSERT`. Fenced or missing communities still map to `AccessDenied`. | | Live scoped-table discovery calls `community_write_fence_excluded_table()` | Uses a Rust `EXCLUDED_SCOPED_TABLES` list. A unit test pins it equal to the SQL list while both exist. | | The deletion executor compares the live `fenced_tables` against the frozen inventory | Only tenant-table equality is compared. Inventories **still record** live fences, so an older executor that validates an inventory frozen by this release still matches. The digest still covers the full frozen manifest. | `lock_community_deletion_shared` now refuses transactions that are not READ COMMITTED, so every lock-then-read admission gets the check: `guard_community_write`, serving-lease finalization (`guard_transaction_with_serving_lease`), lease renewal/verification, owner preparation and owner-mutation admission. The old SQL assertion did this check for the lease path, and the trigger backed up the rest. The isolation level comes back from the existing `pg_advisory_xact_lock_shared` statement, so the hot path makes no extra round trip. The refusal comes before any lifecycle-state read. ## Mixed versions | Combination | Why it is safe | |---|---| | Previous release, this DB | No schema change, so all old checks still see the 32 fences. | | This release, DB with fence objects | Extra objects are tolerated, and the trigger backstop still applies. | | This release **plus #8318**, DB without fence objects | Serving, readiness, lease admission and deletion proven by `migration_schema_fence_objects_are_optional_for_serving_and_deletion`, which drops them inline. Without #8318, serving and deletion still work, but the push matcher, exhausted-job reaper, invite retention, ephemeral reaper and startup d_tag backfill fail with `42883 community_write_allowed(uuid) does not exist` (seen in deploy-stack E2E). | | Older executor, inventory frozen by this release | `fenced_tables` is still populated from the live catalog (32). | This PR does not depend on #8318 to merge and can ship alongside it, but it is not drop-safe alone. The drop PR (#8323) needs #8219, #8318 and this PR deployed on every pod, and every rollback image after the drop must include #8318. ## Tests - `store::deletion::postgres_tests::migration_schema_fence_objects_are_optional_for_serving_and_deletion` migrates to 0060. It then runs the following with the fence objects present, then again after dropping them inline: - serving startup - readiness - the deletion catalog check - `inventory_schema` - lease acquire/release - `AccessDenied` for quiescing, fenced, tombstoned and missing communities - `store::deletion::postgres_tests::deletion_lock_admission_rejects_pinned_snapshot_isolation` checks that REPEATABLE READ and SERIALIZABLE are refused by both `guard_community_write` and `guard_transaction_with_serving_lease`. The lease case uses a never-acquired lease, so if the isolation check is skipped it fails as stale instead. - `buzz-deletion` `postgres_tests::approved_legacy_fence_inventory_accepts_current_catalog` checks that a legacy digest with 32 fences validates against a live catalog with none, and that an unknown tenant table is still rejected. - `store::deletion::tests::scoped_table_exclusions_match_sql_during_transition` checks exclusion-list parity. - `store::deletion::postgres_tests::serving_lease_acquire_waits_for_fence_lock_then_refuses` holds the exclusive deletion lock and checks that lease acquisition stays pending. It then commits `fenced` and requires `AccessDenied` with no lease row persisted. Removing the lease path's `guard_community_write` call fails this test, while the other four lease/admission tests above still pass. - `buzz-deletion` `serving_guard_heartbeats_long_operation_through_quiescing` now requires `AccessDenied` for a lease acquired during quiescing, rather than any error. Each test was confirmed to fail when its fix is removed (rc 100, 0/1): the isolation refusal, the three validators restored to the base version, the fence-list comparison restored, and one name dropped from the Rust list. ## What got simpler The lease path loses its special SQL assertion and its error-code translation and reuses the ordinary app guard. The serving and deletion catalog checks each lose one requirement. No new abstraction. The `fenced_tables` reader stays only for mixed-version executors; the drop PR deletes it. **Identified debt (pre-existing, outside this diff):** `runtime::postgres_tests::writer_pool_rejects_non_read_committed_database_default` hangs on clean `177551dda` too. ## Verification (Blox, tree identical to this commit) | Gate | Result | |---|---| | `cargo fmt --all -- --check` | pass | | `cargo clippy --workspace --all-targets -- -D warnings` | pass | | `cargo nextest run -p buzz-db --lib` | 167/167 | | `scripts/test-postgres-test-discovery.sh` | pass | | `scripts/postgres-test-run.sh -p` buzz-admin / audit / deletion / push-gateway / relay / search / workflow | all rc 0 (4, 7, 12, 11, 402, 20, 2) | | `scripts/postgres-test-run.sh -p buzz-db` | 510 pass, 1 fail: the pre-existing hang above, interrupted | | Round 2 (this head): `-p buzz-db` / `-p buzz-deletion` | 511 pass + the same pre-existing hang interrupted / 12/12 | The new PG tests are selected by the `postgres-ci` filter `(test(/postgres_tests::/) or binary(/^postgres_/)) and not test(/(^|::)external_infra[^:]*::/)`. buzz-review-completed --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-16 15:28:54 -04:00
Brad Seiler	2d0f78c6e1	feat(auth): support explicit mesh JWKS transport (#8382) ... ## Summary Allow an issuer to opt into mesh HTTP for its JSON Web Key Set (JWKS). Public HTTPS remains the default. This supports deployments whose service mesh authenticates and protects the key endpoint. The sealed source contract selects transport and contributes to policy identity. Mesh mode requires a DNS hostname and checks every resolved address against an explicit allow-list before pinning one address. Both modes disable proxies, reject redirects, cap responses, and enforce a whole-fetch deadline. The relay cannot verify mesh security: operators must enforce mutual TLS, endpoint authorization, and egress restrictions. ### Related issue No duplicate mesh-JWKS issue or PR found. Existing NIP-FI relay enforcement: #7224. No deployment change is included. ### Testing - At `eccb8cb19420abda44cf6e5e62b1361e2843bd1d`: full auth package and auth/relay all-target clippy passed. Full relay package passed: 1,457 library tests, 37 binary tests, and 9 boot tests; 446 tests were ignored. - At prior head `113939838cd81d5e8d0b0f8783c42ffaae44a4d3`: the full relay library suite failed the demo echo test (1,455 passed, 1 failed, 440 ignored). Independent checks reproduced that timeout at base `d61fe2071d2b3d8e56dd931947f16bfe0d6173fe`; the receive-cancellation defect is outside this change. A later pass does not establish that defect is fixed. - Three falsifying mutations at `eccb8cb194` were caught: mixed DNS answers, proxy bypass, and whole-fetch deadline. The full locked auth package passed after restoring the exact source. - At `eccb8cb194`, `cargo build --locked --release -p buzz-relay -p buzz-cli -p buzz-admin` passed with command-local `CARGO_PROFILE_RELEASE_STRIP=none`. This avoids a macOS loader error without a source, lockfile, or project-profile change. - Earlier hooks failed the ACP signal test and Tauri login-shell PATH test. The ACP failure also reproduced on unmodified main `93ffc37c`. Main merged without conflict. The full Tauri package then passed with `RUST_TEST_THREADS=1`: 3,422 library tests and 3 integration tests, 19 ignored. All pre-push hooks passed with that same command-local setting; no hook or test was skipped. - At `eccb8cb194`, isolated `just test` passed all 107 runner steps. The monolithic `just ci` run timed out after 1,120 seconds; all recipes passed across that run and separate `just desktop-tauri-test`, `just web-build`, and `just mobile-test` invocations. No monolithic pass is claimed. - Exact-head release live-local checks passed: loopback HTTP mesh JWKS warm; valid assertion 101; expired/wrong issuer/wrong audience 403; valid signed disconnect 200; wrong command audience 403. Redirect-only JWKS returned assertion 503 without a request to its target. Both omitted/default and explicit public transport rejected HTTP at startup. - A separate NIP-FI-off release CLI smoke test created an open channel, sent an accepted message, read it back, and fetched its thread. The assertion probe closes after upgrade; it does not prove a complete NIP-42 session. Local loopback tests do not prove deployment mesh authentication, endpoint authorization, egress controls, or real `240/4` routing. Generated with Codex ## Human Testing valid assertion / configured community audience: HTTP 101 (expected 101) expired assertion: HTTP 403 (expected 403) wrong issuer: HTTP 403 (expected 403) wrong community audience: HTTP 403 (expected 403) valid command JWT / configured command audience: HTTP 200, body=b'{"disconnected": true}' (expected 200) valid command signature / wrong audience: HTTP 403 (expected 403) --------- Signed-off-by: seiler <seiler@squareup.com> Signed-off-by: Local Builder <1c476cb1127ea270cb900bdc3891060140bdd1173f91498030ee9b4db01c2f20@buzz.block.builderlab.xyz> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Local Builder <1c476cb1127ea270cb900bdc3891060140bdd1173f91498030ee9b4db01c2f20@buzz.block.builderlab.xyz>	2026-10-16 14:26:08 -04:00
Luke Tornquist	c0ab789499	feat(buzz-db): retire the channel-TTL trigger (#8210) ... ## Summary This drops the channel-TTL event trigger. The application already refreshes ephemeral-channel deadlines in the event transaction (`event_follow_up::refresh_channel_ttls`, from #8135), so the trigger only repeats that work. This PR originally retired both follow-up triggers. #8352 has since retired the push-match trigger in migration 0063, so after the rebase this PR covers only the TTL half. Migration `0065_retire_channel_ttl_trigger.sql`: - `SET LOCAL lock_timeout = '5s'` (the same bound as 0063) - drops `events_refresh_channel_ttl` and `refresh_channel_ttl_after_event_insert()` Dropping the trigger on the partitioned parent also drops its partition clones. Published migrations, including 0022 and 0024, stay checksum-frozen. The same objects are removed from `schema/` (`schema/functions/…`, `schema/triggers/events.sql`, `schema/schema.sql`) and from the pgschema reconcile script. After this, migrated and desired-state databases have no event follow-up triggers. Tests: - `follow_up_retirement_tests.rs` (renamed from `push_retirement_tests.rs`): the push-retirement upgrade test now stops at 0063. A new test upgrades 0064 → 0065 and checks that the TTL trigger is gone from the parent and every partition, that the function is gone, and that the published checksums haven't changed. - `event_follow_up_postgres_tests.rs`: the contract now has two arms. `AppOnly` is the shipped schema, where both triggers are absent. `Dual` reconstructs the historical overlap of both triggers. The `PushRetired` arm (the push trigger dropped, the TTL trigger kept) was the shipped state before this PR. Its tests are removed because `Dual` and `AppOnly` together cover it. - `embedded_migrator_contains_consolidated_initial_schema` pins the 0065 statements and checks that both objects are absent from the desired schema. **Deploy gate:** do not run 0065 against a database still served by a relay older than #8135. After 0065, raw SQL writers no longer refresh TTL deadlines. `docs/push-gateway-deployment.md` is updated to match. ### What got simpler The trigger and the app no longer both refresh TTL deadlines; the app is now the only producer. One plpgsql function and one trigger are gone from both schema paths, and 8 per-partition drops are gone from the reconcile script. The follow-up contract drops from three arms to two. ### Related issue BUZZ-313 (Linear), part of the trigger-retirement work. ### Testing Head is `4f7d0baf0`. Results below are at `678927730` unless noted. `678927730..4f7d0baf0` changes only one test in `crates/buzz-db/src/store/event.rs`, described in the first bullet. - **Fixed test, `permanent_channel_event_commits_do_not_lock_the_channel_row`** (`4f7d0baf0`). - **Why it changed:** after 0065, its raw-pool inserts no longer ran any TTL refresh, so the test passed whatever `refresh_channel_ttls` did. This was raised in review. - **What it does now:** it inserts through the production `AdmittedTx` path and keeps the barrier and the tuple-lock probe. It commits all N transactions while an external transaction holds `FOR SHARE` on the channel row. It also asserts that no `buzz_db_channel_ttl_refresh_failures_total` is counted. - **Mutation check:** with `SELECT … FOR UPDATE` injected into `refresh_channel_ttl`, the new test fails (`must not wait on the channel row`, timeout at 10s). The old test passes under the same mutation. - **Blox:** fmt and `buzz-db` clippy pass. - **GitHub CI at `4f7d0baf0`:** every required check passes (56 pass, 0 fail). CI builds the merge ref with `main` `93ffc37cb`. - **New and renamed tests, checked by name in the CI logs at `4f7d0baf0`.** These pass in PostgreSQL Domain / PostgreSQL Tests ([117033725006](https://github.com/block/buzz/actions/runs/38915998692/job/117033725006)): - `permanent_channel_event_commits_do_not_lock_the_channel_row` - `migration_schema_ttl_retirement_drops_every_partition_trigger_and_keeps_checksums` - `migration_schema_push_retirement_preserves_ttl_and_published_checksums` - `event_insert_ttl_refresh_handles_permanent_ephemeral_duplicate_and_activation_race` - `{desired,migration_schema}_event_follow_up_contract_{dual,app_only}` (four tests) `embedded_migrator_contains_consolidated_initial_schema` passes in Rust / Unit Tests ([117024125423](https://github.com/block/buzz/actions/runs/38915998692/job/117024125423)). No `*_push_retired` test runs anywhere. - **Blox gates at `678927730`** (private PG16 + Redis, on a workstation shared with other sessions): - `cargo fmt --check`, workspace `clippy -D warnings` and `buzz-db --lib` all pass. - The `buzz-admin`, `buzz-audit`, `buzz-deletion`, `buzz-push-gateway`, `buzz-search` and `buzz-workflow` Postgres packages pass. - `buzz-db`: 517 of 519 tests passed. The 2 that hung, both `writer_pool_*` tests, were killed. `buzz-relay`: 413 of 426 passed. The 13 failures are covered below. - **Uncontended reruns at `4f7d0baf0`** on a fresh, idle workstation (PG16.15, `competing=0` for every arm): - **`writer_pool_rejects_non_read_committed_database_default` predates this PR.** It timed out 3/3 at head and 3/3 at `main` `93ffc37cb`, with a 300s timeout per run. - **`writer_pool_metrics_stop_after_session_timeout_setup_failure` also predates this PR.** It timed out 2/3 at head and 2/3 at `main`, and passed in the remaining run on each side. - Both tests pass in CI. The gate accepts them as a named exception for this PR, as a pre-existing hang on the PG16 test environment. - **`buzz-db`:** 518 of 519 tests passed. The single failure is `writer_pool_rejects_non_read_committed_database_default`, killed after it hung (the exception above). - **`buzz-relay`: still open.** 413 of 426 tests passed. The same 13 `handlers::group_state_postgres_tests` failed as on the shared box, each panicking at once in its fixture's `ensure community` step with `Sqlx(PoolTimedOut)`. Contention does not explain them. The fixture's community setup is in `buzz-db`, which this PR changes. A head/`main` comparison of the full `buzz-relay` package on the same idle box is running to find out whether this predates the PR. All 13 pass in CI at this head. - **E2E (Gimli, Blox)** at `050e593af`. `050e593af..678927730` changes only tests and docs; the migration, schema and runtime code are byte-identical. Gimli's audited verdict: **EXERCISED AND WORKS**, for DB and relay runtime only. Gimli ran it at `050e593af`; this head's runtime is byte-identical but was not freshly executed. - **Upgrade:** a real relay startup took the schema from 0064 to 0065. That removed the TTL function and the trigger from the parent table and from all 8 partitions. Checksums for migrations 1–64 are unchanged. - **Lock-timeout rollback:** with a real lock held, the migration failed with `55P03` at 5.0007s and the schema rolled back fully to 64. Once the lock was released, a retry applied 65. - **Fresh desired schema:** after pgschema plus reconcile, the retired objects are absent. - **Real signed kind-9 sends:** on both schema paths, these refreshed ephemeral-channel deadlines to about +600s. Permanent channels stayed NULL. - **Raw SQL insert:** at 64 it refreshed the deadline. After 65 it left the deadline unchanged, on both the upgraded and the fresh schema. - **Not covered:** native or device testing, and deploy. - **Review:** Legolas found 0 P0 and 0 P1. His three optional P2s are fixed in `678927730`. - **No human app test:** this is a DB-only change. - **Migration numbering:** #8237 and #8372 also add a `0065_*` migration. Whichever PR merges second must renumber. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-16 11:22:21 -04:00
Bradley Axen	d039d9a2d5	fix(relay): confirm requested channels in one membership query (#8378) ... ## What A request that names channels with `#h` used to check each channel missing from the cached access set with its own `is_member` query. These checks ran one after another, so the database cost of a request grew with the number of channels it named. That cost is why the relay capped explicit `#h` values at 128, and why mobile splits large communities into many requests ([#8342](https://github.com/block/buzz/pull/8342)). This PR makes the check one query, and then raises the cap. - **One membership query.** WS REQ, WS COUNT, and HTTP `/query` and `/count` now share `confirm_requested_channel_access`. It checks every unconfirmed channel with one `membership_pairs` statement (`channel_id = ANY($2)`), so one request costs one query, whatever number of channels it names. `membership_pairs` uses the same SQL rules as `is_member`: deleted channels and removed members don't count, and it runs on the writer pool. - **Hash-set lookups.** Comparing the requested channels with the accessible set no longer costs requested × accessible comparisons. This covers `apply_channel_scope_to_query`, the system-channel scope helpers, the COUNT and bridge loops, and `#h` dedupe. - **`MAX_EXPLICIT_CHANNEL_VALUES` goes from 128 to 2048.** Huddle liveness used the same constant to limit session IDs. It now has its own constant, `MAX_HUDDLE_LIVENESS_SESSIONS`, which stays at 128, because each linked session still costs a separate mesh-directory lookup. Authorization rules are unchanged: - A scoped token still sets the upper limit. A channel outside the token is never checked. - Channels already cached and readable system channels are skipped. - Repaired channels are added in request order, each once, so `keep_repaired_system_channels_named` still sees the same repaired tail. - A database error still closes the request (or returns 500 over HTTP). One difference: the old loop could leave a partial repair if a later query failed, and that can no longer happen. ## Not in this PR - **Client chunk sizes.** `desktop/src/shared/api/relayClientShared.ts:39` and `mobile/lib/shared/relay/nostr_filters.dart:8` stay at 128. They can go up only after every relay they talk to runs this change. - **Per-connection limit on live channel scope.** Each authorized channel in a live WS subscription still gets a fan-out index entry for each kind, plus a Redis topic retain. Real membership limits this, not the cap. Still, an account in at least 2048 channels could now register up to 1024 × 2048 channel scopes on one connection; before, the most was 1024 × 128. If we want to keep the old worst case, the fix is a limit on the total channel scope per connection, not per REQ. ## Testing - `cargo clippy -p buzz-relay --tests` and `cargo fmt` are clean. - `cargo test -p buzz-relay --lib`: 1449 passed, 6 failed. All 6 are in `api::media::tests` and fail because the local Postgres on :5432 has no `buzz` role. Nothing in this change touches them. - I ran the Postgres lane (`scripts/postgres-test-run.sh`) for the bridge, req, count, group-state, and system-channel tests on local PG14: 55 passed. That includes the new `named_channels_missing_from_the_cache_are_confirmed_together`, which checks that the reader's channels missing from the cache are added in request order, each once, and that a stranger's channel stays out. - Not done yet: a live run against a local relay sending a REQ with more than 128 `#h` values. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Bradley Axen <baxen@squareup.com> Co-authored-by: murderbot <3754f8729004d95654c46dbab3129e4ab9ef05cc2534e2a3fbfc155983bd637b@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 17:22:37 -07:00
Logan Johnson	7852b1670c	fix(search): normalize profile name tiers in one place (#8334) ... 🤖 ## Summary Profile name search (`POST /profiles/search`, added in #8301) sorts results into tiers: an exact name first, then a name that starts with the query, then a name with a later word that starts with it. This PR stacks on #8301 and fixes two review items. **Tier normalization in one place.** Before this change, the query was lower-cased in Rust (`to_lowercase`), and the names were lower-cased in SQL (`lower()`). These can disagree. Under a `C` collation, PostgreSQL's `lower()` changes only ASCII letters, while Rust also lower-cases letters like `É`. So a search for `Émile` did not put the profile named `Émile` in the exact tier. Now the query and both name fields go through the same SQL expression: lower-case, collapse whitespace runs to one space, then trim. The query is bound once and both sides compare equal by construction. The trim also moved after the collapse, so surrounding whitespace that the regex turns into a space is now trimmed too. **Ranking wording.** The doc comment and `docs/profile-search.md` said "ties keep the NIP-50 order". That was wrong. Ties are ordered by `ts_rank_cd` over the two name fields only (not the whole profile, as in NIP-50), then newest first, then event id. Out of scope, not changed here: matching names that contain escaped Unicode or newlines, and any new name index. ## Related issue Review on #8301: https://github.com/block/buzz/pull/8301#pullrequestreview-5524491230 ## Testing - New case in `profile_names_match_name_words_only`: profiles `Émile` and a newer `Émilee`, query `Émile`. The exact match must come first. - Ran the Postgres integration tests against a `postgres:17-alpine` database created with `--locale=C`: - this branch: `profile_names_match_name_words_only` and `profile_names_survive_unparsable_content` pass; - #8301 code with the new case: `profile_names_match_name_words_only` fails (`Émilee` sorts before `Émile`). - Nine other tests in that file fail locally on both this branch and #8301 because they reuse one database (duplicate community hosts). CI gives each test its own database. - `cargo fmt --check`, `cargo clippy -p buzz-search --all-targets -D warnings`, and `cargo test -p buzz-search`: clean. - The pre-push hook's `rust-tests` and `desktop-tauri-checks` passed. Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-15 18:02:10 -04:00
Logan Johnson	f148c0f22c	feat(relay): add profile search over name and display_name only (#8301) ... 🤖 > **Review as a pair with block/buzz-app#1084** (the app change that uses this endpoint). Either can merge first: the app falls back to the old search on relays without this endpoint. ## Summary Searching for a person by name over the relay's general search returns profiles that only *mention* the text somewhere else: in an avatar image embedded as text, in the bio, or in the verification address. A search for "wes" could fill every result page with profiles whose avatar data happens to contain "Wes", and never return the person actually named Wes. This adds a relay endpoint, `POST /profiles/search`, that looks only at a profile's **name** and **display name**. Every other profile field is ignored. **Matching and paging are the same as the relay's existing full-text profile search** (NIP-50 search over kind:0 profiles with `search_mode: "prefix"`). A client can switch to this endpoint without a visible change in which people a query finds: - Each query word must match a whole word of the name; the last word can be a prefix. Words can be in any order, so "johnson wes" finds "Wes Johnson". Case is ignored. - Pages work the same way: up to 100 results per page, up to page 1000. - The results are exactly the old results minus profiles that match only outside their names. The order is better for people search: exact name first, then names that start with the query, then names with a later word that starts with it, then other matches. Ties are ordered by `ts_rank_cd` over the two name fields only (not the whole profile, as in NIP-50), then newest first, then event id. The response is a JSON array of the original signed profile events (Nostr kind:0), so clients can verify them as they do today. The buzz-app client change is block/buzz-app#1084. ## Details - Request: `{"query": "wes", "limit": 20, "page": 1}`. `query` is 1–100 characters after trimming; `limit` is 1–100 (default 20); `page` is 1–1000 (default 1); unknown fields are rejected with 400. - The search reuses the full-text search's own query builder (`buzz-search`), so the word rule cannot drift from it. The existing full-text index on events stays the access path; the name check runs only on rows the index already matched. There is no new index. One small migration (`0061`) adds the function `profile_name_fields()`; it reads no table data, so it runs instantly. - Only current profiles count: a replaced profile stops matching. Profiles that are not a JSON object, or whose name fields are not strings, never match. Content that PostgreSQL cannot read as `jsonb` (for example a `\^@` escape or very deep nesting) also never matches, and it cannot make the search fail for everyone else. - The handler loads the matched events through the normal routed event read and applies the same reader visibility check as other reads, in search order. - Access uses the same checks as the relay's GIF search endpoint: a signed HTTP request (NIP-98), identity admission (NIP-FI), rate limit, replay protection, and relay membership. That code moved out of `gifs.rs` into one shared helper in `bridge.rs`; GIF behavior is unchanged. - The route is added to the existing protected-route and NIP-FI shadow/enforce route tables. Docs: `docs/profile-search.md`. - Two test-only lines get `sadscan:disable` comments, as other fixtures already do: an existing public ES256 test key in `router.rs`, and the local Postgres test URL in the search integration test. ### Related issue None found. ### Testing - Postgres test for the search query (`profile_names_match_name_words_only`): only name fields match (avatar data URL, `about`, `nip05`, `banner`, non-string names and non-object content are excluded); words in any order and across both name fields; whole words only for completed words; ranking order; pages split without overlap or gaps; replaced profile; other community; `%` literal. - Postgres end-to-end test through the router: an admitted member gets only the name-matching signed event, and the returned event still verifies. - Request validation unit tests; existing GIF admission tests and router route-table tests pass. - Local end-to-end run with buzz-app#1084 against a relay built from this head, with about 195 seeded profiles. For every query tried, the endpoint's results were a subset of the full-text search's results; the only profiles missing were avatar-only matches. --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-15 17:35:33 -04:00
Kalvin C	cdfc38ecbd	feat(search): add newest-first sort to bridge search (#8371) ... ## Summary Adds an opt-in newest-first order to bridge search, so clients can offer "Most recent" alongside relevance. - `buzz-search`: `SearchQuery` gains `sort: SearchSort`. `Relevance` keeps the existing `ts_rank_cd` ordering. `Newest` orders hits by `created_at DESC, id`, and `page` steps through that order. Newest skips the exact-profile-lexeme prioritization, which only applies to relevance ranking. - Bridge `POST /query`: search filters accept `search_sort: "newest"`. When the field is missing or has another value, results stay in relevance order. Community scope, channel intersection and per-hit authorization are unchanged. - NIP-11 lists `buzz-search-sort` in `supported_extensions`. Clients only send the field to relays that advertise it, so they never get relevance results labelled as newest. - WebSocket NIP-50 search stays relevance-only. `NOSTR.md` documents the bridge extension. ### Related issue N/A ### Testing - New Postgres integration test `newest_sort_pages_by_time_then_id_within_scope`. It pages a channel in newest order, including an equal-timestamp tie split across a page boundary and a newer hit outside the channel scope. It checks that the pages join up in `(created_at desc, id asc)` order and that relevance still ranks the strongest match first. - Unit tests: - `bridge_search_sort_extension_selects_newest_only_when_named` - `advertises_nip_cl_and_search_sort_extensions` - Ran the buzz-search Postgres integration suite, the relay `nip11` and `bridge_search` unit tests, `cargo fmt --check`, and `cargo clippy -p buzz-search -p buzz-relay --all-targets -D warnings` on Linux. - Local relay exercised through signed `POST /query`: - NIP-11 lists `buzz-search-sort`. - Relevance and newest return different orders. - Newest pages 1–3 return 25 hits with no gaps or duplicates, including same-second ties across page boundaries. - A non-member gets no hits from a private channel; a member gets all of them. - Client integration is in a companion block/buzz-app change. It was exercised end to end against this relay through the production broker: switching sort, paging, scope changes and capability fallback. Deployment note: clients only offer newest sorting after a relay with this change advertises `buzz-search-sort`. --------- Signed-off-by: Kalvin Chau <kalvin@block.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 13:27:49 -07:00
Tom Brow	6766174673	chore(push): retire push notification database trigger (#8352) ... Buzz must schedule mobile push notifications when it saves an eligible message. The relay, the server that receives and stores messages, now does this in application code. An older database trigger still exists. This trigger is a rule that runs automatically when a message is saved. This PR removes the old trigger and its supporting database function. The relay remains responsible for scheduling notifications. Migration 0063 makes this change in existing databases; updated setup scripts apply it to new databases. The change does not turn on push notifications. Production migration and deployment need separate approval. Before deployment, verify the release images and all pending database migrations. If push needs to be stopped after release, disable it with the feature flag and keep the qualified application version and database schema. Do not downgrade the application image. ### Validation - Local checks passed: 519 isolated database integration tests, 166 database unit tests, 17 source-policy tests, and the relay push-runtime tests. - Tests verify that the migration removes the old trigger, preserves earlier migration records, and lets the application schedule push work correctly. A concurrency test exercises the real application path; deliberately removing its lock makes the test fail. - [Current-head CI passed on retry](https://github.com/block/buzz/actions/runs/38752849383/attempts/2). The first attempt had one database-connection timeout during test setup; the failed jobs passed when rerun on the unchanged commit. - Initial local CI and push checks hit unrelated ACP test failures, which passed when run separately. Only the Rust-test push hook was skipped under the repository's local-check exception; other applicable hooks passed. ### Before merge - [x] Migration 0063 is available and the push trigger remains on [main at 5397ea4](5397ea49e2). Recheck if main advances before merge. --------- Signed-off-by: Tom Brow <tomb@block.xyz>	2026-10-15 08:37:55 -07:00
Tom Brow	5397ea49e2	fix(push): verify extended App Attest proofs with a focused owned module (#8329) ... ## Summary Apple SDK 27.1 device captures include authenticator extension data that the current verifier rejects. Replace the general appattest dependency with a private gateway module that verifies the complete original bytes, frames enrollment and assertions separately, and accepts well-formed optional extensions without SDK-specific length or flag allowlists. The module retains established signature, P-256, CBOR, and certificate-chain libraries. App identity, exact enrollment environment, credential/key binding, nonce, challenge, and increasing-counter checks remain enforced. Bounded parsing rejects ambiguous duplicate keys, truncation, trailing data, and oversized inputs. Fixture helpers are test-only. Adapted code retains both upstream MIT notices and source provenance. Category/version properties remain authenticated observations under the chosen compatibility-release policy. This does not implement Apple's additional distribution-category/version enforcement; introducing those restrictions requires a separately defined deployment policy. The [verification contract](https://github.com/block/buzz/blob/app-attest-owned-verifier/docs/app-attest-compatibility.md) records this and other limits. Related: https://github.com/block/buzz/pull/8020 addresses assertion extension handling. This change also covers enrollment extensions, credential-key parsing, and certificate validation. ### Validation - Gateway suites: 77 default tests and 79 personal-development tests passed, plus startup policy and Clippy. - Added signed enrollment/certificate, generated CBOR mutation, maximum-counter, and PostgreSQL race/replay coverage. All 12 isolated Postgres tests passed. Apple's historical fixture is tested structurally, not accepted as a current enrollment. Positive synthetic leaf certificates are generated at test time. - Protected SDK 26.5/27.1 enrollments and assertions pass the verifier, including the same key across SDK upgrades; development enrollment rejects under production policy. Captures are personal-development identities, not supported production-identity evidence. - Full local CI passed Rust tests and reached 6,828 passing desktop JS tests before an unchanged 10 ms focus-timer test failed; its five-test file passed in isolation. The pre-push native suite had 3,421 passes and an unchanged 300 ms child-PID test failure, also passing in isolation. Prior ACP timing flakes and these unrelated hook exceptions do not waive required CI. - Windows CI exposed a missing system OpenSSL installation for the test fixture generator. The test dependency now builds bundled OpenSSL; all 30 focused App Attest tests pass locally. [Current-head CI](https://github.com/block/buzz/actions/runs/38636062347) passed, including Windows Rust, PostgreSQL, and desktop integration tests. - [Prior-head Codex review](https://github.com/block/buzz/pull/8329#issuecomment-6177515598) found no major issues. Carl subsequently identified a dropped credential-length workaround; fixed in [27f9f26](27f9f2624f), with 34 focused/protected tests and Clippy passing. [Current-head Codex review](https://github.com/block/buzz/pull/8329#issuecomment-6178361143) found no major issues; physical iOS 26 testing was explicitly waived by Tom on October 14, 2026. No iOS 26 device test was performed. The SDK comparison used the same iOS 27.0.1 device, so it demonstrates SDK-linked variation without establishing OS-version causation. - Isolated candidate HTTP/device validation passed with separate authority state and no APNs delivery: SDK 26.5 enrollment/delegation, SDK 27.1 delegation with the exact same key and installation, and fresh SDK 27.1 enrollment/delegation. Assertions changed from 37 to 102 bytes across the upgrade; accepted counter increased from 1 to 3 and delegation generation from 1 to 2. All three runs rejected tampering and replay (401) and consumed challenges (404). Existing Buzz app builds were preserved. No production rollout is included. - Existing-installation validation passed against the current-head Dev gateway: a guarded SDK 27.1 diagnostic reused the stored key and installation, advanced the assertion counter from 88 to 89 and delegation from 89 to 90, and published the existing active relay lease at generation 9 (previously 8). Independent database reads confirmed the changes and preserved key, epoch, and expiry. The original app build was restored in place with current data and keys intact. --------- Signed-off-by: Tom Brow <tomb@block.xyz>	2026-10-15 07:15:19 -07:00
Logan Johnson	d28f6b2f70	feat(relay): let channel-state filters opt in to system channels (#8182) ... 🤖 ## Summary After part 2, a client can read a system channel only if it already knows the ID. This PR lets a client find system channels on purpose, with an ordinary Nostr filter and no new filter key. A filter on the channel-state kinds (39000–39003) now also returns the system channels the reader can read when it has either: - `#t:["system"]`, the channel type tag that every channel-state event already carries; or - any `#P`, the creator tag from #8174. Asking for the channels of a named creator returns a small result, so it does not fill generic channel lists. The filter then matches as usual. Existing clients send neither on these kinds, so they still never see a system channel. | Reader wants | Filter | |---|---| | My system channels | `{kinds:[39002], #p:[me], #t:["system"]}`, then their kind:39000 by `#d` | | Open system channels | `{kinds:[39000], #t:["system"]}` | | One creator's labeled channel, system or not | `{kinds:[39000], #P:[creator], #t:[label]}` | Private system channels stay hidden from non-members, as for any private channel. Values in one tag filter combine with OR, so `#t:["system", label]` means `system` **or** the label; use `#P` plus the label instead. Part 3 of 3, stacked on part 2. ### Related issue None found. ### Details - The opt-in is decided per filter in `handlers/system_channels.rs` and applies on REQ, COUNT and the HTTP bridge. It applies only when every kind in the filter is 39000–39003. - A label alone is not an opt-in. - `NOSTR.md` documents the system channel query rules. - **Spec.** NIP-CL says how a filter asks for system channels. - The relay information document (NIP-11) now lists `"nip-cl"` in `supported_extensions`. Clients use this to tell if a relay supports labels and system channels, and refuse those calls on a relay that does not. NIP-CL documents this. ### Testing - End-to-end relay test: `#t:["system"]` finds both system channels for a member and only the open one for a non-member; 39002 `#p:[me]` with `#t:["system"]` lists the member's system channels; `#P` includes a creator's system channels; creator plus label returns exactly one channel with `limit: 1` and excludes another creator's channel with the same label; a label alone does not return the system channel; COUNT and HTTP `/query` and `/count` agree. - The test fails when the opt-in is disabled. - Unit tests for which filters opt in. ### Stack These are two stacks. Merge them in this order. Client support (creating, labeling and finding these channels) will be in block/buzz-app, not the buzz SDK and CLI. **Stack 1: channel labels and creator lookup (#8174–#8176)** 1. #8174 channel type and creator tags, and the channel creation time 2. #8175 channel labels 3. #8176 database-side label and creator match, and the rule for choosing one channel **Stack 2: system channels (#8180–#8182)**, stacked on stack 1 because it uses labels and the creator tag 4. #8180 the system channel type 5. #8181 leave system channels out of reads that do not name them 6. #8182 channel-state filters opt in to system channels ← this PR --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-15 08:40:16 -04:00
Logan Johnson	cdf03f618a	feat(relay): leave system channels out of reads that do not name them (#8181) ... 🤖 ## Summary After part 1, a system channel is still returned by every query that returns ordinary channels, so existing clients would show it as a chat. This PR makes the relay leave system channels out of every read that does not name them. Clients do not need to filter, so clients that know nothing about system channels stay correct. | Query | Before | After | |---|---|---| | A query that does not name a channel (no `#h`), COUNT, full-text search, the HTTP `/query` and `/count` routes, artifact queries (`{"artifact": ...}`), the sidebar | Includes system channels | Leaves them out | | "Channels I belong to" (kind:39002 `#p:[me]`) | Includes them | Leaves them out | | A filter that names the channel with `#h`, or with `#d` on kinds 39000–39003 | Ordinary read rules | Unchanged: members can read it, and anyone can read it if it is open | Push notifications and "you were added to a channel" notifications (kinds 44100/44101) skip system channels, because a client that received one would show the channel. A client that is given a system channel's ID, for example in a pasted link, can still open it. Naming the ID is the opt-in. This keeps system channels ordinary NIP-29 groups. Part 2 of 3, stacked on part 1. ### Related issue None found. ### Details - The rule lives in one place: `get_accessible_channel_ids`, the set of channels a query may read when it does not name one. It is the union of the reader's memberships and every open channel; both halves now exclude `channel_type = 'system'`. Because the filter is in SQL before `LIMIT`, pages stay full. - A filter that names a system channel adds it to that filter's scope only (`handlers/system_channels.rs`). A sibling filter in the same request keeps the implicit set, so naming one system channel cannot widen another filter. - Thread windows and the workflows API name their channel, so they follow the same named-channel rule. - Artifact queries have their own read check in SQL. It reads a system channel only when the query's `#h` names it. - When a request re-checks membership for a named channel that its first lookup missed, a confirmed system channel joins only the filters that name it. Before, a reader who joined a system channel during the request could have it added to a sibling filter's scope. - The system-channel lookup runs only when a filter names a channel outside the implicit set, so ordinary queries do not pay for it. - **Spec.** NIP-CL lists the reads that leave system channels out, and says when a filter names one. `AGENTS.md` notes this for `get_channels`. ### Testing - End-to-end relay test (Postgres and Redis) with a private system channel, an open system channel and an ordinary channel, as a member and as a non-member: REQ without `#h`, NIP-50 search, COUNT, HTTP `/query`, HTTP search, HTTP `/count`, the channel list, and 39002 `#p:[me]` return only the ordinary channel. Named reads by `#h` and `#d` follow member-or-open on REQ, COUNT, search and HTTP. - Relay test: adding a member to a system channel sends no kind:44100; an ordinary channel still does. - Both tests fail when the SQL exclusion or the named-channel scope is removed. - Database test: an artifact in an open system channel is missing from `{"artifact":"current"}` and present with `#h:[<system channel>]`, for a member and a non-member. It fails without the channel-type condition. - Relay test: the membership re-check confirms a system channel that the system-channel lookup missed. The channel goes to the filter that names it, and a sibling filter's scope does not include it. It fails without the fix. - Unit tests for the per-filter scope. They now run in `just test-unit` and `scripts/run-tests.sh`. ### Stack These are two stacks. Merge them in this order. Client support (creating, labeling and finding these channels) will be in block/buzz-app, not the buzz SDK and CLI. **Stack 1: channel labels and creator lookup (#8174–#8176)** 1. #8174 channel type and creator tags, and the channel creation time 2. #8175 channel labels 3. #8176 database-side label and creator match, and the rule for choosing one channel **Stack 2: system channels (#8180–#8182)**, stacked on stack 1 because it uses labels and the creator tag 4. #8180 the system channel type 5. #8181 leave system channels out of reads that do not name them ← this PR 6. #8182 channel-state filters opt in to system channels --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-15 08:40:15 -04:00
Wes	7b11c6147d	test: fix recurring CI flakes in desktop and Postgres lanes (#8343) ... Fixes the CI flakes that failed most often across `main` and unrelated branches from Oct 5 to Oct 14. The counts come from 150 failed `CI` runs. Every change except the nextest config is test-only. | Flake | Runs hit | Root cause | Fix | |---|---|---|---| | `probe_codex_acp_version_…descendant_holds_pipe_open` | 5 | The test writes its script in-process and then execs it. If a sibling test spawns a process in that window, the child inherits the open write fd and Linux rejects the exec with `ETXTBSY`. The probe swallows that error as `None`. | Write probe scripts from a short-lived `/bin/sh` (`write_executable_script`) | | `empty-edit-delete.spec.ts` (`:51`, `:74`, `:95`) | 11 | The seeded `mock-general-welcome` id isn't hex, so `formatTimelineMessages` can't overlay its edit. The post-subscribe window refetch then reverts the row. | Emit a per-test message with a real id after the exact channel subscription is live | | `huddle-transcription.spec.ts:798` | 9 | The composer's delayed initial autofocus (TipTap `focus()` runs in rAF) lands after the voice popover opens and dismisses it. | Wait for `message-input` focus before opening the popover | | `voice-note.spec.ts:906` | 6 | The fixture `ping.mp3` is about 1s long, so playback can end before `paused === false` is observed. | Loop the fixture audio | | `workflows.spec.ts:291` | 5 | The `Message text` fill runs before the inspector swaps to the new step, so it fills the trigger's condition instead. | Scope the fill to the step inspector after its `Action` menu appears | | `cluster_global_probe_commits_tokens_and_sessions_prove_coverage` | 3 | The `max-threads = 1` group only serializes these tests with each other. Other tests' sessions still mask cluster-wide `pg_stat_activity` (`MaskedActivity`). | `threads-required = "num-test-threads"` | | `test_usage_metrics_lock_has_single_owner_and_releases_on_drop` | 1 (this PR) | The test retries the lock right after dropping the leader. Postgres releases a session advisory lock only after the backend notices the disconnect, so the retry can run first. | Poll for the successor, bounded at 5s | ### Validation - **ETXTBSY:** a Linux experiment with Rust 1.95 using the same write→chmod→spawn pattern under concurrent spawns hit `ETXTBSY` 4–7% of the time. Using the `sh` writer, it hit 0 of 6,000. I did not catch it inside the full suite locally, so this fix is based on the mechanism, not an in-suite repro. All 11 probe tests pass on Linux and macOS. - **empty-edit-delete:** the baseline failed 16 of 45 runs (`--repeat-each`, 6 workers, no retries). The spec fix alone brings that to 4/45. The remaining failures come from a separate product race, described below. - **huddle:** the baseline failed 5 of 10 runs. With the fix, 60 of 60 passed without retries. - **nextest:** checked with a throwaway crate on 0.9.136. A grouped test with `num-test-threads` ran alone, even with `max-threads = 1`. - I have not stress-run the voice-note and workflows fixes; they rely on CI. ### Not in this PR - **Product bug in `useChannelLinks`:** the debounced `#channel` autocomplete close lets Enter accept a stale suggestion for about 120ms. That accounts for the remaining `empty-edit-delete` failures, and it affects users too. With the spec fix plus a synchronous close, the spec passed 120 of 120 runs. This needs a product decision, so it's held for a separate PR. - **Composer autofocus race:** the delayed autofocus can close an overlay the user just opened. The real fix is to revalidate when the delayed focus runs. Overlaps #6530. - **`thread-head-stale-edit.spec.ts:55`:** a forced click on a pointer-events-none action bar is suspected, but I couldn't reproduce it. - **Remaining ETXTBSY exposure:** about 40 other Rust tests write and then exec scripts, and 17 specs still edit the non-hex seed message. Root-cause lanes by Princess Donut, Mongo and Mordecai. Carl integrated and verified them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Wes <wesbillman@users.noreply.github.com> Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz> Co-authored-by: Princess Donut <5d97ac8c272fa949af56c71e586146b1706e5c4c3daa125db5b4625e12e0686c@buzz.block.builderlab.xyz> Co-authored-by: Mongo <81cabd2ca1792494372ed410283c00770c0cf35afca052e940fa522883212b45@buzz.block.builderlab.xyz> Co-authored-by: Mordecai <f314b6033b6e13df94c8e9e1187d3666b6b9ce54e66ee986e31cdaa909e8dcbc@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 20:32:00 -06:00
Bradley Axen	8b60dcf8e3	feat(relay): deliver full signed events to operator listeners (#8309) ... ## Summary Operator-listener mention notifications now carry the full signed event, so a listener such as kgoose can wake an agent without first fetching the event (one signed round trip less on time-to-first-reply). Because the notification now carries message content, two mitigations come with it, as agreed in #buzz-security: 1. **Each target signs its own registration.** `POST /operator/listener/pubkeys` takes `{"registrations": [proof, …]}`. Each proof is a kind `24245` event signed by the target pubkey, with exactly one `p` tag (the listener pubkey) and one `u` tag (`{RELAY_OPERATOR_API_ORIGIN}/operator/listener/pubkeys`), and a `created_at` within ±60s. The old unsigned `{"pubkeys": […]}` POST body is rejected with 400. `DELETE` is unchanged. 2. **Only notify a target that can read the event.** Before each delivery the relay re-checks, on the writer, that: - the registration still exists; - the event isn't deleted; - neither the target nor its recorded NIP-OA owner is banned; - the target passes the relay-membership gate (with the owner path when `allow_nip_oa_auth` is on); - the channel is open, or the target is an active member. A delivery that fails the check retries with backoff instead of being dropped, because a mention can commit just before the target is added to the channel. It fails permanently, with no HTTP request, once attempts are exhausted. The payload is now `v: 2`: the v1 fields plus `event`, the full signed NIP-01 event. There's no migration. Existing registrations stay in place and age out under the 30-day retention; only a configured listener could have created them. Listeners must sign proofs to register new targets, so receivers have to accept `v: 2`. Companion kgoose change: squareup/cash-server#136794 signs registrations with the agent key. The kgoose receiver in squareup/cash-server#136004 currently accepts only `v: 1` and bodies up to 4 KB, so it needs to accept v2 and a larger body. ## Testing - Unit tests: - proof verification (forged signer, wrong kind, stale or future, other or ambiguous listener, other or missing endpoint); - signed-only request bodies; - delivery payload carries a verifiable event; - unreadable or failed loads retry, then fail, without an HTTP request. - Postgres tests: - every access predicate (open, member and non-member private, channel-less, closed community with target or owner membership, owner ban, removed member, deleted channel or event, removed registration); - the route test (signed registration accepted; a proof for another listener and the unsigned body both rejected). - Mutation runs confirmed that the membership, owner, ban and reader-visibility guards each fail a test when removed. - Local relay exercise with a fake listener: - signed registration returned 200; unsigned, stale and wrong-listener proofs returned 400; - open-channel mentions delivered v2 with an event whose id and signature verify independently; - a private-channel mention of a non-member retried without delivering, then delivered once the target was added. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Bradley Axen <baxen@squareup.com> Co-authored-by: murderbot <3754f8729004d95654c46dbab3129e4ab9ef05cc2534e2a3fbfc155983bd637b@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 17:53:25 -07:00
Will Pfleger	e645918c63	fix(relay): match pushed-down tags by position, not containment (#8335) ... 🤖 The relay pushes several tag filters into SQL as JSONB containment (`tags @> [[name, value]]`). Containment ignores element position, so a tag like `["x","e",<id>]` or `["meta","t",v]` satisfies `#e:[<id>]` or `#t:[v]`, and `#t:["t"]` matches `["t","other"]`. Because the predicate runs before `LIMIT`, a newer event that only matches by containment can take a `limit: 1` slot; the exact NIP-01 post-filter then drops it and the real match is never returned. Where no post-filter follows, the decoy is returned or counted outright. `push_exact_tag_match` in `buzz-db/src/store/event.rs` keeps containment as the GIN-indexed prefilter (one array-bound `tags @> ANY(...)`, so the e-tag planner shape is unchanged) and adds a positional `EXISTS (... tag->>0 = name AND tag->>1 = ANY(values))`. It now backs every `EventQuery` tag pushdown: | Pushdown | Before | Effect of over-match | |---|---|---| | `#e` (`e_tags`, query and COUNT) | containment | REQ `limit` slot theft; COUNT has no post-filter and counted decoys; thread aux closure returned decoy reactions/replies | | artifact `#d` (`d_tag_values`) | containment | `limit` slot theft on NIP-AR identity lookups | | `custom_tag` (bridge `#buzz-channel`) | containment | no post-filter: a decoy project/repository was returned for its channel | | shared-gated visibility (`shared_gated_reader`) | containment of `["shared","true"]` | a private event tagged `["x","shared","true"]` took a page slot before `event_visible_to_reader` dropped it, so the page came back short. Never a visibility leak: the post-filter stays in place | **Bridge `/count` honors `#buzz-channel`.** `POST /query` scoped project/repository reads by `#buzz-channel`, but `POST /count` ignored it, so a count for one channel counted matching projects in every channel the reader can see. `/count` now sets the same `custom_tag` on both its channel-scoped and access-scoped paths, and the COUNT SQL builder pushes `custom_tag` down exactly like the query builder, so `/count` and `/query` select the same events on both the fast count and the post-filtered fallback. Left unchanged: the retention-free purge predicate for mesh status (`tags @> [["k","buzz-mesh-status"]]`). It mirrors the migration 0019 trigger, has no `LIMIT`, and only affects how the author's own deleted event is purged. REQ `#t` outside group-state kinds is still post-filtered only. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz>	2026-10-15 00:37:11 +00:00
Logan Johnson	71d7aa0436	feat(relay): add the system channel type (#8180) ... 🤖 ## Summary Buzz uses channel membership to decide who can read what, and every channel a person belongs to shows up in their client as a chat. So there is no way to make a group that exists only for access control, for example "only this agent and its owner can read the agent's config". Such a channel would appear in both people's sidebars, search results and unread counts. This stack adds a **system channel**: an ordinary channel that queries leave out unless they ask for it. This first PR adds the type itself. - `kind:9007` (create channel) with `["channel_type", "system"]` creates a system channel. - It is private unless the creator sends `["visibility", "open"]`. - While it is private, its channel metadata event (kind:39000) carries the NIP-29 `hidden` marker, which means "hide this group's metadata from non-members". Until now the relay set `hidden` only for DMs. On its own this PR does not yet hide system channels from queries. Part 2 does that. Part 1 of 3, stacked on #8176 (the channel labels and creator lookup stack, which this feature uses). ### Related issue None found. ### Details - Migration 0062 adds `system` to the `channel_type` Postgres enum. No new columns. - `buzz_core::channel::ChannelType::System`, and `system` is now a reserved label name (labels cannot equal a channel type name). - `buzz-admin` reconcile writes `hidden` with the same rule as the relay. Usage metrics count system channels. - **Rollback.** A relay older than this change cannot read `channel_type = 'system'`. Do not roll back past it while system channels exist. - **Spec.** NIP-CL gets a "System channels" section. `ARCHITECTURE.md` lists the `System` type. ### Testing - Relay test: a system channel is private and `hidden` by default; opening it removes `hidden`; its first `t` tag is `system`. - Migration embed and usage-metrics tests. ### Stack These are two stacks. Merge them in this order. Client support (creating, labeling and finding these channels) will be in block/buzz-app, not the buzz SDK and CLI. **Stack 1: channel labels and creator lookup (#8174–#8176)** 1. #8174 channel type and creator tags, and the channel creation time 2. #8175 channel labels 3. #8176 database-side label and creator match, and the rule for choosing one channel **Stack 2: system channels (#8180–#8182)**, stacked on stack 1 because it uses labels and the creator tag 4. #8180 the system channel type ← this PR 5. #8181 leave system channels out of reads that do not name them 6. #8182 channel-state filters opt in to system channels --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-14 19:03:34 -04:00
Logan Johnson	a3909acc67	feat(relay): match label and creator filters before the page limit (#8176) ... 🤖 ## Summary After parts 1 and 2, a client can ask for "channels with label X" or "channels created by key K". But the relay applied these tag filters after it read one page of the newest channel events. So a request such as `{kinds:[39000], #t:["workspace"], limit:10}` could return fewer than 10 channels, or none, while matching channels existed. This PR moves the label and creator match into the database query, before the page limit. Part 3 of 3, stacked on part 2. ### Related issue None found. ### Details - Applies only when every kind in the filter is a relay-signed channel-state kind (39000–39003). Other kinds keep today's behavior. - The database match uses JSONB containment on `tags`, which is GIN-indexed. Containment can match a little too much (for example, a tag that contains both strings in another position), never too little. So the relay still checks each returned event with the normal Nostr filter rules, and `COUNT` requests keep that check too. - The same match is applied to the query path and the count path. - **Spec.** NIP-CL now says that `#t` and `#P` match before `limit`. - **Choosing one channel.** The relay does not make labels unique, so two creates for one label can both succeed, for example when two processes of one agent start at the same time. NIP-CL gets a "Choosing one channel" section: a client that needs one channel per creator and label looks it up with `#P` + `#t`, creates it only if none exists, and otherwise uses the oldest one (the lowest `created_at` tag, then the lowest channel ID). Every reader that follows this rule picks the same channel, even after an edit. ### Testing - Relay test: many channels, one match, small `limit`. The request returns the match for both `#t` and `#P`. This test fails without the change. - Unit test: `#t` and `#P` reach the database query only for channel-state kinds. - Relay test: one creator sends two creates for the same label at once. Both channels exist, a `#P` + `#t` lookup finds both, and the oldest-channel rule picks the same one in either order, even after the older channel's metadata is re-published by an edit. ### Stack These are two stacks. Merge them in this order. Client support (creating, labeling and finding these channels) will be in block/buzz-app, not the buzz SDK and CLI. **Stack 1: channel labels and creator lookup (#8174–#8176)** 1. #8174 channel type and creator tags, and the channel creation time 2. #8175 channel labels 3. #8176 database-side label and creator match, and the rule for choosing one channel ← this PR **Stack 2: system channels (#8180–#8182)**, stacked on stack 1 because it uses labels and the creator tag 4. #8180 the system channel type 5. #8181 leave system channels out of reads that do not name them 6. #8182 channel-state filters opt in to system channels --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-14 19:03:34 -04:00
Logan Johnson	2f79374f48	feat(relay): channel labels (#8175) ... 🤖 ## Summary A plugin or agent often needs to tag its own channels and then find them again, for example "my attention-config channel" or "all channels of this workspace". Today a channel has no labels: the relay ignores any `t` tags in create or edit requests. This PR adds channel labels. - A create request can carry up to 8 labels as `["t", "<label>"]` tags. - An edit request with `t` tags replaces the label set. `["t", ""]` alone clears it, the same convention as `["ttl", ""]`. Only owners and admins can change labels, as for the name. - The relay publishes each label as a `t` tag on the channel's state events, after the channel type tag. Part 2 of 3, stacked on part 1. ### Related issue None found. ### Details - **Rules.** A label uses 1–64 characters from `a-z`, `0-9`, `.`, `:` and `-`. The relay refuses a label equal to a channel type name (`stream`, `forum`, `dm`, `workflow`, `system`), so a label cannot make a channel look like another type. - **No trust from labels.** Any admin can set a label. Only the relay-written creator tag (part 1) says who made a channel. - **Storage.** New migration `0061_channel_labels.sql` adds `channels.labels TEXT[] NOT NULL DEFAULT '{}'`. Channels without labels look and behave as before. - **Readers.** The CLI and `buzz-acp` read the channel type from the first `t` tag. Before this PR they used the last one, which would now be a label. - **Spec.** NIP-CL gets a "Channel labels" section with these rules. ### Testing - Relay tests: labels at creation appear after the type tag, edit replaces and clears them, invalid labels are refused, and members who are not admins cannot change them. - Unit tests for the label rules: allowed characters, length, reserved type names, the 8-label cap and the clear marker. - `buzz-acp` test: the channel type comes from the first `t` tag, not a label. ### Stack These are two stacks. Merge them in this order. Client support (creating, labeling and finding these channels) will be in block/buzz-app, not the buzz SDK and CLI. **Stack 1: channel labels and creator lookup (#8174–#8176)** 1. #8174 channel type and creator tags, and the channel creation time 2. #8175 channel labels ← this PR 3. #8176 database-side label and creator match, and the rule for choosing one channel **Stack 2: system channels (#8180–#8182)**, stacked on stack 1 because it uses labels and the creator tag 4. #8180 the system channel type 5. #8181 leave system channels out of reads that do not name them 6. #8182 channel-state filters opt in to system channels --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-14 17:25:46 -04:00
Will Pfleger	b3795236cc	fix(relay): reject unreadable write targets as missing (#8249) ... 🤖 A sender who cannot use or read a referenced event, artifact or workflow now gets exactly the response it would get for an ID that does not exist. #8197 (merged) set this rule for deletions; this applies it to the other writes that look up a target, plus the workflow read and webhook endpoints. Artifact `create` and workflow definition saves are the exceptions, described below. Access is checked before any message that depends on the target, readability goes through `event_visible_to_reader` (including #8197's private-deletion rule), and soft-deleted targets keep matching missing ones. | Path | Checks, in order | Response for any unusable target | |---|---|---| | Reaction (kind 7) | target lookup → `event_visible_to_reader` → every channel gate for the derived channel, including archived | `invalid: reaction target event not found` | | Edit (kind 40003) | target in the edit's `h` channel and readable → author / agent-owner check, including an author since removed from a private channel | `invalid: edit target not found or not editable by you` | | Forum vote (kind 45002) | target in the vote's `h` channel and readable → kind check | `invalid: vote target event not found` | | Report by event (kind 1984 `e`) | `event_visible_to_reader` → token channel scope → channel membership or open | `invalid: report target event not found` | | Reply (NIP-10 and workflow `send_message` replies) | parent in the reply's channel and readable | `reply parent not found` | | Artifact update / delete / restore / move | head exists and is homed in `h` (move: in the source the sender can write) → revision, type, deleted and root checks | `conflict: artifact head unavailable` | | Workflow trigger (kind 46020) | workflow exists and sender is its owner | `invalid: workflow not found or not triggerable by you` | | `GET /workflows/{id}/runs` and `/runs/{run_id}/approvals` | workflow exists, has a channel, and the caller can read it | `404 {"error":"workflow not found"}` | | `POST /hooks/{id}` | workflow exists and the provided secret matches its stored secret → definition and trigger-type checks | `404 {"error":"workflow not found"}` | Two of these reject writes that used to be accepted: a reaction to an event the sender cannot read (such as someone else's author-only event), and a report on an event the reporter cannot read. Reports by pubkey and by blob are unchanged. In an archived channel, writes whose channel comes from their target (reactions and deletions) get that target's not-found response, and an artifact move out of an archived source channel gets `conflict: artifact head unavailable` like any other source the sender cannot write; every other write keeps `invalid: channel is archived`. A webhook call with an absent or wrong secret, or to a workflow without a webhook secret, now gets `404` instead of `401`/`400`. Artifact `create` and workflow definition saves still report a taken ID, as NIP-AR allows for artifacts. Removed messages, none of which any client depends on from these endpoints: `edit target event not found`, `must be event author to edit`, `target event belongs to a different channel`, `target event has no channel`, `parent event belongs to a different channel`, `parent event has no channel association`, `artifact home changed`, `workflow is not accessible`, `workflow is not channel-scoped`, the webhook's `authentication failed` and `webhook secret required but not configured`, `forbidden: not authorized to trigger this workflow` for non-owners (it remains for an owner whose channel authority lapsed), and `restricted: not a channel member` for an edit by an author removed from a private channel. `e2e_existence_oracle` checks each event write over HTTP and WS, and the workflow read and webhook endpoints over HTTP, against missing, soft-deleted, other-channel, private-channel, archived, author-only and wrong-kind targets as applicable, with a positive control per path. It runs in the Backend Integration job alongside `e2e_author_only_deletion`. `removed_author_edit_does_not_reveal_target` checks that a removed author's edits of their remaining message, their deleted message and a missing ID get identical answers. In the PostgreSQL suite, `head_outside_authorized_channel_matches_missing_head` covers the artifact transaction, `workflow_reply_to_unusable_parent_matches_missing` covers workflow replies through `RelayActionSink::send_message`, `scoped_token_report_does_not_reveal_target` and `scoped_token_reaction_does_not_reveal_target` cover channel-scoped tokens, and `channelless_workflow_runs_read_as_missing` covers the runs read for a workflow with no channel. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz>	2026-10-14 17:18:38 -04:00
Logan Johnson	ea24a3ee3d	fix(ci): remove Detect Changed Paths timeout and deletion test race (#8300) ... 🤖 ## Summary This PR removes two CI flakes that failed relay-stack runs on 2026-10-13. Both passed when rerun. - **Detect Changed Paths timeout.** This first CI job had a 2-minute limit. Its steps take about 75 seconds, but GitHub sometimes finishes the job minutes after the last step. In three runs, every step had finished, but GitHub still cancelled the job at the limit, and all later jobs failed. The limit is now 10 minutes. - **PostgreSQL deletion test.** `approval_boundary_blocks_claim_until_exact_inventory_is_approved` sometimes could not claim an approved deletion request. This was a race in the test, not a product bug. How the test race happens: - The test sends an UPDATE that the database must reject. Then it claims the same row. PostgreSQL sends the error to the client before it rolls back the statement and releases its row lock. sqlx returns as soon as it gets the error. - The claim runs on another pooled connection with `FOR UPDATE ... SKIP LOCKED`. It can skip the row while the lock is still held, and return no claim. - The test now runs the rejected UPDATE on its own connection. Then it sends `SELECT 1` on that connection before it claims. That round trip completes only after PostgreSQL has finished the rollback. Production code does not change. A claim that skips a row locked by another transaction is correct behavior. I checked the other tests in this file. Only this test has a rejected write on the request row immediately before a `SKIP LOCKED` claim. In the other tests, a blocking `approve` call runs before the claim. ### Related issue N/A. No existing PR found. Failing runs: [timeout #8174](https://github.com/block/buzz/actions/runs/38468285648/job/115540525751), [timeout #8176](https://github.com/block/buzz/actions/runs/38468321774/job/115541550557), [timeout #8182](https://github.com/block/buzz/actions/runs/38468329504/job/115541197994), [deletion test #8182](https://github.com/block/buzz/actions/runs/38468329504/job/115550003726). ### Testing - I confirmed the order in the PostgreSQL 17 source: `EmitErrorReport()` (which flushes to the client) runs before `AbortCurrentTransaction()` in `tcop/postgres.c`. A BEFORE UPDATE trigger locks the row first (`GetTupleForTrigger` in `commands/trigger.c`). - I could not reproduce the failure locally: 0 failures in 410 runs, including runs on a CPU-limited PostgreSQL container. The fixed test passes against PostgreSQL 17. Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-10-14 15:31:05 -04:00
Ian Oberst	2270b66e14	test(buzz-db): drop partition scratch schemas on panic (#8293) ... ## Summary The partition Postgres tests dropped their scratch schema with a `drop_schema` call at the end of each test body. When a test panicked, that call never ran, so the schema leaked into the shared database, along with any `{schema}_*` sibling schemas or reader role the test created. A reviewer on #7993 hit this locally: the leaked schema plus a database-wide catalog query made later runs fail. CI hides it because the test wrapper gives each process its own database. `scratch_pool` / `scratch_pool_with_max_connections` now return `(pool, ScratchSchema)`. The guard owns the admin pool and schema name and cleans up on `Drop`: - **Fresh thread and connection.** `Drop` runs on the test's runtime thread, which can't drive the test's pools while it blocks, so cleanup runs on its own thread with its own runtime and connection. - **Terminates the test's backends first.** Both pools record each connection's backend pid. A test that panics mid-transaction keeps its locks, because the rollback would only run on the blocked runtime. Without this step `DROP SCHEMA` waits on those locks. - **Drops `{name}` plus every schema and role named `{name}_*`.** This replaces the per-test sibling-schema and `DROP ROLE` cleanup. The reader role is now named `{schema}_reader`. - **Bounded.** `lock_timeout = '10s'`. Failures print a warning and never panic inside `Drop`. The 52 call sites are migrated mechanically. Test-only change. ### Related issue Follow-up from review on #7993. No existing issue found. ### Testing - New `scratch_schema_is_dropped_when_a_test_panics_holding_a_lock`: the test body creates a sibling schema, takes `ACCESS EXCLUSIVE` on a scratch table in an open transaction, and panics. The test then asserts no `{name}` / `{name}_*` schema remains. - Proved it can fail: with the backend-termination step disabled, cleanup hits the lock timeout and the test fails, reporting both leaked schemas. - Partition Postgres lane: `scripts/postgres-test-run.sh -p buzz-db --lib -E 'test(partition)'`, 53/53 passed. - The same filter run directly against the shared `buzz` database: 53/53 passed, then 0 `partition_audit_test_%` schemas and 0 `partition_audit_%` roles remained. - `cargo clippy -p buzz-db --tests` clean. Pre-push hooks passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Ian Oberst <ioberst@block.xyz> Co-authored-by: Ian Oberst <ioberst@block.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 11:36:26 -07:00