16 Commits

Author	SHA1	Message	Date
Logan Johnson	6e6d290030	test(db): drop the stale roster probe note from the guard exceptions ... The startup no longer probes the roster trigger, so the comment on the roster guard exceptions now names only the replica-floor probe. Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Logan Johnson <loganj@squareup.com>	2026-10-26 16:57:07 -04:00
Logan Johnson	3f81a0f573	Stop requiring the 0032 roster trigger at startup ... The relay refused to boot unless migration 0032's roster-fence trigger was present and behaved, and it rechecked the trigger catalog after migrating. Since #8316 every roster writer runs the same check in Rust (validate_channel_roster_snapshot), so the trigger is only a backstop. Remove the startup probe and the post-migration catalog check so a later migration can drop the trigger. Under the docs/database.md retirement sequence this release must be fully deployed before that drop ships. Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Logan Johnson <loganj@squareup.com>	2026-10-26 16:03:11 -04:00
Luke Tornquist	039e1c8d59	feat(relay): write the NIP-43 member list in the membership transaction (#8482) ... ## Summary The relay-signed NIP-43 member list (kind:13534) was published after each membership change had already committed. If that publish failed, or if a change never republished at all (startup `bootstrap_owner`), the list stayed wrong until the background reconcile sweep found and fixed it. #8467 moved that sweep off the startup path. This PR removes the reason the sweep exists. Now every `buzz-db` function that changes `relay_members` writes the new list **in the same transaction**, as its last statement before commit: - It takes the per-community list lock last, reads the members, signs the list, retires the old one and inserts the new one. - If the list write fails, the membership change rolls back and the caller gets an error. - A call that changes nothing (already a member, role mismatch, host exists, limit reached) writes no list. The functions return `MembershipWrite<T> { outcome, snapshot }`. Callers push `snapshot` to live subscribers after commit. That push is the only thing left outside the transaction; if it fails, live clients see the change on their next fetch, and the stored list is already correct. Write paths covered (RPR #2, every write path): `add_relay_member`, `claim_relay_membership`, `claim_relay_invite`, `remove_relay_member`, `remove_relay_member_if_role`, `update_relay_member_role`, `bootstrap_owner` / `provision_owner`, `transfer_ownership`, `backfill_from_allowlist`, and the owner insert in `create_community_with_owner[_and_nip_fi_issuers]`. Community deletion removes the members together with the community, so it writes no list. The relay keypair is passed in explicitly instead of being stored on `Db`, so the compiler finds every caller. ### Behavior changes - **Open relays get an accurate list too.** Provisioning and ownership transfer used to publish only when `require_relay_membership` was on. The write is now unconditional. - **Same-second changes can't reuse an id.** The list's `created_at` is one second past the newest earlier list for that community, including retired ones, or the current time if that is later. Before, adding and then removing a member in the same second could produce an event with the same id as an earlier list. The insert then failed, and the stale list stayed live. A burst of changes can push `created_at` ahead of the clock by up to one second per change; it returns to the clock once the burst ends. - **`buzz-admin add-member` / `remove-member`** no longer build their own list (`publish_membership_list_with_bump` is deleted). They push the list the database wrote. Concurrent CLI runs now serialize on the database lock rather than depending on `run.sh` running one at a time. - **Owner mutations lock the community row `FOR NO KEY UPDATE`.** `transfer_ownership` and `bootstrap_owner` used `FOR UPDATE`. The list insert takes `FOR KEY SHARE` on that row through `events.community_id`'s foreign key, so a repair holding the list lock, or a removal or role update holding a member row, deadlocked with a concurrent owner change. `FOR NO KEY UPDATE` still conflicts with archive, unarchive and deletion admission, so the lifecycle ordering is unchanged. - **Startup delivers its list to other replicas.** When backfill or `bootstrap_owner` changes members at startup, the relay fans the newest list out through pub/sub once its state is ready, so clients connected to other replicas during a rolling deploy see an owner rotation without a refetch. - **New `buzz-admin republish-members`.** Rewrites the community's list from the stored members in one community-write transaction and pushes it live. This replaces the sweep as the repair for a list that is missing or stale: a `relay_members` change made directly in SQL, a restore, or a deployment that turns on `BUZZ_REQUIRE_RELAY_MEMBERSHIP` after running without it. ## Rollout Nothing rewrites a list automatically once the sweep is gone. A community whose list drifted before this deploys stays wrong until its next membership change or until an operator runs `buzz-admin republish-members`. Either deploy #8467 and let its sweep complete one pass first, or run `republish-members` for each community after this deploys. The command repairs only the community that `RELAY_URL` maps to, so on a relay hosting many communities, run it once per community host. ## What got simpler Deleted: - `reconcile_nip43_membership_snapshots`, `spawn_nip43_membership_reconciler`, its leader lock key and #8467's standby retry - the three `nip43_membership_snapshot_needs_reconciliation*` drift reads (one already deprecated) - `publish_nip43_membership_list` / `publish_nip43_membership_locked`, the separate after-commit republish at every call site, and `buzz-admin`'s duplicate publisher - the reconcile metrics `buzz_nip43_membership_reconciliations_total` and `buzz_nip43_membership_reconciliation_failures_total` - the after-commit publish metrics `buzz_nip43_membership_publications_total` and `buzz_nip43_membership_publication_seconds`, and the `publish_nip43_membership_locked` transaction metric label. A failed list write is now a failed membership write and shows up in the caller's existing error handling and the transaction metrics of its write path. - the `BUZZ_NIP43_RECONCILE_INTERVAL_SECS` setting There is now one way the list gets written, and it can't disagree with `relay_members` after commit. **Unused deploy config (outside this repo):** hosted deployments that set `BUZZ_NIP43_RECONCILE_INTERVAL_SECS` can drop it. It is now ignored. Dashboards or alerts on the four deleted metrics will stop receiving data. ## Testing New Postgres tests (`buzz-db`): - `relay_members::every_membership_write_commits_its_list`: for each write path, the returned list is the single live kind:13534 and names exactly the committed members. No-op calls return no list and leave the live list alone. - `relay_members::a_failed_list_write_rolls_back_the_membership_change`: holds the list lock, waits for the join to block on it, cancels that statement, and checks that the member was not added and the previous list is still live. - `relay_members::concurrent_joins_all_land_in_the_list`: 8 concurrent joins; the final list names all 8. - `relay_members::same_second_return_to_an_earlier_membership_writes_a_new_list`: add, remove, add, remove within a second; every list is newer than the last and no id repeats. - `relay_members::owner_transfer_and_list_repair_both_commit` and `relay_members::owner_transfer_and_member_removal_both_commit`: hold a repair (list lock) or removal (member row) open, wait until a transfer blocks behind it, then finish it. Both transactions commit; with `FOR UPDATE` back in place, one of them fails. - `community::create_community_with_owner_is_atomic_and_create_only` and `relay_invite::bounded_claim_exhausts_and_existing_member_retry_does_not_consume` now also check the list. Each new test was checked to fail against a mutation of the new code that brings back the old behavior (details in the thread). Stacked on #8467. Merge order: #8467, then #8468, then this. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-20 11:39:13 -04:00
Luke Tornquist	c99872f749	fix(relay): delete the startup NIP-33 d_tag backfill (#8468) ... ## Summary Delete the NIP-33 `d_tag` backfill that ran on every relay boot. On every boot, before the listener opened, the relay ran: ```sql SELECT DISTINCT community_id FROM events WHERE kind BETWEEN 30000 AND 39999 AND d_tag IS NULL ``` No index covers `d_tag IS NULL`. The only `d_tag` index is the partial `idx_events_parameterized`, which is `WHERE d_tag IS NOT NULL`. So this query scanned every partition of `events`. In production it averaged about 11s per boot, max 19s (`buzz_startup_phase_seconds{phase="d_tag_backfill"}`), and found nothing. These rows shouldn't exist anymore: - The backfill was the one-time migration for #246, the change that added `d_tag`. - In `crates/buzz-db/src`, every production write path sets the column for kinds 30000–39999 through `extract_d_tag`: the main insert, `replaceable.rs`, `channel_members.rs`, `relay_members.rs`, and `push.rs`. A new source test, `every_production_event_insert_names_d_tag`, now enforces that. - The `events` inserts that omit `d_tag` are all test fixtures. `scripts/backfill-d-tag.sql` is still there as the manual fallback for any database that predates #246 and somehow never booted a relay since. ### What got simpler - Removed `Db::backfill_d_tags` and its admission test. - Removed the source-shape assertion in `observability_source.rs`. - Removed the `d_tag_backfill` startup phase. - `StartupStep::ALL` is now a slice, so removing a step no longer means editing a hand-kept array length. ### Notes - The `buzz_startup_phase_seconds{phase="d_tag_backfill"}` series stops reporting. - **Rebased onto main after #8467 merged.** Both remove a startup phase. With the fixed-length `ALL` array the two merged cleanly in git into a 17-entry `[Self; 18]`, which doesn't compile. This PR makes `ALL` a slice. - **Before merging**, run this read-only query against production and confirm it returns `0`: `SELECT count(*) FROM events WHERE kind BETWEEN 30000 AND 39999 AND d_tag IS NULL`. It scans every partition, so use a read replica. If it isn't `0`, run `scripts/backfill-d-tag.sql` first. The cutover script copies legacy `d_tag` values as-is, so only this query proves historical data is clean. - **Self-hosted installs** should run the same check before upgrading. ### Related issue None found. Companion to #8467, which also targets relay startup time. ### Testing - New `every_production_event_insert_names_d_tag` (`crates/buzz-db/tests/observability_source.rs`): every production `INSERT INTO events` must name `d_tag` in its column list. Dropping `d_tag` from `push.rs` makes it fail and name the function. - Runtime E2E against a real relay: boot has no `d_tag_backfill` phase; kind:30023 replacement via WebSocket and HTTP keeps only the newer event, an unrelated `d` survives, and an older replay can't win (confirmed in SQL). - Stacked on #8467 at `501dfd83`: a fresh two-replica boot runs all enabled startup phases, with neither `nip43_reconcile` nor `d_tag_backfill` present. Both replicas pass readiness and return the same signed message over authenticated HTTP and WebSocket, and outsiders are denied. - Blox, at `501dfd83`: `just fmt-check clippy security-review-check file-size-check` passes. Every `test-unit` step and `just test` pass except the `buzz-acp` `git_bootstrap` native-git tests, which fail the same way at base `b6aa5539` on Blox, and one `buzz-agent` catalog timing test that fails only under full-suite load and passes when run alone. - `e27139de` adds only a doc comment on `every_production_event_insert_names_d_tag`, saying which SQL shapes the text match covers. No code changed, so the results above still apply. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-20 11:22:09 -04:00
Luke Tornquist	e6c56ee5ee	fix(relay): take NIP-43 snapshot reconcile off the startup path (#8467) ... ## Summary Before the relay opened its listener, it reconciled the NIP-43 membership snapshot for every active community. Each community costs two writer round trips, one after the other. On a production deployment with about 80k communities, this one step averaged 174s of a roughly 180s boot (max 324s). Pods in an availability zone farther from the writer took 3–4× longer than pods in the writer's zone. The step doesn't need to block startup. The snapshots live in the shared database, and the replicas that are already running serve the same snapshots. A new replica has nothing of its own to repair before it serves. The sweep before bind did the same work as the background maintenance sweep; only the metric labels differed. This PR: - Removes the NIP-43 sweep that ran before bind. - Moves the background loop into `side_effects::spawn_nip43_membership_reconciler`. Its first sweep now runs as soon as the task starts. Before, the loop skipped its first tick, so the first maintenance sweep waited a full `BUZZ_NIP43_RECONCILE_INTERVAL_SECS`. - Runs that loop on one replica only. The replica holding the `NIP43_RECONCILE_LOCK_KEY` session advisory lock sweeps; the others retry the lock every `NIP43_STANDBY_RETRY` (5s, capped at the interval). This reuses the usage-metrics leader lease (a detached Postgres session), renamed from `UsageMetricsLeader`/`try_lock_usage_metrics` to the generic `SessionLeader`/`try_lock_session_leader`. - Waits a full interval after each sweep (sleep after sweep), so a sweep longer than the interval never runs straight into the next one. - `reconcile_nip43_membership_snapshots` is now the only entry point and keeps its original signature (`-> Result<usize>`). It's no longer deprecated. ### What got simpler Deleted: - `Nip43ReconciliationPurpose` (the `Bootstrap`/`Maintenance` purposes) and `reconcile_nip43_membership_snapshots_with_purpose` - `Db::bootstrap_community_hosts` and `Db::nip43_membership_snapshot_needs_reconciliation_for_bootstrap` - The startup progress log (`NIP-43 startup reconciliation progress`, `nip43_progress_due`) and its test - The `nip43_reconcile` startup phase: `StartupStep::ALL` goes from 19 to 18 steps The leader lock reuses the existing usage-metrics lease rather than adding a second leader primitive. `StorageAccountingLeader` is still a near-duplicate of it; folding that in is left as identified debt. ### Trade-offs - **Repair before serving.** A newly started replica no longer guarantees it has repaired stale snapshots before serving. The running fleet doesn't guarantee that today either: the background sweep is what repairs drift, and the leader starts it as soon as it takes the lock. - **Rollout load.** Before, every replica swept before binding, and every replica also ran the background loop. Now there is one sweeper in steady state, so writer load from this job drops from one sweep per replica to one per deployment, and rollouts no longer stack boot sweeps. Standbys add one lock attempt every 5s each. - **Leader handoff.** When the leader's process exits (including SIGKILL), its connection closes, Postgres releases the session lock, and a standby retries the lock at least every `NIP43_STANDBY_RETRY` (5s) and sweeps as soon as it gets it. Each attempt also waits for a writer connection, so under database pressure takeover can take longer than 5s. Before this, a standby waited a full interval (3600s in production), so after a rollout replaced the leader nothing swept for up to an hour. If the leader is lost with its host or network instead, the lock is held until the database server's TCP keepalive drops the session, and nobody sweeps meanwhile. The usage-metrics leader already has the same exposure. The lease is checked only between sweeps, so a leader that loses its session mid-sweep finishes that sweep while the new leader starts one; the per-community publication lock keeps that overlap safe. - **Snapshot window.** A missing or stale kind:13534 snapshot is served until the leader's sweep reaches that community; before, a booting replica blocked until it was repaired. Admission reads canonical `relay_members`, so this window doesn't change who's admitted. It does affect clients that read the snapshot: desktop onboarding shows the membership-denied page to a member missing from it, the desktop/mobile member lists and join alerts lag, and desktop mesh discovery keeps its existing allowlist when the snapshot is missing. In production the inline publications that keep it current have not failed since 2026-10-07, and the sweep repaired no snapshots in the last 30 days. - **Dashboards and alerts.** The `buzz_startup_phase_seconds{phase="nip43_reconcile"}` series stops reporting. Anything charting or alerting on it should move to the sweep's existing counters (`buzz_nip43_membership_reconciliations_total`, `buzz_nip43_membership_reconciliation_failures_total`). ### Related issue None found. ### Merge order #8468 is stacked on this PR. Both remove a startup phase, and with the old fixed-length `StartupStep::ALL` array, merging both would have built a 17-entry `[Self; 18]` that doesn't compile. Merge this PR first; #8468 makes `ALL` a slice. ### Testing - `background_reconciler_sweeps_immediately` (private scratch database): spawns the production reconciler with a 24h interval and 24h standby retry, and requires the first sweep to publish a missing snapshot. Fails if the skipped first tick is restored. - `background_reconciler_standby_takes_over_on_short_retry` (private scratch database): with a 24h interval and 100ms standby retry, nothing publishes while another session holds the leader lock; once it's released the standby must publish within 10s. Fails if the standby sleeps the full interval again (checked by reverting that line). - `test_usage_metrics_lock_has_single_owner_and_releases_on_drop` covers the renamed lease. - `maintenance_sweep_skips_archived_and_deleted_communities` now calls the single entry point. - Runtime E2E at `574b28b3`: two real relays sharing one database, 3600s sweep interval, production 5s standby retry. While the leader was alive, the standby made only lock attempts (no sweep or snapshot-comparison reads). SIGKILL of the actual lock holder, 3/3 trials: the survivor took the lock in 4.81–4.86s and repaired the snapshot within 10ms after that. Both replicas served authenticated reads while the initial publication was stalled. - Blox, at `574b28b3`: `just fmt-check clippy security-review-check file-size-check` passes. Every `test-unit` step and `just test` (118 suites) passes except the `buzz-acp` `git_bootstrap` native-git tests, which fail the same way at base `b6aa5539` on Blox (environment, not this change). --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-20 10:19:46 -04:00
Will Pfleger	6286a54321	feat(relay): store per-community NIP-FI issuer bindings (#8372) ... Stack: this PR → #8377 → #8399 → #8401 bb-public creates communities all the time, so a hand-edited `BUZZ_NIP_FI_COMMUNITIES` map can't list them. This PR stores an explicit NIP-FI issuer allowlist for each community in the database and writes it when the community is created. Login checks don't read the table yet; that comes in #8377 together with the database-backed resolver. Operator steps are in [`docs/operator-nip-fi-bindings.md`](docs/operator-nip-fi-bindings.md). **Storage.** Migration `0066_community_nip_fi_issuers` adds `community_nip_fi_issuers (community_id, issuer, created_at)`, with primary key `(community_id, issuer)` and the community write fence attached. The assertion audience is `https://<communities.host>`, so the table has no `aud` column. Zero rows means no binding. The table is included in whole-community deletion and purge. **One bindable rule.** `BUZZ_NIP_FI_PROVISION_BINDING` (`{"host_suffix": ..., "authorized_issuers": [...]}`) is parsed in `buzz-auth` with `BUZZ_NIP_FI_ISSUERS`, whatever `BUZZ_NIP_FI_MODE` is set to, and both the relay and `buzz-admin` use the same parser. Each authorized issuer must be a configured issuer (1 to 2048 bytes), and the suffix must not cover any host declared in `BUZZ_NIP_FI_COMMUNITIES`; an unreadable declaration refuses startup even in mode `off`. `ProvisionBindingPolicy::is_bindable(host)` is true for a host exactly one label under the suffix that is not reserved. It decides creation-time binding, explicit `bind`, and what the lookup route reports. **Binding on creation.** Rows are written in the same transaction as the community insert on all three creation paths (`POST /operator/communities` `create_only`, its legacy convergence branch, and startup bootstrap), and only when the call actually created the community, so existing rows are never rewritten. **Reserved host.** A host whose `https://<host>` equals a configured `command_audiences` value never holds a binding. Operator provisioning of it is rejected with a 400 and the `reserved_host` error code, bootstrap creates it unbound, and the availability route reports it unavailable with an error code. `main.rs` calls `verify_reserved_hosts_unbound` at startup; if a reserved host has rows the relay refuses to start and names the fix (`buzz-admin nip-fi unbind --host <h> --issuer <i>`, or revert the command audience). A database failure during that check is a separate `Check` error. **Operator commands.** - `buzz-admin nip-fi backfill` pages unbound communities by host (1000 per page) and binds each page with one `INSERT ... SELECT` limited to active communities. It skips communities being deleted, always prints `{rows_bound, unbound, unbound_hosts, truncated}`, and exits non-zero while any bindable community is still unbound. Archived communities are bound on purpose. Both bind paths take the shared deletion lock and then lock the community row (`FOR NO KEY UPDATE`, in id order for a batch), the same order deletion uses, and the backfill insert also requires the community to still have no binding, so an operator `bind` that lands after the backfill listed a community wins and the default issuer is not added beside it. - `buzz-admin nip-fi bind --host --issuer` refuses a reserved host, a host outside the suffix, or an unconfigured issuer. - `buzz-admin nip-fi unbind --host --issuer` refuses to remove a community's last issuer (archive or delete the community instead), with one exception: when a valid provisioning policy shows that backfill can't bind the host (it's reserved, or outside the suffix), the last issuer can be removed. That is the reserved-host recovery path in the runbook. A missing or malformed policy doesn't block `unbind`; it just means the exception never applies. **Lookup route.** `GET /operator/communities/nip-fi-binding?host=` (NIP-98, `RELAY_OPERATOR_PUBKEYS`) reads the database and returns `{host, active, issuers[]}`. `host` is the stored host. It returns 404 for an unknown host, 200 with `active:false` for an archived or quiescing community, and `issuers:[]` when no provisioning policy is set. It lists only issuers that are configured and on a bindable host. kgoose depends on this contract; it is also documented in the runbook. **NIP-FA.** The `relay_url` bullet now also says: "An adapter MAY obtain its exact relay list from a configured, authenticated source; it MUST still compare `relay_url` exactly." **Deployment dependency.** The kgoose config side is already handled by [cash-server#137671](https://github.com/squareup/cash-server/pull/137671). In public-community mode it requires the command audience not to be a community audience. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>	2026-10-17 13:13:49 -04:00
Luke Tornquist	97e1f6a082	fix(db): validate channel roster snapshots in the app writer (#8316) ... ## Summary Phase 1 of retiring the channel roster snapshot trigger (`trg_events_guard_channel_roster_snapshot`, migration 0032). Every admitted writer that can store a kind 39002 roster snapshot now runs the same check the trigger runs, on the same transaction and under the same lock: - `LockedMemberSnapshot::replace_member_event` (relay's live roster publisher) validates the snapshot it already holds the membership lock for, before soft-deleting the old head. - `insert_event_on` and `insert_event_with_thread_metadata_tx` (generic event writes). - `replace_parameterized_event_in_transaction_impl` (inside its existing savepoint) and `Db::replace_addressable_event` (admin reconciliation path), after dominance checks and before the old head is deleted. The validator (`validate_channel_roster_snapshot`) is a direct port of `guard_channel_roster_snapshot()`: four-field `p` tags, 64-char hex pubkey, empty relay field, valid role, and an exact match against active `channel_members` rows. It raises the same `23514` messages, so callers still see `DbError::Sqlx(sqlx::Error::Database(..))` with the same code and text. **Lock order:** replacement key, then channel-membership key, then the canonical read, then the old-head `UPDATE` and the write. Legacy writers took the membership key later, at the trigger's INSERT, after the old-head row locks. Taking it before the `UPDATE` is not the same order, but it is strictly safer: it removes the row-lock/membership-key cycle. The membership writers in `channel_members.rs` take the same membership key before they change rows. Not every membership writer does yet: `execute_kick_with_marker` (reached from report enforcement) and `deploy_kick_member` in `relay_admin_actions.rs` set `removed_at` without the key, so a kick can commit after a roster passes validation and before that roster commits, leaving the removed member listed. The retained trigger has the same gap, so this PR neither causes nor closes it. When the trigger also runs it re-takes a key this transaction already holds; transaction advisory locks are re-entrant, so this adds no wait. The validator is one verdict query on the happy path. On failure it raises the trigger's exact error via a `DO` block, so mixed-version behavior is identical; phase 2 can switch to a typed `DbError`. **Unchanged on purpose:** the trigger, its function, migration 0032, and the startup probe `verify_channel_roster_fence_behavior`. Pods still on the old version depend on them. Phase 2 drops them once this version is running everywhere. ## Tests New PG tests in `channel_members.rs` `postgres_tests`, each with only the roster trigger disabled (the helper asserts the catalog check then fails): - `app_roster_guard_matches_trigger_for_roster_input_table`: 19 shape and membership cases (3/5-field tags, short and non-hex pubkeys, relay field, unknown role, duplicate, role mismatch, missing/extra member, uppercase hex, non-`p` tags, empty rosters, all roles). For each one, the app outcome through `Db::replace_addressable_event` must equal the trigger's outcome on a raw INSERT, including SQLSTATE and message. - `app_roster_guard_locked_writer_rejects_stale_and_accepts_current_without_trigger` - `app_roster_guard_covers_generic_and_parameterized_writers_without_trigger` - `app_roster_guard_rechecks_after_concurrent_membership_commit_without_trigger`: the 0032 race. A newer membership commits while an older snapshot waits on the membership key. The barrier is `pg_locks` with a bound, not sleeps. `stale_legacy_roster_cannot_replace_new_locked_snapshot` (`replaceable.rs`) and `desired_schema_rejects_stale_legacy_roster_role` (`channel_members.rs`) now simulate the old binary inline (replacement key, old-head `UPDATE`, raw INSERT) instead of calling the app API, so they test the retained trigger again. The replaceable.rs one keeps its one-connection pool, so it still catches self-deadlock. `observability_source.rs` gains `every_production_event_insert_runs_the_roster_guard`, which requires a real guard call in every production `INSERT INTO events` function, with explicit exceptions (NIP-43 membership and push lease rows, both `channel_id` NULL, and the two rolled-back startup probes), plus a self-test. Falsified: removing one guard call fails the source policy; dropping the role clause or `lower()` fails the parity table; disabling the trigger fails both legacy tests. The PG tests are picked up by the `postgres-ci` nextest filter (`test(/postgres_tests::/)`); the source policy runs in `just test-unit`. ### Coverage limits (review follow-up) - `every_production_event_insert_runs_the_roster_guard` is a presence check. It rejects missing, commented, and quoted calls, but it does not prove a call is awaited, reachable, or passed the event's real kind and channel. Behavior is covered by the PG tests above. The test now says this in its doc comment. - Exemptions match by function header. A kind 39002 insert added inside one of the four exempt functions would pass the policy. This is documented next to the exemption list. - The 19-case parity table and the race test drive `Db::replace_addressable_event` only. The generic, parameterized, and locked writers get the mismatch case only, and the parameterized path's membership-key-before-delete ordering is not exercised concurrently. All five writers call the same validator, so this PR doesn't widen these tests. - `disable_roster_trigger` commits `DISABLE TRIGGER` on the test database. That is safe under the per-test database harness the `postgres-ci` profile uses, but not against a shared database. Its doc comment now says so. ## Validation Run on a Blox workstation (private PG 16 and Redis) against a tree byte-identical to this commit: - `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo nextest run -p buzz-db --lib`: pass. - `cargo nextest run -p buzz-db --test observability_source`: 19 passed. - `scripts/postgres-test-run.sh -p <pkg>` for buzz-admin, buzz-audit, buzz-deletion, buzz-push-gateway, buzz-relay, buzz-search, buzz-workflow: pass. - buzz-db: the full run passed 513 and was stopped at a 240 s deadline on `writer_pool_rejects_non_read_committed_database_default`, which hangs the same way on the unmodified base. A second run excluding the three host-flaky runtime tests already reproduced on the base passed all 511 others. ## Simpler? Not yet. While the trigger is still in place, the check runs twice. Phase 2 deletes the trigger, its function, the startup catalog and behavior probes, the migration.rs equivalence asserts, and the pgschema reconcile entry, which leaves one check in Rust. ## Phase 2 (separate PR, after fleet rollout) **Prerequisite before claiming complete roster serialization (phase 2 or anywhere else):** make both kick paths take the channel membership key before they change rows, and add a concurrent kick-vs-roster-publication test. The retained trigger does not prove this; it has the same gap. A new migration drops the trigger from `events` and its partitions, then drops `guard_channel_roster_snapshot()`. It also removes the schema files and includes, `verify_channel_roster_fence*` and the relay startup call, the migration.rs asserts, and the probe tests, and updates `scripts/reconcile-schema-after-pgschema.sql`. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-16 17:07:25 -04:00
Luke Tornquist	60de1db0b8	fix(db): admit fleet sweeps per tenant instead of in-SQL fence predicate (#8318) ... ## What Removes every production call to the SQL fence predicate `community_write_allowed()` from application SQL, so no application mutation depends on the boolean predicate anymore. This is one prerequisite for BUZZ-315 part B; #8322 (tolerating absent fence objects) is a separate one. The triggers and SQL functions stay in this PR because mixed-version pods still depend on them. The predicate isn't inlined. It's a VOLATILE plpgsql function that takes the deletion lock and *then* reads lifecycle state on a fresh snapshot, and an inline expression would read state from the statement's start snapshot. A separate earlier check would race deletion. So each sweep now: 1. reads its candidate community IDs with a plain read, bounded the same way the original statement was, 2. opens one `AdmittedTx` per community (`begin_community_sweep_transaction`, a thin wrapper over `begin_community_write_transaction` that turns an admission refusal into "skip this tenant"; the four periodic sweeps go through `sweep_admitted_communities`, which also owns their error handling), 3. runs the original mutation scoped to `community_id = $1` and commits. | Site | Change | | --- | --- | | `runtime/push_enqueue.rs` | Deleted the redundant predicate. This transaction already calls `guard_transaction`. | | `store/relay_invite.rs` `reap_expired_relay_invites` | Keeps the global expiry order and the 1,000-row ceiling. Candidates are pre-filtered to active tenants so fenced invites can't fill the batch. Admission is still what authorizes the write. | | `store/workflow.rs` `prune_scheduled_workflow_fires_before` | Prunes per admitted community and sums the row counts. | | `store/channel.rs` `reap_expired_ephemeral_channels` | Archives per admitted community and returns the committed rows, so the relay still runs their system and discovery follow-ups. The candidate read has no lifecycle pre-filter (it has no `LIMIT`, so admission alone decides). | | `store/event.rs` `backfill_d_tags` | Backfills per admitted community at startup. | | `store/push.rs` `claim_due_match_batch_with_loader` | Picks the earliest due active community, admits it, and claims under the same `FOR UPDATE OF q SKIP LOCKED` and batch limit. It skips at most 16 tenants that fence between the read and admission. | | `store/push.rs` `reap_exhausted_matches` | Reaps per admitted community. | **Tenant failures.** The four periodic sweeps (invite reap, workflow fire prune, channel reap, exhausted match reap) log a warning and continue when one tenant fails. That tenant's rows are left unmodified and stay eligible, so the next tick retries them. A tenant error doesn't abort processing of the other tenants already in the selected candidate page. Invite cleanup keeps its oldest-first 1,000-row selection, so if one tenant owns that entire page and its delete keeps failing, tenants outside the page wait. No persistent tenant-local failure for that delete is known in the current schema, and base aborted the whole page on any error. The returned count or rows cover the tenants that committed. Claim, enqueue and the startup backfill still return the error to the caller. **Database outages.** A connection-level error (pool acquire timeout, closed pool, socket I/O; `DbError::is_connection_unavailable`) isn't tenant-local, so it stops the periodic sweep at that tenant instead of moving on. Without this, an outage that began after the candidate read cost one writer acquire timeout and one warning per remaining tenant, and invite cleanup runs inside the usage-metrics leader tick. Rows already committed are still returned, so reaped channels keep their follow-ups. Remaining tenants retry next tick. **Known limit (pre-existing, tracked separately):** - *Claim at a locked head.* `claim_due_match_batch_with_loader` returns `None` once an admitted tenant claims zero rows. If every due row in the oldest active tenant is row-locked, younger tenants aren't tried on that call. Base behaved the same way. This PR doesn't change it. ## Tests All tests use a one-session fixture (`test_support::admission_fixture`) with `session_replication_role = replica`, so the trigger backstop is off and only application admission protects the fenced tenants. The fixture shadows `community_write_allowed` with a function that raises, in a per-call `admission_test_<uuid>` schema. - One test per site: a quiescing tenant's rows are untouched while an active tenant in the same sweep still progresses. - Race tests for invite reap, channel reap and claim. These are the sites where the candidate read already filters out fenced tenants. Each test holds the tenant's deletion lock, waits until the sweep is blocked in admission, fences the tenant in that transaction, and asserts its rows are untouched. The claim test also asserts that a second active tenant is claimed in the same call. - `channel_reap_returns_committed_rows_after_earlier_tenant_error` and `periodic_reaps_continue_after_tenant_error_and_retry_retained_rows` inject a per-tenant failure. They check that later tenants commit, that the failed tenant's rows are retained, and that the next call processes them. - `fleet_sweep_stops_at_first_unavailable_connection` runs the sweep helper against a closed pool with three candidates and asserts only the first is attempted. `fleet_sweep_returns_committed_results_when_database_drops_mid_page` (PostgreSQL) commits the first tenant, closes the pool, and asserts the first tenant's result is returned and the sweep stops at the second of three candidates. `connection_unavailable_covers_only_pool_and_io_failures` pins the classification; the per-tenant failure tests above confirm a database-raised error still continues. Falsification, each with the tests failing as expected: | Mutation | Result | | --- | --- | | Original SQL restored at invite sweep, push claim, d-tag backfill | 0/3 pass (`legacy SQL fence predicate called`) | | `guard_community_write` returns `Ok(())` after taking the lock | race tests + channel reap test 0/4 pass | | Per-tenant errors abort the sweep again | 0/2 pass | | Sweep helper `continue`s instead of stopping on a connection-level error | `fleet_sweep_stops_at_first_unavailable_connection` and `fleet_sweep_returns_committed_results_when_database_drops_mid_page` fail | | Sweep helper drops already-committed results when it stops (`return Vec::new()`) | `fleet_sweep_returns_committed_results_when_database_drops_mid_page` fails | | Sweep helper removed from the source-policy chokepoint check | `serving_table_writes_expose_syntactic_chokepoint_or_guarded_tx_routes` fails | The PostgreSQL tests are under `postgres_tests` modules with `#[ignore = "requires Postgres"]`, which puts them in the `.config/nextest.toml` PostgreSQL filter. CI evidence is under Verification. ## Verification CI at head `5d37fbda` ([run 38925996275](https://github.com/block/buzz/actions/runs/38925996275)) is green. The PostgreSQL job ([job 117062895683](https://github.com/block/buzz/actions/runs/38925996275/job/117062895683)) passed 1017/1017. That includes named PASS lines for all 30 `admission_postgres_tests`, `fleet_sweep_returns_committed_results_when_database_drops_mid_page`, and both writer-pool tests listed below. Unit Tests ([job 117058136482](https://github.com/block/buzz/actions/runs/38925996275/job/117058136482)) has PASS lines for `connection_unavailable_covers_only_pool_and_io_failures` and `fleet_sweep_stops_at_first_unavailable_connection`. Local gates ran on a Linux cloud workstation against private PostgreSQL 16 and Redis. The tree was `735170df`, which is this head rebased on `main` `c0ab789`, so the gates built the merge result. | Gate | Result | | --- | --- | | `cargo fmt --all -- --check` | pass | | `cargo clippy --workspace --all-targets -- -D warnings` | pass | | `cargo nextest run -p buzz-db --lib --tests` | 185 passed | | `scripts/test-postgres-test-discovery.sh` | pass | | PostgreSQL lane for buzz-db (excluding the two writer-pool tests below) | 530 passed | | PostgreSQL lanes for buzz-admin, buzz-audit, buzz-deletion, buzz-push-gateway, buzz-search, buzz-workflow | all pass | | PostgreSQL lane for buzz-relay | 413/426. The 13 `group_state_postgres_tests` fail with `PoolTimedOut` on unmodified `main` `c0ab789` too (see below) | | The three mutations in the falsification table above (`continue` ×2, `return Vec::new()`) | each fails its target test | These failures happen only on the workstation, and each one passes in CI's PostgreSQL job at this head: - `writer_pool_metrics_stop_after_session_timeout_setup_failure` stalls. It's an unchanged writer-pool configuration test, and it stalled the same way at earlier heads. - The 13 relay `group_state_postgres_tests` fail at the same point, `ensure community: Sqlx(PoolTimedOut)`, on unmodified `main` `c0ab789`. This PR doesn't touch `buzz-relay`. ## What got simpler - No application mutation calls the boolean predicate `community_write_allowed()` anymore. Dropping the fence objects still has other dependencies: serving-write lease acquisition calls `assert_community_write_allowed()` (`store/deletion.rs:3068`), catalog validation requires both functions (`store/deletion.rs:1195-1198`), and the triggers and compatibility SQL still use them. Those belong to #8322 and part B, not this PR. - The claim's `target` CTE is gone, and every sweep uses the same admission-and-skip helper. - The four periodic sweeps share one loop (`sweep_admitted_communities`) for admission, skip, continue-on-tenant-error and stop-on-outage, instead of four copies of it. - The source policy treats that helper as a tenant-local chokepoint, so `backfill_d_tags` no longer needs a reviewed exception. - The per-tenant loops add code, because each community now gets its own admitted transaction. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-16 15:46:37 -04:00
Luke Tornquist	c0ab789499	feat(buzz-db): retire the channel-TTL trigger (#8210) ... ## Summary This drops the channel-TTL event trigger. The application already refreshes ephemeral-channel deadlines in the event transaction (`event_follow_up::refresh_channel_ttls`, from #8135), so the trigger only repeats that work. This PR originally retired both follow-up triggers. #8352 has since retired the push-match trigger in migration 0063, so after the rebase this PR covers only the TTL half. Migration `0065_retire_channel_ttl_trigger.sql`: - `SET LOCAL lock_timeout = '5s'` (the same bound as 0063) - drops `events_refresh_channel_ttl` and `refresh_channel_ttl_after_event_insert()` Dropping the trigger on the partitioned parent also drops its partition clones. Published migrations, including 0022 and 0024, stay checksum-frozen. The same objects are removed from `schema/` (`schema/functions/…`, `schema/triggers/events.sql`, `schema/schema.sql`) and from the pgschema reconcile script. After this, migrated and desired-state databases have no event follow-up triggers. Tests: - `follow_up_retirement_tests.rs` (renamed from `push_retirement_tests.rs`): the push-retirement upgrade test now stops at 0063. A new test upgrades 0064 → 0065 and checks that the TTL trigger is gone from the parent and every partition, that the function is gone, and that the published checksums haven't changed. - `event_follow_up_postgres_tests.rs`: the contract now has two arms. `AppOnly` is the shipped schema, where both triggers are absent. `Dual` reconstructs the historical overlap of both triggers. The `PushRetired` arm (the push trigger dropped, the TTL trigger kept) was the shipped state before this PR. Its tests are removed because `Dual` and `AppOnly` together cover it. - `embedded_migrator_contains_consolidated_initial_schema` pins the 0065 statements and checks that both objects are absent from the desired schema. **Deploy gate:** do not run 0065 against a database still served by a relay older than #8135. After 0065, raw SQL writers no longer refresh TTL deadlines. `docs/push-gateway-deployment.md` is updated to match. ### What got simpler The trigger and the app no longer both refresh TTL deadlines; the app is now the only producer. One plpgsql function and one trigger are gone from both schema paths, and 8 per-partition drops are gone from the reconcile script. The follow-up contract drops from three arms to two. ### Related issue BUZZ-313 (Linear), part of the trigger-retirement work. ### Testing Head is `4f7d0baf0`. Results below are at `678927730` unless noted. `678927730..4f7d0baf0` changes only one test in `crates/buzz-db/src/store/event.rs`, described in the first bullet. - **Fixed test, `permanent_channel_event_commits_do_not_lock_the_channel_row`** (`4f7d0baf0`). - **Why it changed:** after 0065, its raw-pool inserts no longer ran any TTL refresh, so the test passed whatever `refresh_channel_ttls` did. This was raised in review. - **What it does now:** it inserts through the production `AdmittedTx` path and keeps the barrier and the tuple-lock probe. It commits all N transactions while an external transaction holds `FOR SHARE` on the channel row. It also asserts that no `buzz_db_channel_ttl_refresh_failures_total` is counted. - **Mutation check:** with `SELECT … FOR UPDATE` injected into `refresh_channel_ttl`, the new test fails (`must not wait on the channel row`, timeout at 10s). The old test passes under the same mutation. - **Blox:** fmt and `buzz-db` clippy pass. - **GitHub CI at `4f7d0baf0`:** every required check passes (56 pass, 0 fail). CI builds the merge ref with `main` `93ffc37cb`. - **New and renamed tests, checked by name in the CI logs at `4f7d0baf0`.** These pass in PostgreSQL Domain / PostgreSQL Tests ([117033725006](https://github.com/block/buzz/actions/runs/38915998692/job/117033725006)): - `permanent_channel_event_commits_do_not_lock_the_channel_row` - `migration_schema_ttl_retirement_drops_every_partition_trigger_and_keeps_checksums` - `migration_schema_push_retirement_preserves_ttl_and_published_checksums` - `event_insert_ttl_refresh_handles_permanent_ephemeral_duplicate_and_activation_race` - `{desired,migration_schema}_event_follow_up_contract_{dual,app_only}` (four tests) `embedded_migrator_contains_consolidated_initial_schema` passes in Rust / Unit Tests ([117024125423](https://github.com/block/buzz/actions/runs/38915998692/job/117024125423)). No `*_push_retired` test runs anywhere. - **Blox gates at `678927730`** (private PG16 + Redis, on a workstation shared with other sessions): - `cargo fmt --check`, workspace `clippy -D warnings` and `buzz-db --lib` all pass. - The `buzz-admin`, `buzz-audit`, `buzz-deletion`, `buzz-push-gateway`, `buzz-search` and `buzz-workflow` Postgres packages pass. - `buzz-db`: 517 of 519 tests passed. The 2 that hung, both `writer_pool_*` tests, were killed. `buzz-relay`: 413 of 426 passed. The 13 failures are covered below. - **Uncontended reruns at `4f7d0baf0`** on a fresh, idle workstation (PG16.15, `competing=0` for every arm): - **`writer_pool_rejects_non_read_committed_database_default` predates this PR.** It timed out 3/3 at head and 3/3 at `main` `93ffc37cb`, with a 300s timeout per run. - **`writer_pool_metrics_stop_after_session_timeout_setup_failure` also predates this PR.** It timed out 2/3 at head and 2/3 at `main`, and passed in the remaining run on each side. - Both tests pass in CI. The gate accepts them as a named exception for this PR, as a pre-existing hang on the PG16 test environment. - **`buzz-db`:** 518 of 519 tests passed. The single failure is `writer_pool_rejects_non_read_committed_database_default`, killed after it hung (the exception above). - **`buzz-relay`: still open.** 413 of 426 tests passed. The same 13 `handlers::group_state_postgres_tests` failed as on the shared box, each panicking at once in its fixture's `ensure community` step with `Sqlx(PoolTimedOut)`. Contention does not explain them. The fixture's community setup is in `buzz-db`, which this PR changes. A head/`main` comparison of the full `buzz-relay` package on the same idle box is running to find out whether this predates the PR. All 13 pass in CI at this head. - **E2E (Gimli, Blox)** at `050e593af`. `050e593af..678927730` changes only tests and docs; the migration, schema and runtime code are byte-identical. Gimli's audited verdict: **EXERCISED AND WORKS**, for DB and relay runtime only. Gimli ran it at `050e593af`; this head's runtime is byte-identical but was not freshly executed. - **Upgrade:** a real relay startup took the schema from 0064 to 0065. That removed the TTL function and the trigger from the parent table and from all 8 partitions. Checksums for migrations 1–64 are unchanged. - **Lock-timeout rollback:** with a real lock held, the migration failed with `55P03` at 5.0007s and the schema rolled back fully to 64. Once the lock was released, a retry applied 65. - **Fresh desired schema:** after pgschema plus reconcile, the retired objects are absent. - **Real signed kind-9 sends:** on both schema paths, these refreshed ephemeral-channel deadlines to about +600s. Permanent channels stayed NULL. - **Raw SQL insert:** at 64 it refreshed the deadline. After 65 it left the deadline unchanged, on both the upgraded and the fresh schema. - **Not covered:** native or device testing, and deploy. - **Review:** Legolas found 0 P0 and 0 P1. His three optional P2s are fixed in `678927730`. - **No human app test:** this is a DB-only change. - **Migration numbering:** #8237 and #8372 also add a `0065_*` migration. Whichever PR merges second must renumber. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-16 11:22:21 -04:00
Luke Tornquist	d25384a4f3	buzz-db: admit every fenced-table writer in app code (#8219) ... ## Summary This makes every serving writer of a community-write-fenced table take community admission in application code. That's the "Dual enforcement" step of the Supported Writer Contract in `ARCHITECTURE.md`. Admission means the shared community deletion lock plus the active/not-deleted check from `deletion::guard_community_write`. It's taken before the transaction's first tenant write or row lock and held until commit. The 32 `community_write_fence_*` triggers, the six fence SQL functions and `validate_serving_catalog` are unchanged. They stay authoritative. Removing them is a separate migration that will be checked table by table. ## Changes - **New admission entry point.** `begin_community_write_transaction` (`runtime/mod.rs`) builds an `AdmittedTx` for non-event writers. It sits next to the existing event-write entry points and shares their implementation. - **Writers moved onto it:** - `api_tokens`, `archived_identities`, `channels`, `channel_members` (including DM and admin paths), `community_bans`, `git_repo_names` - `moderation_actions` and `moderation_reports`, `personal_read_accounts` and `personal_read_frontiers`, `pubkey_allowlist` - `push_leases`, `push_match_queue`, `push_wake_outbox` - `relay_invites`, `relay_members` (including the new-community owner bootstrap and the startup `pubkey_allowlist` backfill), `join_policy_acceptances` - `thread_metadata`, `users`, `workflows`, `workflow_runs`, `workflow_approvals`, `scheduled_workflow_fires` - Personal-read helpers now require `&mut AdmittedTx`. - **`buzz-audit`.** An append now passes admission first, then takes a transaction-scoped audit-chain lock (`pg_advisory_xact_lock`). This replaces the session lock and the `catch_unwind` unlock cleanup. The relay audit worker retries a `lock_timeout` on either lock and keeps the entry; an admission refusal is terminal, as the trigger's refusal was. - **Fleet-wide statements.** These keep `community_write_allowed(...)` in the same SQL statement, because there's no single community to admit up front: - scheduled-fire prune - invite expiry - push matcher claim and exhausted-job cleanup - startup `d_tag` backfill - channel TTL cleanup - **Relay-member removal.** `remove_relay_member` and `remove_relay_member_if_role` run `delete_member_with_acceptances` inside their admitted transaction. The helper no longer opens its own transaction, so the member lock, the `join_policy_acceptances` delete and the member delete all hold admission until commit. `delete_workflow` and `delete_workflow_for_owner` are gone on `main`, and the coordinate deletion path they used to back was already admitted. - **Unchanged exceptions.** The deletion-executor generation exception and the serving-lease path work as before. - **Tables with no production writer.** `delivery_log` and `subscriptions` are only written by a historical cutover script and a test fixture. Their triggers stay. - **No cross-community moves.** No production `community_id` move exists on a fenced table. The only `SET community_id = NULL` is the deletion-executor purge. So no dual-community lock ordering was needed. ## What got simpler - Per-store ad-hoc writer helpers are gone: `begin_event_write_transaction` and `acquire_event_write_connection` in `channel.rs`, and `begin_operation_transaction` in `push.rs`. - Audit append drops the session advisory unlock, the `catch_unwind`/`resume_unwind` panic handling, `log_inner` and the `futures-util` dependency. ## Notes for reviewers - **New crate dependency.** `buzz-audit` now depends on `buzz-db` only to call the free function `buzz_db::deletion::guard_community_write`. `ARCHITECTURE.md` names this as the one sanctioned subsystem edge, and its audit section is updated to match. - **Hot paths.** `ensure_user` first checks whether the user exists and returns without admission when it does (no write happens). Only a real insert takes admission. `touch_api_token` has no production caller today. - **No-op writers stay read-only.** The startup allowlist backfill returns early on plain reads when the legacy table is gone, it has no rows for the community, or the community already has members, so a quiescing deployment community still starts. It re-checks under admission before inserting. - **Owner mutations.** When admission refuses a transfer or a provisioning and a deletion request exists, both report `DeletionPending`, as before. For a transfer, the operator API keeps its coded `deletion_lifecycle_conflict` 409. For provisioning, it keeps the existing uncoded 409, `owner_conflict: community deletion is pending`. - **Per-community worker loops.** `claim_due_push_wakes` and similar now return `AccessDenied` for a community that goes quiescing between the active-host scan and the claim. The worker logs it at `debug` and the next scan skips the community. - **Kick side effects.** `membership_removal_fence` admits before the membership lock to keep the lock order of `add_member`. A community that leaves `active` between the kick and the fence skips live subscription eviction; the membership cache is already invalidated and fan-out re-checks membership. This is documented on the function. ## Testing - **New refusal tests.** 19 tests: the 18 in `store::admission_postgres_tests`, plus `audit_append_requires_application_admission_with_triggers_bypassed`. Each runs with ordinary triggers disabled (`session_replication_role = replica`), checks that a raw fenced insert would succeed, and asserts that the writer is refused for both `quiescing` and `fenced` communities. Active writes still succeed. - **The tests catch a missing check.** Measured in review at `6fc073c3f`: with the guard in `AdmittedTx::admit` bypassed, 25 tests fail. That includes 16 of the 18 database admission tests. The audit test stays green there, because audit calls `guard_community_write` directly. - **Affected PG suites, with triggers on:** 183/183 passed across 16 store suites, push 18/18 run isolated, audit 7/7. The existing artifact, reaction and replaceable admission tests pass. - **Review fixes, each checked to fail without its fix:** - `allowlist_backfill_requires_application_admission` and `owner_mutations_report_pending_deletion_when_admission_refuses` fail with the old `relay_members.rs`. - `empty_allowlist_backfill_skips_admission` fails (`AccessDenied … write-fenced (quiescing)`) without the allowlist-rows early return. - `audit_worker_retries_admission_lock_timeout_until_original_entry_is_appended_once` holds the community deletion lock exclusive and fails without the admission retry arm. - The existing audit-chain retry test now looks for the transaction-scoped lock and passes. - The two owner-revoke mini-schema tests (`bridge.rs`, `ingest.rs`) now include `communities` and `community_deletion_lock_key`, which admission reads; both failed without that. - **Static checks:** `cargo fmt --check` and workspace `cargo clippy --all-targets -D warnings` pass. - **Not run locally:** the `mobile-check` step of `just ci` (the build machine couldn't reach pub.dev, and clean base fails the same way) and the full `just test` lane. CI on this PR covers both. `mesh_demo::demo_join_forwarded_arm_round_trips_echo` fails on the build machine at base `394d4a434` as well; it uses no database. Refs BUZZ-314. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-14 10:58:12 -04:00
Luke Tornquist	16eb0b6685	feat(buzz-db): run push enqueue and channel TTL refresh in app code (#8135) ... ## Summary Moves the producer side of the two event follow-up triggers into application code. The triggers stay in place, so both producers run until a later change drops the triggers after this ships to every pod. Builds on #8130 (`AdmittedTx`). Refs BUZZ-176. ### Push match enqueue New `store::event_follow_up::enqueue_push_match` replaces the `events_enqueue_push_match` trigger (`schema/functions/enqueue_push_match_job.sql`, migration 0023) and matches it exactly: - Same kind allowlist (9, 40002, 45001, 45003), now `PUSH_MATCH_KINDS`. - Takes the same SHARED `buzz_push_gate:<community>` advisory lock as its own statement, so under READ COMMITTED the lease check gets a snapshot taken after the lock is granted. Lease activations still take it EXCLUSIVE in `push.rs`. - Same lease predicate (`active AND endpoint_enabled AND expires_at > now`), then `INSERT push_match_queue ... ON CONFLICT DO NOTHING`. - Fails closed: any error rejects the event, as the trigger did. It runs right after the INSERT, only when the insert affected a row. The consumer side (claims, `lease_until`, attempts, reaper) is already app-owned and is unchanged. ### Channel TTL refresh `AdmittedTx` now records every channel that received an event (except kind 9007 and channel-less events). `AdmittedTx::commit()` refreshes their `ttl_deadline` as the last statement before COMMIT, which keeps the commit-time behavior of the deferred `events_refresh_channel_ttl` trigger (migration 0024): - One savepoint per channel, in sorted order, under the same SHARED `buzz_channel_ttl:<community>:<channel>` lock. - Runs the same `UPDATE` with the same `archived_at`/`deleted_at` filters as its own statement after the lock, so its snapshot follows the lock grant. Its `ttl_seconds IS NOT NULL` qual filters a permanent channel before any tuple lock, so a permanent channel's row is never locked. An ephemeral channel's row lock is held through the remaining refreshes and the COMMIT round trip (the trigger held it only inside COMMIT). - Lock keys come from one builder per protocol (`channel::channel_ttl_lock_key`, `push::push_gate_lock_key`), shared by the SHARED and EXCLUSIVE sides. The shared push-gate wait is observed through `observe_advisory_lock(LockType::PushGate, …)`. - On failure it rolls back the savepoint, logs a warning, and increments `buzz_db_channel_ttl_refresh_failures_total{reason}`, where `reason` is `lock_timeout` or `error`. The event still commits, matching the trigger's `EXCEPTION WHEN OTHERS`. A cancelled statement (`57014`) still rejects the transaction, because plpgsql never caught that either. - While both producers run, an app refresh that hits `lock_timeout` is counted, and then the trigger retries it inside COMMIT and may succeed. So during the dual window the failure counter can over-report and the worst-case wait doubles. Both go away when the retirement PR drops the triggers, after which the counter is exact. ### Call sites Every production `INSERT INTO events` calls `event_follow_up::after_admitted_insert`: - `insert_event_on` - `insert_event_with_thread_metadata_tx` - `LockedMemberSnapshot::replace_member_event` - `replace_addressable_event` - `publish_nip43_membership_locked` - `accept_lease_event` (kind 30350, a no-op today, kept so the policy has no exceptions) `replace_parameterized_event_in_transaction_impl` inserts inside the `parameterized_replace` savepoint. It enqueues the push job inside the savepoint, so a rollback to that savepoint also drops the job, and records the TTL channel only after the savepoint is released. The two rolled-back verification probes (`verify_floor_guard_behavior`, `verify_channel_roster_fence_behavior`) are named exceptions, because they never commit. ### Source policy `tests/observability_source.rs` adds `every_production_event_insert_runs_the_follow_up_hook`. It scans every `crates/*/src` and fails on any function that inserts into `events` without the hook. Matching is normalized (letter case, whitespace, `\` continuations, `\n`/`\t`/`\r` escapes, optional `public.`). The file is masked once first: comments (`//`, nested `/* */`) and string, raw-string (including `br"`/`cr"`), and char literals become spaces. `#[cfg(test)]` markers, `fn` signatures and the hook are all found in that masked view, so a marker in a comment (even a multiline block comment) strips nothing, an `fn` line inside a multiline string doesn't split a function, and a commented-out or quoted hook doesn't count. SQL is still matched in the original text. A table test pins the masker's output for each literal form. A fixture test pins what the scan rejects (no hook, push without TTL, a hook in a line comment, block comment, or string, lowercase or line-broken or escaped SQL, writers after test-only items or under comment mentions) and what it ignores (writers in `#[cfg(test)] mod`). A floor of seven hooked writers catches a scan that silently stops seeing writers. ### Tests `src/store/event_follow_up_postgres_tests.rs` is the contract suite from the BUZZ-176 discovery work, ported to the `AdmittedTx` API. It is a crate-internal `#[cfg(test)]` module because it runs `SET LOCAL` timeouts and in-transaction reads through the crate-private `AdmittedTx::conn()`. It runs on both the desired and migration schemas, in two arms: - **dual**: production during the migration window, with the triggers and the app code both running. - **app-only**: drops both triggers in the disposable database (and asserts no partition clones remain), the way the retirement migration will. The trigger-only baseline is the base branch, where the same suite passed before this change. The suite also pins the two branches that differ from "swallow everything": a `lock_timeout` failure increments `buzz_db_channel_ttl_refresh_failures_total{reason="lock_timeout"}` exactly once, and a statement cancel (`57014`) during the refresh rejects the event with no row, job, or deadline change. ## What got simpler Nothing yet. This adds a parallel producer during the migration window. It is what lets the follow-up PR delete two plpgsql functions, two triggers, and the "keep in sync with migrations/00xx" duplication between `schema/functions/` and `push.rs`. ## Verification Rebased onto `main` @ `745501962`. All runs used a Linux cloud workstation with the pinned toolchain (rustc/clippy 1.95.0, hermit), and the tree was clean before and after each run. The final head, `c5a06a82`, changes only `crates/buzz-db/tests/observability_source.rs` compared with `1eb11bde` (via `fbc9a598`, `7c0baa41`, `9199992c` and `d405b2f6`). Production source is identical between the two. | Check | SHA | Result | |---|---|---| | `cargo fmt --check`, `cargo clippy -p buzz-db --all-targets -D warnings` | `fbc9a598` | pass | | `cargo clippy` (workspace) | `1eb11bde` | pass | | source policy (`--test observability_source`), fmt, buzz-db clippy | `c5a06a82` | 17/17, pass, pass | | `cargo test --locked -p buzz-db` (non-PG) | `c5a06a82` | 148 lib + 17 policy + 5 doctests | | source policy (`--test observability_source`) | `fbc9a598` | 16/16 | | `cargo test --locked -p buzz-db` (non-PG) | `fbc9a598` | 148 lib + 16 policy + 5 doctests | | contract suite (`store::event_follow_up_postgres_tests`, desired/migration × dual/app-only) | `1eb11bde` | 4/4 | | buzz-db PG full | `1eb11bde` | 464/464 | | buzz-relay PG | `1eb11bde` | 378/378 | | GitHub CI | `fbc9a598` | 53 pass, 0 fail (Codex security review pending) | **Falsifications** (each mutation made the named test fail, then was reverted): - dropping the `relay_members` hook → policy names `publish_nip43_membership_locked` - no-op push producer → app-only arms fail, dual arms pass (trigger covers) - skipping the TTL refresh → app-only fails on the deadline; dual fails on the failure-metric assertion - swallowing `57014` → all four arms fail - dropping the metric increment → all four arms fail - unanchored `#[cfg(test)]` match, or no escape handling → fixture test fails - a `#[cfg(test)]` constructor that skips admission, added inside `impl AdmittedTx` (indented or as a column-0 `mod`) → `admitted_tx_is_constructed_only_by_admitting_constructors` fails; putting the strip back into the constructor scan → its fixture test fails - at `7c0baa41`: `accept_lease_event`'s hook commented out and the real call made through an alias → policy names `accept_lease_event` (the same mutation passes under the `fbc9a598` policy); hook matching on unmasked text → the comment/string fixtures fail - at `9199992c`/`d405b2f6`: no nested block-comment depth, no string-escape skip, no `br"`/`cr"` raw-literal prefix, or no raw-string branch at all in the masker → the follow-up fixture test fails - at `c5a06a82`: finding `#[cfg(test)]` in the unmasked text → `writer_after_block_commented_cfg_test` is no longer reported; finding `fn` signatures in the unmasked text → `writer_after_raw_string_example` is no longer reported; no `cr` raw prefix → the masker table test fails on `cr"` - unhooked writers appended to `side_effects.rs` (after test-only items, under a doc-comment mention, with an escaped newline) → policy names each **Live relay E2E**: an independent reviewer ran this at `1eb11bde` against a real relay and CLI, with PostgreSQL 17 and Redis 7, the follow-up triggers absent, and no partition clones. - A kind 9 on a TTL channel advanced the shortened deadline by about 580 s. On a permanent channel, the deadline stayed NULL. - Each accepted event produced exactly one pending `push_match_queue` row with the same event id. - An exact signed replay was a no-op. - While the exclusive TTL lock was held, the event and its job stayed invisible and the deadline didn't change. After release, both persisted once. - Roster create/join/add/remove kept SQL and wire state in step (1→2→3→2), with one current 39002 head. - NIP-33 replace, duplicate, stale, failure and retry cases all passed. Limits: the push lease was a SQL fixture, and push consumers were disabled. Kind 30023 is global, so channel-scoped TTL on the parameterized path is covered only by the PG contracts. No live relay rerun was done at `fbc9a598`, because that delta is test-only. **Latency** (measured at pre-rebase `f8380927`, same production delta; debug build on loopback, 1,000 samples per cell, EVENT→OK): head app-only p50/p99 3.80/4.58 ms (permanent) and 3.87/4.64 ms (TTL), against base trigger-only 3.49–3.53/4.00–4.03 ms and 3.56–3.62/4.30–4.46 ms. About +0.3 ms p50, from the extra client round trips (lock, UPDATE, savepoints, push lock/insert) that the triggers ran server-side. One debug loopback run, not a production p99. Not measured: contention from many writers on one hot TTL channel, where each writer now waits on the channel row lock for about two more client round trips (RELEASE SAVEPOINT and COMMIT) than with the trigger. In review, a burst of 8 concurrent posts to one TTL channel committed with no deadlock or lock timeout. Watch send p99 and `pg_locks` waits on `channels` during rollout. **Known pre-existing, not touched here:** `api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo` is flaky when Redis is present. It returns a 504 in 4 of 5 runs at base `745501962` and 2 of 5 at `fbc9a598`, always at the same assertion. No CI lane selects it. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-09 12:33:31 -04:00
Luke Tornquist	f8bf2c529c	refactor(db): type-level admitted event-write transactions (AdmittedTx) (#8130)	2026-10-08 08:47:07 -04:00
Luke Tornquist	d0fd96fca6	BUZZ-175: Route serving event writes through the tenant lock (#7828) ... ## Summary Route every serving event write through one community-admitted transaction entry point. `Db::begin_event_write_transaction(community)` is now the only `Db` constructor for event-write transactions: it takes the shared community admission lock before returning, so callers that use it cannot take domain or row locks ahead of tenant admission, and a quiescing community rejects the write at entry. Database triggers and commit-time fences remain the authoritative backstop. This migrates normal, replaceable, roster, relay membership, reaction, reminder, admin-delete, push-owned, artifact-accept, huddle-join (48101), workflow-deletion, and command-executor event writes. Coupled mention writes stay in the same transaction; pool-level `insert_mentions` now opens its own admitted transaction. **Review fixes (since `950d5f6`):** - The unguarded event-write constructor is gone. Removed with it: the deprecated `begin_transaction` alias, the test-only replica-floor constructor, and `begin_community_write_transaction` (now identical to the single constructor). Artifact accept, the huddle join, and both workflow-deletion paths now admit before any domain or row lock; `persist_command_event` no longer hand-rolls `guard_transaction`; read-only `query_artifacts` no longer opens an event-write transaction. - `insert_channel_head_checked` is now `insert_canvas_head_checked`: it rejects non-canvas kinds before opening a transaction and always takes the coordinate lock. - Source policy now also scans `buzz-relay`: any function that hands a transaction to an event-write helper must have opened it through the admitted constructor. The scan drops each `#[cfg(test)]` item instead of stopping at the first one: it tracks brace depth and ends the item on a line ending in `;` or `}` (ignoring a trailing `//` comment, even after a string that contains `//`), with a column-0 `}` as fallback. Fixtures pin a unit struct, a multi-line static, a `const { … }` block, a struct with fields, a module, and one-line items with trailing comments, including after `"https://…"` and `"ws://…"` strings; a raw opener after each must still be flagged. Known limit: braces inside string or char literals or comments are still counted, and only `//` trailing comments are recognized, so a test-only item with unbalanced braces in a literal or comment, or with a trailing `/* */`, would hide what follows; none exists today. The scan checks that admission is present; the per-path PostgreSQL tests pin that it comes before domain locks. - `just test-unit` (and the non-nextest fallback in `scripts/run-tests.sh`) now runs `buzz-db --test observability_source`. Before this, no CI job ran these source-policy tests: the unit lane ran `buzz-db --lib` only and the PostgreSQL lane runs only ignored tests. - Renamed or removed since `950d5f6`: `begin_transaction`, `begin_community_write_transaction`, and the replica-floor test constructor are gone; `insert_channel_head_checked` → `insert_canvas_head_checked`; test helper `quiesce_community_for_tests` now delegates to `main`'s `set_deletion_state`, which reads the real `deletion_fence_generation` instead of hard-coding `'0'`; `runtime::insert_mentions` is pinned to the admitted constructor by `pool_level_insert_mentions_opens_the_tenant_local_chokepoint`. **What got simpler:** four transaction constructors collapse to one, and the hand-rolled admission guard is deleted. The compiler rejects callers of the removed constructors, but a transaction opened from `Db::pool()` can still reach the public `*_in_transaction` helpers; only the source-policy scan and the database fences catch that until BUZZ-254. **Behavior note:** #7932 made `insert_event` skip the transaction for kinds that are not listener mentions. Every serving event write now has to run in a transaction that holds the community admission lock, so `insert_event` always opens the admitted transaction again. `main`'s `soft_delete_event_and_update_thread` / `_in_tx` split and its removal-marker guard are kept; the outer function now opens the admitted transaction. ### Related issue [BUZZ-175](https://linear.app/squareup/issue/BUZZ-175). Caller-migration stage following #7706 (merged); rebased onto `main` at `5173fad66`. Follow-up: [BUZZ-254](https://linear.app/squareup/issue/BUZZ-254) makes event-write helpers require a type-level admitted transaction. **Known debt, out of scope:** `create_channel`, `update_channel`, and `insert_thread_metadata` write channel and thread rows (not events) outside the admitted constructor. ### Testing At `efe4a94623dd69973780a3c39f16dd972cf7cb76` on a cloud workstation: fmt; `buzz-db` clippy (all targets, `-D warnings`); `observability_source` 14/14; `just --dry-run test-unit` includes the new step. Reverting just the end-of-item check to an earlier rule makes a trailing-comment fixture fail. At `eaf190ae59d1595741c3d19dc4c23ce492b5f4d4` (the commit after it only changes this test file, `Justfile`, and `scripts/run-tests.sh`): full GitHub CI green; on the workstation, clippy for `buzz-db` and `buzz-relay`, ordinary `buzz-db` 154/154, `just test-unit` green, and the PostgreSQL lane 805 of 806 (exception below). An end-to-end run against a local relay and the built desktop app covered artifact accept, huddle join, and workflow deletion for both a healthy and a quiescing community, plus p-tag reaction rollback and replay of the same signed event. New tests: one fenced-community test per migrated path (artifact accept, huddle join, both workflow deletions) puts the community into `quiescing`, holds the lock the path takes next, and asserts rejection at entry with nothing stored; a DB-free test pins the non-canvas rejection. Exceptions: - `runtime::postgres_tests::writer_pool_rejects_non_read_committed_database_default` hangs on that workstation's local Postgres 16 at both this head and an earlier `main` base, `f0eb5575f` (2/2 runs each). The test, `Db::new`, and the pool setup are unchanged by this PR, and the test passes in GitHub CI. Live traces put the hang in the client, not the database: the 1 s acquire timeout stops firing after sqlx's `after_connect` rejection path under tokio's `current_thread` runtime. No database lock waits were observed. - Two `observability_source` assertions were already failing on `main`, unnoticed because no CI job ran that test binary: a stale `fetch_optional` pin after #8005, and lease-seam attribution that now lives in the `bounded_serving_lease_sql` helper. Both are corrected here, and the binary now runs in `just test-unit`. Generated with Codex --------- Signed-off-by: tornquist <tornquist@squareup.com> Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-06 10:24:41 -04:00
ravarora2	91ab9d31b8	Add operation-aware database pool acquisition metrics (#7195) ... ## Why Buzz already reports coarse writer/reader database checkout waits, but those signals cannot explain which startup or serving operation is blocked by pool pressure. That makes rollout diagnosis and postmortems ambiguous: a readiness probe, NIP-42 authentication, authorization check, reconnect history repair, event write, and background maintenance can all wait on the same pool while appearing identical. This PR implements Package 2A of the pod-handoff plan: operation-aware pool borrow causality. It is observability-only; it does not change configured pool sizes, SQL semantics, transaction ordering, or timeout policy. Physical DNS/TCP/TLS/Postgres authentication and session initialization remain the separate Package 2B boundary. ## Metric contract The final contract separates three questions: | Question | Metric | |---|---| | How long did checkout wait? | `buzz_db_pool_acquire_duration_seconds{pool_role,operation}` | | How did the attempt end? | `buzz_db_pool_acquire_attempts_total{pool_role,operation,outcome}` | | Who is waiting now for a tracked operation? | `buzz_db_pool_waiters{pool_role,operation}` | Outcomes are `success`, `timeout`, `error`, and `cancelled`. Operations are `bootstrap`, `readiness`, `tenant_resolution`, `authentication`, `authorization`, `subscription_history`, `event_write`, and `maintenance`. Only these eleven pool/operation pairs are constructible: ```text writer/bootstrap reader/bootstrap writer/readiness writer/tenant_resolution writer/authentication writer/authorization reader/authorization writer/subscription_history reader/subscription_history writer/event_write writer/maintenance ``` The duration histogram intentionally does not carry `outcome`. Result data remains available on the terminal counter for historical counts and rates, without multiplying the expensive histogram family. Nine finite buckets plus `+Inf`, sum, and count produce 12 duration series per valid pair. Together with four outcome counters and one waiter gauge, the new-family ceiling is exactly 187 raw Prometheus series per pod, asserted from the production exporter. The existing coarse acquisition families remain temporarily for dashboard compatibility while the new series are validated in staging. The operation-specific waiter family covers the explicitly routed, deployment-critical operations above; it is not a census of every possible SQLx checkout in the process. The dashboard pairs it with SQLx pool active/idle/max gauges for whole-pool capacity context, and treats a missing operation series as unknown rather than healthy zero. ## What changed ### Cancellation-safe acquisition ownership - Add writer- and reader-specific typed operation APIs so invalid label pairs cannot be constructed and store modules cannot emit reader labels. - Own every polled acquisition with one RAII terminal guard. - Record exactly one duration and terminal outcome for success, timeout, error, or cancellation. - Emit nothing for a future that is created but never polled. - Balance the operation-specific waiter count exactly once on every terminal or dropped future. - Periodically refresh every expected waiter pair, including healthy zero, so missing telemetry is not presented as zero. Reader pairs are emitted only when a distinct read pool is configured; a writer-only pod cannot fabricate healthy reader-zero state. ### Production attribution Route the deployment-critical acquisition paths through caller-owned semantic entry points, including: - writer and reader bootstrap; - the real post-#7149 readiness acquisition and deletion-catalog validation; - tenant resolution and community lifecycle checks; - NIP-42 allowlist authentication; - membership, moderation, invite, operator, Git, agent-owner, and policy authorization; - operator community create/list/archive/unarchive, reverse host/channel tenant resolution, and REQ row-community conformance lookups; - writer/reader subscription history, feed, thread, and routed fallback paths; - primary and command event writes, replaceable events, mention indexing, reaction/channel/member/archive side effects, and thread metadata; - push matching, usage rollups/leadership, replica-fence startup and recurring probes, periodic reconciliation, channel/deletion reapers, partitions, and other bounded maintenance/bootstrap paths. Shared helpers now accept caller-owned intent or expose named semantic variants instead of assigning one misleading operation to every caller. No known P0 path uses `other`. ### Readiness and size-one-pool correctness - Rebase on the post-#7149 readiness implementation and instrument the actual `Db::readiness_check` acquisition rather than the superseded ping-only seam. - Acquire once for deletion-catalog validation and run its queries on that connection, preserving the shared readiness deadline. - Scope the channel-roster catalog checkout before the behavior probe so a size-one writer pool cannot self-deadlock during startup verification. ### Exporter, documentation, and CI - Register metric HELP/type/unit metadata through the production Prometheus builder. - Configure dedicated checkout buckets at 1ms, 5ms, 10ms, 25ms, 50ms, 150ms, 500ms, 1s, and 3s. - Add a production scrape-contract test for exact names, labels, buckets, valid pairs, sensitive-label exclusion, and the 187-series ceiling. - Add source mutation guards for the P0 semantic entry points and raw-checkout bypasses. - Add an exact backend-integration CI selector for the production attribution, cancellation, readiness, and size-one-pool PostgreSQL tests. - Document the frozen label vocabulary, valid combinations, semantics, and cardinality budget in the Helm chart README. ## Dashboard intent The new Stage 2 row in **Buzz Startup & Rollout Safety** is deployment-first: - baseline-versus-candidate attempts, failure rates, cancellation rates, and maximum wait by operation; - outcome counts and percentages over time by SHA/ReplicaSet; - acquisition wait heatmap, average, and maximum through the rollout; - historical waiter pressure beside writer active/idle/max context; - per-pod postmortem drilldown, including terminated pods; - a smaller current-waiter table with explicit stale/missing semantics. Percentile widgets remain disabled until Datadog metadata confirms percentile support for the new distribution. Current gauges use no fill, interpolation, or `default_zero`; missing means unknown. ## Risk assessment Moderate. The patch touches many database acquisition call sites, but preserves the selected physical pool and executes the same SQL on the acquired connection. The main risks are incorrect semantic attribution, cancellation double-counting, and a helper accidentally acquiring twice. Typed APIs, production-method PostgreSQL tests, source guards, the raw scrape contract, and the size-one-pool regression cover those risks. No tenant, community, user, pubkey, event, channel, SQL, URL, pod, version, ReplicaSet, or request-controlled value is emitted as an application metric label. Deployment identity is supplied by infrastructure enrichment. ## Verification - `cargo fmt --all -- --check` — passed. - `cargo clippy -p buzz-db -p buzz-relay --all-targets --all-features -- -D warnings` — passed. - `cargo test -p buzz-db` — 122 passed, 0 failed, 263 ignored; source-contract integration test: 3 passed, 0 failed. - Focused relay compatibility, metric-contract, and readiness tests — passed. - `scripts/test-postgres-test-discovery.sh` — passed. - Full `buzz-relay` package run from the identical tree reached 1,015 passes; the six media-test failures all stopped in their shared local PostgreSQL setup with `Sqlx(PoolTimedOut)` because Docker/PostgreSQL was unavailable. The same six failed in isolation, while every changed exact test passed. - Exact implementation head: `f92910b353086e9edf85918ca5f72190edbbe22f`. - Exact multi-architecture staging image: `dev-sha-f92910b353086e9edf85918ca5f72190edbbe22f-run-33607968668-1` (`sha256:161712c8ed2e265a15df9b63e02248d5973481f875ff129d7d2ae78a09d487a2`). - Focused staging GitOps PR: <https://github.com/squareup/builderbot-platform-core-infrastructure/pull/299> — merged after renderer, inventory, infrastructure test, Kargo, Semgrep, and Intersect gates passed; the source/generated-artifact diff was exactly two image lines. - Exact GitHub head reports 47 terminal checks: 35 successful and 12 intentionally skipped. PostgreSQL, unit, lint, security, both server cross-compiles, backend integration, relay E2E, desktop, mobile, image, Helm, Semgrep, zizmor, and DCO gates are green. - Datadog readback identifies two exact-image pods, `buzz-d79c8d8f7-ckv2l` and `buzz-d79c8d8f7-qzqdp`, in ReplicaSet `buzz-d79c8d8f7`; both report the full source SHA above. - Both pods report all eleven allowed pool/operation waiter pairs at current zero, with no invalid pair. The acceptance window observed nonzero success receipts for readiness, tenant resolution, authorization, subscription history, event write, and maintenance, and no timeout, error, or cancelled outcome. Maximum observed wait was about 101 ms for maintenance and 50 ms for reader subscription history. The main **Buzz Startup & Rollout Safety** dashboard now has a live Stage 2 database row with eight widgets and nineteen fully scoped queries. Final readback preserved all seven top-level groups, found zero under-scoped Row 6 queries, and confirmed the tracked-operation waiter boundary in the panel descriptions. Generated with Codex. --------- Signed-off-by: Ravneet Arora <rarora@squareup.com>	2026-09-02 17:04:34 -04:00
Luke Tornquist	a3730784fc	refactor(db): extract domain stores from database runtime (#6987) ... ## Summary Finish the remaining database-store extraction tracked by [TheSentinel454/buzz#2](https://github.com/TheSentinel454/buzz/issues/2) in one reviewable PR. This consolidates the previously stacked domain slices after #6782 merged. It preserves the runtime/store boundary established by #6660, #6668, #6700, and #6782 while separating database runtime infrastructure from domain-owned persistence: - `runtime/` owns pool construction and sizing, writer/reader routing, read sessions and route proofs, transaction infrastructure, observability primitives, replica fencing, health support, migrations, and cross-cutting runtime tests. - `store/` owns domain records, SQL, row parsing, locks and invariants, `Db` domain methods, focused tests, and logical-operation datastore spans. - `lib.rs` remains a 57-line compatibility facade that preserves existing crate-root paths and `Db` method signatures through re-exports. Domain coverage includes API tokens, authentication allowlists, reminders, event queries, threads, reactions, feeds, users and DMs, push, workflows/runs/approvals, relay membership and invites, product feedback, moderation/admin moderation, relay admin actions/operators, git repositories, archived identities, usage, partition maintenance, deletion, channel membership inherited from merged #6782, and the final runtime/store layout. The branch has been rebased onto current `main`. Database changes that landed there were incorporated rather than overwritten: `relay_admin_actions.rs` and `relay_operators.rs` now live under `store/`, their 27 public `Db` wrappers and existing behavior remain intact, and every wrapper has exactly one fixed-name datastore span. Concurrent changes to migration, moderation, admin moderation, and error handling are also retained. ### Exact base and head - Base: `main` at `ed11c8d8bf0a17402be5cf243724f89471530d2f` - Head: `codex/issue-2-store-extraction` at `be24430472d1a87ac5c0d6026c620cd6caea3537` ### Related issue - Structural tracker: [TheSentinel454/buzz#2](https://github.com/TheSentinel454/buzz/issues/2) - Domain trackers: [#6](https://github.com/TheSentinel454/buzz/issues/6), [#7](https://github.com/TheSentinel454/buzz/issues/7), [#12](https://github.com/TheSentinel454/buzz/issues/12), [#13](https://github.com/TheSentinel454/buzz/issues/13) - Acceptance trackers: [#17](https://github.com/TheSentinel454/buzz/issues/17), [#19](https://github.com/TheSentinel454/buzz/issues/19) This supersedes #6783, #6784, #6787, #6788, #6789, #6792, #6820, #6794, #6796, #6797, #6798, #6799, #6804, #6805, #6806, #6808, #6809, #6811, #6812, #6813, #6814, #6815, and #6890. Their discussions remain available for review history. ### #17 / #19 acceptance - Preserves the metric names, fixed labels, transaction/lock timing boundaries, and privacy/cardinality constraints introduced by #6700. - Keeps exactly one datastore span per public logical operation, including the 27 relay-admin wrappers added on `main`. - Removes `store_ownership.rs`; physical ownership and focused source guards now enforce the boundary directly. - Leaves no `impl Db`, domain SQL, focused domain test group, or datastore span in `lib.rs`. - Preserves existing public paths such as `buzz_db::channel`, `buzz_db::event`, and `buzz_db::workflow` through crate-root re-exports while keeping internal `runtime` and `store` namespaces private. ### Non-goals - No SQL, schema, locking, transaction, retry, timeout, or client-visible behavior changes. - No generic store traits, domain handles, broad `PgExecutor` migration, new store crate, raw pool accessor, or broader directory reorganization. - No tracker issues are closed by this PR. ### Risk The cumulative diff is large but structural. Risk is primarily module-path, ownership, or conflict-resolution drift. It is mitigated by preserving public re-exports, comparing the newly moved `main` implementations to their upstream source, source guards, touched-crate compilation, PostgreSQL-backed test coverage, and an independent exact-head review on a separate clean Blox workstation. ### Testing Author workstation `buzz-tornquist-pr-6987-rebase`, rebased branch ending at exact head `be24430472d1a87ac5c0d6026c620cd6caea3537`: - `cargo fmt --all --check` - `cargo clippy -p buzz-db -p buzz-relay --all-targets -- -D warnings` - `cargo test -p buzz-db --lib` — 113 passed, 240 PostgreSQL tests intentionally ignored - `cargo test -p buzz-db --test observability_source` — 2 passed - PostgreSQL-backed `buzz-db` coverage under native PostgreSQL — 235 passed in the shared serial run; the five shared-state/config-sensitive cases passed as isolated reruns against fresh schemas, including the two owner-limit tests with their fixture's `BUZZ_MAX_COMMUNITIES_PER_OWNER=3` - `cargo test -p buzz-relay --lib -- --test-threads=1` under native PostgreSQL/Redis — 991 passed; the three current-month partition-sensitive identity-archive cases passed after provisioning the August 2026 test partition; 87 infrastructure-marked tests remained ignored - Source/diff guards — relay-admin implementation bodies match current `main`; all 27 public wrapper signatures are retained; exactly one datastore span wraps each wrapper; `lib.rs` has zero `impl Db` blocks and zero datastore spans; no duplicate top-level relay-admin modules or `store_ownership.rs`; `error.rs` matches current `main` Independent clean review workstation `buzz-tornquist-pr-6987-review`, detached at exact head `be24430472d1a87ac5c0d6026c620cd6caea3537`: - `cargo fmt --all --check` - `cargo clippy -p buzz-db -p buzz-relay --all-targets -- -D warnings` - `cargo test -p buzz-db --lib` — 113 passed, 240 ignored - `cargo test -p buzz-db --test observability_source` — 2 passed - Exact-head ownership/re-export/instrumentation audit — no remaining actionable findings --------- Signed-off-by: OpenAI Codex <codex@openai.com> Signed-off-by: tornquist <tornquist@squareup.com> Co-authored-by: OpenAI Codex <codex@openai.com>	2026-08-28 15:24:32 -04:00
Luke Tornquist	113a33b7e4	Add database pressure observability (#6700) ... ## Why Database pressure currently collapses several distinct delays into one symptom. This adds the evidence layer needed to distinguish pool acquisition wait, logical database operation time, advisory-lock wait, and selected transaction duration before changing timeout or retry policy. This is the phase 2 Lane A observability bundle for [#26](https://github.com/TheSentinel454/buzz/issues/26), [#28](https://github.com/TheSentinel454/buzz/issues/28), and [#33](https://github.com/TheSentinel454/buzz/issues/33). It is stacked on #6668. ## What - Record explicit reader/writer checkout wait and acquisition outcomes with `buzz_db_pool_acquire_wait_seconds` and `buzz_db_pool_acquisitions_total`. - Extend the compile-time `#[datastore_span(name = "...")]` seam with `buzz_db_operation_duration_seconds`, so operation labels remain static source literals instead of request data. - Route correctness-critical replacement, membership, push-gate, deletion, and migration/schema-safety advisory locks through one observer without changing their SQL, order, scope, or blocking behavior. - Measure six internally owned transaction lifetimes with `buzz_db_transaction_duration_seconds`, starting after `BEGIN` succeeds and ending after explicit commit/rollback or scope exit. - Emit root slow-operation warnings at 500 ms, logging the first slow completion and then 1/100 per call site with only `operation`, `outcome`, and `elapsed_ms`. - Document names, units, fixed label vocabularies, measurement boundaries, and blind spots in this PR description. Fixed labels are deliberately small: - `pool_role`: `writer`, `reader` - `lock_type`: `replacement`, `membership`, `push_gate`, `deletion`, `migration_schema_safety` - `outcome`: `success`, `error`, `timeout` where SQLx/PostgreSQL can distinguish it accurately - `operation`: compile-time datastore names plus the six closed transaction operation names documented in the runbook No metric or slow warning contains community IDs, event IDs, event kinds, coordinates, d-tags, SQL/query text, query IDs, returned errors, or event content. ## Coverage boundaries - Operation duration is the complete annotated logical function body, not pure SQL execution; it may include implicit checkout, lock wait, nested operations, and application work. Cancelled futures do not reach its completion hook. - Pool timing covers explicit helper checkouts, including proved-reader routing and selected writer-owned transactions. Implicit SQLx checkout through `&PgPool` remains folded into operation duration. - Lock timing covers application-side blocking locks in the five named families. Trigger/stored-procedure locks, channel-TTL locking, the usage try-lock, and the audit service session lock remain outside this slice. - Transaction timing covers only the six wholly owned boundaries documented in the runbook. It excludes pool wait, `BEGIN`, asynchronous rollback cleanup after an early return, and caller-owned `Db::begin_transaction` lifetime. ## Relationship to #6229 #6229 is the incident-driven timeout precursor. This PR does not add or change `statement_timeout`, `lock_timeout`, `idle_in_transaction_session_timeout`, retries, audit durability, or client-visible conflicts. It provides the missing distributions needed to evaluate those policies later and intentionally leaves #6229's open audit retry/durability finding untouched. The branches overlap in `crates/buzz-db/src/lib.rs` and `crates/buzz-db/src/migration.rs`, so a later rebase may need textual conflict resolution, but the behavior is complementary rather than duplicated. ## Risk assessment Moderate-low. The primary risk is instrumentation overhead and added static series. Cardinality is source-bounded, slow logs are sampled/redacted root events, and the lock/transaction changes wrap existing awaits without changing policy or ordering. ## Verification Author workstation: `buzz-tornquist-db-pressure-observability` (`2010927`), exact head `d7cf833e26c528adfcde3917ded80daf6f4ddac9`, parent `6f50e6b2b2a996349149af61d35bdd6a355f77fd`. - `cargo fmt --all --check` — passed - `cargo clippy -p buzz-datastore-tracing -p buzz-db -p buzz-audit -p buzz-search -p buzz-relay --all-targets -- -D warnings` — passed - `cargo test -p buzz-datastore-tracing --quiet` — 4 passed - `cargo test -p buzz-db --quiet` — 109 passed, 200 ignored - `cargo test -p buzz-audit -p buzz-search --quiet` — 16 passed, 25 ignored - `cargo test -p buzz-relay --lib --quiet -- --test-threads=1` — 906 passed, 48 ignored - Native PostgreSQL focused tests for pool success/timeout/error, lock success/contention/timeout/error, replacement, membership serialization, push ordering, deletion fencing, migration/schema exclusion, and reader fallback — 8 passed The default-parallel relay run passed once; subsequent runs exposed the existing load-sensitive `api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo` 504 at the end of the suite. That test passes in isolation and the full relay suite passes serially. Independent exact-head review workstation: `buzz-tornquist-db-pressure-observability-review` (`2013067`). Formatting, the same all-target clippy command, datastore instrumentation tests, DB unit tests, source privacy guards, and diff/non-goal audits passed; no review findings. Generated with Codex --------- Signed-off-by: tornquist <tornquist@squareup.com>	2026-08-25 13:00:22 -04:00