4 Commits

Author	SHA1	Message	Date
Luke Tornquist	c0ab789499	feat(buzz-db): retire the channel-TTL trigger (#8210) ... ## Summary This drops the channel-TTL event trigger. The application already refreshes ephemeral-channel deadlines in the event transaction (`event_follow_up::refresh_channel_ttls`, from #8135), so the trigger only repeats that work. This PR originally retired both follow-up triggers. #8352 has since retired the push-match trigger in migration 0063, so after the rebase this PR covers only the TTL half. Migration `0065_retire_channel_ttl_trigger.sql`: - `SET LOCAL lock_timeout = '5s'` (the same bound as 0063) - drops `events_refresh_channel_ttl` and `refresh_channel_ttl_after_event_insert()` Dropping the trigger on the partitioned parent also drops its partition clones. Published migrations, including 0022 and 0024, stay checksum-frozen. The same objects are removed from `schema/` (`schema/functions/…`, `schema/triggers/events.sql`, `schema/schema.sql`) and from the pgschema reconcile script. After this, migrated and desired-state databases have no event follow-up triggers. Tests: - `follow_up_retirement_tests.rs` (renamed from `push_retirement_tests.rs`): the push-retirement upgrade test now stops at 0063. A new test upgrades 0064 → 0065 and checks that the TTL trigger is gone from the parent and every partition, that the function is gone, and that the published checksums haven't changed. - `event_follow_up_postgres_tests.rs`: the contract now has two arms. `AppOnly` is the shipped schema, where both triggers are absent. `Dual` reconstructs the historical overlap of both triggers. The `PushRetired` arm (the push trigger dropped, the TTL trigger kept) was the shipped state before this PR. Its tests are removed because `Dual` and `AppOnly` together cover it. - `embedded_migrator_contains_consolidated_initial_schema` pins the 0065 statements and checks that both objects are absent from the desired schema. **Deploy gate:** do not run 0065 against a database still served by a relay older than #8135. After 0065, raw SQL writers no longer refresh TTL deadlines. `docs/push-gateway-deployment.md` is updated to match. ### What got simpler The trigger and the app no longer both refresh TTL deadlines; the app is now the only producer. One plpgsql function and one trigger are gone from both schema paths, and 8 per-partition drops are gone from the reconcile script. The follow-up contract drops from three arms to two. ### Related issue BUZZ-313 (Linear), part of the trigger-retirement work. ### Testing Head is `4f7d0baf0`. Results below are at `678927730` unless noted. `678927730..4f7d0baf0` changes only one test in `crates/buzz-db/src/store/event.rs`, described in the first bullet. - **Fixed test, `permanent_channel_event_commits_do_not_lock_the_channel_row`** (`4f7d0baf0`). - **Why it changed:** after 0065, its raw-pool inserts no longer ran any TTL refresh, so the test passed whatever `refresh_channel_ttls` did. This was raised in review. - **What it does now:** it inserts through the production `AdmittedTx` path and keeps the barrier and the tuple-lock probe. It commits all N transactions while an external transaction holds `FOR SHARE` on the channel row. It also asserts that no `buzz_db_channel_ttl_refresh_failures_total` is counted. - **Mutation check:** with `SELECT … FOR UPDATE` injected into `refresh_channel_ttl`, the new test fails (`must not wait on the channel row`, timeout at 10s). The old test passes under the same mutation. - **Blox:** fmt and `buzz-db` clippy pass. - **GitHub CI at `4f7d0baf0`:** every required check passes (56 pass, 0 fail). CI builds the merge ref with `main` `93ffc37cb`. - **New and renamed tests, checked by name in the CI logs at `4f7d0baf0`.** These pass in PostgreSQL Domain / PostgreSQL Tests ([117033725006](https://github.com/block/buzz/actions/runs/38915998692/job/117033725006)): - `permanent_channel_event_commits_do_not_lock_the_channel_row` - `migration_schema_ttl_retirement_drops_every_partition_trigger_and_keeps_checksums` - `migration_schema_push_retirement_preserves_ttl_and_published_checksums` - `event_insert_ttl_refresh_handles_permanent_ephemeral_duplicate_and_activation_race` - `{desired,migration_schema}_event_follow_up_contract_{dual,app_only}` (four tests) `embedded_migrator_contains_consolidated_initial_schema` passes in Rust / Unit Tests ([117024125423](https://github.com/block/buzz/actions/runs/38915998692/job/117024125423)). No `*_push_retired` test runs anywhere. - **Blox gates at `678927730`** (private PG16 + Redis, on a workstation shared with other sessions): - `cargo fmt --check`, workspace `clippy -D warnings` and `buzz-db --lib` all pass. - The `buzz-admin`, `buzz-audit`, `buzz-deletion`, `buzz-push-gateway`, `buzz-search` and `buzz-workflow` Postgres packages pass. - `buzz-db`: 517 of 519 tests passed. The 2 that hung, both `writer_pool_*` tests, were killed. `buzz-relay`: 413 of 426 passed. The 13 failures are covered below. - **Uncontended reruns at `4f7d0baf0`** on a fresh, idle workstation (PG16.15, `competing=0` for every arm): - **`writer_pool_rejects_non_read_committed_database_default` predates this PR.** It timed out 3/3 at head and 3/3 at `main` `93ffc37cb`, with a 300s timeout per run. - **`writer_pool_metrics_stop_after_session_timeout_setup_failure` also predates this PR.** It timed out 2/3 at head and 2/3 at `main`, and passed in the remaining run on each side. - Both tests pass in CI. The gate accepts them as a named exception for this PR, as a pre-existing hang on the PG16 test environment. - **`buzz-db`:** 518 of 519 tests passed. The single failure is `writer_pool_rejects_non_read_committed_database_default`, killed after it hung (the exception above). - **`buzz-relay`: still open.** 413 of 426 tests passed. The same 13 `handlers::group_state_postgres_tests` failed as on the shared box, each panicking at once in its fixture's `ensure community` step with `Sqlx(PoolTimedOut)`. Contention does not explain them. The fixture's community setup is in `buzz-db`, which this PR changes. A head/`main` comparison of the full `buzz-relay` package on the same idle box is running to find out whether this predates the PR. All 13 pass in CI at this head. - **E2E (Gimli, Blox)** at `050e593af`. `050e593af..678927730` changes only tests and docs; the migration, schema and runtime code are byte-identical. Gimli's audited verdict: **EXERCISED AND WORKS**, for DB and relay runtime only. Gimli ran it at `050e593af`; this head's runtime is byte-identical but was not freshly executed. - **Upgrade:** a real relay startup took the schema from 0064 to 0065. That removed the TTL function and the trigger from the parent table and from all 8 partitions. Checksums for migrations 1–64 are unchanged. - **Lock-timeout rollback:** with a real lock held, the migration failed with `55P03` at 5.0007s and the schema rolled back fully to 64. Once the lock was released, a retry applied 65. - **Fresh desired schema:** after pgschema plus reconcile, the retired objects are absent. - **Real signed kind-9 sends:** on both schema paths, these refreshed ephemeral-channel deadlines to about +600s. Permanent channels stayed NULL. - **Raw SQL insert:** at 64 it refreshed the deadline. After 65 it left the deadline unchanged, on both the upgraded and the fresh schema. - **Not covered:** native or device testing, and deploy. - **Review:** Legolas found 0 P0 and 0 P1. His three optional P2s are fixed in `678927730`. - **No human app test:** this is a DB-only change. - **Migration numbering:** #8237 and #8372 also add a `0065_*` migration. Whichever PR merges second must renumber. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-16 11:22:21 -04:00
Tom Brow	6766174673	chore(push): retire push notification database trigger (#8352) ... Buzz must schedule mobile push notifications when it saves an eligible message. The relay, the server that receives and stores messages, now does this in application code. An older database trigger still exists. This trigger is a rule that runs automatically when a message is saved. This PR removes the old trigger and its supporting database function. The relay remains responsible for scheduling notifications. Migration 0063 makes this change in existing databases; updated setup scripts apply it to new databases. The change does not turn on push notifications. Production migration and deployment need separate approval. Before deployment, verify the release images and all pending database migrations. If push needs to be stopped after release, disable it with the feature flag and keep the qualified application version and database schema. Do not downgrade the application image. ### Validation - Local checks passed: 519 isolated database integration tests, 166 database unit tests, 17 source-policy tests, and the relay push-runtime tests. - Tests verify that the migration removes the old trigger, preserves earlier migration records, and lets the application schedule push work correctly. A concurrency test exercises the real application path; deliberately removing its lock makes the test fail. - [Current-head CI passed on retry](https://github.com/block/buzz/actions/runs/38752849383/attempts/2). The first attempt had one database-connection timeout during test setup; the failed jobs passed when rerun on the unchanged commit. - Initial local CI and push checks hit unrelated ACP test failures, which passed when run separately. Only the Rust-test push hook was skipped under the repository's local-check exception; other applicable hooks passed. ### Before merge - [x] Migration 0063 is available and the push trigger remains on [main at 5397ea4](5397ea49e2). Recheck if main advances before merge. --------- Signed-off-by: Tom Brow <tomb@block.xyz>	2026-10-15 08:37:55 -07:00
Will Pfleger	020193f8e3	feat(relay): add direct staff ban/timeout/delete with staff guard (#7883) ... Relay staff (the `RELAY_OPERATOR_PUBKEYS` config roster, the owner fallback, and DB operator/moderator rows) can now ban, time out, and delete without a report through the admin API. Staff never join the community transport: these are admin-host, NIP-98 signed routes that drive the existing report enforcement state machine. Community admission, bans, and right-click menus are unchanged from `main`. Stack: this PR → #7904 ## Routes | Route | Body | |---|---| | `POST /api/admin/v1/members/{pubkey}/ban?communityHost=` | `{requestId, reason?}` | | `POST /api/admin/v1/members/{pubkey}/timeout?communityHost=` | `{requestId, expirationSecs, reason?}` | | `POST /api/admin/v1/events/{eventId}/delete?communityHost=` | `{requestId, reason?}` | Responses: `200 {actionId, state: "succeeded", replayed}`, `202 {state: "pending"}` when the action is accepted but not yet converged (the recovery worker finishes it; the same `requestId` replays), `409 request_id_conflict`, `409 target_is_staff`, `404 event_not_in_community`, `400 invalid_community_host` / `unknown_community_host`, and `422 enforcement_failed`. `422` is now the single failed-enforcement code for both the report and direct paths (shared `enforcement_failed()` mapping). ## Design - **Migration 0055.** `relay_admin_actions.report_id` becomes nullable. A direct row has `report_id NULL`, and `report_community_id` stays the tenant. New columns: `enforcement_target_event_id` and `timeout_secs`. The `relay_admin_actions_direct_shape` CHECK spells out each direct action: timeout needs both `timeout_secs` and `timeout_until`, ban and delete need neither, and delete needs both the event id and the author. A partial unique index on `(report_community_id, request_id) WHERE report_id IS NULL` makes direct actions idempotent. `pgschema` drops multi-column CHECKs, so `scripts/reconcile-schema-after-pgschema.sql` re-adds `relay_admin_actions_direct_shape` on desired-state databases. - **Atomic acceptance.** `claim_direct_action` inserts the action row and its `moderation_actions` audit row (`ban` / `timeout` / `delete_message`, `relay_operator` / `relay_moderator`) in one transaction. `ON CONFLICT DO NOTHING` sends a concurrent same-key request to the comparison. Retries compare actor, action, reason, `timeout_secs`, and the client-named target: the event id for delete, the pubkey otherwise. The event author, channel, and `timeout_until` are derived at acceptance and never compared. - **Retry racing its own first attempt.** Handler order: stored-action lookup → validation → staff guard → accept. If validation fails (for example B's event lookup runs after A already deleted the event), the handler re-reads the stored action. It returns a replay or 409 rather than 404. - **Report-less enforcement.** `finalize_success(report_id: Option<Uuid>)` skips the report CAS and the `reporter_notice` for direct actions. Tombstone and affected-user notices are unchanged. `recover_one` drives a direct action from its persisted row. `deliver_reporter_notice` treats a missing `report_id` as a terminal error instead of panicking. - **Permanent bans clear old expiry.** The ban upsert shared by direct and report-backed bans sets `ban_expires_at = NULL`, matching the kind-9040 path, so a permanent ban over an active or lapsed timed ban is in effect with no deadline. Timeouts are untouched. - **Community purge removes admin-action rows.** `purge_postgres` deletes the community's `relay_admin_outbox` rows (joined through their action), then its `relay_admin_actions` rows by `report_community_id`, before `moderation_reports`. That order satisfies both FKs and covers report-backed and direct (`report_id NULL`) rows in one pass. The same keyed row sources (`RELAY_ADMIN_TABLES`) feed the deletion inventory, which now records row counts for both tables, and post-purge logical verification, which now fails if either table still has a row for the community. Before this, a community with any admin-resolved report could never finish deletion (23503 on `relay_admin_actions_report_community_id_report_id_fkey`, stuck at `bindings_removed`). The `_operator_global_tables` registry reasons for both tables (seeded in migration 0036) are updated in 0055 and `schema.sql` to say community deletion purges them explicitly. - **Staff bans disconnect clusterwide.** The shared enforcement driver (`drive_enforcement`, used by the report and direct paths, fresh and recovery) calls `disconnect_pubkey_clusterwide` for `ban` before finalizing, matching the kind-9040 ban. A crash before finalize leaves the action non-terminal, so recovery re-runs the disconnect (idempotent) even when the ban step marker is already set. Timeouts do not disconnect, matching in-community timeout. - **Staff guard (D2).** `lookup_admin_principal` returns `Result<Option<AdminPrincipal>>`, with the same precedence as `resolve_admin_principal` (now a wrapper). `Ok(None)` means not staff. A roster lookup error is a 500, so the guard fails closed. Ban and timeout (direct) and ban/timeout/kick (via `resolve_report`) refuse any effective-roster target with `409 target_is_staff`, including the actor. The report path refuses before claim, so the report stays `open`. Delete is not guarded on either path. The check runs only when an action is first accepted: a replay or crash recovery completes an accepted action even if its target became staff afterwards. - **`communityHost` grammar (D4).** `buzz_core::tenant::validate_community_host` accepts only DNS labels and an optional port, and rejects schemes, paths, queries, userinfo, IP brackets, and whitespace before tenant binding. ## Tests binding fixes (red under mutation → green) Postgres tests are in `api/admin/mod.rs` `postgres_tests`, run against a DB migrated through 0055. Every rejection test seeds an existing restriction and asserts, through the shared `assert_no_effects` helper, that restriction values and action/audit/outbox/restriction/event counts are unchanged. | Fix | Test | Mutation → red | |---|---|---| | Racing retry replays | `direct_delete_retry_racing_first_attempt_replays_not_404` (barrier-paused HTTP interleaving; same-intent → 200 replay, conflicting reason → 409; the retry is checked with `assert_no_effects` against a snapshot taken after the first attempt, with the event author's restriction seeded) | drop the post-failure re-read → `left: 404 right: 200`; conflicting retry updates a restriction on the re-read path → `raced retry` snapshot mismatch (`ban_reason: "mutated"` vs `"seeded ban"`) | | Per-action retry comparison | `direct_ban_replays_same_request_and_conflicts_on_changed_intent` | drop `reason` from the comparison → `left: (200, None) right: (409, Some("request_id_conflict"))` | | Lookup fails closed (helper-level, calls `refuse_staff_target` directly) | `staff_lookup_failure_fails_closed` | `.unwrap_or(None)` on lookup error → panics on `unwrap_err` (guard passed) | | Direct staff guard | `direct_ban_and_timeout_refuse_staff_targets` (config/self and DB staff over HTTP with seeded restrictions unchanged; owner fallback checked at helper level via `refuse_staff_target`) | remove `refuse_staff_target` → `left: (200, None) right: (409, Some("target_is_staff"))` | | Report-path guard incl. kick | `resolve_report_refuses_kick_and_ban_of_staff_target` | remove report-path guard → `left: (200, None) right: (409, Some("target_is_staff"))` | | Accept-time-only check | `recovery_completes_accepted_ban_after_target_became_staff` | re-check staff in `drive_direct_action` → `left: "pending" right: "succeeded"` | | Timeout CHECK | `migration_0055_rejects_half_filled_direct_timeout` (buzz-db, migrated schema) and `direct_timeout_shape_check_holds_on_desired_state_schema` (buzz-relay, `pgschema` + reconcile schema) | loosen timeout branch of the CHECK, or drop the reconcile block → half-filled insert accepted | | No reporter notice for direct | `direct_ban_replays_…` (outbox assertion) | enqueue `reporter_notice` unconditionally → assertion at the `reporter_notice` check fails | | Delete unguarded | `direct_delete_removes_staff_authored_event` | — (behavioral control) | | Unauthorized direct requests have no effect | `direct_routes_refuse_unauthorized_requests_without_effects` (nonstaff, community owner, disabled auth, wrong host, wrong URL, altered body; restrictions on the targeted member and on the delete target's author, plus action/audit/outbox rows, unchanged) | roster bypass / drop mutation-principal check / admit owner / skip host check → `200 vs 403`; drop URL binding / payload hash → `200 vs 401` | | Foreign-community delete | `direct_delete_of_foreign_community_event_has_no_effect` | unscoped event lookup → `(200, None) vs (404, "event_not_in_community")` | | Banned staff actor still acts | `community_banned_staff_actor_still_acts_under_membership_enforcement` (membership enforcement on) | gate the actor on community bans → `403 vs 200` | | Timeout idempotency | `direct_timeout_keeps_expiry_on_retry_and_conflicts_on_changed_intent` | compare derived `timeout_until` → `409 vs 200`; compare action only → code `None vs Some("request_id_conflict")` | | Competing claims | `competing_direct_claims_accept_once` (identical → existing, conflicting → conflict; one action + one audit row each) | join skips compare → `[claimed, existing] vs [claimed, conflict]`; join always conflicts → the reverse | | Aborted acceptance rolls back | `aborted_direct_acceptance_rolls_back` | accept outside the transaction → effects `(1,0,0,0,0) vs (0,0,0,0,0)` | | Recovery exactly once | `recovery_after_committed_mutation_finalizes_exactly_once` (one recovery, `succeeded`, one `affected_user_notice`; each `recover_one` bounded by a 30s timeout) | ignore the enforcement step marker → `recovery did not converge: Elapsed(())` after 30s | | Permanent ban over a timed ban | `direct_ban_clears_prior_ban_expiry` (active and expired prior ban; asserts `ban_expires_at` NULL, admission `moderation_restriction_state` banned, timeout unchanged) | drop `ban_expires_at = NULL` → active: `left: Some(+1 day) right: None`; expired: `left: Some(-1 day) right: None` | | Rejections never rewrite restrictions | `direct_ban_and_timeout_refuse_staff_targets`, `resolve_report_refuses_kick_and_ban_of_staff_target` (value snapshot via `assert_no_effects`) | staff guard updates the target's existing restriction before returning 409 → counts equal but `ban_reason: "mutated"` vs `"seeded ban"` | | Purge with admin-action rows | `purge_postgres_succeeds_with_relay_admin_action_rows` (buzz-db; report-backed and direct rows with outbox rows; inventory counts both tables; deleted community left with none, second community untouched; a straggler direct row fails logical verification) | remove the new purge deletes → `23503 … violates foreign key constraint "relay_admin_actions_report_community_id_report_id_fkey"`; drop the relay admin tables from verification → `straggler relay admin action must fail verification` | | Ban disconnects, timeout doesn't | `direct_ban_disconnects_target_and_timeout_does_not` | remove the disconnect call (run alone) → `ban must close the target's socket` | | Recovery still disconnects | `recovery_past_ban_step_still_disconnects_target` | remove the disconnect call → `recovery must disconnect the banned target` | | Purge result keyed by table | `checkpointed_resume_is_idempotent_and_tombstone_blocks_name_reuse` compares returned keys to the scoped tables plus the two relay admin tables | — (existing regression, updated) | | Failed replay | `direct_replay_of_failed_action_returns_422_without_effects` | — (coverage) | | Pending under contention | `direct_action_under_contention_returns_202_pending` | — (coverage) | | Timeout/delete recovery | `recovery_completes_accepted_direct_timeout_and_delete` (test-only `claim_stranded` acquires just the requested action through `acquire_action_lease`, then runs `recover_one`, so tests don't claim each other's rows) | — (coverage) | ## Rollout note Before rolling the relay back to a binary without this PR, drain non-terminal direct rows (`report_id IS NULL AND state IN ('pending','enforcing')`). An older binary cannot decode `report_id NULL`, and recovery batches containing such rows stall. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>	2026-09-30 14:23:54 -04:00
Luke Tornquist	0ccf934b88	fix(db): disable heartbeat vacuum truncation (#6898) ... ## Why `replica_heartbeat` is updated continuously, but autovacuum heap truncation can take an `ACCESS EXCLUSIVE` lock whose replay cancels concurrent hot-standby reads. Truncating a fixed, single-row table provides no meaningful space benefit. ## What - Add migration 0034 to set `vacuum_truncate = false` on `replica_heartbeat` - Keep the desired-state schema aligned for fresh installations - Extend migration coverage so both paths retain the reloption ## Risk Assessment Low — this disables only autovacuum's heap-truncation phase for a single-row table; normal vacuum cleanup remains enabled. The tradeoff is retaining at most the table's otherwise-truncatable tail pages. ## References - [Investigation thread](https://sq-block.slack.com/archives/C0B2WT43BDH/p1787760831375989) - [PostgreSQL table storage parameters](https://www.postgresql.org/docs/current/sql-createtable.html#SQL-CREATETABLE-STORAGE-PARAMETERS) Generated with Codex --- **Update Aug 27, 2026:** Hardened fresh `pgschema` bootstraps after review. - Renamed the shared post-apply script to `reconcile-schema-after-pgschema.sql` and made it restore the heartbeat reloption and singleton row idempotently. - Added live catalog and row assertions that stop bootstrap when the database does not match the required state. - Updated every repository `pgschema apply` caller, including the desktop release smoke path. - Added a regression test that requires future `pgschema apply` callers to run reconciliation. - Documented the `pgschema` DML and storage-parameter limitation in `AGENTS.md` and at the reconciliation entry point. - Merged current `main` and renumbered this migration to 0034 after `main` added migration 0033. --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com>	2026-08-27 13:01:13 -04:00