Files
buzz/schema/functions/enqueue_push_match_job.sql
Tom Brow 5b907b838a fix(push): isolate push enqueue and stop workers on rollback (#8216)
Push enqueueing previously ran inside the message transaction, so a push
lock or database error could block or reject a message. Configured relay
writers now submit bounded best-effort work only after commit. A
dedicated connection and bounded queue isolate enqueue waits and
failures from message-write connections. Overload, errors, process loss,
or shutdown may drop a notification; the committed message remains
authoritative.

`BUZZ_PUSH_ENABLED=false` disables the application producer and workers.
Updated writer connections always bypass any surviving enqueue trigger.
The overlap migration updates an existing function without resurrecting
retired objects. Apply it before relying on isolation with a surviving
trigger, and replace every old enabled process and writer connection:
missing settings preserve legacy behavior during overlap.

SIGTERM/Ctrl-C cancels enqueue, matcher, and delivery work before
listener drain. Durable queues, claims, leases, preferences, and
delivery configuration remain intact. Claims expire normally; gateway
requests already accepted may still deliver. Reactivation requires an
explicit operator decision about retained work. There is no automatic
purge, startup interlock, or backfill of messages written while
disabled.

Built on merged [#8135](https://github.com/block/buzz/pull/8135),
including [#8178](https://github.com/block/buzz/pull/8178). Trigger
retirement [#8210](https://github.com/block/buzz/pull/8210) remains
open; this change is tested with the trigger present and after removal.
Operational details are in [the rollback and reactivation
guide](https://github.com/block/buzz/blob/relay-push-rollback/docs/push-gateway-deployment.md#relay-rollback-and-reactivation).

### Validation


[9d3d710](9d3d710dc1)
changes documentation and Rust comments only, with no executable or
assertion changes. Formatting and diff checks passed;
[12f1d1e](12f1d1e814)
additionally restores the reviewer-requested caveat that active lease
renewal may suppress a notification. This is also documentation-only;
Owner approval was confirmed on 12f1d1e.
[a4f602d](a4f602df49)
subsequently merges main to resolve a migration-count conflict after
main added migration 0058. Both 0058 and rollback migration 0059 are
retained; the focused embedded-migrator test, formatting, and applicable
desktop pre-push checks passed. Current-head [owner
approval](https://github.com/block/buzz/pull/8216#pullrequestreview-5474958334)
is confirmed. [CI attempt 2
passed](https://github.com/block/buzz/actions/runs/37985516452/attempts/2),
with required checks passing or appropriately skipped. The first attempt
failed only the unchanged replica-floor probe test with `MaskedActivity
{ masked: 1 }`; one failed-jobs retry passed without code changes. All
four push contract tests passed. The current-head [advisory security
review](https://github.com/block/buzz/actions/runs/37985514346/job/114006457769)
has remained in its review step since 20:18:51 UTC without a published
result for over 25 minutes. It is stalled under the advisory-review
policy, not a clean security review.

[Required CI checks passed or were
skipped](https://github.com/block/buzz/actions/runs/37964035783) on
[3164b25](3164b25cde).
The advisory security review was cancelled without a published result.
Codex's durable-retry suggestion was [rebutted and
resolved](https://github.com/block/buzz/pull/8216#discussion_r4232674980)
under the explicit best-effort notification contract. The [lease
activation-order
observation](https://github.com/block/buzz/pull/8216#discussion_r4233051799)
is correct. Under the clarified product requirement, either notification
outcome is acceptable during overlapping registration and message
arrival. Exact commit ordering is not required; deliberate historical
backfill remains prohibited. The bot thread is resolved against that
clarification; [Owner review accepted the clarified overlap
contract](https://github.com/block/buzz/pull/8216#pullrequestreview-5474720278)
on 9d3d710.

All 488 PostgreSQL tests passed. The final four-way producer contracts
passed on desired/migrated schemas, with/without triggers. Added
coverage verifies message commit under push-gate and queue-table locks,
recovery after enqueue failure, in-flight enqueue cancellation, the
lease timestamp filter (not production activation ordering),
serving-lease writers, rollback preservation, reconnects, and
reactivation without backfill. Queue saturation and all three relay
worker shutdown tests passed. Rust lint and all 17 source-policy checks
passed.

Full local CI/repository runs encountered the unchanged ACP
`keepalive_resets_idle_past_deadline` timing failure. Earlier desktop
subprocess timing tests also failed under parallel load and passed as an
isolated module. Under Tom's local pre-push exception, this update
excluded `rust-tests` and `desktop-tauri-checks`; remaining hooks stayed
enabled. Required CI validation remains necessary before merge. A
focused run competing with the full database suite hit wall-clock
deadlines; tests now use a short database lock timeout to distinguish
unwanted push dependencies, and their final isolated run passed.

A local process-replacement rehearsal on 3164b25 used the real relay
binary and a mock gateway holding a delivery request open. SIGTERM
closed that request in under 1 ms; the relay exited in 5.02 seconds.
Restarting the same binary with push disabled produced no further
gateway requests over 35 seconds, beyond the 30-second claim expiry. The
outbox row, lease, and its subscriptions remained byte-for-byte
unchanged. This used a seeded outbox and did not exercise APNs or a
mobile device.

Production writer inventory, shutdown deadline, and retained-queue
disposition still require operational validation before rollout. The PR
is marked ready for review. The overlap behavior is accepted by the
product requirement; owner review accepted that contract. No production
deployment has occurred.

---------

Signed-off-by: Tom Brow <tomb@block.xyz>
2026-10-09 13:48:06 -07:00

37 lines
1.6 KiB
PL/PgSQL

-- T1b push gate (migrations 0023 and 0059). Enqueue only when the
-- community has an active, endpoint-enabled, unexpired lease; the shared
-- advisory lock pairs with the exclusive lock taken by lease activations
-- (crates/buzz-db/src/push.rs) to close the lost-wake race.
CREATE FUNCTION enqueue_push_match_job() RETURNS trigger
LANGUAGE plpgsql AS $$
BEGIN
-- Legacy connections without the setting preserve their existing behavior.
-- Updated writer pools always set it, including on reconnect. Check before
-- taking push locks or touching leases/queues so disabled chat stays healthy.
IF COALESCE(current_setting('buzz.push_async_enqueue', true), '') = 'on' THEN
RETURN NEW;
END IF;
IF NOT COALESCE(NULLIF(current_setting('buzz.push_enabled', true), '')::boolean, true) THEN
RETURN NEW;
END IF;
-- Keep this allowlist identical to the relay's validated NIP-PL descriptor.
-- This compatibility path covers legacy writers until they are replaced.
IF NEW.kind IN (9, 40002, 45001, 45003) THEN
PERFORM pg_advisory_xact_lock_shared(
hashtextextended('buzz_push_gate:' || NEW.community_id::text, 0));
IF EXISTS (
SELECT 1 FROM push_leases
WHERE community_id = NEW.community_id
AND active
AND endpoint_enabled
AND expires_at > EXTRACT(EPOCH FROM now())::bigint
) THEN
INSERT INTO push_match_queue (community_id, event_id)
VALUES (NEW.community_id, NEW.id)
ON CONFLICT DO NOTHING;
END IF;
END IF;
RETURN NEW;
END
$$;