mirror of
https://github.com/block/buzz.git
synced 2026-10-31 06:21:34 +00:00
ef4130da4b
## Summary
When the user-deletion worker dropped a lease heartbeat, its pooled
connection could be left inside an open transaction. The worker's final
lease `release` then reported success but never committed, and a rerun
saw the request as "leased by another worker" for up to 60 s. The
heartbeat is now one autocommit `UPDATE`.
The PR also fixes six PostgreSQL-lane test flakes from the 21–28 Oct
2026 CI audit. It moves one relay test off the 100 ms-acquire test
helper before that test starts flaking the same way.
This is one of seven PRs split out of the `flakey-tests` branch
(siblings below). It touches only `buzz-db`, `buzz-deletion` and
`buzz-relay`. Every relay file except `connection.rs` and
`boot_lifecycle.rs` changes only for the helper rename in the last row.
### Related issue
None found. Nothing on `main` (eda26b942) or in an open PR covers these.
G, a single hermit `pgschema` 504 that needs a workflow change, is out
of scope.
### Flaky tests fixed
| Test | Failing runs | Cause | Change |
|---|---|---|---|
| `buzz-deletion`
`user::worker::postgres_tests::dummy_executor_failure_retries_same_community_stage`
(:47) and `next_step_advances_one_handler_at_a_time_through_completion`
(:131) | 40439041234 (main), 40399635380#3, 40413732817#1 | **Product
bug.** `run_stage_with_heartbeat` drops the renewal future when the
stage handler wins `select!`. The heartbeat opened a transaction. sqlx
0.9 bumps `transaction_depth` only after `BEGIN` returns, so a `begin()`
dropped mid-`BEGIN` queues no `ROLLBACK`, and the connection goes back
to the pool inside an open transaction. The worker's autocommit
`release` then ran inside it and never committed. |
`EntityDeletionStore::heartbeat` is one autocommit `UPDATE`, guarded by
`lock_owned`'s ownership/status/expiry predicate. A superseded lease is
never renewed. A new regression test,
`dropped_heartbeat_leaves_no_open_transaction_for_the_release`, drops
the real heartbeat mid-`BEGIN` through a pausing proxy. |
| `buzz-db`
`session_statement_timeouts_bound_writer_and_reader_sessions`
(runtime/tests.rs:3138, `PoolTimedOut`) | 40128463987, 40314950864#1,
40374286094 (main), 40394821009#1, 40413732817#1 | Each config's lazy
reader pool dialled cold inside the 150 ms `READER_ACQUIRE_TIMEOUT`.
`Db::new` never runs the boot ping. | Warm the reader the way the relay
boots it (`read_pool_boot_ping_for_tests` + `await_idle_floor`). |
| `buzz-relay` `startup_owns_the_dependency_sampler` (boot_lifecycle.rs)
| Torn scrape: 39865872891 (main), 39624074244#2, 40385799750#1,
40413293801#1. Early exit: 39704588623 | metrics-exporter-prometheus
0.18.3 snapshots counters, then gauges. The scrape that first shows the
completion gauge can therefore predate the `checks_total` counters. |
Assert on a second scrape, taken after the gauge is seen. On an early
relay exit, the helper now panics with the exit status, stdout and
stderr. The early exit itself was not reproduced. |
| `buzz-db`
`sample_writer_waits_for_shared_floor_lock_before_sampling_time`
(replica_fence.rs:1086) and
`replica_floor_probe_waits_for_shared_writer_and_records_after_release`
(runtime/tests.rs:3452) | 39601763602 (main), 40391445284#1,
39825846222, 40301667418 | Both scan cluster-wide `pg_stat_activity` but
ran outside the serialized `postgres-cluster-global` group. Connection
churn from other tests masked activity (`MaskedActivity { masked: 1 }`).
| Rename both with a `cluster_global_` prefix. `nextest show-config
test-groups --profile postgres-ci` lists both in the group. |
| `buzz-db`
`stalled_reader_sessions_release_their_slots_for_new_connections`
(reader_pool_postgres_tests.rs:354) | 40399635380#2 | The floor warms
one connection, so the second warm-up acquire dials cold inside 150 ms.
| Retry only that warm-up acquire on `PoolTimedOut`, for up to 5 s
(#8472's precedent). The stall under test keeps the production budget. |
| `buzz-db`
`readiness_check_cancellation_balances_waiter_and_inflight_connection`
(runtime/tests.rs:391, `Elapsed`) | 39785370353 | The test aborted
`pg_sleep(5)` after a fixed 25 ms. Recovery lands at server start + 5 s,
but the bound is abort + 5 s, a ~22 ms margin locally. | Poll
`pg_stat_activity` until the query is active, then abort. Use
`pg_sleep(1)` within the unchanged 5 s bound. |
| `buzz-relay` `upgrade_would_deny_line_is_written_at_auth_while_open`
(connection.rs) | None yet (preventive, added by #8634) | It used the
100 ms-acquire `test_state_with_database_url` against the real test
database, the same shape as the `PoolTimedOut` flakes above. | Use
`test_state_with_database_url_and_pool_options` with sqlx's default
timeout. The 100 ms helper is renamed
`test_state_with_unreachable_database_url`; all its remaining callers
use closed ports. |
Local before/after counts are from the original `flakey-tests` commit
(1f0451915), on macOS with OrbStack Postgres:
- Regression test: 3/3 FAIL before, 3/3 PASS after.
- Worker stress: 3 failures in 288 iterations before; 2,100 passing runs
after.
- Statement timeouts: 44/50 FAIL before, 50/50 PASS after.
- Sampler: 2/200 FAIL before, 200/200 PASS after.
- `MaskedActivity` under `pgbench -C`: 2/50 iterations FAIL before.
After, the lane passed 3× (613/613) but was not re-run under pgbench.
- Stalled reader: 10/30 FAIL before, 30/30 PASS after.
- Forced late-`Execute` cancellation: 10/10 FAIL before, 10/10 PASS
after.
### Verification
This branch was pushed without local re-verification, by the user's
choice: local test runs were overloading the machine and failing for
that reason. **Hosted CI is the check.** It was pushed with `git push
--no-verify`, because the pre-push hook runs clippy and unit tests.
- Every file on this branch is byte-identical to its final version on
`flakey-tests` (67185df6a), where the counts above were measured.
- Not re-run for this PR: clippy, nextest and the Postgres lanes.
- Checked for this PR: `git diff --check`, `rustfmt --check` on the
changed files, and a conflict-free merge with all six siblings onto
`main`.
### Readiness
This PR has a product fix (the `buzz-db` lease heartbeat). Under
AGENTS.md "Before opening a PR", it needs an agent review and a **human
check against a local relay** before `buzz-review-completed` is added.
Suggested check:
1. With local Postgres and a relay, submit and approve a user deletion
with `buzz-admin user-deletions submit` / `approve`.
2. Run `buzz-admin user-deletions run` until it completes, then run it
again at once.
3. Working looks like this: both runs complete, neither reports the
request as leased by another worker, and `buzz-admin user-deletions
inspect <id>` shows no leftover lease. The original race is rare and is
not expected to reproduce by hand; this check confirms releases still
commit.
### Merge order
Independent of the other six. It merges cleanly onto `main`, and with
every sibling, in any order.
### Sibling PRs
Split from `flakey-tests` (one PR per area): #8677 (relay E2E helper +
Windows shell), #8678 (mobile), #8679 (Tauri apt cache + Windows Rust
cap), #8680 (desktop product races), #8681 (desktop E2E state gates),
#8682 (desktop E2E composer + row actions).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>