## Summary
The scheduled `buzz-admin deletions drain` Job never gets past its first
database connection in a hosted staging deployment. Every run exits 5
after exactly 3 seconds with `pool timed out while waiting for an open
connection`.
The drain starts in a fresh pod with no mesh sidecar, and that pod's
first connection can take longer than the relay's 3-second acquisition
budget. `connect_store()` in `buzz-deletion` inherits that budget from
`DbConfig::default()`, with no retry. In the same pod, the same command
run 15 seconds later succeeds. TCP to Postgres, Redis, and S3 from a
sidecar-less pod is fine.
#7770 fixed the same failure for `buzz-admin storage-snapshot`, but only
inside `buzz-admin`. This PR moves that connector into `buzz-db` as
`Db::connect_cold_start` and uses it from both run-once workers:
- **Pool settings:** a 30s acquisition budget, and no idle spare
connections (`min_connections: 0`).
- **Retries:** startup timeouts and transient transport errors retry at
most twice, after 2s and 5s, so startup waits at most 97s.
- **No retry:** configuration, authentication, TLS, and protocol errors
fail immediately, and the error message stays free of credentials.
- **Per-worker settings:** each worker keeps its own pool size.
Storage-snapshot stays at 1 connection, and deletion stays at
`BUZZ_DB_POOL_SIZE` (default 20). Each worker also keeps its session
timeouts.
- **Startup events:** the stderr events carry a worker prefix.
Storage-snapshot still emits `storage_snapshot_db_connect_*`, and the
deletion worker now emits `deletion_db_connect_*`.
- **Error text:** the exhaustion message is now `{worker} database
startup failed after …`. For storage-snapshot that reads
`storage_snapshot database startup failed …` instead of `storage
snapshot database startup failed …`. Nothing in this repo matches on
that string.
- **Session-setup failures:** SQLx retries a failed `after_connect` hook
(writer floor guard, session timeouts, isolation assertion) until the
acquire deadline. Those failures therefore surface as `timeout`, are
retried, and fail closed after at most 97s instead of immediately.
Relay pool defaults are unchanged. The deletion CLI's operator commands
(`list`, `approve`, …) use the same connector. They normally run inside
the warm relay container, where the first attempt succeeds.
**What got simpler:** there is now one cold-start connector instead of a
`buzz-admin`-private copy that the deletion worker couldn't reach.
`storage_snapshot_startup.rs` is gone, and its unit tests moved with the
code. The two Postgres startup tests now share one harness with the
drain's tests (`postgres_storage_snapshot.rs` →
`postgres_cold_start.rs`).
### Related issue
Follow-up to #7770 (same failure, other worker) and #7827 (the drain
CronJob). No duplicate found.
### Testing
All at head `1545de7b5`:
- **GitHub CI:** green on attempt 1, including `PostgreSQL Domain /
PostgreSQL Tests`.
- **Blox:**
- fmt, `clippy --workspace --all-targets -D warnings`, and the `buzz-db`
/ `buzz-admin` / `buzz-deletion` tests pass.
- All four `postgres_cold_start` tests pass, each against a freshly
migrated database.
- Falsification: with main's `connect_store` restored, the drain retry
test fails.
- **Agent review:** two rounds. All findings are fixed or disclosed; the
error-text change is listed above.
- **Agent exercise:** real `buzz-admin deletions list` / `deletions
drain` against local Postgres 17 behind a fault proxy.
- **5s delay:** succeeds on attempt 1 (3/3). Main fails at about 3.5s.
- **Dropped handshake, then 5s delay:** attempt 1 fails with
`UnexpectedEof`, then a 2s backoff, then attempt 2 succeeds (3/3).
- **Wrong password, missing database or malformed URL:** exit 5 after
one attempt.
- **Persistent rejection:** exit 5 after three attempts, with 2s and 5s
backoffs.
- The deletion queue stayed empty throughout.
- **Not covered:** an approved deletion, real S3 / Pod Identity, TLS and
session-setup failures, and a live 97s stall.
---------
Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
## Summary
The standalone S3 accounting worker can exit before its scan begins
because database startup inherits the relay’s three-second connection
budget. A cold DNS probe took about five seconds.
Give `buzz-admin storage-snapshot` its own pool: one connection, no
minimum idle connections, and a 30-second acquisition budget. Retry
startup timeouts and transport failures at most twice, after two and
five seconds. Total startup wait is bounded to 97 seconds.
Configuration, authentication, TLS, and protocol errors fail
immediately.
Log each startup attempt’s elapsed time and error category without
connection credentials. Relay defaults stay unchanged. Retries stop
before lock acquisition; the same lock-owning database session still
publishes the completed snapshot.
### Related issue
Follow-up to #7543. No duplicate startup fix found.
### Testing
Archived-command check: built a nextest archive, temporarily moved
`target/debug/buzz-admin` out of the checkout, and ran
`scripts/postgres-test-run.sh --archive-file <archive>`. Both tests
failed before the path fix and passed afterward. The tests use nextest’s
relocated executable path, with the original Cargo path retained for
local runs.
Manually restored the old startup call and ran
`scripts/postgres-test-run.sh -p buzz-admin --test
postgres_storage_snapshot -E 'test(storage_snapshot_retries_then)'`. It
failed before reaching S3; restoring the fix passed. This confirms the
test exercises the actual worker command.
Local `just ci` passed all stages before mobile tests, which stopped at
SDK discovery in the `objective_c` native hook. The default Xcode tools
report an unaccepted license. Two existing PostgreSQL tests hit
connection timeouts in the parallel run and passed individually.
After merge, build a worker image and repeat the cold-pod staging smoke
test through snapshot publication before changing relay metrics mode.
Generated with Codex
---------
Signed-off-by: Ravneet Arora <rarora@squareup.com>
