feat(db): expose connection setup metrics (#7286)

## Why

Current pool metrics show checkout outcomes and pool state after the
fact. They do not show when a checkout began or which writer-connection
setup step failed. During startup and pool growth, operators need to
distinguish pool saturation from a slow or unsafe connection setup.

## What

- Count every instrumented database checkout when it starts, with a
fixed operation label.
- Measure writer-pool creation and every physical writer connection
across `physical_connect`, `created_at_floor`, `session_timeouts`,
`isolation`, and `ready`.
- Publish fixed-cardinality start, terminal-outcome, duration, and
waiter metrics without database URLs, SQL, raw errors, connection
ordinals, or per-connection lifecycle logs.
- Document how to identify the active setup bottleneck during a rollout
or incident.

## How

The existing typed checkout wrapper records starts and current waiters.
The huddle-history path uses that wrapper instead of a raw pool
checkout.

The production SQLx `after_connect` hook records start, terminal
outcome, and duration for each sequential safety step. A drop guard
records cancellation once if setup exits before a terminal result. Fixed
enums keep every label bounded.

For a phase, `started_total - sum(attempts_total)` is the number of
in-progress attempts on that pod. Because setup is sequential, a later
phase starting also proves the earlier phases succeeded.

The database path emits metrics only. This PR intentionally does not add
per-connection lifecycle receipts or direct `stderr` writes.

## Risk

Medium. This changes the production writer-pool `after_connect` hook and
adds one counter update to instrumented checkout paths. It does not
change the database safety statements or their failure behavior. The
telemetry has fixed labels and no per-connection identifiers.

## Testing

At exact head `ffb5fcb114a9c986ffd0cb2cfc9d413a32d5b1aa`, a release
relay was started against isolated local PostgreSQL, Redis, and MinIO
services. The main health endpoint returned `ok`, readiness returned
`{"status":"ready"}`, the connection-step metrics were exported, and no
database lifecycle receipts were emitted. A release CLI then created a
channel, sent a message, and read the same event back successfully.

The earlier staging deployment used pre-rebase head
`f58e9480a4f068db0c591f604fd6800fdd4bfc45`. The deployed
multi-architecture image came from [GitHub Actions run
33780888255](https://github.com/block/buzz/actions/runs/33780888255),
manifest
`sha256:b9351fa644e08376cbe1999f9bee311d33d1799a68eadef7929c4f862a832fec`.

The [staging
deployment](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/317)
brought both pods in ReplicaSet `buzz-6c8758bd7d` to Ready with zero
restarts, and the connection metrics produced data in the [rollout
dashboard](https://app.datadoghq.com/dashboard/tm7-qxr-wt2/buzz-startup--rollout-safety).
That staging image predates the cleanup that removed per-connection
lifecycle logs; the metric schema and database safety statements are
unchanged.

## Verification

- `cargo test -p buzz-db -- --test-threads=1`: 128 tests passed across
the package and integration target; 255 opt-in tests remained ignored.
- Four opt-in production-path PostgreSQL regressions passed: initial
minimum connections, post-startup pool growth, isolation failure, and
session-timeout setup failure.
- All 35 relay media tests passed against the isolated PostgreSQL
database.
- Rust formatting, workspace clippy, desktop/Tauri clippy, web checks,
mobile analysis, security checks, and file-size policy checks passed.
- The repository-wide unit stage also exposed three unrelated existing
`buzz-acp` failures: one timing-sensitive keepalive test and two
environment-default tests.

## Bigger picture

This is the database-metrics part of the startup and rollout
observability work. The early-startup lifecycle foundation merged in
#7258, so the rebase removed that duplicate commit from this PR.

Originating discussion:
buzz://message?channel=6ac85131-70cd-4bda-a031-38d34114934e&id=fa2bed181c092697210a60bb6eedc55a665d5c1c6cabc1413a04686647011f71

Generated with Codex

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>

deploy/charts/buzz/README.md

@@ -148,8 +148,9 @@ state to zero; it does not fabricate dependency failures or latency samples.
 
 ### Operation-aware database pool acquisition contract
 
-The operation-aware families separate three questions: who is waiting now,
-how completed/abandoned attempts ended, and how long checkout waits took.
+The operation-aware families separate four questions: when an operation asked
+for a connection, who is waiting now, how completed/abandoned attempts ended,
+and how long checkout waits took.
 Outcome remains on the terminal counter for historical deployment comparison;
 it is intentionally absent from the expensive duration histogram.
 
@@ -161,6 +162,7 @@ maximum gauges when diagnosing total capacity pressure.
 
 | Metric | Type | Labels |
 |--------|------|--------|
+| `buzz_db_pool_acquire_started_total` | counter | `pool_role`, `operation` |
 | `buzz_db_pool_acquire_duration_seconds` | histogram | `pool_role`, `operation` |
 | `buzz_db_pool_acquire_attempts_total` | counter | `pool_role`, `operation`, `outcome` |
 | `buzz_db_pool_waiters` | gauge | `pool_role`, `operation`; tracked operations only, periodically refreshed including zero |
@@ -182,12 +184,49 @@ writer/maintenance
 ```
 
 Nine finite checkout buckets plus `+Inf`, sum, and count yield 12 histogram
-series per valid pair. The new contract therefore has a hard ceiling of 187
-raw Prometheus series per pod: `11 × (12 + 4 + 1)`. The two legacy acquisition
+series per valid pair. The new contract therefore has a hard ceiling of 198
+raw Prometheus series per pod: `11 × (1 + 12 + 4 + 1)`. The two legacy acquisition
 families remain temporarily for dashboard compatibility and are not part of
 that new-family budget. No `other` operation or request-controlled/sensitive
 label is valid.
 
+### Writer connection setup contract
+
+Writer connection setup is separate from checkout. At boot, SQLx constructs
+the writer pool and creates its minimum physical connections. Later it may
+create more when the pool grows or replaces a broken or expired connection.
+Every connected writer session must install the created-at floor, install the
+session timeouts, verify READ COMMITTED isolation, and reach `ready` before SQLx
+can give it to a caller.
+
+| Metric | Type | Labels |
+|--------|------|--------|
+| `buzz_db_connection_step_started_total` | counter | `pool_role`, `step` |
+| `buzz_db_connection_step_duration_seconds` | histogram | `pool_role`, `step` |
+| `buzz_db_connection_step_attempts_total` | counter | `pool_role`, `step`, `outcome` |
+
+The fixed steps are `writer_pool`, `physical_connect`, `created_at_floor`,
+`session_timeouts`, `isolation`, and `ready`. Measurable phases emit start,
+duration, and terminal evidence. `physical_connect` and `ready` are success
+milestones: SQLx 0.9 exposes `after_connect` only after DNS, network, TLS, and
+authentication finish, so Buzz does not invent separate timings for those
+internal phases. Raw connect failures before `after_connect` are classified on
+the aggregate `writer_pool` phase during initial construction. Outcomes are
+`succeeded`, `failed`, `timed_out`, and `cancelled` where valid.
+
+For a measurable phase on one pod, subtract all terminal outcomes from its
+start counter to derive the number of attempts currently in progress. The
+session phases run in order, so a later phase start also proves the earlier
+phases succeeded for that attempt.
+
+Four start counters, four 13-series histograms, and fifteen terminal counters
+create a hard ceiling of 71 raw Prometheus series per pod. Connection ordinals,
+database URLs, hosts, usernames, SQL, and raw errors are forbidden as metric
+labels.
+
+When audit logging is enabled, the `buzz_db_connection_*` metric totals combine
+the main writer pool and the separate audit writer pool.
+
 ### Physical database pool utilization and configuration contract
 
 Every physical Postgres pool the relay owns reports current utilization,