feat(relay): add detailed readiness metrics (#7149)

## Summary

This PR makes relay readiness failures diagnosable without weakening the
existing fail-closed readiness contract. It distinguishes Postgres pool
acquisition from query execution, Redis pool acquisition,
deletion-catalog validation, and the overall two-second deadline;
exports a bounded Prometheus contract for rollout dashboards; and fixes
the concurrency, shutdown, and listener-boundary semantics needed for
those signals to be trustworthy.

## Why

The previous `/_readiness` implementation exposed only an aggregate
ready/not-ready result. During a rollout, operators could not tell
whether a pod was blocked on:

- acquiring a Postgres connection;
- executing the Postgres readiness query;
- acquiring a Redis connection;
- validating the deletion catalog; or
- the shared readiness deadline.

Adding metrics to the existing handler also exposed three correctness
hazards that this PR resolves:

1. the same handler is mounted on both the public application listener
and the private Kubernetes health listener, so public requests could
otherwise distort rollout telemetry;
2. concurrent probes can finish out of order, allowing an older result
to overwrite newer current-state gauges; and
3. a probe started before SIGTERM can finish afterward, incorrectly
return `200 ready`, and resurrect ready gauges while the process is
draining.

## Behavior

### Readiness evaluation

- Postgres, Redis, and deletion-catalog checks still run under one
shared two-second deadline.
- Postgres distinguishes pool acquisition timeout/error from query
timeout/error.
- Redis distinguishes pool acquisition timeout/error. This does not
claim a Redis command round trip.
- The deletion catalog distinguishes operation timeout/error.
- Multiple failures are reported as `multiple_dependencies_failed`;
exhaustion of the shared deadline is reported as `overall_timeout` when
no more specific completed outcome wins.
- Readiness remains fail-closed: every dependency must succeed for `200
{"status":"ready"}`.

### Ordered publication and shutdown

`ReadinessCoordinator` is process-owned and uses one mutex as the
linearization point for probe generations, current-state publication,
and terminal shutdown.

- Every completed dependency attempt may contribute its truthful counter
and duration observation.
- Only the newest admissible probe generation may publish current-state
gauges.
- An older, slower probe cannot overwrite a newer probe's gauges.
- `begin_shutdown()` and probe commit serialize through the same
coordinator.
- Once shutdown commits, an in-flight probe cannot return ready or
publish ready/current dependency gauges, even if its dependency work
later succeeds.

Shutdown without dependency evaluation records only:

- `buzz_readiness_checks_total{reason="shutting_down"}`; and
- `buzz_readiness_state{check="overall"} = 0`.

It does **not** fabricate dependency failures, dependency state changes,
or zero-duration latency samples. If shutdown wins after an in-flight
evaluation actually ran, those completed dependency attempts may remain
as attempt telemetry, but they cannot overwrite shutdown-dominant
current state.

### Listener and response contract

- The private health listener's `/_readiness` route is the sole
authority for rollout readiness telemetry.
- The public application listener retains `/_readiness` for
compatibility, evaluates the same dependencies, and preserves the
existing response shape, but it emits no `buzz_readiness_*` metrics.
- Ready responses remain `200 {"status":"ready"}`.
- Shutdown responses remain `503 {"status":"shutting_down"}`.
- Failed private-health responses include the bounded `reason` plus
`postgres`, `redis`, and `deletion_catalog` booleans.
- Failed public compatibility responses retain the dependency booleans
but omit the new detailed reason.
- No header, query parameter, path value, or other request-controlled
value becomes a metric label.

## Prometheus contract

The final schema is intentionally capped at **99 raw Prometheus series
per pod**.

| Metric | Type | Labels | Raw series/pod |
|---|---|---|---:|
| `buzz_readiness_checks_total` | counter | `reason` | 12 |
| `buzz_readiness_dependency_checks_total` | counter | `dependency`,
typed `outcome` | 11 |
| `buzz_readiness_check_duration_seconds` | histogram | `check` | 72 |
| `buzz_readiness_state` | gauge | `check` | 4 |
| **Total** |  |  | **99** |

### Closed label sets

`reason`:

```text
ready
shutting_down
postgres_pool_timeout
postgres_pool_error
postgres_query_timeout
postgres_query_error
redis_pool_timeout
redis_pool_error
deletion_catalog_timeout
deletion_catalog_error
overall_timeout
multiple_dependencies_failed
```

Valid `dependency` / `outcome` pairs are enforced by typed enums:

- `postgres`: `success | pool_timeout | pool_error | operation_timeout |
operation_error`
- `redis`: `success | pool_timeout | pool_error`
- `deletion_catalog`: `success | operation_timeout | operation_error`

`check` is `overall | postgres | redis | deletion_catalog`.

The readiness histogram has 15 configured buckets concentrated around
the two-second deadline:

```text
0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5,
0.75, 1.0, 1.25, 1.5, 1.75, 2.0, 2.5, +Inf
```

The contract deliberately removes the redundant overall `result` label
and the histogram `outcome` label. Pod, ReplicaSet, version, rollout,
raw error, SQL, URL, tenant, user, community, pubkey, and
request-controlled values are prohibited application labels;
infrastructure enrichment can supply deployment identity outside the
application metric.

## Postgres production seam and CI

The real `Db::readiness_check` path now supports deterministic
production-seam testing while preserving the same acquisition/query
implementation used by the relay. The isolated PostgreSQL lane
automatically discovers and executes three ignored integration tests
covering:

- a held sole connection causing pool timeout, followed by recovery
after release;
- closed-pool acquisition error;
- acquisition success followed by query timeout;
- classified query error;
- cancellation while waiting for a connection;
- cancellation during an in-flight query; and
- eventual pool recovery with waiter/in-flight state balanced.

This prevents the central SQLx/Postgres behavior from being merely
compiled but never executed in CI.

## Tests and verification

- `cargo fmt --all -- --check`
- `./scripts/test-postgres-test-discovery.sh`
- complete `cargo test -p buzz-db`
- focused `cargo test -p buzz-relay readiness`
- real-router `cargo test -p buzz-relay real_health_route_`
- controlled PostgreSQL execution of all three ignored readiness tests
- production health router -> real `GET /_readiness` -> Prometheus
render assertions
- public-router requests proving zero rollout telemetry
- deterministic out-of-order A/B probe tests
- SIGTERM-during-probe tests proving shutdown dominance
- exported metric name/type/exact-label/bucket assertions
- exported raw-series allowlist and 99-series ceiling assertion

The route-to-scrape regression test uses the production Prometheus
builder and verifies the `2`, `2.5`, and `+Inf` readiness buckets. It
fails if the health recording call, health route, public/health
boundary, generation fence, shutdown fence, or bucket override is
removed.

## Risk assessment

**Medium.** This changes the live readiness publication path and adds
synchronization around probe commit/shutdown. The risk is bounded by:

- preserving the existing dependencies and shared two-second deadline;
- keeping readiness fail-closed;
- using a short, process-local mutex only at begin/commit linearization
points, not across dependency awaits;
- retaining the public compatibility endpoint while isolating its
telemetry;
- enforcing typed, low-cardinality labels and an exported series
ceiling; and
- covering the production router, Prometheus exposition, real PostgreSQL
seam, concurrency ordering, and shutdown races.

## Operational notes

- Dashboards should treat `buzz_readiness_state` as the latest sampled
current state, not as an event stream.
- `shutting_down` should be excluded from dependency-failure alerts
because no dependency failure is implied.
- Do not use `default_zero()` for missing current-state data;
missing/stale is unknown, not healthy.
- Use histogram buckets, heatmaps, max, or average until the Datadog
distribution metadata confirms percentiles are enabled; do not title a
widget p95 before that live readback.

## Non-goals

This PR does not add the broader process-startup lifecycle,
worker/listener supervision, shutdown coordinator, WebSocket/huddle
handoff, client recovery telemetry, dashboard mutations, Datadog
configuration changes, or deployment changes. Those remain separate
follow-up work.

## References

- [Staging dev relay image
runbook](https://github.com/block/buzz/blob/main/docs/staging-dev-relay-images.md)
- Readiness telemetry contract: `deploy/charts/buzz/README.md`

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>

deploy/charts/buzz/README.md

@@ -115,6 +115,27 @@ disables that probe through `relay.extraEnv`, `/_readiness` does not test object
 storage; configuration is still parsed strictly, but reachability and addressing
 errors surface on the first storage operation.
 
+### Readiness telemetry contract
+
+Only requests served by the private health listener (`BUZZ_HEALTH_PORT`) emit
+rollout readiness telemetry. The compatibility `/_readiness` route on the public
+app listener returns health but does not change these metrics.
+
+| Metric | Type | Labels |
+|--------|------|--------|
+| `buzz_readiness_checks_total` | counter | `reason` from the closed readiness-reason set |
+| `buzz_readiness_dependency_checks_total` | counter | `dependency`, typed bounded `outcome` |
+| `buzz_readiness_check_duration_seconds` | histogram | `check` only |
+| `buzz_readiness_state` | gauge | `check` only; latest publishable generation |
+
+The schema has a ceiling of 99 raw Prometheus series per pod: 12 overall
+reasons, 11 valid dependency/outcome pairs, 72 histogram series, and 4 gauges.
+Do not add pod, ReplicaSet, version, rollout, error text, SQL, URL, tenant,
+user, community, pubkey, header, query, or other request-controlled labels.
+Shutdown without dependency evaluation increments only
+`buzz_readiness_checks_total{reason="shutting_down"}` and sets the overall
+state to zero; it does not fabricate dependency failures or latency samples.
+
 ## Relay Pod extensions
 
 The chart exposes narrow extension points for init containers, volumes, relay