Make relay readiness process-local (#7341)

## Summary

Make Kubernetes readiness depend only on the relay process lifecycle, so
a shared Postgres or Redis slowdown cannot withdraw every pod at once.

Move dependency checks to one fixed-cadence loop per pod. The loop
permits only one evaluation at a time, emits metrics after each sample,
and caches a timestamped report. `/_status` reads that cache without
calling Postgres, Redis, or the deletion catalog. It reports
`not_yet_sampled`, `fresh`, or `stale`. Each completed sample also
writes a Unix timestamp gauge, so Datadog can compute sample age at
query time.

Fail closed when the community lifecycle lookup errors. An inconclusive
tenant-state lookup no longer admits a socket to AUTH or REQ handling.

### Related issue

None found. This addresses the elevated relay HTTP 500 and reconnect
incident investigated on 2026-09-03.

### Tradeoff

The readiness gauge now reports the latest private-probe observation,
not the shutdown transition itself. After shutdown starts, a scrape can
still see `1` until the next readiness probe refreshes the gauge to `0`.
Kubernetes routing is unaffected because it uses the readiness response
from the authoritative process state.

A terminal `0` was never guaranteed: even a transition-owned gauge could
update in process and then exit before Prometheus scraped it. If no
scrape occurs before exit, both designs miss the terminal sample. The
new design adds at most the interval until the next readiness probe when
a scrape occurs during shutdown.

Dependency status is now sampled every 30 seconds rather than on demand.
A slow evaluation delays the next sample instead of stacking more work.
The status payload exposes the cached report age. The completion
timestamp stays unchanged when sampling stops, so `time() - timestamp`
crosses the stale threshold without a server aging loop. Outcome
counters and duration histograms remain cumulative and cannot provide
no-data detection.

### Testing

Gimli exercised exact SHA `edff1ec4a5362e132e367bf7449d2b0a68f1e9a9` on
an isolated Blox stack. Repeated `/_status` bursts caused no dependency
calls, metrics advanced without status traffic, stale status remained
visible, and lifecycle lookup failures admitted no AUTH or REQ handling.
Follow-up verification at `fdbb394ed598c94cde85ed01cb11e9de0ee8f00d`
confirmed sampler-owned outcome and duration emission without status
traffic and bound the real startup seam. The final timestamp-gauge
change is covered by targeted red/green tests and the same boot
regression at `c0f36e22b527c2fd6a63a548d036d006ffa17171`.

### Complexity

Readiness remains one process-local sample per probe. Dependency I/O now
has one owner and one execution path; request volume cannot increase
dependency-check load. One completion timestamp replaces the prior
server-side age update path; Datadog computes age without another loop.
Community admission also returns to one fail-closed rule for inactive
and unknown tenant state.

Generated with [Claude Code](https://claude.ai/code)

---------

Signed-off-by: tornquist <tornquist@squareup.com>
Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Amp <amp@ampcode.com>

docs/deployment-identity.md

@@ -47,6 +47,15 @@ The relay health listener exposes intrinsic build identity at `/_status`:
     "source_sha": "<40-character-source-sha>",
     "id": "github-actions:<run-id>:<attempt>",
     "url": "https://github.com/block/buzz/actions/runs/<run-id>/attempts/<attempt>"
+  },
+  "dependencies": {
+    "sample": "fresh",
+    "sample_interval_seconds": 30,
+    "sample_age_seconds": 12,
+    "postgres": true,
+    "redis": true,
+    "deletion_catalog": true,
+    "reason": "ready"
   }
 }
 ```
@@ -54,6 +63,16 @@ The relay health listener exposes intrinsic build identity at `/_status`:
 Non-CI builds report stable `unknown` or `local` fallback values instead of
 claiming provenance they do not have.
 
+`dependencies` is a cached diagnostic snapshot of shared-dependency health. A
+per-pod background loop evaluates the dependencies every 30 seconds; the
+endpoint only reads the latest report and never contacts a dependency itself,
+so polling it costs nothing. `sample` is always present and reports whether
+that cached verdict is `fresh`, `stale`, or `not_yet_sampled` — before the
+first cycle completes the health fields are absent rather than defaulted.
+`/_readiness` does not consult any of this — see
+[the readiness contract](../deploy/charts/buzz/README.md#readiness-contract) —
+so this endpoint must never be wired to a Kubernetes probe.
+
 ## Helm digest pinning
 
 Buzz chart `0.1.8` and newer accept an immutable image digest: