## Summary
Make Kubernetes readiness depend only on the relay process lifecycle, so
a shared Postgres or Redis slowdown cannot withdraw every pod at once.
Move dependency checks to one fixed-cadence loop per pod. The loop
permits only one evaluation at a time, emits metrics after each sample,
and caches a timestamped report. `/_status` reads that cache without
calling Postgres, Redis, or the deletion catalog. It reports
`not_yet_sampled`, `fresh`, or `stale`. Each completed sample also
writes a Unix timestamp gauge, so Datadog can compute sample age at
query time.
Fail closed when the community lifecycle lookup errors. An inconclusive
tenant-state lookup no longer admits a socket to AUTH or REQ handling.
### Related issue
None found. This addresses the elevated relay HTTP 500 and reconnect
incident investigated on 2026-09-03.
### Tradeoff
The readiness gauge now reports the latest private-probe observation,
not the shutdown transition itself. After shutdown starts, a scrape can
still see `1` until the next readiness probe refreshes the gauge to `0`.
Kubernetes routing is unaffected because it uses the readiness response
from the authoritative process state.
A terminal `0` was never guaranteed: even a transition-owned gauge could
update in process and then exit before Prometheus scraped it. If no
scrape occurs before exit, both designs miss the terminal sample. The
new design adds at most the interval until the next readiness probe when
a scrape occurs during shutdown.
Dependency status is now sampled every 30 seconds rather than on demand.
A slow evaluation delays the next sample instead of stacking more work.
The status payload exposes the cached report age. The completion
timestamp stays unchanged when sampling stops, so `time() - timestamp`
crosses the stale threshold without a server aging loop. Outcome
counters and duration histograms remain cumulative and cannot provide
no-data detection.
### Testing
Gimli exercised exact SHA `edff1ec4a5362e132e367bf7449d2b0a68f1e9a9` on
an isolated Blox stack. Repeated `/_status` bursts caused no dependency
calls, metrics advanced without status traffic, stale status remained
visible, and lifecycle lookup failures admitted no AUTH or REQ handling.
Follow-up verification at `fdbb394ed598c94cde85ed01cb11e9de0ee8f00d`
confirmed sampler-owned outcome and duration emission without status
traffic and bound the real startup seam. The final timestamp-gauge
change is covered by targeted red/green tests and the same boot
regression at `c0f36e22b527c2fd6a63a548d036d006ffa17171`.
### Complexity
Readiness remains one process-local sample per probe. Dependency I/O now
has one owner and one execution path; request volume cannot increase
dependency-check load. One completion timestamp replaces the prior
server-side age update path; Datadog computes age without another loop.
Community admission also returns to one fail-closed rule for inactive
and unknown tenant state.
Generated with [Claude Code](https://claude.ai/code)
---------
Signed-off-by: tornquist <tornquist@squareup.com>
Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Amp <amp@ampcode.com>
## Why
Phase 1 of progressive delivery needs an explicit eligibility boundary
before deployment automation can safely consume relay artifacts. Today
the canonical Docker workflow can publish independently of the same-SHA
CI result, the relay does not expose its source/build identity, and the
Helm chart only renders mutable tags.
The existing `ghcr.io/block/buzz-staging-dev` workflow in #6709 remains
a manual pre-merge preview lane. It is deliberately not a canonical
promotion input.
## What changed
- Compile the full source SHA and GitHub Actions build identity into
canonical relay binaries and expose it at `/_status`.
- Gate creation of tagged multi-architecture manifests on the latest
exact-SHA `CI` push run succeeding on `main` or `release`.
- Preserve the signed SLSA provenance and add a signed, release-only
deployment-eligibility predicate with source, build run, qualifying CI
run, and exact compatible Buzz chart version.
- Keep failed or pending builds untagged and without an eligibility
predicate; preview-package artifacts are excluded by package and signer
contract.
- Add optional `image.digest` support to Buzz chart `0.1.8`, retaining
the existing tag fallback.
- Document verification and the intentional boundary between image/chart
compatibility and backwards-compatible database migrations.
## Risk and rollout
Canonical tag publication now waits for same-SHA CI and fails closed
when the newest matching run fails. Per-architecture blobs may remain
untagged after a failed qualification; they are not promotion inputs.
Debug manifests are also CI-gated but do not receive the release
eligibility predicate.
There are no migration or schema-orchestration changes. The companion
BuilderBot infrastructure PR will pin a separately verified canonical
main digest and expose a read-only running-deployment inspection
command. Signed eligibility is available only for artifacts published
after this workflow lands.
## Verification
- Blox: workflow contract fixtures, including stale-green rejection and
predicate structure
- Blox: `actionlint` for `ci.yml` and `docker.yml`
- Blox: 46 Helm unit tests, chart lint, and production/quickstart render
fixtures
- Blox: relay status unit test, workspace Rust formatting, and workspace
all-target clippy
- Live evidence: exact-SHA CI selector chose successful run
`32857345153` for source `9d1e4b257657f382d3111ce748f3da8d063b7671`
- Existing canonical digest provenance verified with `gh attestation
verify`
## References
- #6709 — developer-preview precursor; intentionally separate from
canonical eligibility
- `docs/deployment-identity.md`
*Generated with Codex*
---------
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
