b37e477211
## Summary Make Kubernetes readiness depend only on the relay process lifecycle, so a shared Postgres or Redis slowdown cannot withdraw every pod at once. Move dependency checks to one fixed-cadence loop per pod. The loop permits only one evaluation at a time, emits metrics after each sample, and caches a timestamped report. `/_status` reads that cache without calling Postgres, Redis, or the deletion catalog. It reports `not_yet_sampled`, `fresh`, or `stale`. Each completed sample also writes a Unix timestamp gauge, so Datadog can compute sample age at query time. Fail closed when the community lifecycle lookup errors. An inconclusive tenant-state lookup no longer admits a socket to AUTH or REQ handling. ### Related issue None found. This addresses the elevated relay HTTP 500 and reconnect incident investigated on 2026-09-03. ### Tradeoff The readiness gauge now reports the latest private-probe observation, not the shutdown transition itself. After shutdown starts, a scrape can still see `1` until the next readiness probe refreshes the gauge to `0`. Kubernetes routing is unaffected because it uses the readiness response from the authoritative process state. A terminal `0` was never guaranteed: even a transition-owned gauge could update in process and then exit before Prometheus scraped it. If no scrape occurs before exit, both designs miss the terminal sample. The new design adds at most the interval until the next readiness probe when a scrape occurs during shutdown. Dependency status is now sampled every 30 seconds rather than on demand. A slow evaluation delays the next sample instead of stacking more work. The status payload exposes the cached report age. The completion timestamp stays unchanged when sampling stops, so `time() - timestamp` crosses the stale threshold without a server aging loop. Outcome counters and duration histograms remain cumulative and cannot provide no-data detection. ### Testing Gimli exercised exact SHA `edff1ec4a5362e132e367bf7449d2b0a68f1e9a9` on an isolated Blox stack. Repeated `/_status` bursts caused no dependency calls, metrics advanced without status traffic, stale status remained visible, and lifecycle lookup failures admitted no AUTH or REQ handling. Follow-up verification at `fdbb394ed598c94cde85ed01cb11e9de0ee8f00d` confirmed sampler-owned outcome and duration emission without status traffic and bound the real startup seam. The final timestamp-gauge change is covered by targeted red/green tests and the same boot regression at `c0f36e22b527c2fd6a63a548d036d006ffa17171`. ### Complexity Readiness remains one process-local sample per probe. Dependency I/O now has one owner and one execution path; request volume cannot increase dependency-check load. One completion timestamp replaces the prior server-side age update path; Datadog computes age without another loop. Community admission also returns to one fail-closed rule for inactive and unknown tenant state. Generated with [Claude Code](https://claude.ai/code) --------- Signed-off-by: tornquist <tornquist@squareup.com> Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Amp <amp@ampcode.com>
3.1 KiB
Relay deployment identity
Canonical relay images from ghcr.io/block/buzz carry two signed
attestations:
- SLSA build provenance maps the immutable image digest to the source commit and Docker workflow run.
- The Buzz deployment-eligibility predicate records the successful same-SHA CI run and the exact Buzz Helm chart version from that source commit.
The Docker workflow creates tagged multi-architecture manifests only after the
same full source SHA has a successful CI push run on main or release.
Architecture-specific build manifests may exist without tags while CI is
running or after it fails; they do not receive the deployment-eligibility
predicate and are not promotion inputs.
Verify a canonical eligible digest with:
gh attestation verify \
oci://ghcr.io/block/buzz@sha256:<digest> \
--repo block/buzz \
--signer-workflow block/buzz/.github/workflows/docker.yml \
--predicate-type https://buzz.block.xyz/attestations/deployment-eligibility/v1 \
--source-ref refs/heads/main
The predicate's helm_chart.compatible_version is image-to-chart metadata. It
does not describe database schema compatibility and does not relax Buzz's rule
that migrations remain backwards compatible.
The manual pre-merge workflow publishes only to
ghcr.io/block/buzz-staging-dev. Those preview images are intentionally
ineligible: they use a different package, may name non-main source, and do not
receive the canonical deployment-eligibility predicate.
Runtime inspection
The relay health listener exposes intrinsic build identity at /_status:
{
"service": "buzz-relay",
"version": "0.2.1",
"uptime_seconds": 123,
"build": {
"source_sha": "<40-character-source-sha>",
"id": "github-actions:<run-id>:<attempt>",
"url": "https://github.com/block/buzz/actions/runs/<run-id>/attempts/<attempt>"
},
"dependencies": {
"sample": "fresh",
"sample_interval_seconds": 30,
"sample_age_seconds": 12,
"postgres": true,
"redis": true,
"deletion_catalog": true,
"reason": "ready"
}
}
Non-CI builds report stable unknown or local fallback values instead of
claiming provenance they do not have.
dependencies is a cached diagnostic snapshot of shared-dependency health. A
per-pod background loop evaluates the dependencies every 30 seconds; the
endpoint only reads the latest report and never contacts a dependency itself,
so polling it costs nothing. sample is always present and reports whether
that cached verdict is fresh, stale, or not_yet_sampled — before the
first cycle completes the health fields are absent rather than defaulted.
/_readiness does not consult any of this — see
the readiness contract —
so this endpoint must never be wired to a Kubernetes probe.
Helm digest pinning
Buzz chart 0.1.8 and newer accept an immutable image digest:
image:
repository: ghcr.io/block/buzz
digest: sha256:<64-lowercase-hex-characters>
When image.digest is set, the chart renders repository@digest and ignores
image.tag. Existing tag-only values remain backwards compatible.