mirror of
https://github.com/block/buzz.git
synced 2026-10-27 12:40:30 +00:00
b37e477211
## Summary Make Kubernetes readiness depend only on the relay process lifecycle, so a shared Postgres or Redis slowdown cannot withdraw every pod at once. Move dependency checks to one fixed-cadence loop per pod. The loop permits only one evaluation at a time, emits metrics after each sample, and caches a timestamped report. `/_status` reads that cache without calling Postgres, Redis, or the deletion catalog. It reports `not_yet_sampled`, `fresh`, or `stale`. Each completed sample also writes a Unix timestamp gauge, so Datadog can compute sample age at query time. Fail closed when the community lifecycle lookup errors. An inconclusive tenant-state lookup no longer admits a socket to AUTH or REQ handling. ### Related issue None found. This addresses the elevated relay HTTP 500 and reconnect incident investigated on 2026-09-03. ### Tradeoff The readiness gauge now reports the latest private-probe observation, not the shutdown transition itself. After shutdown starts, a scrape can still see `1` until the next readiness probe refreshes the gauge to `0`. Kubernetes routing is unaffected because it uses the readiness response from the authoritative process state. A terminal `0` was never guaranteed: even a transition-owned gauge could update in process and then exit before Prometheus scraped it. If no scrape occurs before exit, both designs miss the terminal sample. The new design adds at most the interval until the next readiness probe when a scrape occurs during shutdown. Dependency status is now sampled every 30 seconds rather than on demand. A slow evaluation delays the next sample instead of stacking more work. The status payload exposes the cached report age. The completion timestamp stays unchanged when sampling stops, so `time() - timestamp` crosses the stale threshold without a server aging loop. Outcome counters and duration histograms remain cumulative and cannot provide no-data detection. ### Testing Gimli exercised exact SHA `edff1ec4a5362e132e367bf7449d2b0a68f1e9a9` on an isolated Blox stack. Repeated `/_status` bursts caused no dependency calls, metrics advanced without status traffic, stale status remained visible, and lifecycle lookup failures admitted no AUTH or REQ handling. Follow-up verification at `fdbb394ed598c94cde85ed01cb11e9de0ee8f00d` confirmed sampler-owned outcome and duration emission without status traffic and bound the real startup seam. The final timestamp-gauge change is covered by targeted red/green tests and the same boot regression at `c0f36e22b527c2fd6a63a548d036d006ffa17171`. ### Complexity Readiness remains one process-local sample per probe. Dependency I/O now has one owner and one execution path; request volume cannot increase dependency-check load. One completion timestamp replaces the prior server-side age update path; Datadog computes age without another loop. Community admission also returns to one fail-closed rule for inactive and unknown tenant state. Generated with [Claude Code](https://claude.ai/code) --------- Signed-off-by: tornquist <tornquist@squareup.com> Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Amp <amp@ampcode.com>
88 lines
3.1 KiB
Markdown
88 lines
3.1 KiB
Markdown
# Relay deployment identity
|
|
|
|
Canonical relay images from `ghcr.io/block/buzz` carry two signed
|
|
attestations:
|
|
|
|
- SLSA build provenance maps the immutable image digest to the source commit
|
|
and Docker workflow run.
|
|
- The Buzz deployment-eligibility predicate records the successful same-SHA
|
|
CI run and the exact Buzz Helm chart version from that source commit.
|
|
|
|
The Docker workflow creates tagged multi-architecture manifests only after the
|
|
same full source SHA has a successful `CI` push run on `main` or `release`.
|
|
Architecture-specific build manifests may exist without tags while CI is
|
|
running or after it fails; they do not receive the deployment-eligibility
|
|
predicate and are not promotion inputs.
|
|
|
|
Verify a canonical eligible digest with:
|
|
|
|
```bash
|
|
gh attestation verify \
|
|
oci://ghcr.io/block/buzz@sha256:<digest> \
|
|
--repo block/buzz \
|
|
--signer-workflow block/buzz/.github/workflows/docker.yml \
|
|
--predicate-type https://buzz.block.xyz/attestations/deployment-eligibility/v1 \
|
|
--source-ref refs/heads/main
|
|
```
|
|
|
|
The predicate's `helm_chart.compatible_version` is image-to-chart metadata. It
|
|
does not describe database schema compatibility and does not relax Buzz's rule
|
|
that migrations remain backwards compatible.
|
|
|
|
The manual pre-merge workflow publishes only to
|
|
`ghcr.io/block/buzz-staging-dev`. Those preview images are intentionally
|
|
ineligible: they use a different package, may name non-main source, and do not
|
|
receive the canonical deployment-eligibility predicate.
|
|
|
|
## Runtime inspection
|
|
|
|
The relay health listener exposes intrinsic build identity at `/_status`:
|
|
|
|
```json
|
|
{
|
|
"service": "buzz-relay",
|
|
"version": "0.2.1",
|
|
"uptime_seconds": 123,
|
|
"build": {
|
|
"source_sha": "<40-character-source-sha>",
|
|
"id": "github-actions:<run-id>:<attempt>",
|
|
"url": "https://github.com/block/buzz/actions/runs/<run-id>/attempts/<attempt>"
|
|
},
|
|
"dependencies": {
|
|
"sample": "fresh",
|
|
"sample_interval_seconds": 30,
|
|
"sample_age_seconds": 12,
|
|
"postgres": true,
|
|
"redis": true,
|
|
"deletion_catalog": true,
|
|
"reason": "ready"
|
|
}
|
|
}
|
|
```
|
|
|
|
Non-CI builds report stable `unknown` or `local` fallback values instead of
|
|
claiming provenance they do not have.
|
|
|
|
`dependencies` is a cached diagnostic snapshot of shared-dependency health. A
|
|
per-pod background loop evaluates the dependencies every 30 seconds; the
|
|
endpoint only reads the latest report and never contacts a dependency itself,
|
|
so polling it costs nothing. `sample` is always present and reports whether
|
|
that cached verdict is `fresh`, `stale`, or `not_yet_sampled` — before the
|
|
first cycle completes the health fields are absent rather than defaulted.
|
|
`/_readiness` does not consult any of this — see
|
|
[the readiness contract](../deploy/charts/buzz/README.md#readiness-contract) —
|
|
so this endpoint must never be wired to a Kubernetes probe.
|
|
|
|
## Helm digest pinning
|
|
|
|
Buzz chart `0.1.8` and newer accept an immutable image digest:
|
|
|
|
```yaml
|
|
image:
|
|
repository: ghcr.io/block/buzz
|
|
digest: sha256:<64-lowercase-hex-characters>
|
|
```
|
|
|
|
When `image.digest` is set, the chart renders `repository@digest` and ignores
|
|
`image.tag`. Existing tag-only values remain backwards compatible.
|