mirror of
https://github.com/block/buzz.git
synced 2026-10-27 12:40:30 +00:00
main
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Luke Tornquist
|
5a93e862ee |
fix(chart): own the Datadog version label on every workload Pod (#7876)
## Summary
Every chart-managed Pod now carries a chart-owned
`tags.datadoghq.com/version` label derived from the deployed image,
instead of a value each wrapper restates once per workload.
Covered Pod templates:
- relay Deployment (`templates/deployment.yaml`)
- pairing relay (`templates/pairing-relay.yaml`)
- storage-accounting CronJob
(`templates/storage-accounting-cronjob.yaml`)
- operator jobs, including the deletion drain
(`templates/_operator-jobs.tpl`)
**Source order:** `image.tag`, then `image.digest`, then
`Chart.AppVersion` (`buzz.imageVersionSource`). Tag-first is deliberate:
- The rendered value matches the `sha-<commit>` pins wrappers already
carry, so upgrading to this chart does not roll any Pods.
- An image promotion only has to write `image.tag` + `image.digest`. No
per-workload label edit is needed.
Exact runtime identity stays exact in two places. The image reference is
`repository@digest` when a digest is pinned.
`BUZZ_STORAGE_SNAPSHOT_CODE_SHA` (persisted as `code_sha`) is still
digest-first.
**Label safety:** `buzz.imageVersionLabel` maps any OCI tag or digest to
a valid label value.
- Valid values already in Datadog's normal form (lowercase, no `__`)
pass through byte for byte, so `sha-<hex>` and `1.2.3-rc.4` are
unchanged. Datadog lowercases tag values and collapses `__`, so any
other value is hashed rather than allowed to report the same version as
a different revision.
- A digest keeps 63 hex characters.
- Anything else is replaced by the first 63 hex characters of its
SHA-256.
- Exactly 63 lowercase hex characters is reserved for the hashed and
digest forms, so a rendered label cannot be fed back as a tag to make
two revisions collide.
- When both tag and digest are set, the label names the tag even if they
disagree; the digest still pins what runs.
**Precedence:** a `tags.datadoghq.com/version` supplied through any
workload's `podLabels` is ignored. Every other label, including
`tags.datadoghq.com/env` and `/service`, passes through unchanged.
Overriding instead of failing means a wrapper with a stale pin is
corrected by the upgrade itself. The chart bump and the wrapper cleanup
don't have to land together.
**What got simpler:** one image identity instead of N. Wrappers delete
their per-workload version pins. Image promotion tooling no longer needs
a "mirror the tag into a label path" feature. One shared helper
(`buzz.podLabelsWithVersion`) replaces four `podLabels` passthroughs.
Chart `0.1.10 → 0.1.11`, with an `artifacthub.io/changes` entry.
Documented in the chart README ("Datadog version label") and
`docs/deployment-identity.md`.
Note: a vendor-namespaced label now renders unconditionally in an OSS
chart. This is in tension with
[VISION_SOVEREIGN.md](../blob/main/VISION_SOVEREIGN.md). The label is
inert without the operator-supplied `tags.datadoghq.com/service`/`env`,
and the chart already ships vendor-specific defaults (e.g.
`sidecar.istio.io/inject`).
### Testing
Run on a Blox (Linux) workstation at tree `41634b7ca` (helm v3.16.4,
helm-unittest 0.8.2, the CI versions):
- `helm lint deploy/charts/buzz`: 0 failed
- `helm unittest deploy/charts/buzz`: 12 suites, **85/85 passed**
New `tests/version_label_test.yaml` checks the relay, pairing relay, and
deletion drain: tag wins over a stale `podLabels` pin, the label is
stamped with no `podLabels`, and digest-only images get a label-safe
value. `tests/storage_accounting_test.yaml` covers the same cases for
the CronJob, plus the `Chart.AppVersion` fallback.
Mutation checks:
- Reverting any one template to the plain `podLabels` passthrough fails
tests.
- Switching the source order to digest-first fails tests.
- Restoring the case-insensitive passthrough fails the three
Datadog-normalization tests (uppercase, `__`, uppercased 63-hex).
No Rust, desktop, web, or mobile sources changed.
## Linear tracking
Related work: [BUZZ-215](https://linear.app/squareup/issue/BUZZ-215).
---------
Signed-off-by: tornquist <tornquist@squareup.com>
Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
|
||
|
Luke Tornquist
|
b37e477211 |
Make relay readiness process-local (#7341)
## Summary Make Kubernetes readiness depend only on the relay process lifecycle, so a shared Postgres or Redis slowdown cannot withdraw every pod at once. Move dependency checks to one fixed-cadence loop per pod. The loop permits only one evaluation at a time, emits metrics after each sample, and caches a timestamped report. `/_status` reads that cache without calling Postgres, Redis, or the deletion catalog. It reports `not_yet_sampled`, `fresh`, or `stale`. Each completed sample also writes a Unix timestamp gauge, so Datadog can compute sample age at query time. Fail closed when the community lifecycle lookup errors. An inconclusive tenant-state lookup no longer admits a socket to AUTH or REQ handling. ### Related issue None found. This addresses the elevated relay HTTP 500 and reconnect incident investigated on 2026-09-03. ### Tradeoff The readiness gauge now reports the latest private-probe observation, not the shutdown transition itself. After shutdown starts, a scrape can still see `1` until the next readiness probe refreshes the gauge to `0`. Kubernetes routing is unaffected because it uses the readiness response from the authoritative process state. A terminal `0` was never guaranteed: even a transition-owned gauge could update in process and then exit before Prometheus scraped it. If no scrape occurs before exit, both designs miss the terminal sample. The new design adds at most the interval until the next readiness probe when a scrape occurs during shutdown. Dependency status is now sampled every 30 seconds rather than on demand. A slow evaluation delays the next sample instead of stacking more work. The status payload exposes the cached report age. The completion timestamp stays unchanged when sampling stops, so `time() - timestamp` crosses the stale threshold without a server aging loop. Outcome counters and duration histograms remain cumulative and cannot provide no-data detection. ### Testing Gimli exercised exact SHA `edff1ec4a5362e132e367bf7449d2b0a68f1e9a9` on an isolated Blox stack. Repeated `/_status` bursts caused no dependency calls, metrics advanced without status traffic, stale status remained visible, and lifecycle lookup failures admitted no AUTH or REQ handling. Follow-up verification at `fdbb394ed598c94cde85ed01cb11e9de0ee8f00d` confirmed sampler-owned outcome and duration emission without status traffic and bound the real startup seam. The final timestamp-gauge change is covered by targeted red/green tests and the same boot regression at `c0f36e22b527c2fd6a63a548d036d006ffa17171`. ### Complexity Readiness remains one process-local sample per probe. Dependency I/O now has one owner and one execution path; request volume cannot increase dependency-check load. One completion timestamp replaces the prior server-side age update path; Datadog computes age without another loop. Community admission also returns to one fail-closed rule for inactive and unknown tenant state. Generated with [Claude Code](https://claude.ai/code) --------- Signed-off-by: tornquist <tornquist@squareup.com> Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Amp <amp@ampcode.com> |
||
|
Luke Tornquist
|
f249710331 |
Qualify canonical relay images for staged delivery (#6781)
## Why Phase 1 of progressive delivery needs an explicit eligibility boundary before deployment automation can safely consume relay artifacts. Today the canonical Docker workflow can publish independently of the same-SHA CI result, the relay does not expose its source/build identity, and the Helm chart only renders mutable tags. The existing `ghcr.io/block/buzz-staging-dev` workflow in #6709 remains a manual pre-merge preview lane. It is deliberately not a canonical promotion input. ## What changed - Compile the full source SHA and GitHub Actions build identity into canonical relay binaries and expose it at `/_status`. - Gate creation of tagged multi-architecture manifests on the latest exact-SHA `CI` push run succeeding on `main` or `release`. - Preserve the signed SLSA provenance and add a signed, release-only deployment-eligibility predicate with source, build run, qualifying CI run, and exact compatible Buzz chart version. - Keep failed or pending builds untagged and without an eligibility predicate; preview-package artifacts are excluded by package and signer contract. - Add optional `image.digest` support to Buzz chart `0.1.8`, retaining the existing tag fallback. - Document verification and the intentional boundary between image/chart compatibility and backwards-compatible database migrations. ## Risk and rollout Canonical tag publication now waits for same-SHA CI and fails closed when the newest matching run fails. Per-architecture blobs may remain untagged after a failed qualification; they are not promotion inputs. Debug manifests are also CI-gated but do not receive the release eligibility predicate. There are no migration or schema-orchestration changes. The companion BuilderBot infrastructure PR will pin a separately verified canonical main digest and expose a read-only running-deployment inspection command. Signed eligibility is available only for artifacts published after this workflow lands. ## Verification - Blox: workflow contract fixtures, including stale-green rejection and predicate structure - Blox: `actionlint` for `ci.yml` and `docker.yml` - Blox: 46 Helm unit tests, chart lint, and production/quickstart render fixtures - Blox: relay status unit test, workspace Rust formatting, and workspace all-target clippy - Live evidence: exact-SHA CI selector chose successful run `32857345153` for source `9d1e4b257657f382d3111ce748f3da8d063b7671` - Existing canonical digest provenance verified with `gh attestation verify` ## References - #6709 — developer-preview precursor; intentionally separate from canonical eligibility - `docs/deployment-identity.md` *Generated with Codex* --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> |