Files
buzz/docs/deployment-identity.md
Luke Tornquist 5a93e862ee fix(chart): own the Datadog version label on every workload Pod (#7876)
## Summary

Every chart-managed Pod now carries a chart-owned
`tags.datadoghq.com/version` label derived from the deployed image,
instead of a value each wrapper restates once per workload.

Covered Pod templates:
- relay Deployment (`templates/deployment.yaml`)
- pairing relay (`templates/pairing-relay.yaml`)
- storage-accounting CronJob
(`templates/storage-accounting-cronjob.yaml`)
- operator jobs, including the deletion drain
(`templates/_operator-jobs.tpl`)

**Source order:** `image.tag`, then `image.digest`, then
`Chart.AppVersion` (`buzz.imageVersionSource`). Tag-first is deliberate:
- The rendered value matches the `sha-<commit>` pins wrappers already
carry, so upgrading to this chart does not roll any Pods.
- An image promotion only has to write `image.tag` + `image.digest`. No
per-workload label edit is needed.

Exact runtime identity stays exact in two places. The image reference is
`repository@digest` when a digest is pinned.
`BUZZ_STORAGE_SNAPSHOT_CODE_SHA` (persisted as `code_sha`) is still
digest-first.

**Label safety:** `buzz.imageVersionLabel` maps any OCI tag or digest to
a valid label value.
- Valid values already in Datadog's normal form (lowercase, no `__`)
pass through byte for byte, so `sha-<hex>` and `1.2.3-rc.4` are
unchanged. Datadog lowercases tag values and collapses `__`, so any
other value is hashed rather than allowed to report the same version as
a different revision.
- A digest keeps 63 hex characters.
- Anything else is replaced by the first 63 hex characters of its
SHA-256.
- Exactly 63 lowercase hex characters is reserved for the hashed and
digest forms, so a rendered label cannot be fed back as a tag to make
two revisions collide.
- When both tag and digest are set, the label names the tag even if they
disagree; the digest still pins what runs.

**Precedence:** a `tags.datadoghq.com/version` supplied through any
workload's `podLabels` is ignored. Every other label, including
`tags.datadoghq.com/env` and `/service`, passes through unchanged.
Overriding instead of failing means a wrapper with a stale pin is
corrected by the upgrade itself. The chart bump and the wrapper cleanup
don't have to land together.

**What got simpler:** one image identity instead of N. Wrappers delete
their per-workload version pins. Image promotion tooling no longer needs
a "mirror the tag into a label path" feature. One shared helper
(`buzz.podLabelsWithVersion`) replaces four `podLabels` passthroughs.

Chart `0.1.10 → 0.1.11`, with an `artifacthub.io/changes` entry.
Documented in the chart README ("Datadog version label") and
`docs/deployment-identity.md`.

Note: a vendor-namespaced label now renders unconditionally in an OSS
chart. This is in tension with
[VISION_SOVEREIGN.md](../blob/main/VISION_SOVEREIGN.md). The label is
inert without the operator-supplied `tags.datadoghq.com/service`/`env`,
and the chart already ships vendor-specific defaults (e.g.
`sidecar.istio.io/inject`).

### Testing

Run on a Blox (Linux) workstation at tree `41634b7ca` (helm v3.16.4,
helm-unittest 0.8.2, the CI versions):
- `helm lint deploy/charts/buzz`: 0 failed
- `helm unittest deploy/charts/buzz`: 12 suites, **85/85 passed**

New `tests/version_label_test.yaml` checks the relay, pairing relay, and
deletion drain: tag wins over a stale `podLabels` pin, the label is
stamped with no `podLabels`, and digest-only images get a label-safe
value. `tests/storage_accounting_test.yaml` covers the same cases for
the CronJob, plus the `Chart.AppVersion` fallback.

Mutation checks:
- Reverting any one template to the plain `podLabels` passthrough fails
tests.
- Switching the source order to digest-first fails tests.
- Restoring the case-insensitive passthrough fails the three
Datadog-normalization tests (uppercase, `__`, uppercased 63-hex).

No Rust, desktop, web, or mobile sources changed.

## Linear tracking

Related work: [BUZZ-215](https://linear.app/squareup/issue/BUZZ-215).

---------

Signed-off-by: tornquist <tornquist@squareup.com>
Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
2026-10-08 15:12:00 -04:00

5.1 KiB

Relay deployment identity

Canonical relay images from ghcr.io/block/buzz carry two signed attestations:

  • SLSA build provenance maps the immutable image digest to the source commit and Docker workflow run.
  • The Buzz deployment-eligibility predicate records the successful same-SHA CI run and the exact Buzz Helm chart version from that source commit.

The Docker workflow creates tagged multi-architecture manifests only after the same full source SHA has a successful CI push run on main or release. Architecture-specific build manifests may exist without tags while CI is running or after it fails; they do not receive the deployment-eligibility predicate and are not promotion inputs.

Verify a canonical eligible digest with:

gh attestation verify \
  oci://ghcr.io/block/buzz@sha256:<digest> \
  --repo block/buzz \
  --signer-workflow block/buzz/.github/workflows/docker.yml \
  --predicate-type https://buzz.block.xyz/attestations/deployment-eligibility/v1 \
  --source-ref refs/heads/main

The predicate's helm_chart.compatible_version is image-to-chart metadata. It does not describe database schema compatibility and does not relax Buzz's rule that migrations remain backwards compatible.

The manual pre-merge workflow publishes only to ghcr.io/block/buzz-staging-dev. Those preview images are intentionally ineligible: they use a different package, may name non-main source, and do not receive the canonical deployment-eligibility predicate.

Runtime inspection

The relay health listener exposes intrinsic build identity at /_status:

{
  "service": "buzz-relay",
  "version": "0.2.1",
  "uptime_seconds": 123,
  "build": {
    "source_sha": "<40-character-source-sha>",
    "id": "github-actions:<run-id>:<attempt>",
    "url": "https://github.com/block/buzz/actions/runs/<run-id>/attempts/<attempt>"
  },
  "dependencies": {
    "sample": "fresh",
    "sample_interval_seconds": 30,
    "sample_age_seconds": 12,
    "postgres": true,
    "redis": true,
    "deletion_catalog": true,
    "reason": "ready"
  }
}

Non-CI builds report stable unknown or local fallback values instead of claiming provenance they do not have.

dependencies is a cached diagnostic snapshot of shared-dependency health. A per-pod background loop evaluates the dependencies every 30 seconds; the endpoint only reads the latest report and never contacts a dependency itself, so polling it costs nothing. sample is always present and reports whether that cached verdict is fresh, stale, or not_yet_sampled — before the first cycle completes the health fields are absent rather than defaulted. /_readiness does not consult any of this — see the readiness contract — so this endpoint must never be wired to a Kubernetes probe.

Helm digest pinning

Buzz chart 0.1.8 and newer accept an immutable image digest:

image:
  repository: ghcr.io/block/buzz
  digest: sha256:<64-lowercase-hex-characters>

When image.digest is set, the chart renders repository@digest and ignores image.tag. Existing tag-only values remain backwards compatible.

Derived deployment labels

Every chart-managed Pod — the relay, the pairing relay, the storage-accounting CronJob, and operator jobs such as the deletion drain — carries a chart-owned tags.datadoghq.com/version label derived from the deployed image, so a wrapper never restates the image identity once per workload. The label names image.tag when it is set (the readable sha-<commit> build name that promotion writes together with image.digest), otherwise image.digest, otherwise Chart.AppVersion.

The exact runtime identity stays exact elsewhere: the Pod's image reference is repository@digest when a digest is pinned, and the storage-accounting BUZZ_STORAGE_SNAPSHOT_CODE_SHA (persisted as code_sha on every snapshot row) is the digest when set, otherwise the tag, otherwise Chart.AppVersion.

The label value has to be sanitized: a label value is capped at 63 bytes, must begin and end with an alphanumeric, and may otherwise contain only [-._a-zA-Z0-9], while image.tag accepts any OCI tag. Because that domain is larger than the label codomain, no mapping onto it is injective; the chart provides a deterministic, collision-resistant one. A digest keeps the first 63 characters of its hex, a revision that is already a valid label value in Datadog's normal form (lowercase, no __) is preserved byte for byte, and anything else (a leading _, an uppercase letter, a __ run, a byte outside the label alphabet, more than 63 bytes) is replaced by the first 63 hex characters of its SHA-256 — 252 retained bits, the same margin as the digest case, and no readable prefix.

Exactly 63 lowercase hex characters is reserved for those hashed and digest forms: a tag of that shape is hashed instead of preserved, so a rendered label cannot be copied into image.tag to make two revisions report one version. Shorter hex tags and 40-character git SHAs pass through unchanged.

Any tags.datadoghq.com/version supplied through a workload's podLabels is ignored; see the chart README's "Datadog version label" section.