feat(relay): add early startup lifecycle logs (#7258)

## Why

Early relay failures can currently appear as a container restart without
a trustworthy in-process account of whether crypto, structured logging,
configuration, relay identity, or the metrics listener failed. Most of
those steps happen before the Prometheus exporter exists, so their
chronology belongs in logs rather than metrics.

Implements the logs-only early-startup slice of #7238. Post-bind
Prometheus exporter supervision is tracked separately in #7284.

## What changed

- create a process lifecycle recorder before the Tokio runtime and emit
a fixed, versioned JSON schema directly to stderr;
- record started and exactly one terminal event for `crypto_init`,
`tracing_init`, `config_load`, `key_load`, `metrics_bind`, and the
aggregate `process_telemetry` phase;
- keep every status and reason bounded and suppress raw errors that
could contain credentials, keys, URLs, or other secrets;
- return typed metrics-install errors so `metrics_bind` can be
classified without logging raw values, while preserving the existing
public `metrics::install` API;
- document the logs-only evidence contract and add real child-process
regressions for success and failure paths.

This PR adds **no startup metric families** and no dashboard contract.
Existing application metrics remain unchanged.

## Verification

Exact head: `8faf7526822a119efa035e58b2b3c59aa67fc81d`

- `cargo fmt --all -- --check`
- `cargo clippy -p buzz-relay --all-targets -- -D warnings`
- relay binary target: 13 passed, 1 PostgreSQL-only test ignored
- real relay child-process lifecycle target: 9 passed
- full relay package library target: 1,023 passed, 89 ignored; the same
six media tests failed at `crates/buzz-relay/src/api/media.rs:1145` with
`Sqlx(PoolTimedOut)` because local PostgreSQL is unavailable
- three independent exact-head reviews found no correctness, security,
compatibility, lifecycle-accounting, logs-only-scope, or test-adequacy
finding

All exact-head GitHub CI gates are green, including lint, unit tests,
PostgreSQL, relay/backend/desktop integration, both Linux server
cross-compiles, Windows/macOS builds, and security checks.

## Staging verification

- exact multi-architecture image:
`dev-sha-8faf7526822a119efa035e58b2b3c59aa67fc81d-run-33708188952-1`
- immutable manifest:
`sha256:26cad28266a6bb0b0e7081eb6091d374e5489f8bb78c475a4a65737dee86cc67`
- image workflow: https://github.com/block/buzz/actions/runs/33708188952
- focused staging deployment:
https://github.com/squareup/builderbot-platform-core-infrastructure/pull/314
- replacement ReplicaSet `buzz-d68764bc7` has two Ready pods with zero
restarts
- Datadog received one complete, contiguous sequence 1-12 from each pod;
both end with `process_telemetry/terminal/succeeded` at 3 ms
- queries scoped to the replacement ReplicaSet return no data for the
removed `buzz_startup_phase_terminal` or
`buzz_startup_phase_duration_seconds` families

The experimental Row 7 was removed from the Buzz Startup & Rollout
Safety dashboard. This logs-only PR deliberately adds no replacement
dashboard row.

---

**Update Sep 3, 12:26 ET:** Clarified the review boundary: this PR does
not close the broader #7238. Later exporter-task termination is
pre-existing runtime behavior and is now explicitly tracked in #7284; no
production code or staged image changed in this update.

Generated with Codex

Signed-off-by: Ravneet Arora <rarora@squareup.com>
This commit is contained in:
ravarora2
2026-09-03 13:37:51 -04:00
committed by GitHub
parent 752cbfc037
commit 88687876f7
9 changed files with 1320 additions and 25 deletions

View File

@@ -115,6 +115,16 @@ disables that probe through `relay.extraEnv`, `/_readiness` does not test object
storage; configuration is still parsed strictly, but reachability and addressing
errors surface on the first storage operation.
### Early-startup telemetry contract
`buzz_process_lifecycle` JSON records are the authoritative history for the
fixed phases `crypto_init`, `tracing_init`, `config_load`, `key_load`, and
`metrics_bind`, plus the aggregate `process_telemetry` result. They use bounded
status/reason values and never contain raw configuration, keys, URLs, or errors.
These phases intentionally do not emit metrics. Most run before the Prometheus
exporter exists, and one uniform log-only contract preserves every phase's real
event time and failure without assigning an eventual scrape time to earlier work.
### Readiness telemetry contract
Only requests served by the private health listener (`BUZZ_HEALTH_PORT`) emit