jerry/buzz - buzz - Gitea: Git with a cup of tea

jerry/buzz

Fork 0

mirror of https://github.com/block/buzz.git synced 2026-10-29 13:31:44 +00:00

Commit Graph

Author	SHA1	Message	Date
Luke Tornquist	b37e477211	Make relay readiness process-local (#7341 ) ## Summary Make Kubernetes readiness depend only on the relay process lifecycle, so a shared Postgres or Redis slowdown cannot withdraw every pod at once. Move dependency checks to one fixed-cadence loop per pod. The loop permits only one evaluation at a time, emits metrics after each sample, and caches a timestamped report. `/_status` reads that cache without calling Postgres, Redis, or the deletion catalog. It reports `not_yet_sampled`, `fresh`, or `stale`. Each completed sample also writes a Unix timestamp gauge, so Datadog can compute sample age at query time. Fail closed when the community lifecycle lookup errors. An inconclusive tenant-state lookup no longer admits a socket to AUTH or REQ handling. ### Related issue None found. This addresses the elevated relay HTTP 500 and reconnect incident investigated on 2026-09-03. ### Tradeoff The readiness gauge now reports the latest private-probe observation, not the shutdown transition itself. After shutdown starts, a scrape can still see `1` until the next readiness probe refreshes the gauge to `0`. Kubernetes routing is unaffected because it uses the readiness response from the authoritative process state. A terminal `0` was never guaranteed: even a transition-owned gauge could update in process and then exit before Prometheus scraped it. If no scrape occurs before exit, both designs miss the terminal sample. The new design adds at most the interval until the next readiness probe when a scrape occurs during shutdown. Dependency status is now sampled every 30 seconds rather than on demand. A slow evaluation delays the next sample instead of stacking more work. The status payload exposes the cached report age. The completion timestamp stays unchanged when sampling stops, so `time() - timestamp` crosses the stale threshold without a server aging loop. Outcome counters and duration histograms remain cumulative and cannot provide no-data detection. ### Testing Gimli exercised exact SHA `edff1ec4a5362e132e367bf7449d2b0a68f1e9a9` on an isolated Blox stack. Repeated `/_status` bursts caused no dependency calls, metrics advanced without status traffic, stale status remained visible, and lifecycle lookup failures admitted no AUTH or REQ handling. Follow-up verification at `fdbb394ed598c94cde85ed01cb11e9de0ee8f00d` confirmed sampler-owned outcome and duration emission without status traffic and bound the real startup seam. The final timestamp-gauge change is covered by targeted red/green tests and the same boot regression at `c0f36e22b527c2fd6a63a548d036d006ffa17171`. ### Complexity Readiness remains one process-local sample per probe. Dependency I/O now has one owner and one execution path; request volume cannot increase dependency-check load. One completion timestamp replaces the prior server-side age update path; Datadog computes age without another loop. Community admission also returns to one fail-closed rule for inactive and unknown tenant state. Generated with [Claude Code](https://claude.ai/code) --------- Signed-off-by: tornquist <tornquist@squareup.com> Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Amp <amp@ampcode.com>	2026-09-28 14:02:12 -04:00
ravarora2	891208914e	Add authenticated WebSocket recovery telemetry (#7546 ) ## Summary Argo rollout recovery needs to distinguish an open WebSocket from a connection that completed NIP-42 authentication and is usable. The relay currently exposes total WebSocket connections, but it cannot answer whether authenticated clients recovered after a pod drain or why authentication failed. This change adds a bounded recovery contract: - `buzz_auth_attempts_total{method="nip42"}` - `buzz_auth_outcomes_total{method="nip42",outcome}` - `buzz_auth_duration_seconds{method="nip42",outcome}` - `buzz_ws_authenticated_connections_active` - `buzz_auth_post_terminal_frames_total{state}` `buzz_auth_attempts_total` now has one consistent recovery-oriented unit: a challenge lifecycle successfully queued to the connection writer. This intentionally replaces the historical unit of AUTH frames that reached the pending handler. AUTH frames received after a lifecycle is already authenticated or failed are protocol noise; they use the separate bounded post-terminal counter and cannot inflate rollout-gating attempts or outcomes. Each issued challenge reaches exactly one terminal outcome. Authentication transitions and active-gauge changes share one short synchronous state lock, cancellation can claim a pending database-backed AUTH before a late handler result, and a drop guard reconciles aborted or panicked connection futures. Accounting is terminalized before writer joins, while terminal socket delivery has a one-second best-effort bound so a stalled sink cannot retain the authenticated gauge or connection permit indefinitely. Dependency failures are distinct from policy verdicts: allowlist and relay-membership lookup errors now emit `allowlist_check_error` and `relay_membership_check_error`, never `allowlist_denied` or `not_relay_member`. The fail-closed allow/deny policy is unchanged. All rollout-facing labels come from fixed enums; identities, challenges, URLs, and raw error strings are never exported. All fixed counter and histogram label sets, including zero histogram buckets, counts, and sums, are present on the first real exporter scrape. This is the first implementation slice of the rollout-reliability plan. A later PR will consume these metrics from opt-in Argo Rollout analysis; merging this PR alone does not change deployment behavior. ### Related issue None found. ### Testing Validated at final head `f9992cac835284c25ee0f99e17ddba0b932d7c69`, which contains current `origin/main`: - `cargo fmt --all -- --check` - `cargo clippy -p buzz-relay --all-targets -- -D warnings` - compiled every `buzz-relay` test target - release builds for `buzz-relay`, `buzz`, `buzz-admin`, and `buzz-test-cli` - 16/16 connection lifecycle/writer/production-dispatch tests - 7/7 AUTH handler tests - bounded metric-contract exporter test - 13/13 relay main-target tests (1 PostgreSQL-only test ignored) - 9/9 real boot-lifecycle tests, including a production install and first Prometheus scrape Mutation checks proved that the production regressions fail when any of these are removed or reintroduced: - challenge attempt start in `handle_active_connection` - authentication series initialization in the real metric installer - separation of post-terminal AUTH floods from authoritative attempts The exact-head release binaries were also exercised against the isolated review database: 1. Started the release relay and verified health/readiness. 2. Created a channel with the regular CLI. 3. Held one authenticated WebSocket subscription open and authenticated a second WebSocket publisher. 4. Published a message and read it back through the regular CLI. 5. Observed `attempt_delta=2`, `success_delta=2`, `active_while_subscribed=1`, and `active_after_disconnect=0`. 6. Sent SIGTERM and waited for graceful relay shutdown. ### Local caveat The full relay library target consistently reached 1,049 passes with 89 intentionally ignored tests, but the untouched `api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo` intermittently returned HTTP 504 instead of 200. One isolated rerun passed; later isolated and full reruns reproduced the 504. No mesh files are changed by this PR, and all changed-path, main-target, boot-lifecycle, lint, build, mutation, and live-local checks above passed at the final head. Generated with Codex --------- Signed-off-by: Ravneet Arora <rarora@squareup.com>	2026-09-16 16:06:33 -04:00
ravarora2	88687876f7	feat(relay): add early startup lifecycle logs (#7258 ) ## Why Early relay failures can currently appear as a container restart without a trustworthy in-process account of whether crypto, structured logging, configuration, relay identity, or the metrics listener failed. Most of those steps happen before the Prometheus exporter exists, so their chronology belongs in logs rather than metrics. Implements the logs-only early-startup slice of #7238. Post-bind Prometheus exporter supervision is tracked separately in #7284. ## What changed - create a process lifecycle recorder before the Tokio runtime and emit a fixed, versioned JSON schema directly to stderr; - record started and exactly one terminal event for `crypto_init`, `tracing_init`, `config_load`, `key_load`, `metrics_bind`, and the aggregate `process_telemetry` phase; - keep every status and reason bounded and suppress raw errors that could contain credentials, keys, URLs, or other secrets; - return typed metrics-install errors so `metrics_bind` can be classified without logging raw values, while preserving the existing public `metrics::install` API; - document the logs-only evidence contract and add real child-process regressions for success and failure paths. This PR adds no startup metric families and no dashboard contract. Existing application metrics remain unchanged. ## Verification Exact head: `8faf7526822a119efa035e58b2b3c59aa67fc81d` - `cargo fmt --all -- --check` - `cargo clippy -p buzz-relay --all-targets -- -D warnings` - relay binary target: 13 passed, 1 PostgreSQL-only test ignored - real relay child-process lifecycle target: 9 passed - full relay package library target: 1,023 passed, 89 ignored; the same six media tests failed at `crates/buzz-relay/src/api/media.rs:1145` with `Sqlx(PoolTimedOut)` because local PostgreSQL is unavailable - three independent exact-head reviews found no correctness, security, compatibility, lifecycle-accounting, logs-only-scope, or test-adequacy finding All exact-head GitHub CI gates are green, including lint, unit tests, PostgreSQL, relay/backend/desktop integration, both Linux server cross-compiles, Windows/macOS builds, and security checks. ## Staging verification - exact multi-architecture image: `dev-sha-8faf7526822a119efa035e58b2b3c59aa67fc81d-run-33708188952-1` - immutable manifest: `sha256:26cad28266a6bb0b0e7081eb6091d374e5489f8bb78c475a4a65737dee86cc67` - image workflow: https://github.com/block/buzz/actions/runs/33708188952 - focused staging deployment: https://github.com/squareup/builderbot-platform-core-infrastructure/pull/314 - replacement ReplicaSet `buzz-d68764bc7` has two Ready pods with zero restarts - Datadog received one complete, contiguous sequence 1-12 from each pod; both end with `process_telemetry/terminal/succeeded` at 3 ms - queries scoped to the replacement ReplicaSet return no data for the removed `buzz_startup_phase_terminal` or `buzz_startup_phase_duration_seconds` families The experimental Row 7 was removed from the Buzz Startup & Rollout Safety dashboard. This logs-only PR deliberately adds no replacement dashboard row. --- Update Sep 3, 12:26 ET: Clarified the review boundary: this PR does not close the broader #7238. Later exporter-task termination is pre-existing runtime behavior and is now explicitly tracked in #7284; no production code or staged image changed in this update. Generated with Codex Signed-off-by: Ravneet Arora <rarora@squareup.com>	2026-09-03 13:37:51 -04:00

Author

SHA1

Message

Date

Luke Tornquist

b37e477211

Make relay readiness process-local (#7341 )

## Summary

Make Kubernetes readiness depend only on the relay process lifecycle, so
a shared Postgres or Redis slowdown cannot withdraw every pod at once.

Move dependency checks to one fixed-cadence loop per pod. The loop
permits only one evaluation at a time, emits metrics after each sample,
and caches a timestamped report. `/_status` reads that cache without
calling Postgres, Redis, or the deletion catalog. It reports
`not_yet_sampled`, `fresh`, or `stale`. Each completed sample also
writes a Unix timestamp gauge, so Datadog can compute sample age at
query time.

Fail closed when the community lifecycle lookup errors. An inconclusive
tenant-state lookup no longer admits a socket to AUTH or REQ handling.

### Related issue

None found. This addresses the elevated relay HTTP 500 and reconnect
incident investigated on 2026-09-03.

### Tradeoff

The readiness gauge now reports the latest private-probe observation,
not the shutdown transition itself. After shutdown starts, a scrape can
still see `1` until the next readiness probe refreshes the gauge to `0`.
Kubernetes routing is unaffected because it uses the readiness response
from the authoritative process state.

A terminal `0` was never guaranteed: even a transition-owned gauge could
update in process and then exit before Prometheus scraped it. If no
scrape occurs before exit, both designs miss the terminal sample. The
new design adds at most the interval until the next readiness probe when
a scrape occurs during shutdown.

Dependency status is now sampled every 30 seconds rather than on demand.
A slow evaluation delays the next sample instead of stacking more work.
The status payload exposes the cached report age. The completion
timestamp stays unchanged when sampling stops, so `time() - timestamp`
crosses the stale threshold without a server aging loop. Outcome
counters and duration histograms remain cumulative and cannot provide
no-data detection.

### Testing

Gimli exercised exact SHA `edff1ec4a5362e132e367bf7449d2b0a68f1e9a9` on
an isolated Blox stack. Repeated `/_status` bursts caused no dependency
calls, metrics advanced without status traffic, stale status remained
visible, and lifecycle lookup failures admitted no AUTH or REQ handling.
Follow-up verification at `fdbb394ed598c94cde85ed01cb11e9de0ee8f00d`
confirmed sampler-owned outcome and duration emission without status
traffic and bound the real startup seam. The final timestamp-gauge
change is covered by targeted red/green tests and the same boot
regression at `c0f36e22b527c2fd6a63a548d036d006ffa17171`.

### Complexity

Readiness remains one process-local sample per probe. Dependency I/O now
has one owner and one execution path; request volume cannot increase
dependency-check load. One completion timestamp replaces the prior
server-side age update path; Datadog computes age without another loop.
Community admission also returns to one fail-closed rule for inactive
and unknown tenant state.

Generated with [Claude Code](https://claude.ai/code)

---------

Signed-off-by: tornquist <tornquist@squareup.com>
Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Amp <amp@ampcode.com>

2026-09-28 14:02:12 -04:00

ravarora2

891208914e

Add authenticated WebSocket recovery telemetry (#7546 )

## Summary

Argo rollout recovery needs to distinguish an open WebSocket from a
connection that completed NIP-42 authentication and is usable. The relay
currently exposes total WebSocket connections, but it cannot answer
whether authenticated clients recovered after a pod drain or why
authentication failed.

This change adds a bounded recovery contract:

- `buzz_auth_attempts_total{method="nip42"}`
- `buzz_auth_outcomes_total{method="nip42",outcome}`
- `buzz_auth_duration_seconds{method="nip42",outcome}`
- `buzz_ws_authenticated_connections_active`
- `buzz_auth_post_terminal_frames_total{state}`

`buzz_auth_attempts_total` now has one consistent recovery-oriented
unit: a challenge lifecycle successfully queued to the connection
writer. This intentionally replaces the historical unit of AUTH frames
that reached the pending handler. AUTH frames received after a lifecycle
is already authenticated or failed are protocol noise; they use the
separate bounded post-terminal counter and cannot inflate rollout-gating
attempts or outcomes.

Each issued challenge reaches exactly one terminal outcome.
Authentication transitions and active-gauge changes share one short
synchronous state lock, cancellation can claim a pending database-backed
AUTH before a late handler result, and a drop guard reconciles aborted
or panicked connection futures. Accounting is terminalized before writer
joins, while terminal socket delivery has a one-second best-effort bound
so a stalled sink cannot retain the authenticated gauge or connection
permit indefinitely.

Dependency failures are distinct from policy verdicts: allowlist and
relay-membership lookup errors now emit `allowlist_check_error` and
`relay_membership_check_error`, never `allowlist_denied` or
`not_relay_member`. The fail-closed allow/deny policy is unchanged. All
rollout-facing labels come from fixed enums; identities, challenges,
URLs, and raw error strings are never exported.

All fixed counter and histogram label sets, including zero histogram
buckets, counts, and sums, are present on the first real exporter
scrape. This is the first implementation slice of the
rollout-reliability plan. A later PR will consume these metrics from
opt-in Argo Rollout analysis; merging this PR alone does not change
deployment behavior.

### Related issue

None found.

### Testing

Validated at final head `f9992cac835284c25ee0f99e17ddba0b932d7c69`,
which contains current `origin/main`:

- `cargo fmt --all -- --check`
- `cargo clippy -p buzz-relay --all-targets -- -D warnings`
- compiled every `buzz-relay` test target
- release builds for `buzz-relay`, `buzz`, `buzz-admin`, and
`buzz-test-cli`
- 16/16 connection lifecycle/writer/production-dispatch tests
- 7/7 AUTH handler tests
- bounded metric-contract exporter test
- 13/13 relay main-target tests (1 PostgreSQL-only test ignored)
- 9/9 real boot-lifecycle tests, including a production install and
first Prometheus scrape

Mutation checks proved that the production regressions fail when any of
these are removed or reintroduced:

- challenge attempt start in `handle_active_connection`
- authentication series initialization in the real metric installer
- separation of post-terminal AUTH floods from authoritative attempts

The exact-head release binaries were also exercised against the isolated
review database:

1. Started the release relay and verified health/readiness.
2. Created a channel with the regular CLI.
3. Held one authenticated WebSocket subscription open and authenticated
a second WebSocket publisher.
4. Published a message and read it back through the regular CLI.
5. Observed `attempt_delta=2`, `success_delta=2`,
`active_while_subscribed=1`, and `active_after_disconnect=0`.
6. Sent SIGTERM and waited for graceful relay shutdown.

### Local caveat

The full relay library target consistently reached 1,049 passes with 89
intentionally ignored tests, but the untouched
`api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo`
intermittently returned HTTP 504 instead of 200. One isolated rerun
passed; later isolated and full reruns reproduced the 504. No mesh files
are changed by this PR, and all changed-path, main-target,
boot-lifecycle, lint, build, mutation, and live-local checks above
passed at the final head.

Generated with Codex

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>

2026-09-16 16:06:33 -04:00

ravarora2

88687876f7

feat(relay): add early startup lifecycle logs (#7258 )

## Why

Early relay failures can currently appear as a container restart without
a trustworthy in-process account of whether crypto, structured logging,
configuration, relay identity, or the metrics listener failed. Most of
those steps happen before the Prometheus exporter exists, so their
chronology belongs in logs rather than metrics.

Implements the logs-only early-startup slice of #7238. Post-bind
Prometheus exporter supervision is tracked separately in #7284.

## What changed

- create a process lifecycle recorder before the Tokio runtime and emit
a fixed, versioned JSON schema directly to stderr;
- record started and exactly one terminal event for `crypto_init`,
`tracing_init`, `config_load`, `key_load`, `metrics_bind`, and the
aggregate `process_telemetry` phase;
- keep every status and reason bounded and suppress raw errors that
could contain credentials, keys, URLs, or other secrets;
- return typed metrics-install errors so `metrics_bind` can be
classified without logging raw values, while preserving the existing
public `metrics::install` API;
- document the logs-only evidence contract and add real child-process
regressions for success and failure paths.

This PR adds **no startup metric families** and no dashboard contract.
Existing application metrics remain unchanged.

## Verification

Exact head: `8faf7526822a119efa035e58b2b3c59aa67fc81d`

- `cargo fmt --all -- --check`
- `cargo clippy -p buzz-relay --all-targets -- -D warnings`
- relay binary target: 13 passed, 1 PostgreSQL-only test ignored
- real relay child-process lifecycle target: 9 passed
- full relay package library target: 1,023 passed, 89 ignored; the same
six media tests failed at `crates/buzz-relay/src/api/media.rs:1145` with
`Sqlx(PoolTimedOut)` because local PostgreSQL is unavailable
- three independent exact-head reviews found no correctness, security,
compatibility, lifecycle-accounting, logs-only-scope, or test-adequacy
finding

All exact-head GitHub CI gates are green, including lint, unit tests,
PostgreSQL, relay/backend/desktop integration, both Linux server
cross-compiles, Windows/macOS builds, and security checks.

## Staging verification

- exact multi-architecture image:
`dev-sha-8faf7526822a119efa035e58b2b3c59aa67fc81d-run-33708188952-1`
- immutable manifest:
`sha256:26cad28266a6bb0b0e7081eb6091d374e5489f8bb78c475a4a65737dee86cc67`
- image workflow: https://github.com/block/buzz/actions/runs/33708188952
- focused staging deployment:
https://github.com/squareup/builderbot-platform-core-infrastructure/pull/314
- replacement ReplicaSet `buzz-d68764bc7` has two Ready pods with zero
restarts
- Datadog received one complete, contiguous sequence 1-12 from each pod;
both end with `process_telemetry/terminal/succeeded` at 3 ms
- queries scoped to the replacement ReplicaSet return no data for the
removed `buzz_startup_phase_terminal` or
`buzz_startup_phase_duration_seconds` families

The experimental Row 7 was removed from the Buzz Startup & Rollout
Safety dashboard. This logs-only PR deliberately adds no replacement
dashboard row.

---

**Update Sep 3, 12:26 ET:** Clarified the review boundary: this PR does
not close the broader #7238. Later exporter-task termination is
pre-existing runtime behavior and is now explicitly tracked in #7284; no
production code or staged image changed in this update.

Generated with Codex

Signed-off-by: Ravneet Arora <rarora@squareup.com>

2026-09-03 13:37:51 -04:00

3 Commits