mirror of
https://github.com/block/buzz.git
synced 2026-11-06 17:31:19 +00:00
e097c42996
## Summary
Relay pods on large deployments can take over 5 minutes to open their
health port, and right now nothing shows which startup step is slow.
Startup timing ends at `metrics_bind`, and the slow work all happens
after it.
This PR times every startup step from Postgres connect through public
listener bind. It does not change startup behavior.
- Each step logs `Startup phase started` and `Startup phase finished`,
with `phase`, `elapsed_ms`, and `status` (`succeeded` / `degraded` /
`failed`, the same values `buzz_process_lifecycle` records use).
- Gauge `buzz_startup_phase_current{phase}` is `1` while a step runs and
`0` after it ends. A pod that is still starting shows exactly which step
it is on.
- Gauge `buzz_startup_phase_seconds{phase}` records each finished step's
duration.
- The startup NIP-43 membership reconcile logs `NIP-43 startup
reconciliation progress` every 5,000 communities, with `done`, `total`,
`republished`, `failed`, and `elapsed_ms`.
These are gauges rather than counters on purpose. Each value is written
once per boot, and a monotonic-counter scrape drops a series' first
sample, so a one-shot counter never shows up.
## Contract change
`deploy/charts/buzz/README.md` ("Early-startup telemetry contract") said
the early lifecycle phases are log-only. That still holds: `crypto_init`
through `metrics_bind` are unchanged and emit no metrics.
The README now also documents the post-bind steps. The guard in
`boot_lifecycle.rs` now allows exactly the two `buzz_startup_phase_*`
families. It still rejects any other startup, boot, or lifecycle family,
and it also rejects any exported label value that names an early phase.
## Design
- `crates/buzz-relay/src/startup_steps.rs` adds a closed `StartupStep`
vocabulary (18 steps) and a `StepTimer` guard.
- A step ends exactly once:
- `finish()` → `succeeded`
- `degrade()` then `finish()` → `degraded` (a non-fatal error branch)
- dropped without finishing → `failed` (covers `?` early returns and
panics)
- The NIP-43 and large-roster reconciles now return `ReconcileSummary {
repaired, failed }`. A sweep where any item fails ends its step
`degraded`, and the final log line includes the `failed` count. NIP-FI
warm ends `degraded` when any issuer did not warm. The deprecated
`reconcile_nip43_membership_snapshots` wrapper still returns `repaired`.
- Steps that are disabled by config (migrate, audit connect, NIP-FI
warm, git probe, NIP-43 reconcile) are not timed, so their series never
appear.
- The gauges are set once, so they age out after the existing gauge idle
timeout (2700 s with default intervals; logged as `idle_timeout_secs`).
A step that runs longer than that loses its `_current` series. The
README says so.
- Progress logging runs only for the `Bootstrap` reconcile. The 60 s
maintenance sweep stays quiet.
## What got simpler
Little got simpler in the code: this mostly adds instrumentation. One CI
gap closed along the way: the 54 `config` tests and the 9 non-ignored
`boot_lifecycle` tests, which ran in no CI lane, now run in `test-unit`.
The instrumentation is scoped as small as I could make it, and it is the
evidence needed for the follow-up that should delete code: moving the
per-community NIP-43 reconcile off the pre-bind path, after which the
900 s startup-probe override can go back to the chart default.
## Testing
- `startup_steps` unit tests (`DebuggingRecorder`):
- a running step is current and has no duration
- a finished step clears current and records its elapsed time
- dropped and degraded steps clear current and record a duration
- log capture: `finish`, `degrade`+`finish`, drop, and panic each emit
exactly one `Startup phase finished` record with the right `status`.
Mutating `degrade()` to a no-op and removing the ends-once guard makes
these tests fail.
- the vocabulary is unique and shares no names with the early lifecycle
phases
- Tests that capture logs or check callsite interest now share
`test_support::tracing_dispatch_lock()`. These are the `startup_steps`
helpers, two `telemetry` tests, and the `config` and `bridge`
log-capture helpers. tracing-core caches each callsite's interest across
the whole process. Without the lock, the new panic test lost its record
in 60 of 150 parallel runs of `telemetry:: startup_steps:: config::`.
With the lock it failed 0 of 300. Removing the lock from `gauges()`
brings the failures back (53 of 100 runs).
- `nip43_progress_logs_every_interval_inside_the_startup_sweep_only`
checks the cadence at 79,942 communities (15 progress lines, none at the
start or end) and table-tests that the maintenance sweep never logs
progress.
-
`boot_lifecycle::successful_main_emits_complete_lifecycle_and_reports_current_startup_step`
boots the real relay binary against a TCP peer that never speaks
Postgres. It asserts that `/metrics` shows
`buzz_startup_phase_current{phase="db_connect"} 1` and no finished-step
durations, and that no early phase leaks into metrics.
- **CI routing.** `just test-unit` picks relay tests by hand-maintained
filter. Before this PR, CI's filters selected none of the
`startup_steps`, `telemetry`, `config` or non-ignored `boot_lifecycle`
tests above; the NIP-43 cadence test is picked up by the existing
`handlers::side_effects::tests::` clause. Its infra-free relay stanza
now selects `startup_steps::tests::`, `telemetry::tests::` and
`config::tests::` (anchored so `nip_fi_config` is not selected twice),
and a new `cargo nextest run -p buzz-relay --test boot_lifecycle` line
runs the non-ignored `boot_lifecycle` tests (they spawn the real binary
and need no Postgres or Redis; the Postgres cases stay `#[ignore]`d).
`scripts/run-tests.sh` mirrors both, running `telemetry` and `config`
single-threaded because they mutate process env.
### Manual check
1. `just relay` (or run `buzz-relay` against local Postgres and Redis).
2. Once the relay is up: `curl -s localhost:9102/metrics | grep
buzz_startup_phase`. Every step that ran has a `_seconds` value, and
every `_current` value is 0.
3. In the relay logs, `"Startup phase finished"` lines show each step's
`elapsed_ms`.
---------
Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
88 lines
3.1 KiB
Rust
88 lines
3.1 KiB
Rust
#![deny(unsafe_code)]
|
|
#![warn(missing_docs)]
|
|
//! NIP-01 WebSocket relay for Buzz private team communication.
|
|
|
|
mod admission;
|
|
mod build_info;
|
|
mod rejection;
|
|
|
|
/// Shared NIP-FI assertion evaluation, denial rendering, and key pairing.
|
|
pub(crate) mod nip_fi_core;
|
|
/// NIP-FI session admission gate — per-connection effect-permit and quiescence barrier.
|
|
pub(crate) mod nip_fi_gate;
|
|
pub(crate) mod nip_fi_session;
|
|
pub(crate) mod nip_fi_shadow;
|
|
pub(crate) mod nip_fi_shadow_session;
|
|
/// NIP-FI test hooks — production barriers for deterministic B1/B2 witnesses.
|
|
#[cfg(test)]
|
|
pub(crate) mod nip_fi_test_hooks;
|
|
/// NIP-FI assertion validation at WebSocket upgrade.
|
|
pub(crate) mod nip_fi_upgrade;
|
|
|
|
/// REST API route handlers.
|
|
pub mod api;
|
|
/// WebSocket audio relay for huddle voice channels.
|
|
pub mod audio;
|
|
/// Relay configuration from environment variables.
|
|
pub mod config;
|
|
/// Runtime conformance harness — abstract trace emission at the
|
|
/// ingest/read accept-reject boundary, replayed against
|
|
/// `docs/spec/MultiTenantRelay.tla` by the independent `buzz-conformance`
|
|
/// checker.
|
|
pub mod conformance;
|
|
/// WebSocket connection lifecycle and state.
|
|
pub mod connection;
|
|
/// Relay error types.
|
|
pub mod error;
|
|
/// WebSocket message handlers for NIP-01 client commands.
|
|
pub mod handlers;
|
|
/// Stateless HMAC-signed relay invite tokens (mint/verify).
|
|
pub mod invite_token;
|
|
/// Fixed-schema evidence for the relay's earliest startup steps.
|
|
pub mod lifecycle;
|
|
/// Inter-relay mesh startup wiring (`BUZZ_MESH` seam).
|
|
pub mod mesh_boot;
|
|
/// Prometheus metrics: recorder, upkeep, HTTP middleware.
|
|
pub mod metrics;
|
|
/// NIP-11 relay information document.
|
|
pub mod nip11;
|
|
mod nip98;
|
|
/// NIP-FI relay configuration: mode, issuer registry, JWKS warm/refresh.
|
|
pub mod nip_fi_config;
|
|
/// NIP-FI HTTP ingress enforcement: assertion extraction, verification,
|
|
/// key-pairing check, and deny-map gate for every protected HTTP surface.
|
|
pub(crate) mod nip_fi_http;
|
|
/// Deployment-global operator-listener mention delivery worker.
|
|
pub mod operator_listener;
|
|
/// NIP-01 client/relay message parsing.
|
|
pub mod protocol;
|
|
/// Durable NIP-PL matcher and delivery worker.
|
|
pub mod push_runtime;
|
|
/// Readiness-probe telemetry and the per-pod dependency sampler behind `/_status`.
|
|
pub mod readiness;
|
|
/// Axum router construction.
|
|
pub mod router;
|
|
/// Timing for startup work between metrics bind and serving.
|
|
pub mod startup_steps;
|
|
/// Shared application state.
|
|
pub mod state;
|
|
pub mod storage_sweep;
|
|
/// Subscription registry with (channel, kind) fan-out index.
|
|
pub mod subscription;
|
|
/// OpenTelemetry tracing initialisation (tracer provider + OTLP exporter).
|
|
pub mod telemetry;
|
|
/// Row-zero host binding: resolve the request community from the connection host.
|
|
pub mod tenant;
|
|
#[cfg(test)]
|
|
mod test_support;
|
|
/// Relay-side tunnel session directory and routing.
|
|
pub mod tunnel;
|
|
/// Webhook secret generation and constant-time comparison.
|
|
pub mod webhook_secret;
|
|
/// Workflow action sink — relay-side implementation of [`buzz_workflow::ActionSink`].
|
|
pub mod workflow_sink;
|
|
|
|
pub use config::Config;
|
|
pub use error::{RelayError, Result};
|
|
pub use state::AppState;
|