jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-10-27 12:40:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Make relay readiness process-local (#7341)

Browse Source 
## Summary

Make Kubernetes readiness depend only on the relay process lifecycle, so
a shared Postgres or Redis slowdown cannot withdraw every pod at once.

Move dependency checks to one fixed-cadence loop per pod. The loop
permits only one evaluation at a time, emits metrics after each sample,
and caches a timestamped report. `/_status` reads that cache without
calling Postgres, Redis, or the deletion catalog. It reports
`not_yet_sampled`, `fresh`, or `stale`. Each completed sample also
writes a Unix timestamp gauge, so Datadog can compute sample age at
query time.

Fail closed when the community lifecycle lookup errors. An inconclusive
tenant-state lookup no longer admits a socket to AUTH or REQ handling.

### Related issue

None found. This addresses the elevated relay HTTP 500 and reconnect
incident investigated on 2026-09-03.

### Tradeoff

The readiness gauge now reports the latest private-probe observation,
not the shutdown transition itself. After shutdown starts, a scrape can
still see `1` until the next readiness probe refreshes the gauge to `0`.
Kubernetes routing is unaffected because it uses the readiness response
from the authoritative process state.

A terminal `0` was never guaranteed: even a transition-owned gauge could
update in process and then exit before Prometheus scraped it. If no
scrape occurs before exit, both designs miss the terminal sample. The
new design adds at most the interval until the next readiness probe when
a scrape occurs during shutdown.

Dependency status is now sampled every 30 seconds rather than on demand.
A slow evaluation delays the next sample instead of stacking more work.
The status payload exposes the cached report age. The completion
timestamp stays unchanged when sampling stops, so `time() - timestamp`
crosses the stale threshold without a server aging loop. Outcome
counters and duration histograms remain cumulative and cannot provide
no-data detection.

### Testing

Gimli exercised exact SHA `edff1ec4a5362e132e367bf7449d2b0a68f1e9a9` on
an isolated Blox stack. Repeated `/_status` bursts caused no dependency
calls, metrics advanced without status traffic, stale status remained
visible, and lifecycle lookup failures admitted no AUTH or REQ handling.
Follow-up verification at `fdbb394ed598c94cde85ed01cb11e9de0ee8f00d`
confirmed sampler-owned outcome and duration emission without status
traffic and bound the real startup seam. The final timestamp-gauge
change is covered by targeted red/green tests and the same boot
regression at `c0f36e22b527c2fd6a63a548d036d006ffa17171`.

### Complexity

Readiness remains one process-local sample per probe. Dependency I/O now
has one owner and one execution path; request volume cannot increase
dependency-check load. One completion timestamp replaces the prior
server-side age update path; Datadog computes age without another loop.
Community admission also returns to one fail-closed rule for inactive
and unknown tenant state.

Generated with [Claude Code](https://claude.ai/code)

---------

Signed-off-by: tornquist <tornquist@squareup.com>
Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Amp <amp@ampcode.com>
This commit is contained in:
Luke Tornquist
2026-09-28 14:02:12 -04:00
committed by GitHub
parent 447951a19c
commit b37e477211
13 changed files with 2434 additions and 795 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.github/workflows/_ci-relay.yml vendored
View File

@@ -198,6 +198,8 @@ jobs:
with:
name: desktop-e2e-relay
path: target/ci
- name: Restore relay executable permission
run: chmod +x ./target/ci/buzz-relay
- name: PostgreSQL-backed tests
env:
BUZZ_POSTGRES_ADMIN_URL: postgres://buzz:${{ env.BUZZ_TEST_POSTGRES_PASSWORD }}@localhost:5432/postgres

2
ARCHITECTURE.md
View File

@@ -644,7 +644,7 @@ pub enum AuthState { Pending { challenge: String }, Authenticated(AuthContext),
| GET | `/.well-known/nostr.json` | NIP-05 identity |
| GET | `/health` | Health check |
| GET | `/_liveness` | Liveness probe |
| GET | `/_readiness` | Readiness probe |
| GET | `/_readiness` | Readiness probe — local process lifecycle only |
| POST | `/events` | Submit a signed Nostr event over HTTP (same ingest path as WebSocket `EVENT`) |
| POST | `/query` | Query Nostr events over HTTP with NIP-01 filters |
| POST | `/count` | Count Nostr events over HTTP with NIP-45 filters |

2
Justfile
View File

@@ -472,7 +472,7 @@ test-unit:
# because they live in the binary target; the nested
# `tests::postgres_tests::` stays in the PostgreSQL lane.
cargo nextest run -p buzz-relay --lib --bin buzz-relay \
-E 'test(/^api::admin::/) + test(/^handlers::channel_authz::/) + test(/^handlers::moderation_authz::/) + test(/^handlers::side_effects::tests::/) + test(/^storage_sweep::tests::/) + test(/^nip_fi_http::tests::/) + test(/^nip_fi_config::tests::/) + test(/^router::tests::/) + test(/^api::parse_query_tests::/) + test(/^api::git::transport::off_mode_precedence_tests::/) + (kind(bin) & (test(/^tests::/) + test(/^composition_tests::/)) - test(/^tests::postgres_tests::/))'
-E 'test(/^api::admin::/) + test(/^handlers::channel_authz::/) + test(/^handlers::moderation_authz::/) + test(/^handlers::side_effects::tests::/) + test(/^storage_sweep::tests::/) + test(/^nip_fi_http::tests::/) + test(/^nip_fi_config::tests::/) + test(/^readiness::tests::/) + test(/^router::tests::/) + test(/^api::parse_query_tests::/) + test(/^api::git::transport::off_mode_precedence_tests::/) + test(=state::tests::neither_a_confirmed_inactive_community_nor_a_failed_lookup_admits_the_socket) + (kind(bin) & (test(/^tests::/) + test(/^composition_tests::/)) - test(/^tests::postgres_tests::/))'
# ACP author-gate and queue tests protect the trust boundary between
# relay events and agent prompts. They are infra-free; ignored lifecycle
# tests remain excluded and run in their dedicated integration lanes.

3
crates/buzz-relay/src/lib.rs
View File

@@ -45,7 +45,8 @@ pub mod operator_listener;
pub mod protocol;
/// Durable NIP-PL matcher and delivery worker.
pub mod push_runtime;
mod readiness;
/// Readiness-probe telemetry and the per-pod dependency sampler behind `/_status`.
pub mod readiness;
/// Axum router construction.
pub mod router;
/// Shared application state.

25
crates/buzz-relay/src/main.rs
View File

@@ -227,6 +227,10 @@ async fn run_relay_main(boot: BootTracker) -> anyhow::Result<()> {
let usage_interval_secs = usage_metrics_interval_secs();
let usage_idle_timeout_secs = usage_metrics_idle_timeout_secs(usage_interval_secs);
let dependency_sample_completion_republish_interval =
buzz_relay::readiness::dependency_sample_completion_republish_interval(
usage_idle_timeout_secs,
);
let (boot, ()) = boot.run_required(
StartupPhase::MetricsBind,
|| relay_metrics::try_install(config.metrics_port, usage_idle_timeout_secs),
@@ -244,6 +248,7 @@ async fn run_relay_main(boot: BootTracker) -> anyhow::Result<()> {
info!(
port = config.metrics_port,
idle_timeout_secs = usage_idle_timeout_secs,
completion_republish_secs = dependency_sample_completion_republish_interval.as_secs(),
"Prometheus metrics exporter started"
);
@@ -455,7 +460,13 @@ async fn run_relay_main(boot: BootTracker) -> anyhow::Result<()> {
cfg.create_pool(Some(deadpool_redis::Runtime::Tokio1))
.map_err(|e| anyhow::anyhow!("Redis pool creation failed: {e}"))?
};
let redis_health_pool = redis_pool.clone(); // cheap Arc clone — shared with readiness handler
let redis_health_pool = redis_pool.clone(); // cheap Arc clone — shared with AppState
// One-time bootstrap gate, deliberately before AppState and therefore before
// the health listener binds. Post-start Redis failures are dependency
// failures and must never move readiness; never having connected at all is
// a broken deployment, not a blip.
buzz_relay::state::verify_redis_command_path(&redis_health_pool).await?;
info!("Redis command path connected");
let pubsub = Arc::new(
PubSubManager::new(&config.redis_url, redis_pool)
.await
@@ -1100,6 +1111,16 @@ async fn run_relay_main(boot: BootTracker) -> anyhow::Result<()> {
));
}
// Per-pod dependency diagnostics runtime: one seam starts the dependency
// sampler and its independent completion-epoch republisher together.
{
let diagnostics_state = Arc::clone(&state);
buzz_relay::readiness::start_dependency_sampler_and_completion_publisher(
diagnostics_state,
dependency_sample_completion_republish_interval,
);
}
// Cross-pod connection-control consumer: receive disconnect commands from
// Redis pub/sub (published by the pod that recorded a ban) and close any
// matching local sockets. A member's live connections may land on any pod,
@@ -1273,6 +1294,8 @@ async fn run_relay_main(boot: BootTracker) -> anyhow::Result<()> {
serve(router, health_router, Arc::clone(&state)).await?;
state.community_revalidator_cancel.cancel();
state.dependency_sampler_cancel.cancel();
state.dependency_completion_publisher_cancel.cancel();
// Signal the audit worker to stop accepting, flush buffered entries, and
// exit. Uses a CancellationToken so it works regardless of how many

41
crates/buzz-relay/src/metrics.rs
View File

@@ -290,6 +290,7 @@ pub fn try_install(port: u16, gauge_idle_timeout_secs: u64) -> Result<(), Metric
metrics::set_global_recorder(recorder)
.map_err(|_error| MetricsInstallError::RecorderConflict)?;
describe_readiness_metrics();
describe_community_admission_metrics();
describe_db_pool_metrics();
describe_auth_metrics();
initialize_auth_metric_series();
@@ -306,24 +307,42 @@ pub fn install(port: u16, gauge_idle_timeout_secs: u64) {
.unwrap_or_else(|error| panic!("metrics exporter must install exactly once: {error}"));
}
/// Register the frozen readiness metric descriptions with the active recorder.
/// Register the frozen readiness and dependency-diagnostic metric descriptions.
///
/// The two `buzz_readiness_*` probe families describe local process lifecycle.
/// The dependency families keep their names for dashboard continuity but are
/// published by the per-pod dependency sampler, not by the Kubernetes probe or
/// by an `/_status` request — a shared-dependency failure no longer deroutes
/// the pod, and nobody has to read the endpoint for the metrics to move.
pub(crate) fn describe_readiness_metrics() {
metrics::describe_counter!(
"buzz_readiness_checks_total",
"Kubernetes health-listener readiness probes by terminal bounded reason"
"Kubernetes health-listener readiness probes by lifecycle reason (ready, shutting_down)"
);
metrics::describe_counter!(
"buzz_readiness_dependency_checks_total",
"Completed readiness dependency attempts by dependency and bounded outcome"
"Completed dependency-sampler attempts by dependency and bounded outcome"
);
metrics::describe_histogram!(
"buzz_readiness_check_duration_seconds",
metrics::Unit::Seconds,
"Completed readiness check duration without outcome label multiplication"
"Completed dependency-sampler check duration without outcome label multiplication"
);
metrics::describe_gauge!(
"buzz_readiness_state",
"Latest publishable readiness state by check, where 1 is ready and 0 is not ready"
"Latest private readiness-probe observation, where 1 is ready and 0 is shutting down"
);
metrics::describe_gauge!(
"buzz_readiness_dependency_sample_completed_timestamp_seconds",
"Unix time the cached /_status dependency report completed, absent until the first sample completes; sampler completion advances it and the publisher re-emits it"
);
}
/// Register the bounded community-admission contract.
pub(crate) fn describe_community_admission_metrics() {
metrics::describe_counter!(
"buzz_community_admission_checks_total",
"Durable community-active checks at socket admission by bounded outcome"
);
}
@@ -541,7 +560,17 @@ pub(crate) fn readiness_test_recorder() -> (
metrics_exporter_prometheus::PrometheusRecorder,
metrics_exporter_prometheus::PrometheusHandle,
) {
let recorder = configured_prometheus_builder(300).build_recorder();
readiness_test_recorder_with_idle_timeout(300)
}
#[cfg(test)]
pub(crate) fn readiness_test_recorder_with_idle_timeout(
gauge_idle_timeout_secs: u64,
) -> (
metrics_exporter_prometheus::PrometheusRecorder,
metrics_exporter_prometheus::PrometheusHandle,
) {
let recorder = configured_prometheus_builder(gauge_idle_timeout_secs).build_recorder();
let handle = recorder.handle();
(recorder, handle)
}

1461
crates/buzz-relay/src/readiness.rs
View File

File diff suppressed because it is too large Load Diff

903
crates/buzz-relay/src/router.rs
View File

File diff suppressed because it is too large Load Diff

283
crates/buzz-relay/src/state.rs
View File

@@ -182,10 +182,99 @@ impl Drop for CommunityConnectionGuard {
}
}
/// Message reported when the one-time Redis bootstrap gate rejects startup.
///
/// Bounded and stable so operators and the boot regression test can match on
/// it without parsing the underlying driver error.
pub const REDIS_BOOTSTRAP_FAILURE: &str = "Redis command path unavailable at startup";
/// Budget for the one-time bootstrap PING. A refused port answers immediately;
/// this only bounds a blackholed address, where hanging forever would be worse
/// than exiting.
const REDIS_BOOTSTRAP_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
/// Keep the Redis driver's per-command timeout outside the relay-owned startup
/// budget so [`REDIS_BOOTSTRAP_TIMEOUT`] remains the one authoritative bound.
const REDIS_BOOTSTRAP_DRIVER_TIMEOUT: std::time::Duration =
REDIS_BOOTSTRAP_TIMEOUT.saturating_mul(2);
/// Proves once, during startup, that the Redis command path this pod will serve
/// from can actually be reached.
///
/// `deadpool_redis` pools dial lazily and `PubSubManager::new` only allocates
/// channels, so without this nothing in boot ever opened a command connection:
/// a relay came up against a dead Redis, bound its health listener, and — since
/// readiness reports local lifecycle only — advertised ready forever. Binding
/// that listener is a one-way latch, so the check has to happen before it, and
/// it is deliberately a *startup* gate: once serving, a Redis blip is a
/// dependency failure and must never change readiness.
pub async fn verify_redis_command_path(pool: &deadpool_redis::Pool) -> anyhow::Result<()> {
let ping = async {
let connection = pool
.get()
.await
.map_err(|error| anyhow::anyhow!("{REDIS_BOOTSTRAP_FAILURE}: {error}"))?;
// This one-shot connection is removed from the pool so extending its
// driver timeout cannot leak into normal serving traffic. The relay's
// outer timeout below must bound both lazy checkout and PING.
let mut connection = deadpool_redis::Connection::take(connection);
connection.set_response_timeout(REDIS_BOOTSTRAP_DRIVER_TIMEOUT);
redis::cmd("PING")
.query_async::<String>(&mut connection)
.await
.map_err(|error| anyhow::anyhow!("{REDIS_BOOTSTRAP_FAILURE}: {error}"))
};
match tokio::time::timeout(REDIS_BOOTSTRAP_TIMEOUT, ping).await {
Err(_) => Err(anyhow::anyhow!(
"{REDIS_BOOTSTRAP_FAILURE}: no response within {REDIS_BOOTSTRAP_TIMEOUT:?}"
)),
Ok(result) => result.map(|_| ()),
}
}
/// Bounded outcome of the durable community-active check run when a socket is
/// admitted.
///
/// `outcome` is the only dimension. Community, tenant, connection, and error
/// text are request-controlled and deliberately absent from the label set.
#[derive(Debug, Clone, Copy)]
enum AdmissionOutcome {
Active,
Inactive,
CheckError,
}
impl AdmissionOutcome {
fn label(self) -> &'static str {
match self {
Self::Active => "active",
Self::Inactive => "inactive",
Self::CheckError => "check_error",
}
}
}
fn record_admission_check(outcome: AdmissionOutcome) {
metrics::counter!(
"buzz_community_admission_checks_total",
"outcome" => outcome.label(),
)
.increment(1);
}
/// Registers a socket, durably revalidates its community, then runs it.
///
/// The ordering is the archival admission invariant: archive-before-query is
/// observed by the query, while archive-after-registration sees the token.
///
/// Admission is fail-closed: only an affirmative `Ok(true)` may serve. Both
/// `Ok(false)` and a lookup `Err` cancel, because neither proves this tenant is
/// currently admitted, and `docs/multi-tenant-relay.md` I5
/// (`Inv_AdmissionFence`) grants capability only to an actor *currently*
/// admitted to that community. The two are still told apart in telemetry
/// (`buzz_community_admission_checks_total{outcome}`) so an operator can
/// separate archival from database pressure.
pub(crate) async fn run_registered_community_connection<Check, CheckFuture, Run, RunFuture>(
registry: &CommunityConnectionRegistry,
connection_id: Uuid,
@@ -201,9 +290,29 @@ pub(crate) async fn run_registered_community_connection<Check, CheckFuture, Run,
{
let cancel = control.cancel.clone();
let _guard = registry.register(connection_id, community_id, control.clone());
if !matches!(check_active().await, Ok(true)) {
cancel.cancel();
return;
match check_active().await {
Ok(true) => record_admission_check(AdmissionOutcome::Active),
Ok(false) => {
record_admission_check(AdmissionOutcome::Inactive);
cancel.cancel();
return;
}
Err(error) => {
// A lookup failure is not an answer, so it cannot authorize one.
// Admitting here would begin serving AUTH and REQ for a tenant
// whose lifecycle is unknown, and the adjacent host-binding seam
// already refuses on exactly this evidence (see
// `router::nip11_or_ws_handler`). The client sees an ordinary dial
// failure and retries.
record_admission_check(AdmissionOutcome::CheckError);
tracing::warn!(
%community_id,
%error,
"community active check failed; refusing the socket"
);
cancel.cancel();
return;
}
}
if cancel.is_cancelled() {
return;
@@ -719,8 +828,14 @@ pub struct AppState {
pub audio_rooms: Arc<AudioRoomManager>,
/// Set to `true` on SIGTERM — readiness probe returns 503.
pub shutting_down: Arc<AtomicBool>,
/// Orders readiness gauge publication against terminal shutdown.
pub(crate) readiness: Arc<crate::readiness::ReadinessCoordinator>,
/// Cached shared-dependency evaluation behind the diagnostic `/_status`
/// endpoint, owned by [`crate::readiness::run_dependency_sampler`]. Never
/// consulted by a Kubernetes probe, and never evaluated by a request.
pub(crate) dependency_diagnostics: Arc<crate::readiness::DependencyDiagnostics>,
/// Stops only the periodic dependency sampler during graceful shutdown.
pub dependency_sampler_cancel: CancellationToken,
/// Stops only the completion-epoch publisher during graceful shutdown.
pub dependency_completion_publisher_cancel: CancellationToken,
/// Process start time — used by `/_status` endpoint.
pub started_at: Instant,
/// Shared, community-scoped NIP-98 replay prevention.
@@ -946,7 +1061,9 @@ impl AppState {
git_pack_cache,
audio_rooms: Arc::new(AudioRoomManager::new()),
shutting_down: Arc::new(AtomicBool::new(false)),
readiness: Arc::new(crate::readiness::ReadinessCoordinator::default()),
dependency_diagnostics: Arc::new(crate::readiness::DependencyDiagnostics::default()),
dependency_sampler_cancel: CancellationToken::new(),
dependency_completion_publisher_cancel: CancellationToken::new(),
started_at: Instant::now(),
nip98_replay,
gif_http_client,
@@ -990,21 +1107,21 @@ impl AppState {
)
}
/// Atomically closes readiness publication before exposing shutdown to
/// the relay's other fast-path lifecycle checks.
/// Withdraws this pod from routing. The lifecycle flag is authoritative for
/// `/_readiness`; the private probe publishes its sampled observation to
/// the readiness gauge on its next request.
pub fn begin_shutdown(&self) {
self.readiness.begin_shutdown();
self.shutting_down.store(true, Ordering::Release);
}
#[cfg(test)]
pub(crate) fn set_readiness_evaluator(
pub(crate) fn set_dependency_evaluator(
&mut self,
evaluator: Arc<dyn crate::readiness::ReadinessEvaluator>,
evaluator: Arc<dyn crate::readiness::DependencyEvaluator>,
) {
self.readiness = Arc::new(crate::readiness::ReadinessCoordinator::with_evaluator(
evaluator,
));
self.dependency_diagnostics = Arc::new(
crate::readiness::DependencyDiagnostics::with_evaluator(evaluator),
);
}
/// Inter-relay mesh handle. `None` ⇒ mesh-off / single-instance: callers
@@ -2096,6 +2213,144 @@ pub(crate) mod tests {
assert!(!started_during.load(Ordering::SeqCst));
}
/// Reads one `buzz_community_admission_checks_total` series by exact label set.
fn admission_counter(
snapshot: &[(
metrics_util::CompositeKey,
Option<metrics::Unit>,
Option<metrics::SharedString>,
metrics_util::debugging::DebugValue,
)],
outcome: &str,
) -> Option<u64> {
snapshot.iter().find_map(|(key, _, _, value)| {
let labels = key
.key()
.labels()
.map(|label| (label.key(), label.value()))
.collect::<Vec<_>>();
if key.key().name() != "buzz_community_admission_checks_total"
|| labels != [("outcome", outcome)]
{
return None;
}
match value {
metrics_util::debugging::DebugValue::Counter(count) => Some(*count),
_ => panic!("community admission checks must be a counter"),
}
})
}
/// Admission is fail-closed on both non-affirmative outcomes. A confirmed
/// `Ok(false)` and a lookup `Err` are different diagnoses — the counter
/// keeps them apart — but neither is proof of current admission, and
/// `docs/multi-tenant-relay.md` I5 (`Inv_AdmissionFence`) grants read or
/// membership capability only to an actor *currently* admitted to that
/// community. Serving AUTH/REQ on an unproven tenant lifecycle is the
/// failure this guards.
#[test]
fn neither_a_confirmed_inactive_community_nor_a_failed_lookup_admits_the_socket() {
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("current-thread runtime");
let recorder = metrics_util::debugging::DebuggingRecorder::new();
let snapshotter = recorder.snapshotter();
let (inactive_cancel, inactive_started, error_cancel, error_started, active_started) =
metrics::with_local_recorder(&recorder, || {
runtime.block_on(async {
let registry = CommunityConnectionRegistry::new();
let community = CommunityId::from_uuid(Uuid::from_u128(0xa));
let inactive_cancel = CancellationToken::new();
let inactive_started = Arc::new(AtomicBool::new(false));
let started = Arc::clone(&inactive_started);
run_registered_community_connection(
&registry,
Uuid::new_v4(),
community,
CommunityConnectionControl::new(inactive_cancel.clone()),
|| async { Ok(false) },
move |_| async move { started.store(true, Ordering::SeqCst) },
)
.await;
let error_cancel = CancellationToken::new();
let error_started = Arc::new(AtomicBool::new(false));
let started = Arc::clone(&error_started);
run_registered_community_connection(
&registry,
Uuid::new_v4(),
community,
CommunityConnectionControl::new(error_cancel.clone()),
|| async { Err(buzz_db::DbError::Sqlx(sqlx::Error::PoolTimedOut)) },
move |_| async move { started.store(true, Ordering::SeqCst) },
)
.await;
let active_started = Arc::new(AtomicBool::new(false));
let started = Arc::clone(&active_started);
run_registered_community_connection(
&registry,
Uuid::new_v4(),
community,
CommunityConnectionControl::new(CancellationToken::new()),
|| async { Ok(true) },
move |_| async move { started.store(true, Ordering::SeqCst) },
)
.await;
(
inactive_cancel,
inactive_started,
error_cancel,
error_started,
active_started,
)
})
});
assert!(
inactive_cancel.is_cancelled(),
"a confirmed-inactive community must still cancel its socket"
);
assert!(
!inactive_started.load(Ordering::SeqCst),
"a confirmed-inactive community must never start the socket body"
);
assert!(
error_cancel.is_cancelled(),
"a failed active check must cancel its socket, not admit it"
);
assert!(
!error_started.load(Ordering::SeqCst),
"a failed active check must never start serving AUTH/REQ on an unproven tenant"
);
assert!(active_started.load(Ordering::SeqCst));
let snapshot = snapshotter.snapshot().into_vec();
assert_eq!(admission_counter(&snapshot, "inactive"), Some(1));
assert_eq!(admission_counter(&snapshot, "check_error"), Some(1));
assert_eq!(admission_counter(&snapshot, "active"), Some(1));
let label_sets = snapshot
.iter()
.filter(|(key, _, _, _)| key.key().name() == "buzz_community_admission_checks_total")
.map(|(key, _, _, _)| {
key.key()
.labels()
.map(|label| label.key().to_owned())
.collect::<Vec<_>>()
})
.collect::<Vec<_>>();
assert_eq!(label_sets.len(), 3, "outcome is the only dimension");
assert!(
label_sets.iter().all(|labels| labels == &["outcome"]),
"admission telemetry must never carry community, tenant, or error labels: {label_sets:?}"
);
}
#[tokio::test]
async fn revalidation_continues_after_one_community_lookup_failure() {
let registry = CommunityConnectionRegistry::new();

350
crates/buzz-relay/tests/boot_lifecycle.rs
View File

@@ -3,6 +3,7 @@ use std::{
io::{Read as _, Write as _},
net::{TcpListener, TcpStream},
process::{Child, Command, ExitStatus, Output, Stdio},
sync::mpsc,
thread::{self, JoinHandle},
time::{Duration, Instant},
};
@@ -10,11 +11,14 @@ use std::{
use serde_json::Value;
use buzz_relay::lifecycle::StartupPhase;
use buzz_relay::state::REDIS_BOOTSTRAP_FAILURE;
const VALID_RELAY_PRIVATE_KEY: &str =
"0000000000000000000000000000000000000000000000000000000000000001";
const CHILD_TIMEOUT: Duration = Duration::from_secs(10);
const MAX_CAPTURE_BYTES: u64 = 1024 * 1024;
/// Bound on every wait for the relay to export a metric family.
const METRICS_SCRAPE_DEADLINE: Duration = Duration::from_secs(8);
struct RelayProcess {
child: Option<Child>,
@@ -156,20 +160,27 @@ fn scrape_metrics(port: u16) -> std::io::Result<String> {
}
fn wait_for_relay_metrics(process: &mut RelayProcess, port: u16) -> String {
let deadline = Instant::now() + Duration::from_secs(8);
wait_for_scraped_metric(process, port, "buzz_audit_enabled")
}
/// Polls the relay's own `/metrics` until `needle` appears, bounded by
/// [`METRICS_SCRAPE_DEADLINE`]. A relay that exits first is a failure, not a
/// timeout, so the panic names the real cause.
fn wait_for_scraped_metric(process: &mut RelayProcess, port: u16, needle: &str) -> String {
let deadline = Instant::now() + METRICS_SCRAPE_DEADLINE;
loop {
assert!(
process.try_wait().is_none(),
"relay exited before its metrics endpoint became usable"
"relay exited before exporting {needle}"
);
if let Ok(response) = scrape_metrics(port) {
if response.contains("buzz_audit_enabled") {
if response.contains(needle) {
return response;
}
}
assert!(
Instant::now() < deadline,
"relay metrics did not become scrapeable within 8s"
"relay did not export {needle} within {METRICS_SCRAPE_DEADLINE:?}"
);
thread::sleep(Duration::from_millis(20));
}
@@ -500,3 +511,334 @@ fn successful_main_emits_complete_lifecycle_without_startup_metrics() {
assert_terminal(&events, "metrics_bind", "succeeded", None);
assert_terminal(&events, "process_telemetry", "succeeded", None);
}
/// Boot gates that need a live Postgres to reach the code under test. Named
/// `postgres_tests` so `.config/nextest.toml`'s `postgres-ci` default filter
/// discovers them structurally; the wrapper hands each test its own database
/// through `DATABASE_URL`.
mod postgres_tests {
use super::*;
const REDIS_BOOTSTRAP_BUDGET: Duration = Duration::from_secs(5);
const REDIS_BOOTSTRAP_SCHEDULING_SLACK: Duration = Duration::from_secs(3);
/// A TCP peer that completes Redis's metadata handshake, then reads and
/// holds PING without replying. This distinguishes a bounded checkout +
/// PING from a refused connection, which returns before the bootstrap
/// timeout is exercised.
struct HangingRedisPeer {
redis_url: String,
request_received: mpsc::Receiver<Vec<u8>>,
stop: mpsc::Sender<()>,
worker: Option<JoinHandle<()>>,
}
impl HangingRedisPeer {
fn spawn() -> Self {
let listener = TcpListener::bind(("127.0.0.1", 0)).expect("bind fake Redis peer");
listener
.set_nonblocking(true)
.expect("set fake Redis listener nonblocking");
let port = listener.local_addr().expect("fake Redis address").port();
let (request_tx, request_received) = mpsc::channel();
let (stop, stop_rx) = mpsc::channel();
let worker = thread::spawn(move || loop {
if stop_rx.try_recv().is_ok() {
return;
}
let (mut stream, _) = match listener.accept() {
Ok(accepted) => accepted,
Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => {
thread::sleep(Duration::from_millis(10));
continue;
}
Err(error) => panic!("accept fake Redis connection: {error}"),
};
stream
.set_read_timeout(Some(Duration::from_millis(100)))
.expect("bound fake Redis read");
let mut request = Vec::new();
loop {
let mut chunk = [0_u8; 4096];
match stream.read(&mut chunk) {
Ok(0) => return,
Ok(read) => {
request.extend_from_slice(&chunk[..read]);
assert!(
request.len() <= 4096,
"fake Redis peer received an oversized request"
);
let setinfo_commands = request
.windows(b"SETINFO".len())
.filter(|window| *window == b"SETINFO")
.count();
if setinfo_commands >= 2 {
// redis-rs pipelines CLIENT SETINFO lib-name
// and lib-ver while establishing a connection.
// Complete that handshake so the relay reaches
// its explicit bootstrap PING, then hold it.
stream
.write_all(b"+OK\r\n+OK\r\n")
.expect("reply to Redis client handshake");
request.clear();
continue;
}
if request
.windows(b"PING".len())
.any(|window| window == b"PING")
{
let _ = request_tx.send(request);
let _ = stop_rx.recv_timeout(CHILD_TIMEOUT);
return;
}
}
Err(error)
if matches!(
error.kind(),
std::io::ErrorKind::WouldBlock | std::io::ErrorKind::TimedOut
) =>
{
if stop_rx.try_recv().is_ok() {
return;
}
}
Err(error) => panic!("read fake Redis request: {error}"),
}
}
});
Self {
redis_url: format!("redis://127.0.0.1:{port}"),
request_received,
stop,
worker: Some(worker),
}
}
fn redis_url(&self) -> &str {
&self.redis_url
}
fn assert_request_received(&self, logs: &str) {
let request = self
.request_received
.recv_timeout(Duration::from_secs(1))
.unwrap_or_else(|_| panic!("relay must reach the fake Redis peer: {logs}"));
assert!(!request.is_empty(), "fake Redis peer read an empty request");
}
}
impl Drop for HangingRedisPeer {
fn drop(&mut self) {
let _ = self.stop.send(());
if let Some(worker) = self.worker.take() {
worker.join().expect("fake Redis peer must not panic");
}
}
}
fn reserve_closed_port() -> u16 {
let reserved = TcpListener::bind(("127.0.0.1", 0)).expect("reserve port");
let port = reserved.local_addr().expect("reserved address").port();
drop(reserved);
port
}
/// Runs the relay until it exits on its own, or kills it once `timeout`
/// passes. Unlike `run_relay`, a relay that keeps serving is a result to
/// assert on rather than a panic, which is the whole point here.
fn run_until_exit(environment: &[(&str, &str)], timeout: Duration) -> (bool, Output) {
let mut process = RelayProcess::spawn(environment);
let deadline = Instant::now() + timeout;
while Instant::now() < deadline {
if process.try_wait().is_some() {
return (true, process.wait(Duration::from_secs(2)));
}
thread::sleep(Duration::from_millis(20));
}
(false, process.terminate())
}
/// Redis is required for pub/sub fan-out, presence, and typing, but nothing
/// in boot ever opened a command connection: `deadpool_redis` pools dial
/// lazily and `PubSubManager::new` only allocates channels, so "Redis
/// pub/sub connected" was logged against a dead port. A relay could
/// therefore boot with Redis unreachable, bind its health listener, and —
/// now that readiness answers from local lifecycle alone — advertise ready
/// for the rest of its life. The bootstrap gate is the one-time proof that
/// the command path has connected at least once, and it has to land before
/// the listener binds, because binding is the one-way latch that makes this
/// pod routable.
///
/// The git conformance probe is disabled so the only remaining startup-fatal
/// gate is the one under test.
#[test]
#[ignore = "requires PostgreSQL"]
fn unreachable_redis_fails_boot_before_the_health_listener_binds() {
let database_url = std::env::var("DATABASE_URL")
.expect("postgres lane provides DATABASE_URL for each test process");
let redis_url = format!("redis://127.0.0.1:{}", reserve_closed_port());
let metrics_port = reserve_closed_port().to_string();
let health_port = reserve_closed_port();
let health_port_value = health_port.to_string();
let (exited, output) = run_until_exit(
&[
("BUZZ_RELAY_PRIVATE_KEY", VALID_RELAY_PRIVATE_KEY),
("BUZZ_METRICS_PORT", &metrics_port),
("BUZZ_HEALTH_PORT", &health_port_value),
("DATABASE_URL", &database_url),
("REDIS_URL", &redis_url),
("BUZZ_GIT_CONFORMANCE_PROBE", "false"),
],
CHILD_TIMEOUT,
);
let logs = format!(
"{}{}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr)
);
assert!(
!logs.contains("Health probe listener started"),
"the Redis bootstrap gate must run before the health listener binds: {logs}"
);
assert!(
exited && !output.status.success(),
"an unreachable Redis command path must be startup-fatal: {logs}"
);
assert!(
logs.contains(REDIS_BOOTSTRAP_FAILURE),
"the failure must name the gate that rejected boot: {logs}"
);
assert!(
TcpListener::bind(("0.0.0.0", health_port)).is_ok(),
"the health port must never have been bound"
);
}
/// A peer that accepts the socket but withholds its Redis response exercises
/// the outer timeout around both lazy pool checkout and PING. Removing or
/// narrowing that timeout makes this test kill a still-running relay at the
/// deadline instead of observing a startup failure.
#[test]
#[ignore = "requires PostgreSQL"]
fn hanging_redis_peer_times_out_before_the_health_listener_binds() {
let database_url = std::env::var("DATABASE_URL")
.expect("postgres lane provides DATABASE_URL for each test process");
let redis_peer = HangingRedisPeer::spawn();
let metrics_port = reserve_closed_port().to_string();
let health_port = reserve_closed_port();
let health_port_value = health_port.to_string();
let timeout = REDIS_BOOTSTRAP_BUDGET + REDIS_BOOTSTRAP_SCHEDULING_SLACK;
let started_at = Instant::now();
let (exited, output) = run_until_exit(
&[
("BUZZ_RELAY_PRIVATE_KEY", VALID_RELAY_PRIVATE_KEY),
("BUZZ_METRICS_PORT", &metrics_port),
("BUZZ_HEALTH_PORT", &health_port_value),
("DATABASE_URL", &database_url),
("REDIS_URL", redis_peer.redis_url()),
("BUZZ_GIT_CONFORMANCE_PROBE", "false"),
],
timeout,
);
let elapsed = started_at.elapsed();
let logs = format!(
"{}{}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr)
);
redis_peer.assert_request_received(&logs);
assert!(
elapsed >= REDIS_BOOTSTRAP_BUDGET,
"the fake peer must hold the request through the bootstrap budget: {elapsed:?}: {logs}"
);
assert!(
exited && elapsed < timeout && !output.status.success(),
"the outer bootstrap timeout must terminate the relay within scheduling slack: {elapsed:?}: {logs}"
);
assert!(
logs.contains(REDIS_BOOTSTRAP_FAILURE),
"the timeout must report the bounded Redis bootstrap failure: {logs}"
);
assert!(
!logs.contains("Health probe listener started"),
"the health listener must not bind before Redis bootstrap succeeds: {logs}"
);
assert!(
TcpListener::bind(("0.0.0.0", health_port)).is_ok(),
"the health port must never have been bound"
);
}
/// Startup is the only owner of dependency evaluation. `/_status` just reads
/// the cache, a readiness probe records no dependency attempt at all, and no
/// request path may start a check — so if `main` stops spawning the sampler,
/// this pod evaluates Postgres, Redis, and the deletion catalog exactly
/// never: the dependency families stay absent from its scrape and `/_status`
/// answers `not_yet_sampled` for the pod's whole life.
///
/// No in-process test can fail on that, because each one drives `sample`
/// itself. This one boots the real binary against real dependencies and
/// reads only the relay's own `/metrics` — no probe, no `/_status`, nothing
/// that could evaluate a dependency on the test's behalf. The first tick
/// fires immediately, so the wait is bounded by
/// [`METRICS_SCRAPE_DEADLINE`] and never a fixed sleep.
///
/// The completion-timestamp gauge is the needle because `sample` writes it
/// last, after the cache already serves the report: observing it proves the
/// whole cycle ran, with no window where the counters have landed but the
/// timestamp has not.
#[test]
#[ignore = "requires PostgreSQL"]
fn startup_owns_the_dependency_sampler() {
let database_url = std::env::var("DATABASE_URL")
.expect("postgres lane provides DATABASE_URL for each test process");
let redis_url = std::env::var("REDIS_URL")
.expect("the postgres lane runs alongside Redis and exports REDIS_URL");
let metrics_port = reserve_closed_port();
let metrics_port_value = metrics_port.to_string();
let health_port_value = reserve_closed_port().to_string();
let bind_addr = format!("127.0.0.1:{}", reserve_closed_port());
let mut process = RelayProcess::spawn(&[
("BUZZ_RELAY_PRIVATE_KEY", VALID_RELAY_PRIVATE_KEY),
("BUZZ_METRICS_PORT", &metrics_port_value),
("BUZZ_HEALTH_PORT", &health_port_value),
("BUZZ_BIND_ADDR", &bind_addr),
("DATABASE_URL", &database_url),
("REDIS_URL", &redis_url),
("BUZZ_GIT_CONFORMANCE_PROBE", "false"),
]);
let scrape = wait_for_scraped_metric(
&mut process,
metrics_port,
"buzz_readiness_dependency_sample_completed_timestamp_seconds",
);
let output = process.terminate();
let logs = format!(
"{}{}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr)
);
assert!(
scrape.contains("buzz_readiness_dependency_checks_total{"),
"the same cycle must publish its per-dependency outcomes: {scrape}"
);
assert!(
scrape.contains("buzz_readiness_check_duration_seconds"),
"a completed evaluation must publish its latency too: {scrape}"
);
assert!(
!scrape.contains("buzz_readiness_checks_total{"),
"no readiness probe was sent, so the sampler alone produced this: {scrape}"
);
assert!(
logs.contains("Health probe listener started"),
"the sampler must be owned by a relay that finished booting: {logs}"
);
}
}

123
deploy/charts/buzz/README.md
View File

@@ -110,10 +110,11 @@ SigV4 signing, so do not put the bucket into `s3.endpoint`; pass Railway's base
Object storage is contacted during relay startup only when
`BUZZ_GIT_CONFORMANCE_PROBE` is enabled (the relay default). A probe failure is
startup-fatal, so Kubernetes readiness never opens. If an operator explicitly
disables that probe through `relay.extraEnv`, `/_readiness` does not test object
storage; configuration is still parsed strictly, but reachability and addressing
errors surface on the first storage operation.
startup-fatal, so the process exits and Kubernetes readiness never opens. If an
operator explicitly disables that probe through `relay.extraEnv`, configuration
is still parsed strictly, but reachability and addressing errors surface on the
first storage operation. `/_readiness` tests no external dependency in either
case — see the readiness contract below.
### Early-startup telemetry contract
@@ -125,26 +126,114 @@ These phases intentionally do not emit metrics. Most run before the Prometheus
exporter exists, and one uniform log-only contract preserves every phase's real
event time and failure without assigning an eventual scrape time to earlier work.
### Readiness contract
**`/_readiness` reports local process lifecycle only.** It performs no
Postgres, Redis, or deletion-catalog I/O: `shutting_down` returns 503, and any
other state returns 200. Shared dependencies are shared by every replica, so
gating the probe on them removed the whole deployment from the load balancer at
once and left a reconnect burst with nowhere to land. There is no separate
"starting" state — the health listener does not bind until the database,
migrations, Redis, and pub/sub are up, so a process that can answer has booted.
Shared-dependency health moved to **`/_status`** on the same private health
listener, under a `dependencies` object carrying the `postgres`, `redis`,
`deletion_catalog`, and aggregate `reason` fields the readiness body used to
return. Do not wire `/_status` to a Kubernetes probe.
**`/_status` is a cached read.** It performs no Postgres, Redis, or
deletion-catalog I/O of its own. One background loop per pod evaluates the
three dependencies **every 30 seconds**, awaiting each evaluation before taking
the next tick, so a pod never has more than one evaluation in flight no matter
how often — or how rarely — the endpoint is read. The loop publishes the
dependency metrics below and caches the report `/_status` serves. Its first
cycle runs at startup, and each evaluation is bounded by a two-second budget.
Every `dependencies` object therefore states how old its report is:
| Field | Meaning |
|-------|---------|
| `sample: "not_yet_sampled"` | the first cycle has not completed; no `postgres`/`redis`/`deletion_catalog`/`reason` fields are present, because there is no observation to report |
| `sample: "fresh"` | the report is at most two cadences (60s) old |
| `sample: "stale"` | the report outlived two cadences, so the sampler missed at least one cycle — read the verdict as history, not as current state |
| `sample_age_seconds` | age of the report at request time (absent when `not_yet_sampled`) |
| `sample_interval_seconds` | the sampling cadence, `30` |
Polling `/_status` more often than the cadence returns the same cached report;
it does not make the data fresher and adds no dependency load.
### Readiness telemetry contract
Only requests served by the private health listener (`BUZZ_HEALTH_PORT`) emit
rollout readiness telemetry. The compatibility `/_readiness` route on the public
app listener returns health but does not change these metrics.
rollout telemetry. The compatibility `/_readiness` route on the public app
listener returns the same lifecycle answer but does not change these metrics.
| Metric | Type | Labels | Source |
|--------|------|--------|--------|
| `buzz_readiness_checks_total` | counter | `reason` ∈ {`ready`, `shutting_down`} | `/_readiness` |
| `buzz_readiness_state` | gauge | `check="overall"`; latest private probe observation, 1 ready or 0 shutting down | `/_readiness` |
| `buzz_readiness_dependency_checks_total` | counter | `dependency`, typed bounded `outcome` | dependency sampler |
| `buzz_readiness_check_duration_seconds` | histogram | `check` only | dependency sampler |
| `buzz_readiness_dependency_sample_completed_timestamp_seconds` | gauge | none; Unix time the cached report completed | completion publisher |
The three dependency families keep their `buzz_readiness_*` names for dashboard
continuity, but nothing about them is request-driven any more: the 30-second
sampler publishes them whether or not anyone reads `/_status`, so a quiet
endpoint no longer produces a flat dashboard during the outage it exists to
explain.
`buzz_readiness_dependency_sample_completed_timestamp_seconds` carries **when
the cached report completed**, in Unix seconds. The relay sampler is the only
owner allowed to advance that epoch, and it writes it immediately after the
cache is replaced. A separate bounded publisher re-emits the stored epoch often
enough to survive local gauge idle-timeout; republishing never advances the
timestamp. If a republish races a newer completion, one scrape can briefly see
the older epoch, but the republish path verifies after writing and repairs to the
newer stored epoch before that republish call returns. So the value stands still
when sampling stops, and the time elapsed since it was written is whatever the
reader computes at read time.
Following the `buzz_storage_sweep_age_seconds` convention, the series is not
emitted until the first sample completes: its absence means "not yet sampled",
not "fresh".
That is the whole server-side contract. Freshness alerting is built from this
gauge in the monitoring provider, and the monitor query, thresholds, and
per-pod tag grouping belong with the deployment's monitor configuration rather
than in this chart — they depend on the provider's query grammar and on the
tags its agent attaches, neither of which this repo owns.
Two properties of the neighboring families are worth knowing when building
that alerting. `buzz_readiness_dependency_checks_total` and
`buzz_readiness_check_duration_seconds` are cumulative, so a sampler that stops
leaves their last values exported and scraped indefinitely: those series stay
present and flat rather than disappearing. And per-report freshness for a human
reading a single pod is already on the `sample`, `sample_age_seconds`, and
`sample_interval_seconds` fields of `/_status` above.
The schema has a ceiling of 87 raw Prometheus series per pod: 2 probe reasons,
11 valid dependency/outcome pairs, 72 histogram series, and 2 gauges (overall lifecycle + completion timestamp). Do not
add pod, ReplicaSet, version, rollout, error text, SQL, URL, tenant, user,
community, pubkey, header, query, or other request-controlled labels. A
readiness probe records no dependency attempt or latency sample at all.
The readiness gauge is not a monotonic lifecycle mirror: shutdown changes the
authoritative lifecycle flag, and the next private readiness probe observes and
publishes that state.
### Community admission telemetry
| Metric | Type | Labels |
|--------|------|--------|
| `buzz_readiness_checks_total` | counter | `reason` from the closed readiness-reason set |
| `buzz_readiness_dependency_checks_total` | counter | `dependency`, typed bounded `outcome` |
| `buzz_readiness_check_duration_seconds` | histogram | `check` only |
| `buzz_readiness_state` | gauge | `check` only; latest publishable generation |
| `buzz_community_admission_checks_total` | counter | `outcome` ∈ {`active`, `inactive`, `check_error`} |
The schema has a ceiling of 99 raw Prometheus series per pod: 12 overall
reasons, 11 valid dependency/outcome pairs, 72 histogram series, and 4 gauges.
Do not add pod, ReplicaSet, version, rollout, error text, SQL, URL, tenant,
user, community, pubkey, header, query, or other request-controlled labels.
Shutdown without dependency evaluation increments only
`buzz_readiness_checks_total{reason="shutting_down"}` and sets the overall
state to zero; it does not fabricate dependency failures or latency samples.
Counts the durable community-active check run before a socket is admitted.
Admission is fail-closed: only `active` serves. `inactive` (a confirmed
archival answer) and `check_error` (the lookup itself failed, so the tenant
lifecycle is unknown) both refuse the socket before any AUTH or REQ frame is
read; the client sees an ordinary dial failure and retries. The two outcomes
stay distinct so a rise in `check_error` reads as database pressure rather than
archival. `outcome` is the only dimension — community and error text are
request-controlled and must never become labels.
### Operation-aware database pool acquisition contract

19
docs/deployment-identity.md
View File

@@ -47,6 +47,15 @@ The relay health listener exposes intrinsic build identity at `/_status`:
"source_sha": "<40-character-source-sha>",
"id": "github-actions:<run-id>:<attempt>",
"url": "https://github.com/block/buzz/actions/runs/<run-id>/attempts/<attempt>"
},
"dependencies": {
"sample": "fresh",
"sample_interval_seconds": 30,
"sample_age_seconds": 12,
"postgres": true,
"redis": true,
"deletion_catalog": true,
"reason": "ready"
}
}
```
@@ -54,6 +63,16 @@ The relay health listener exposes intrinsic build identity at `/_status`:
Non-CI builds report stable `unknown` or `local` fallback values instead of
claiming provenance they do not have.
`dependencies` is a cached diagnostic snapshot of shared-dependency health. A
per-pod background loop evaluates the dependencies every 30 seconds; the
endpoint only reads the latest report and never contacts a dependency itself,
so polling it costs nothing. `sample` is always present and reports whether
that cached verdict is `fresh`, `stale`, or `not_yet_sampled` — before the
first cycle completes the health fields are absent rather than defaulted.
`/_readiness` does not consult any of this — see
[the readiness contract](../deploy/charts/buzz/README.md#readiness-contract) —
so this endpoint must never be wired to a Kubernetes probe.
## Helm digest pinning
Buzz chart `0.1.8` and newer accept an immutable image digest:

15
scripts/run-tests.sh
View File

@@ -155,9 +155,9 @@ run_unit_tests() {
run_test_step "buzz-acp unit tests" \
cargo test -p buzz-acp --lib -- --nocapture
# Mirror the three infra-free relay handler modules in `just test-unit`'s
# nextest expression. Keep the side-effects filter pinned to `::tests::` so
# it does not select the sibling Postgres-backed test module.
# Mirror the relay filters from `just test-unit`: the three handler modules,
# storage-snapshot helpers, readiness and router unit suites, and the single
# scoped admission regression in state::tests.
run_test_step "buzz-relay channel authorization tests" \
cargo test -p buzz-relay --lib handlers::channel_authz:: -- --nocapture
@@ -169,6 +169,15 @@ run_unit_tests() {
run_test_step "buzz-relay storage snapshot tests" \
cargo test -p buzz-relay --lib storage_sweep::tests:: -- --nocapture
run_test_step "buzz-relay readiness tests" \
cargo test -p buzz-relay --lib readiness::tests:: -- --nocapture
run_test_step "buzz-relay router tests" \
cargo test -p buzz-relay --lib router::tests:: -- --nocapture
run_test_step "buzz-relay admission regression test" \
cargo test -p buzz-relay --lib state::tests::neither_a_confirmed_inactive_community_nor_a_failed_lookup_admits_the_socket -- --nocapture
}
# ---- DB / integration tests (infra required) --------------------------------