jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-10-27 04:30:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

Make relay readiness process-local (#7341)

Browse Source 
## Summary

Make Kubernetes readiness depend only on the relay process lifecycle, so
a shared Postgres or Redis slowdown cannot withdraw every pod at once.

Move dependency checks to one fixed-cadence loop per pod. The loop
permits only one evaluation at a time, emits metrics after each sample,
and caches a timestamped report. `/_status` reads that cache without
calling Postgres, Redis, or the deletion catalog. It reports
`not_yet_sampled`, `fresh`, or `stale`. Each completed sample also
writes a Unix timestamp gauge, so Datadog can compute sample age at
query time.

Fail closed when the community lifecycle lookup errors. An inconclusive
tenant-state lookup no longer admits a socket to AUTH or REQ handling.

### Related issue

None found. This addresses the elevated relay HTTP 500 and reconnect
incident investigated on 2026-09-03.

### Tradeoff

The readiness gauge now reports the latest private-probe observation,
not the shutdown transition itself. After shutdown starts, a scrape can
still see `1` until the next readiness probe refreshes the gauge to `0`.
Kubernetes routing is unaffected because it uses the readiness response
from the authoritative process state.

A terminal `0` was never guaranteed: even a transition-owned gauge could
update in process and then exit before Prometheus scraped it. If no
scrape occurs before exit, both designs miss the terminal sample. The
new design adds at most the interval until the next readiness probe when
a scrape occurs during shutdown.

Dependency status is now sampled every 30 seconds rather than on demand.
A slow evaluation delays the next sample instead of stacking more work.
The status payload exposes the cached report age. The completion
timestamp stays unchanged when sampling stops, so `time() - timestamp`
crosses the stale threshold without a server aging loop. Outcome
counters and duration histograms remain cumulative and cannot provide
no-data detection.

### Testing

Gimli exercised exact SHA `edff1ec4a5362e132e367bf7449d2b0a68f1e9a9` on
an isolated Blox stack. Repeated `/_status` bursts caused no dependency
calls, metrics advanced without status traffic, stale status remained
visible, and lifecycle lookup failures admitted no AUTH or REQ handling.
Follow-up verification at `fdbb394ed598c94cde85ed01cb11e9de0ee8f00d`
confirmed sampler-owned outcome and duration emission without status
traffic and bound the real startup seam. The final timestamp-gauge
change is covered by targeted red/green tests and the same boot
regression at `c0f36e22b527c2fd6a63a548d036d006ffa17171`.

### Complexity

Readiness remains one process-local sample per probe. Dependency I/O now
has one owner and one execution path; request volume cannot increase
dependency-check load. One completion timestamp replaces the prior
server-side age update path; Datadog computes age without another loop.
Community admission also returns to one fail-closed rule for inactive
and unknown tenant state.

Generated with [Claude Code](https://claude.ai/code)

---------

Signed-off-by: tornquist <tornquist@squareup.com>
Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Amp <amp@ampcode.com>
This commit is contained in:
Luke Tornquist
2026-09-28 14:02:12 -04:00
committed by GitHub
parent 447951a19c
commit b37e477211
13 changed files with 2434 additions and 795 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
crates/buzz-relay/src/metrics.rs
View File

@@ -290,6 +290,7 @@ pub fn try_install(port: u16, gauge_idle_timeout_secs: u64) -> Result<(), Metric
metrics::set_global_recorder(recorder)
.map_err(|_error| MetricsInstallError::RecorderConflict)?;
describe_readiness_metrics();
describe_community_admission_metrics();
describe_db_pool_metrics();
describe_auth_metrics();
initialize_auth_metric_series();
@@ -306,24 +307,42 @@ pub fn install(port: u16, gauge_idle_timeout_secs: u64) {
.unwrap_or_else(|error| panic!("metrics exporter must install exactly once: {error}"));
}
/// Register the frozen readiness metric descriptions with the active recorder.
/// Register the frozen readiness and dependency-diagnostic metric descriptions.
///
/// The two `buzz_readiness_*` probe families describe local process lifecycle.
/// The dependency families keep their names for dashboard continuity but are
/// published by the per-pod dependency sampler, not by the Kubernetes probe or
/// by an `/_status` request — a shared-dependency failure no longer deroutes
/// the pod, and nobody has to read the endpoint for the metrics to move.
pub(crate) fn describe_readiness_metrics() {
metrics::describe_counter!(
"buzz_readiness_checks_total",
"Kubernetes health-listener readiness probes by terminal bounded reason"
"Kubernetes health-listener readiness probes by lifecycle reason (ready, shutting_down)"
);
metrics::describe_counter!(
"buzz_readiness_dependency_checks_total",
"Completed readiness dependency attempts by dependency and bounded outcome"
"Completed dependency-sampler attempts by dependency and bounded outcome"
);
metrics::describe_histogram!(
"buzz_readiness_check_duration_seconds",
metrics::Unit::Seconds,
"Completed readiness check duration without outcome label multiplication"
"Completed dependency-sampler check duration without outcome label multiplication"
);
metrics::describe_gauge!(
"buzz_readiness_state",
"Latest publishable readiness state by check, where 1 is ready and 0 is not ready"
"Latest private readiness-probe observation, where 1 is ready and 0 is shutting down"
);
metrics::describe_gauge!(
"buzz_readiness_dependency_sample_completed_timestamp_seconds",
"Unix time the cached /_status dependency report completed, absent until the first sample completes; sampler completion advances it and the publisher re-emits it"
);
}
/// Register the bounded community-admission contract.
pub(crate) fn describe_community_admission_metrics() {
metrics::describe_counter!(
"buzz_community_admission_checks_total",
"Durable community-active checks at socket admission by bounded outcome"
);
}
@@ -541,7 +560,17 @@ pub(crate) fn readiness_test_recorder() -> (
metrics_exporter_prometheus::PrometheusRecorder,
metrics_exporter_prometheus::PrometheusHandle,
) {
let recorder = configured_prometheus_builder(300).build_recorder();
readiness_test_recorder_with_idle_timeout(300)
}
#[cfg(test)]
pub(crate) fn readiness_test_recorder_with_idle_timeout(
gauge_idle_timeout_secs: u64,
) -> (
metrics_exporter_prometheus::PrometheusRecorder,
metrics_exporter_prometheus::PrometheusHandle,
) {
let recorder = configured_prometheus_builder(gauge_idle_timeout_secs).build_recorder();
let handle = recorder.handle();
(recorder, handle)
}