Files
buzz/crates/buzz-db/src/runtime/connection_observability.rs
ravarora2 4c2086c084 feat(db): expose connection setup metrics (#7286)
## Why

Current pool metrics show checkout outcomes and pool state after the
fact. They do not show when a checkout began or which writer-connection
setup step failed. During startup and pool growth, operators need to
distinguish pool saturation from a slow or unsafe connection setup.

## What

- Count every instrumented database checkout when it starts, with a
fixed operation label.
- Measure writer-pool creation and every physical writer connection
across `physical_connect`, `created_at_floor`, `session_timeouts`,
`isolation`, and `ready`.
- Publish fixed-cardinality start, terminal-outcome, duration, and
waiter metrics without database URLs, SQL, raw errors, connection
ordinals, or per-connection lifecycle logs.
- Document how to identify the active setup bottleneck during a rollout
or incident.

## How

The existing typed checkout wrapper records starts and current waiters.
The huddle-history path uses that wrapper instead of a raw pool
checkout.

The production SQLx `after_connect` hook records start, terminal
outcome, and duration for each sequential safety step. A drop guard
records cancellation once if setup exits before a terminal result. Fixed
enums keep every label bounded.

For a phase, `started_total - sum(attempts_total)` is the number of
in-progress attempts on that pod. Because setup is sequential, a later
phase starting also proves the earlier phases succeeded.

The database path emits metrics only. This PR intentionally does not add
per-connection lifecycle receipts or direct `stderr` writes.

## Risk

Medium. This changes the production writer-pool `after_connect` hook and
adds one counter update to instrumented checkout paths. It does not
change the database safety statements or their failure behavior. The
telemetry has fixed labels and no per-connection identifiers.

## Testing

At exact head `ffb5fcb114a9c986ffd0cb2cfc9d413a32d5b1aa`, a release
relay was started against isolated local PostgreSQL, Redis, and MinIO
services. The main health endpoint returned `ok`, readiness returned
`{"status":"ready"}`, the connection-step metrics were exported, and no
database lifecycle receipts were emitted. A release CLI then created a
channel, sent a message, and read the same event back successfully.

The earlier staging deployment used pre-rebase head
`f58e9480a4f068db0c591f604fd6800fdd4bfc45`. The deployed
multi-architecture image came from [GitHub Actions run
33780888255](https://github.com/block/buzz/actions/runs/33780888255),
manifest
`sha256:b9351fa644e08376cbe1999f9bee311d33d1799a68eadef7929c4f862a832fec`.

The [staging
deployment](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/317)
brought both pods in ReplicaSet `buzz-6c8758bd7d` to Ready with zero
restarts, and the connection metrics produced data in the [rollout
dashboard](https://app.datadoghq.com/dashboard/tm7-qxr-wt2/buzz-startup--rollout-safety).
That staging image predates the cleanup that removed per-connection
lifecycle logs; the metric schema and database safety statements are
unchanged.

## Verification

- `cargo test -p buzz-db -- --test-threads=1`: 128 tests passed across
the package and integration target; 255 opt-in tests remained ignored.
- Four opt-in production-path PostgreSQL regressions passed: initial
minimum connections, post-startup pool growth, isolation failure, and
session-timeout setup failure.
- All 35 relay media tests passed against the isolated PostgreSQL
database.
- Rust formatting, workspace clippy, desktop/Tauri clippy, web checks,
mobile analysis, security checks, and file-size policy checks passed.
- The repository-wide unit stage also exposed three unrelated existing
`buzz-acp` failures: one timing-sensitive keepalive test and two
environment-default tests.

## Bigger picture

This is the database-metrics part of the startup and rollout
observability work. The early-startup lifecycle foundation merged in
#7258, so the rebase removed that duplicate commit from this PR.

Originating discussion:
buzz://message?channel=6ac85131-70cd-4bda-a031-38d34114934e&id=fa2bed181c092697210a60bb6eedc55a665d5c1c6cabc1413a04686647011f71

Generated with Codex

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>
2026-09-17 11:26:22 -04:00

345 lines
10 KiB
Rust

//! Fixed-vocabulary metrics for writer-pool connection setup.
//!
//! SQLx exposes the point immediately after a physical connection succeeds,
//! but it does not expose a callback immediately before each physical dial.
//! Consequently, `physical_connect` is a success milestone rather than a
//! duration phase. The aggregate `writer_pool` phase owns failures that occur
//! before `after_connect`, while the session phases own their exact failures.
use std::time::{Duration, Instant};
use super::DbPoolRole;
/// Fixed writer connection-setup steps.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum DbConnectionStep {
/// Construct the writer pool and satisfy its initial minimum size.
WriterPool,
/// A physical connection has completed DNS/network/TLS/authentication.
PhysicalConnect,
/// Install the created-at replica-fence floor.
CreatedAtFloor,
/// Install lock, idle-transaction, and statement timeouts.
SessionTimeouts,
/// Verify READ COMMITTED transaction isolation.
Isolation,
/// The physical connection passed every required session premise.
Ready,
}
impl DbConnectionStep {
/// Complete metric-label vocabulary.
pub const ALL: [Self; 6] = [
Self::WriterPool,
Self::PhysicalConnect,
Self::CreatedAtFloor,
Self::SessionTimeouts,
Self::Isolation,
Self::Ready,
];
/// Stable metric label.
pub const fn as_str(self) -> &'static str {
match self {
Self::WriterPool => "writer_pool",
Self::PhysicalConnect => "physical_connect",
Self::CreatedAtFloor => "created_at_floor",
Self::SessionTimeouts => "session_timeouts",
Self::Isolation => "isolation",
Self::Ready => "ready",
}
}
}
/// Bounded terminal outcome for a connection-setup step.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum DbConnectionOutcome {
/// The step completed successfully.
Succeeded,
/// The step failed.
Failed,
/// The aggregate pool deadline expired.
TimedOut,
/// The owning future was dropped before a terminal.
Cancelled,
}
impl DbConnectionOutcome {
/// Stable metric label.
pub const fn as_str(self) -> &'static str {
match self {
Self::Succeeded => "succeeded",
Self::Failed => "failed",
Self::TimedOut => "timed_out",
Self::Cancelled => "cancelled",
}
}
}
/// Valid role/step pairs with an explicit start counter.
pub const CONNECTION_STARTED_STEPS: [(DbPoolRole, DbConnectionStep); 4] = [
(DbPoolRole::Writer, DbConnectionStep::WriterPool),
(DbPoolRole::Writer, DbConnectionStep::CreatedAtFloor),
(DbPoolRole::Writer, DbConnectionStep::SessionTimeouts),
(DbPoolRole::Writer, DbConnectionStep::Isolation),
];
/// Valid role/step pairs with a duration histogram.
pub const CONNECTION_DURATION_STEPS: [(DbPoolRole, DbConnectionStep); 4] = CONNECTION_STARTED_STEPS;
/// Valid role/step/outcome terminal combinations.
pub const CONNECTION_TERMINALS: [(DbPoolRole, DbConnectionStep, DbConnectionOutcome); 15] = [
(
DbPoolRole::Writer,
DbConnectionStep::WriterPool,
DbConnectionOutcome::Succeeded,
),
(
DbPoolRole::Writer,
DbConnectionStep::WriterPool,
DbConnectionOutcome::Failed,
),
(
DbPoolRole::Writer,
DbConnectionStep::WriterPool,
DbConnectionOutcome::TimedOut,
),
(
DbPoolRole::Writer,
DbConnectionStep::WriterPool,
DbConnectionOutcome::Cancelled,
),
(
DbPoolRole::Writer,
DbConnectionStep::PhysicalConnect,
DbConnectionOutcome::Succeeded,
),
(
DbPoolRole::Writer,
DbConnectionStep::CreatedAtFloor,
DbConnectionOutcome::Succeeded,
),
(
DbPoolRole::Writer,
DbConnectionStep::CreatedAtFloor,
DbConnectionOutcome::Failed,
),
(
DbPoolRole::Writer,
DbConnectionStep::CreatedAtFloor,
DbConnectionOutcome::Cancelled,
),
(
DbPoolRole::Writer,
DbConnectionStep::SessionTimeouts,
DbConnectionOutcome::Succeeded,
),
(
DbPoolRole::Writer,
DbConnectionStep::SessionTimeouts,
DbConnectionOutcome::Failed,
),
(
DbPoolRole::Writer,
DbConnectionStep::SessionTimeouts,
DbConnectionOutcome::Cancelled,
),
(
DbPoolRole::Writer,
DbConnectionStep::Isolation,
DbConnectionOutcome::Succeeded,
),
(
DbPoolRole::Writer,
DbConnectionStep::Isolation,
DbConnectionOutcome::Failed,
),
(
DbPoolRole::Writer,
DbConnectionStep::Isolation,
DbConnectionOutcome::Cancelled,
),
(
DbPoolRole::Writer,
DbConnectionStep::Ready,
DbConnectionOutcome::Succeeded,
),
];
/// Four start counters + four 13-series histograms + fifteen terminal counters.
pub const CONNECTION_RAW_SERIES_PER_POD: usize = 4 + (4 * 13) + 15;
pub(crate) struct DbConnectionStepAttempt {
pool_role: DbPoolRole,
step: DbConnectionStep,
started: Instant,
finished: bool,
}
impl DbConnectionStepAttempt {
pub(crate) fn start(pool_role: DbPoolRole, step: DbConnectionStep) -> Self {
metrics::counter!(
"buzz_db_connection_step_started_total",
"pool_role" => pool_role.as_str(),
"step" => step.as_str(),
)
.increment(1);
Self {
pool_role,
step,
started: Instant::now(),
finished: false,
}
}
pub(crate) fn succeed(self) {
self.finish(DbConnectionOutcome::Succeeded);
}
pub(crate) fn fail(self) {
self.finish(DbConnectionOutcome::Failed);
}
pub(crate) fn time_out(self) {
self.finish(DbConnectionOutcome::TimedOut);
}
fn finish(mut self, outcome: DbConnectionOutcome) {
record_terminal(
self.pool_role,
self.step,
outcome,
Some(self.started.elapsed()),
);
self.finished = true;
}
}
impl Drop for DbConnectionStepAttempt {
fn drop(&mut self) {
if self.finished {
return;
}
let outcome = if std::thread::panicking() {
DbConnectionOutcome::Failed
} else {
DbConnectionOutcome::Cancelled
};
record_terminal(
self.pool_role,
self.step,
outcome,
Some(self.started.elapsed()),
);
self.finished = true;
}
}
pub(crate) fn record_milestone(pool_role: DbPoolRole, step: DbConnectionStep) {
record_terminal(pool_role, step, DbConnectionOutcome::Succeeded, None);
}
fn record_terminal(
pool_role: DbPoolRole,
step: DbConnectionStep,
outcome: DbConnectionOutcome,
elapsed: Option<Duration>,
) {
metrics::counter!(
"buzz_db_connection_step_attempts_total",
"pool_role" => pool_role.as_str(),
"step" => step.as_str(),
"outcome" => outcome.as_str(),
)
.increment(1);
if let Some(elapsed) = elapsed {
metrics::histogram!(
"buzz_db_connection_step_duration_seconds",
"pool_role" => pool_role.as_str(),
"step" => step.as_str(),
)
.record(elapsed.as_secs_f64());
}
}
pub(crate) fn classify_pool_outcome(error: &sqlx::Error) -> DbConnectionOutcome {
if matches!(error, sqlx::Error::PoolTimedOut) {
DbConnectionOutcome::TimedOut
} else {
DbConnectionOutcome::Failed
}
}
#[cfg(test)]
mod tests {
use super::*;
use metrics_util::debugging::{DebugValue, DebuggingRecorder};
#[test]
fn vocabulary_and_series_budget_are_frozen() {
assert_eq!(
DbConnectionStep::ALL.map(DbConnectionStep::as_str),
[
"writer_pool",
"physical_connect",
"created_at_floor",
"session_timeouts",
"isolation",
"ready",
]
);
assert_eq!(CONNECTION_RAW_SERIES_PER_POD, 71);
}
#[test]
fn dropped_step_is_cancelled_exactly_once_without_sensitive_labels() {
let recorder = DebuggingRecorder::new();
let snapshotter = recorder.snapshotter();
let _guard = metrics::set_default_local_recorder(&recorder);
let attempt =
DbConnectionStepAttempt::start(DbPoolRole::Writer, DbConnectionStep::CreatedAtFloor);
drop(attempt);
let metrics = snapshotter.snapshot().into_vec();
assert_eq!(metrics.len(), 3);
for (key, _, _, value) in metrics {
let labels = key
.key()
.labels()
.map(|label| (label.key(), label.value()))
.collect::<std::collections::BTreeMap<_, _>>();
assert_eq!(labels.get("pool_role"), Some(&"writer"));
assert_eq!(labels.get("step"), Some(&"created_at_floor"));
assert!(!labels.contains_key("reason"));
assert!(!labels.contains_key("connection_ordinal"));
match value {
DebugValue::Counter(value) => {
assert_eq!(value, 1);
if key.key().name() == "buzz_db_connection_step_attempts_total" {
assert_eq!(labels.get("outcome"), Some(&"cancelled"));
}
}
DebugValue::Histogram(values) => assert_eq!(values.len(), 1),
DebugValue::Gauge(_) => panic!("connection setup has no gauges"),
}
}
}
#[test]
fn pool_timeout_is_distinguished_from_other_failures() {
assert_eq!(
classify_pool_outcome(&sqlx::Error::PoolTimedOut),
DbConnectionOutcome::TimedOut
);
assert_eq!(
classify_pool_outcome(&sqlx::Error::PoolClosed),
DbConnectionOutcome::Failed
);
let io = sqlx::Error::Io(std::io::Error::new(
std::io::ErrorKind::ConnectionRefused,
"postgres://secret-user:secret-password@example.invalid/private",
));
assert_eq!(classify_pool_outcome(&io), DbConnectionOutcome::Failed);
}
}