Files
buzz/crates/buzz-relay/src/api/mod.rs
ravarora2 91ab9d31b8 Add operation-aware database pool acquisition metrics (#7195)
## Why

Buzz already reports coarse writer/reader database checkout waits, but
those
signals cannot explain which startup or serving operation is blocked by
pool
pressure. That makes rollout diagnosis and postmortems ambiguous: a
readiness
probe, NIP-42 authentication, authorization check, reconnect history
repair,
event write, and background maintenance can all wait on the same pool
while
appearing identical.

This PR implements Package 2A of the pod-handoff plan: operation-aware
pool
borrow causality. It is observability-only; it does not change
configured pool
sizes, SQL semantics, transaction ordering, or timeout policy. Physical
DNS/TCP/TLS/Postgres authentication and session initialization remain
the
separate Package 2B boundary.

## Metric contract

The final contract separates three questions:

| Question | Metric |
|---|---|
| How long did checkout wait? |
`buzz_db_pool_acquire_duration_seconds{pool_role,operation}` |
| How did the attempt end? |
`buzz_db_pool_acquire_attempts_total{pool_role,operation,outcome}` |
| Who is waiting now for a tracked operation? |
`buzz_db_pool_waiters{pool_role,operation}` |

Outcomes are `success`, `timeout`, `error`, and `cancelled`. Operations
are
`bootstrap`, `readiness`, `tenant_resolution`, `authentication`,
`authorization`, `subscription_history`, `event_write`, and
`maintenance`.

Only these eleven pool/operation pairs are constructible:

```text
writer/bootstrap                 reader/bootstrap
writer/readiness
writer/tenant_resolution
writer/authentication
writer/authorization             reader/authorization
writer/subscription_history      reader/subscription_history
writer/event_write
writer/maintenance
```

The duration histogram intentionally does not carry `outcome`. Result
data
remains available on the terminal counter for historical counts and
rates,
without multiplying the expensive histogram family. Nine finite buckets
plus
`+Inf`, sum, and count produce 12 duration series per valid pair.
Together with
four outcome counters and one waiter gauge, the new-family ceiling is
exactly
187 raw Prometheus series per pod, asserted from the production
exporter.

The existing coarse acquisition families remain temporarily for
dashboard
compatibility while the new series are validated in staging.

The operation-specific waiter family covers the explicitly routed,
deployment-critical operations above; it is not a census of every
possible
SQLx checkout in the process. The dashboard pairs it with SQLx pool
active/idle/max gauges for whole-pool capacity context, and treats a
missing
operation series as unknown rather than healthy zero.

## What changed

### Cancellation-safe acquisition ownership

- Add writer- and reader-specific typed operation APIs so invalid label
pairs
  cannot be constructed and store modules cannot emit reader labels.
- Own every polled acquisition with one RAII terminal guard.
- Record exactly one duration and terminal outcome for success, timeout,
error,
  or cancellation.
- Emit nothing for a future that is created but never polled.
- Balance the operation-specific waiter count exactly once on every
terminal or
  dropped future.
- Periodically refresh every expected waiter pair, including healthy
zero, so
missing telemetry is not presented as zero. Reader pairs are emitted
only
when a distinct read pool is configured; a writer-only pod cannot
fabricate
  healthy reader-zero state.

### Production attribution

Route the deployment-critical acquisition paths through caller-owned
semantic
entry points, including:

- writer and reader bootstrap;
- the real post-#7149 readiness acquisition and deletion-catalog
validation;
- tenant resolution and community lifecycle checks;
- NIP-42 allowlist authentication;
- membership, moderation, invite, operator, Git, agent-owner, and policy
  authorization;
- operator community create/list/archive/unarchive, reverse host/channel
tenant
  resolution, and REQ row-community conformance lookups;
- writer/reader subscription history, feed, thread, and routed fallback
paths;
- primary and command event writes, replaceable events, mention
indexing,
  reaction/channel/member/archive side effects, and thread metadata;
- push matching, usage rollups/leadership, replica-fence startup and
recurring
probes, periodic reconciliation, channel/deletion reapers, partitions,
and
  other bounded maintenance/bootstrap paths.

Shared helpers now accept caller-owned intent or expose named semantic
variants
instead of assigning one misleading operation to every caller. No known
P0 path
uses `other`.

### Readiness and size-one-pool correctness

- Rebase on the post-#7149 readiness implementation and instrument the
actual
`Db::readiness_check` acquisition rather than the superseded ping-only
seam.
- Acquire once for deletion-catalog validation and run its queries on
that
  connection, preserving the shared readiness deadline.
- Scope the channel-roster catalog checkout before the behavior probe so
a
  size-one writer pool cannot self-deadlock during startup verification.

### Exporter, documentation, and CI

- Register metric HELP/type/unit metadata through the production
Prometheus
  builder.
- Configure dedicated checkout buckets at 1ms, 5ms, 10ms, 25ms, 50ms,
150ms,
  500ms, 1s, and 3s.
- Add a production scrape-contract test for exact names, labels,
buckets,
  valid pairs, sensitive-label exclusion, and the 187-series ceiling.
- Add source mutation guards for the P0 semantic entry points and
raw-checkout
  bypasses.
- Add an exact backend-integration CI selector for the production
attribution,
  cancellation, readiness, and size-one-pool PostgreSQL tests.
- Document the frozen label vocabulary, valid combinations, semantics,
and
  cardinality budget in the Helm chart README.

## Dashboard intent

The new Stage 2 row in **Buzz Startup & Rollout Safety** is
deployment-first:

- baseline-versus-candidate attempts, failure rates, cancellation rates,
and
  maximum wait by operation;
- outcome counts and percentages over time by SHA/ReplicaSet;
- acquisition wait heatmap, average, and maximum through the rollout;
- historical waiter pressure beside writer active/idle/max context;
- per-pod postmortem drilldown, including terminated pods;
- a smaller current-waiter table with explicit stale/missing semantics.

Percentile widgets remain disabled until Datadog metadata confirms
percentile
support for the new distribution. Current gauges use no fill,
interpolation, or
`default_zero`; missing means unknown.

## Risk assessment

Moderate. The patch touches many database acquisition call sites, but
preserves
the selected physical pool and executes the same SQL on the acquired
connection. The main risks are incorrect semantic attribution,
cancellation
double-counting, and a helper accidentally acquiring twice. Typed APIs,
production-method PostgreSQL tests, source guards, the raw scrape
contract, and
the size-one-pool regression cover those risks.

No tenant, community, user, pubkey, event, channel, SQL, URL, pod,
version,
ReplicaSet, or request-controlled value is emitted as an application
metric
label. Deployment identity is supplied by infrastructure enrichment.

## Verification

- `cargo fmt --all -- --check` — passed.
- `cargo clippy -p buzz-db -p buzz-relay --all-targets --all-features --
-D warnings`
  — passed.
- `cargo test -p buzz-db` — 122 passed, 0 failed, 263 ignored;
  source-contract integration test: 3 passed, 0 failed.
- Focused relay compatibility, metric-contract, and readiness tests —
passed.
- `scripts/test-postgres-test-discovery.sh` — passed.
- Full `buzz-relay` package run from the identical tree reached 1,015
passes;
the six media-test failures all stopped in their shared local PostgreSQL
setup with `Sqlx(PoolTimedOut)` because Docker/PostgreSQL was
unavailable.
The same six failed in isolation, while every changed exact test passed.
- Exact implementation head:
  `f92910b353086e9edf85918ca5f72190edbbe22f`.
- Exact multi-architecture staging image:
  `dev-sha-f92910b353086e9edf85918ca5f72190edbbe22f-run-33607968668-1`

(`sha256:161712c8ed2e265a15df9b63e02248d5973481f875ff129d7d2ae78a09d487a2`).
- Focused staging GitOps PR:

<https://github.com/squareup/builderbot-platform-core-infrastructure/pull/299>
— merged after renderer, inventory, infrastructure test, Kargo, Semgrep,
and
Intersect gates passed; the source/generated-artifact diff was exactly
two
  image lines.
- Exact GitHub head reports 47 terminal checks: 35 successful and 12
  intentionally skipped. PostgreSQL, unit, lint, security, both server
cross-compiles, backend integration, relay E2E, desktop, mobile, image,
  Helm, Semgrep, zizmor, and DCO gates are green.
- Datadog readback identifies two exact-image pods,
  `buzz-d79c8d8f7-ckv2l` and `buzz-d79c8d8f7-qzqdp`, in ReplicaSet
  `buzz-d79c8d8f7`; both report the full source SHA above.
- Both pods report all eleven allowed pool/operation waiter pairs at
current
zero, with no invalid pair. The acceptance window observed nonzero
success
  receipts for readiness, tenant resolution, authorization, subscription
history, event write, and maintenance, and no timeout, error, or
cancelled
outcome. Maximum observed wait was about 101 ms for maintenance and 50
ms
  for reader subscription history.

The main **Buzz Startup & Rollout Safety** dashboard now has a live
Stage 2
database row with eight widgets and nineteen fully scoped queries. Final
readback preserved all seven top-level groups, found zero under-scoped
Row 6
queries, and confirmed the tracked-operation waiter boundary in the
panel
descriptions.

Generated with Codex.

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>
2026-09-02 17:04:34 -04:00

386 lines
14 KiB
Rust

//! HTTP API — media, git, NIP-05, and the Nostr HTTP bridge.
pub mod admin;
pub mod bridge;
pub mod events;
pub mod gifs;
pub mod git;
pub mod invites;
pub mod media;
pub mod mesh_demo;
pub mod nip05;
pub mod operator;
pub mod workflows;
// Re-export imeta helpers used by ingest pipeline.
pub use crate::handlers::imeta::{validate_imeta_tags, verify_imeta_blobs};
use axum::{http::StatusCode, response::Json};
/// Standard error envelope.
pub(crate) fn api_error(status: StatusCode, msg: &str) -> (StatusCode, Json<serde_json::Value>) {
(status, Json(serde_json::json!({ "error": msg })))
}
pub(crate) fn internal_error(msg: &str) -> (StatusCode, Json<serde_json::Value>) {
tracing::error!("Internal error: {msg}");
api_error(StatusCode::INTERNAL_SERVER_ERROR, "internal server error")
}
#[allow(dead_code)]
pub(crate) fn not_found(msg: &str) -> (StatusCode, Json<serde_json::Value>) {
api_error(StatusCode::NOT_FOUND, msg)
}
/// Relay membership enforcement — single gate for all authenticated entry points.
///
/// Moved here from the deleted `relay_members` module. Called by `media.rs`, `bridge.rs`,
/// `git/transport.rs`, and `audio/handler.rs`.
pub mod relay_members {
use axum::{
http::{HeaderMap, StatusCode},
response::Json,
};
use buzz_core::{tenant::CommunityId, TenantContext};
use tracing::{debug, info};
use crate::state::AppState;
/// Transport-neutral outcome of a relay-membership check.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum MembershipDecision {
/// Relay membership enforcement is disabled.
OpenRelay,
/// Caller is directly present in `relay_members`.
Member,
/// Caller is admitted through a NIP-OA owner that is a relay member.
ViaOwner(nostr::PublicKey),
/// Caller is not admitted.
Denied,
}
/// Return the sole NIP-OA credential header, if one was supplied.
///
/// Repeated security-sensitive headers are ambiguous across HTTP stacks,
/// so they are treated as no credential instead of silently selecting one.
pub fn extract_auth_tag_header(headers: &HeaderMap) -> Option<&str> {
let mut values = headers.get_all("x-auth-tag").iter();
let (Some(value), None) = (values.next(), values.next()) else {
return None;
};
value.to_str().ok()
}
/// Check relay membership without committing to an HTTP response shape.
///
/// `community` is the server-resolved tenant of the request; membership is
/// scoped to it so admitting a pubkey to community A never admits it to B.
/// A NIP-OA credential is usable only when `signed_auth_created_at` came
/// from the already-verified authentication event carrying that request.
pub async fn check_relay_membership(
state: &AppState,
community: CommunityId,
pubkey_bytes: &[u8],
auth_tag_header: Option<&str>,
signed_auth_created_at: Option<u64>,
) -> Result<MembershipDecision, String> {
if !state.config.require_relay_membership {
return Ok(MembershipDecision::OpenRelay);
}
let pubkey_hex = hex::encode(pubkey_bytes);
let is_member = state
.db
.is_relay_member(community, &pubkey_hex)
.await
.map_err(|e| format!("relay membership check failed: {e}"))?;
if is_member {
return Ok(MembershipDecision::Member);
}
if state.config.allow_nip_oa_auth {
if let Some(tag_json) = auth_tag_header {
let agent_pubkey = nostr::PublicKey::from_slice(pubkey_bytes)
.map_err(|e| format!("invalid agent pubkey for NIP-OA check: {e}"))?;
let Some(auth_created_at) = signed_auth_created_at else {
info!(agent = %pubkey_hex, "NIP-OA auth tag has no verified signed auth timestamp");
return Ok(MembershipDecision::Denied);
};
match buzz_sdk::nip_oa::verify_auth_tag_for_auth_event(
tag_json,
&agent_pubkey,
auth_created_at,
) {
Ok(owner_pubkey) => {
let owner_hex = owner_pubkey.to_hex();
let owner_is_member = state
.db
.is_relay_member(community, &owner_hex)
.await
.map_err(|e| format!("relay membership check (owner) failed: {e}"))?;
if owner_is_member {
debug!(
agent = %pubkey_hex,
owner = %owner_hex,
"NIP-OA membership granted via owner"
);
return Ok(MembershipDecision::ViaOwner(owner_pubkey));
}
}
Err(e) => {
info!(agent = %pubkey_hex, "NIP-OA auth tag invalid: {e}");
}
}
}
}
Ok(MembershipDecision::Denied)
}
/// Enforce relay membership for a pubkey, with NIP-OA agent delegation fallback.
///
/// Returns `Ok(Some(owner_pubkey))` when the agent is not a direct member but
/// its NIP-OA owner *is* — access is granted via delegation.
///
/// On open relays (`require_relay_membership = false`), returns `Ok(None)`
/// immediately — no membership check is performed. Callers that need NIP-OA
/// owner extraction on open relays should call [`extract_nip_oa_owner`] directly.
///
/// Returns `Ok(None)` when the caller is a direct member (closed relay) or when
/// no NIP-OA tag is present/applicable (open relay without auth tag).
pub async fn enforce_relay_membership(
state: &AppState,
community: CommunityId,
pubkey_bytes: &[u8],
auth_tag_header: Option<&str>,
signed_auth_created_at: Option<u64>,
) -> Result<Option<nostr::PublicKey>, (StatusCode, Json<serde_json::Value>)> {
match check_relay_membership(
state,
community,
pubkey_bytes,
auth_tag_header,
signed_auth_created_at,
)
.await
{
Ok(MembershipDecision::OpenRelay) | Ok(MembershipDecision::Member) => Ok(None),
Ok(MembershipDecision::ViaOwner(owner)) => Ok(Some(owner)),
Ok(MembershipDecision::Denied) => Err((
StatusCode::FORBIDDEN,
Json(serde_json::json!({
"error": "relay_membership_required",
"message": "You must be a relay member to access this relay"
})),
)),
Err(e) => {
tracing::error!("relay membership check errored: {e}");
Err(super::internal_error(&e))
}
}
}
/// Extract NIP-OA owner from an auth tag without membership enforcement.
///
/// Used on open relays (`require_relay_membership = false`) to opportunistically
/// extract the owner pubkey for agent→owner backfill. The NIP-OA signature is
/// cryptographically self-proving, so no feature flag is needed. Temporal
/// conditions are evaluated against `signed_auth_created_at`. Returns
/// `None` if the tag, timestamp, or conditions are absent or invalid.
pub fn extract_nip_oa_owner(
pubkey_bytes: &[u8],
auth_tag_header: Option<&str>,
signed_auth_created_at: Option<u64>,
) -> Option<nostr::PublicKey> {
let tag_json = auth_tag_header?;
let auth_created_at = signed_auth_created_at?;
let agent_pubkey = nostr::PublicKey::from_slice(pubkey_bytes).ok()?;
match buzz_sdk::nip_oa::verify_auth_tag_for_auth_event(
tag_json,
&agent_pubkey,
auth_created_at,
) {
Ok(owner) => Some(owner),
Err(e) => {
info!("extract_nip_oa_owner: invalid auth tag: {e}");
None
}
}
}
/// Persist a cryptographically verified NIP-OA agent→owner relationship.
///
/// Both principals are ensured first because `agent_owner_pubkey` has a
/// community-scoped foreign key. The mapping is first-write-wins; an
/// existing mapping is accepted only when it names the same owner.
pub async fn materialize_nip_oa_owner(
state: &AppState,
tenant: &TenantContext,
agent: &nostr::PublicKey,
owner: &nostr::PublicKey,
) -> bool {
for (role, pubkey) in [("agent", agent), ("owner", owner)] {
match state
.db
.ensure_user_for_authorization(tenant.community(), pubkey.as_bytes())
.await
{
Ok(true) => {
metrics::counter!(
"buzz_users_created_total",
"community" => tenant.host().to_owned()
)
.increment(1);
}
Ok(false) => {}
Err(e) => {
tracing::warn!(%role, error = %e, "ensure_user failed during NIP-OA backfill");
return false;
}
}
}
let materialized = match state
.db
.set_agent_owner_for_authorization(
tenant.community(),
agent.as_bytes(),
owner.as_bytes(),
)
.await
{
Ok(true) => true,
Ok(false) => state
.db
.is_agent_owner(tenant.community(), agent.as_bytes(), owner.as_bytes())
.await
.unwrap_or(false),
Err(e) => {
tracing::warn!(error = %e, "failed to backfill agent_owner_pubkey");
false
}
};
if materialized {
state
.author_type_cache
.insert((tenant.community(), agent.to_bytes().to_vec()), true);
state.observer_owner_cache.insert(
(
tenant.community(),
agent.to_bytes().to_vec(),
owner.to_bytes().to_vec(),
),
true,
);
}
materialized
}
#[cfg(test)]
mod tests {
use super::*;
use axum::http::{HeaderMap, HeaderValue};
use buzz_sdk::nip_oa::compute_auth_tag;
use nostr::Keys;
#[test]
fn auth_tag_header_must_be_unique() {
let mut headers = HeaderMap::new();
assert_eq!(extract_auth_tag_header(&headers), None);
headers.insert("x-auth-tag", HeaderValue::from_static("credential-one"));
assert_eq!(extract_auth_tag_header(&headers), Some("credential-one"));
headers.append("x-auth-tag", HeaderValue::from_static("credential-two"));
assert_eq!(extract_auth_tag_header(&headers), None);
}
/// Valid NIP-OA auth tag → returns Some(owner_pubkey).
#[test]
fn valid_nip_oa_returns_owner() {
let owner_keys = Keys::generate();
let agent_keys = Keys::generate();
let agent_pubkey = agent_keys.public_key();
let tag_json = compute_auth_tag(&owner_keys, &agent_pubkey, "")
.expect("compute_auth_tag must succeed");
let result = extract_nip_oa_owner(
&agent_pubkey.to_bytes(),
Some(&tag_json),
Some(nostr::Timestamp::now().as_secs()),
);
assert_eq!(result, Some(owner_keys.public_key()));
}
#[test]
fn nip_oa_time_conditions_use_signed_auth_event_time() {
let owner_keys = Keys::generate();
let agent_pubkey = Keys::generate().public_key();
let expired = compute_auth_tag(&owner_keys, &agent_pubkey, "created_at<200")
.expect("sign expired credential");
assert_eq!(
extract_nip_oa_owner(&agent_pubkey.to_bytes(), Some(&expired), Some(200)),
None
);
let future = compute_auth_tag(&owner_keys, &agent_pubkey, "created_at>200")
.expect("sign future credential");
assert_eq!(
extract_nip_oa_owner(&agent_pubkey.to_bytes(), Some(&future), Some(200)),
None
);
let in_window = compute_auth_tag(
&owner_keys,
&agent_pubkey,
"kind=9&created_at>199&created_at<201",
)
.expect("sign in-window credential");
assert_eq!(
extract_nip_oa_owner(&agent_pubkey.to_bytes(), Some(&in_window), Some(200)),
Some(owner_keys.public_key())
);
assert_eq!(
extract_nip_oa_owner(&agent_pubkey.to_bytes(), Some(&in_window), None),
None,
"a credential without a verified signed auth timestamp must fail closed"
);
}
/// No auth tag → returns None.
#[test]
fn no_auth_tag_returns_none() {
let agent_keys = Keys::generate();
let agent_pubkey = agent_keys.public_key();
let result = extract_nip_oa_owner(
&agent_pubkey.to_bytes(),
None,
Some(nostr::Timestamp::now().as_secs()),
);
assert_eq!(result, None);
}
/// Invalid auth tag → returns None.
#[test]
fn invalid_auth_tag_returns_none() {
let agent_keys = Keys::generate();
let agent_pubkey = agent_keys.public_key();
let result = extract_nip_oa_owner(
&agent_pubkey.to_bytes(),
Some("not valid json"),
Some(nostr::Timestamp::now().as_secs()),
);
assert_eq!(result, None);
}
}
}