jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-10-29 13:31:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,741 Commits 1,216 Branches 239 Tags
c237fd4640bcd3a24fccfe519195ea8564abf16a
Commit Graph

3 Commits

Author SHA1 Message Date
ravarora2
91ab9d31b8 Add operation-aware database pool acquisition metrics (#7195) 
## Why

Buzz already reports coarse writer/reader database checkout waits, but
those
signals cannot explain which startup or serving operation is blocked by
pool
pressure. That makes rollout diagnosis and postmortems ambiguous: a
readiness
probe, NIP-42 authentication, authorization check, reconnect history
repair,
event write, and background maintenance can all wait on the same pool
while
appearing identical.

This PR implements Package 2A of the pod-handoff plan: operation-aware
pool
borrow causality. It is observability-only; it does not change
configured pool
sizes, SQL semantics, transaction ordering, or timeout policy. Physical
DNS/TCP/TLS/Postgres authentication and session initialization remain
the
separate Package 2B boundary.

## Metric contract

The final contract separates three questions:

| Question | Metric |
|---|---|
| How long did checkout wait? |
`buzz_db_pool_acquire_duration_seconds{pool_role,operation}` |
| How did the attempt end? |
`buzz_db_pool_acquire_attempts_total{pool_role,operation,outcome}` |
| Who is waiting now for a tracked operation? |
`buzz_db_pool_waiters{pool_role,operation}` |

Outcomes are `success`, `timeout`, `error`, and `cancelled`. Operations
are
`bootstrap`, `readiness`, `tenant_resolution`, `authentication`,
`authorization`, `subscription_history`, `event_write`, and
`maintenance`.

Only these eleven pool/operation pairs are constructible:

```text
writer/bootstrap                 reader/bootstrap
writer/readiness
writer/tenant_resolution
writer/authentication
writer/authorization             reader/authorization
writer/subscription_history      reader/subscription_history
writer/event_write
writer/maintenance
```

The duration histogram intentionally does not carry `outcome`. Result
data
remains available on the terminal counter for historical counts and
rates,
without multiplying the expensive histogram family. Nine finite buckets
plus
`+Inf`, sum, and count produce 12 duration series per valid pair.
Together with
four outcome counters and one waiter gauge, the new-family ceiling is
exactly
187 raw Prometheus series per pod, asserted from the production
exporter.

The existing coarse acquisition families remain temporarily for
dashboard
compatibility while the new series are validated in staging.

The operation-specific waiter family covers the explicitly routed,
deployment-critical operations above; it is not a census of every
possible
SQLx checkout in the process. The dashboard pairs it with SQLx pool
active/idle/max gauges for whole-pool capacity context, and treats a
missing
operation series as unknown rather than healthy zero.

## What changed

### Cancellation-safe acquisition ownership

- Add writer- and reader-specific typed operation APIs so invalid label
pairs
  cannot be constructed and store modules cannot emit reader labels.
- Own every polled acquisition with one RAII terminal guard.
- Record exactly one duration and terminal outcome for success, timeout,
error,
  or cancellation.
- Emit nothing for a future that is created but never polled.
- Balance the operation-specific waiter count exactly once on every
terminal or
  dropped future.
- Periodically refresh every expected waiter pair, including healthy
zero, so
missing telemetry is not presented as zero. Reader pairs are emitted
only
when a distinct read pool is configured; a writer-only pod cannot
fabricate
  healthy reader-zero state.

### Production attribution

Route the deployment-critical acquisition paths through caller-owned
semantic
entry points, including:

- writer and reader bootstrap;
- the real post-#7149 readiness acquisition and deletion-catalog
validation;
- tenant resolution and community lifecycle checks;
- NIP-42 allowlist authentication;
- membership, moderation, invite, operator, Git, agent-owner, and policy
  authorization;
- operator community create/list/archive/unarchive, reverse host/channel
tenant
  resolution, and REQ row-community conformance lookups;
- writer/reader subscription history, feed, thread, and routed fallback
paths;
- primary and command event writes, replaceable events, mention
indexing,
  reaction/channel/member/archive side effects, and thread metadata;
- push matching, usage rollups/leadership, replica-fence startup and
recurring
probes, periodic reconciliation, channel/deletion reapers, partitions,
and
  other bounded maintenance/bootstrap paths.

Shared helpers now accept caller-owned intent or expose named semantic
variants
instead of assigning one misleading operation to every caller. No known
P0 path
uses `other`.

### Readiness and size-one-pool correctness

- Rebase on the post-#7149 readiness implementation and instrument the
actual
`Db::readiness_check` acquisition rather than the superseded ping-only
seam.
- Acquire once for deletion-catalog validation and run its queries on
that
  connection, preserving the shared readiness deadline.
- Scope the channel-roster catalog checkout before the behavior probe so
a
  size-one writer pool cannot self-deadlock during startup verification.

### Exporter, documentation, and CI

- Register metric HELP/type/unit metadata through the production
Prometheus
  builder.
- Configure dedicated checkout buckets at 1ms, 5ms, 10ms, 25ms, 50ms,
150ms,
  500ms, 1s, and 3s.
- Add a production scrape-contract test for exact names, labels,
buckets,
  valid pairs, sensitive-label exclusion, and the 187-series ceiling.
- Add source mutation guards for the P0 semantic entry points and
raw-checkout
  bypasses.
- Add an exact backend-integration CI selector for the production
attribution,
  cancellation, readiness, and size-one-pool PostgreSQL tests.
- Document the frozen label vocabulary, valid combinations, semantics,
and
  cardinality budget in the Helm chart README.

## Dashboard intent

The new Stage 2 row in **Buzz Startup & Rollout Safety** is
deployment-first:

- baseline-versus-candidate attempts, failure rates, cancellation rates,
and
  maximum wait by operation;
- outcome counts and percentages over time by SHA/ReplicaSet;
- acquisition wait heatmap, average, and maximum through the rollout;
- historical waiter pressure beside writer active/idle/max context;
- per-pod postmortem drilldown, including terminated pods;
- a smaller current-waiter table with explicit stale/missing semantics.

Percentile widgets remain disabled until Datadog metadata confirms
percentile
support for the new distribution. Current gauges use no fill,
interpolation, or
`default_zero`; missing means unknown.

## Risk assessment

Moderate. The patch touches many database acquisition call sites, but
preserves
the selected physical pool and executes the same SQL on the acquired
connection. The main risks are incorrect semantic attribution,
cancellation
double-counting, and a helper accidentally acquiring twice. Typed APIs,
production-method PostgreSQL tests, source guards, the raw scrape
contract, and
the size-one-pool regression cover those risks.

No tenant, community, user, pubkey, event, channel, SQL, URL, pod,
version,
ReplicaSet, or request-controlled value is emitted as an application
metric
label. Deployment identity is supplied by infrastructure enrichment.

## Verification

- `cargo fmt --all -- --check` — passed.
- `cargo clippy -p buzz-db -p buzz-relay --all-targets --all-features --
-D warnings`
  — passed.
- `cargo test -p buzz-db` — 122 passed, 0 failed, 263 ignored;
  source-contract integration test: 3 passed, 0 failed.
- Focused relay compatibility, metric-contract, and readiness tests —
passed.
- `scripts/test-postgres-test-discovery.sh` — passed.
- Full `buzz-relay` package run from the identical tree reached 1,015
passes;
the six media-test failures all stopped in their shared local PostgreSQL
setup with `Sqlx(PoolTimedOut)` because Docker/PostgreSQL was
unavailable.
The same six failed in isolation, while every changed exact test passed.
- Exact implementation head:
  `f92910b353086e9edf85918ca5f72190edbbe22f`.
- Exact multi-architecture staging image:
  `dev-sha-f92910b353086e9edf85918ca5f72190edbbe22f-run-33607968668-1`

(`sha256:161712c8ed2e265a15df9b63e02248d5973481f875ff129d7d2ae78a09d487a2`).
- Focused staging GitOps PR:

<https://github.com/squareup/builderbot-platform-core-infrastructure/pull/299>
— merged after renderer, inventory, infrastructure test, Kargo, Semgrep,
and
Intersect gates passed; the source/generated-artifact diff was exactly
two
  image lines.
- Exact GitHub head reports 47 terminal checks: 35 successful and 12
  intentionally skipped. PostgreSQL, unit, lint, security, both server
cross-compiles, backend integration, relay E2E, desktop, mobile, image,
  Helm, Semgrep, zizmor, and DCO gates are green.
- Datadog readback identifies two exact-image pods,
  `buzz-d79c8d8f7-ckv2l` and `buzz-d79c8d8f7-qzqdp`, in ReplicaSet
  `buzz-d79c8d8f7`; both report the full source SHA above.
- Both pods report all eleven allowed pool/operation waiter pairs at
current
zero, with no invalid pair. The acceptance window observed nonzero
success
  receipts for readiness, tenant resolution, authorization, subscription
history, event write, and maintenance, and no timeout, error, or
cancelled
outcome. Maximum observed wait was about 101 ms for maintenance and 50
ms
  for reader subscription history.

The main **Buzz Startup & Rollout Safety** dashboard now has a live
Stage 2
database row with eight widgets and nineteen fully scoped queries. Final
readback preserved all seven top-level groups, found zero under-scoped
Row 6
queries, and confirmed the tracked-operation waiter boundary in the
panel
descriptions.

Generated with Codex.

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>
 2026-09-02 17:04:34 -04:00
Luke Tornquist
a3730784fc refactor(db): extract domain stores from database runtime (#6987) 
## Summary

Finish the remaining database-store extraction tracked by
[TheSentinel454/buzz#2](https://github.com/TheSentinel454/buzz/issues/2)
in one reviewable PR.

This consolidates the previously stacked domain slices after #6782
merged. It preserves the runtime/store boundary established by #6660,
#6668, #6700, and #6782 while separating database runtime infrastructure
from domain-owned persistence:

- `runtime/` owns pool construction and sizing, writer/reader routing,
read sessions and route proofs, transaction infrastructure,
observability primitives, replica fencing, health support, migrations,
and cross-cutting runtime tests.
- `store/` owns domain records, SQL, row parsing, locks and invariants,
`Db` domain methods, focused tests, and logical-operation datastore
spans.
- `lib.rs` remains a 57-line compatibility facade that preserves
existing crate-root paths and `Db` method signatures through re-exports.

Domain coverage includes API tokens, authentication allowlists,
reminders, event queries, threads, reactions, feeds, users and DMs,
push, workflows/runs/approvals, relay membership and invites, product
feedback, moderation/admin moderation, relay admin actions/operators,
git repositories, archived identities, usage, partition maintenance,
deletion, channel membership inherited from merged #6782, and the final
runtime/store layout.

The branch has been rebased onto current `main`. Database changes that
landed there were incorporated rather than overwritten:
`relay_admin_actions.rs` and `relay_operators.rs` now live under
`store/`, their 27 public `Db` wrappers and existing behavior remain
intact, and every wrapper has exactly one fixed-name datastore span.
Concurrent changes to migration, moderation, admin moderation, and error
handling are also retained.

### Exact base and head

- Base: `main` at `ed11c8d8bf0a17402be5cf243724f89471530d2f`
- Head: `codex/issue-2-store-extraction` at
`be24430472d1a87ac5c0d6026c620cd6caea3537`

### Related issue

- Structural tracker:
[TheSentinel454/buzz#2](https://github.com/TheSentinel454/buzz/issues/2)
- Domain trackers:
[#6](https://github.com/TheSentinel454/buzz/issues/6),
[#7](https://github.com/TheSentinel454/buzz/issues/7),
[#12](https://github.com/TheSentinel454/buzz/issues/12),
[#13](https://github.com/TheSentinel454/buzz/issues/13)
- Acceptance trackers:
[#17](https://github.com/TheSentinel454/buzz/issues/17),
[#19](https://github.com/TheSentinel454/buzz/issues/19)

This supersedes #6783, #6784, #6787, #6788, #6789, #6792, #6820, #6794,
#6796, #6797, #6798, #6799, #6804, #6805, #6806, #6808, #6809, #6811,
#6812, #6813, #6814, #6815, and #6890. Their discussions remain
available for review history.

### #17 / #19 acceptance

- Preserves the metric names, fixed labels, transaction/lock timing
boundaries, and privacy/cardinality constraints introduced by #6700.
- Keeps exactly one datastore span per public logical operation,
including the 27 relay-admin wrappers added on `main`.
- Removes `store_ownership.rs`; physical ownership and focused source
guards now enforce the boundary directly.
- Leaves no `impl Db`, domain SQL, focused domain test group, or
datastore span in `lib.rs`.
- Preserves existing public paths such as `buzz_db::channel`,
`buzz_db::event`, and `buzz_db::workflow` through crate-root re-exports
while keeping internal `runtime` and `store` namespaces private.

### Non-goals

- No SQL, schema, locking, transaction, retry, timeout, or
client-visible behavior changes.
- No generic store traits, domain handles, broad `PgExecutor` migration,
new store crate, raw pool accessor, or broader directory reorganization.
- No tracker issues are closed by this PR.

### Risk

The cumulative diff is large but structural. Risk is primarily
module-path, ownership, or conflict-resolution drift. It is mitigated by
preserving public re-exports, comparing the newly moved `main`
implementations to their upstream source, source guards, touched-crate
compilation, PostgreSQL-backed test coverage, and an independent
exact-head review on a separate clean Blox workstation.

### Testing

Author workstation `buzz-tornquist-pr-6987-rebase`, rebased branch
ending at exact head `be24430472d1a87ac5c0d6026c620cd6caea3537`:

- `cargo fmt --all --check`
- `cargo clippy -p buzz-db -p buzz-relay --all-targets -- -D warnings`
- `cargo test -p buzz-db --lib` — 113 passed, 240 PostgreSQL tests
intentionally ignored
- `cargo test -p buzz-db --test observability_source` — 2 passed
- PostgreSQL-backed `buzz-db` coverage under native PostgreSQL — 235
passed in the shared serial run; the five shared-state/config-sensitive
cases passed as isolated reruns against fresh schemas, including the two
owner-limit tests with their fixture's
`BUZZ_MAX_COMMUNITIES_PER_OWNER=3`
- `cargo test -p buzz-relay --lib -- --test-threads=1` under native
PostgreSQL/Redis — 991 passed; the three current-month
partition-sensitive identity-archive cases passed after provisioning the
August 2026 test partition; 87 infrastructure-marked tests remained
ignored
- Source/diff guards — relay-admin implementation bodies match current
`main`; all 27 public wrapper signatures are retained; exactly one
datastore span wraps each wrapper; `lib.rs` has zero `impl Db` blocks
and zero datastore spans; no duplicate top-level relay-admin modules or
`store_ownership.rs`; `error.rs` matches current `main`

Independent clean review workstation `buzz-tornquist-pr-6987-review`,
detached at exact head `be24430472d1a87ac5c0d6026c620cd6caea3537`:

- `cargo fmt --all --check`
- `cargo clippy -p buzz-db -p buzz-relay --all-targets -- -D warnings`
- `cargo test -p buzz-db --lib` — 113 passed, 240 ignored
- `cargo test -p buzz-db --test observability_source` — 2 passed
- Exact-head ownership/re-export/instrumentation audit — no remaining
actionable findings

---------

Signed-off-by: OpenAI Codex <codex@openai.com>
Signed-off-by: tornquist <tornquist@squareup.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-28 15:24:32 -04:00
Luke Tornquist
113a33b7e4 Add database pressure observability (#6700) 
## Why

Database pressure currently collapses several distinct delays into one
symptom. This adds the evidence layer needed to distinguish pool
acquisition wait, logical database operation time, advisory-lock wait,
and selected transaction duration before changing timeout or retry
policy.

This is the phase 2 Lane A observability bundle for
[#26](https://github.com/TheSentinel454/buzz/issues/26),
[#28](https://github.com/TheSentinel454/buzz/issues/28), and
[#33](https://github.com/TheSentinel454/buzz/issues/33). It is stacked
on #6668.

## What

- Record explicit reader/writer checkout wait and acquisition outcomes
with `buzz_db_pool_acquire_wait_seconds` and
`buzz_db_pool_acquisitions_total`.
- Extend the compile-time `#[datastore_span(name = "...")]` seam with
`buzz_db_operation_duration_seconds`, so operation labels remain static
source literals instead of request data.
- Route correctness-critical replacement, membership, push-gate,
deletion, and migration/schema-safety advisory locks through one
observer without changing their SQL, order, scope, or blocking behavior.
- Measure six internally owned transaction lifetimes with
`buzz_db_transaction_duration_seconds`, starting after `BEGIN` succeeds
and ending after explicit commit/rollback or scope exit.
- Emit root slow-operation warnings at 500 ms, logging the first slow
completion and then 1/100 per call site with only `operation`,
`outcome`, and `elapsed_ms`.
- Document names, units, fixed label vocabularies, measurement
boundaries, and blind spots in this PR description.

Fixed labels are deliberately small:

- `pool_role`: `writer`, `reader`
- `lock_type`: `replacement`, `membership`, `push_gate`, `deletion`,
`migration_schema_safety`
- `outcome`: `success`, `error`, `timeout` where SQLx/PostgreSQL can
distinguish it accurately
- `operation`: compile-time datastore names plus the six closed
transaction operation names documented in the runbook

No metric or slow warning contains community IDs, event IDs, event
kinds, coordinates, d-tags, SQL/query text, query IDs, returned errors,
or event content.

## Coverage boundaries

- Operation duration is the complete annotated logical function body,
not pure SQL execution; it may include implicit checkout, lock wait,
nested operations, and application work. Cancelled futures do not reach
its completion hook.
- Pool timing covers explicit helper checkouts, including proved-reader
routing and selected writer-owned transactions. Implicit SQLx checkout
through `&PgPool` remains folded into operation duration.
- Lock timing covers application-side blocking locks in the five named
families. Trigger/stored-procedure locks, channel-TTL locking, the usage
try-lock, and the audit service session lock remain outside this slice.
- Transaction timing covers only the six wholly owned boundaries
documented in the runbook. It excludes pool wait, `BEGIN`, asynchronous
rollback cleanup after an early return, and caller-owned
`Db::begin_transaction` lifetime.

## Relationship to #6229

#6229 is the incident-driven timeout precursor. This PR does not add or
change `statement_timeout`, `lock_timeout`,
`idle_in_transaction_session_timeout`, retries, audit durability, or
client-visible conflicts. It provides the missing distributions needed
to evaluate those policies later and intentionally leaves #6229's open
audit retry/durability finding untouched.

The branches overlap in `crates/buzz-db/src/lib.rs` and
`crates/buzz-db/src/migration.rs`, so a later rebase may need textual
conflict resolution, but the behavior is complementary rather than
duplicated.

## Risk assessment

Moderate-low. The primary risk is instrumentation overhead and added
static series. Cardinality is source-bounded, slow logs are
sampled/redacted root events, and the lock/transaction changes wrap
existing awaits without changing policy or ordering.

## Verification

Author workstation: `buzz-tornquist-db-pressure-observability`
(`2010927`), exact head `d7cf833e26c528adfcde3917ded80daf6f4ddac9`,
parent `6f50e6b2b2a996349149af61d35bdd6a355f77fd`.

- `cargo fmt --all --check` — passed
- `cargo clippy -p buzz-datastore-tracing -p buzz-db -p buzz-audit -p
buzz-search -p buzz-relay --all-targets -- -D warnings` — passed
- `cargo test -p buzz-datastore-tracing --quiet` — 4 passed
- `cargo test -p buzz-db --quiet` — 109 passed, 200 ignored
- `cargo test -p buzz-audit -p buzz-search --quiet` — 16 passed, 25
ignored
- `cargo test -p buzz-relay --lib --quiet -- --test-threads=1` — 906
passed, 48 ignored
- Native PostgreSQL focused tests for pool success/timeout/error, lock
success/contention/timeout/error, replacement, membership serialization,
push ordering, deletion fencing, migration/schema exclusion, and reader
fallback — 8 passed

The default-parallel relay run passed once; subsequent runs exposed the
existing load-sensitive
`api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo` 504 at
the end of the suite. That test passes in isolation and the full relay
suite passes serially.

Independent exact-head review workstation:
`buzz-tornquist-db-pressure-observability-review` (`2013067`).
Formatting, the same all-target clippy command, datastore
instrumentation tests, DB unit tests, source privacy guards, and
diff/non-goal audits passed; no review findings.

Generated with Codex

---------

Signed-off-by: tornquist <tornquist@squareup.com>
 2026-08-25 13:00:22 -04:00