mirror of
https://github.com/block/buzz.git
synced 2026-10-27 04:30:52 +00:00
6d07a29137
## Why
The usage poller pays memory and query costs proportional to community
count even when per-community emission is disabled. Every pod loads the
full host map, and the leader materializes all per-community aggregates
before suppressing their labels. That makes telemetry collection itself
a scaling constraint.
## What changed
- Fleet-only usage metrics are now the default. `all` remains available
as an explicit rollback mode.
- In fleet-only mode, the poller skips the community host map and all
community-sized query results.
- Exact community counting is replaced with the PostgreSQL planner's
table-size estimate.
- Stock totals are collected as one fixed row, scanning each stock table
once per hour.
- 1d/7d/30d active-user totals come from one bounded 30-day scan per
day.
- Fleet database snapshots run only on a proved read replica, with a
telemetry-specific freshness budget and server-side statement timeouts.
If no such replica is available, the collection is skipped; it never
falls back to the writer.
- Successful fixed-row snapshots are cached and re-emitted on each poll
tick, so gauges stay scrape-visible without raising query frequency.
- `buzz_usage_snapshot_available{family}` reports whether the latest due
refresh of that family succeeded. After a skipped or failed refresh, the
last values stay visible (with `buzz_usage_snapshot_age_seconds`) but
the family reads unavailable until a 60-second retry succeeds.
- The `all` rollback collector publishes the same availability gauges,
so dashboards gated on availability keep working during a rollback.
- Each due refresh outcome is applied by `apply_stock_refresh` /
`apply_activity_refresh`, the single production mapping from `Ok(Some)`
/ `Ok(None)` / `Err` to cache, availability, skip counter, and retry;
the regression test drives that mapping directly.
- `buzz_usage_snapshot_age_seconds` is emitted in the `all` rollback
too, so age-based alerts keep firing through a rollback.
- Every fleet telemetry attempt that completes lands in
`buzz_db_route_decision`; a query error on a proved reader records
`skipped/replica_error`. An attempt abandoned at the relay deadline
records no route decision and appears only as
`buzz_usage_query_skipped_total{reason="timeout"}`. The shared
`proved_reader` warnings no longer claim "routing to writer", because
the telemetry caller never does (serving callers still record their
writer decision in the route counter).
- The PostgreSQL production-method matrix now exercises
`usage_fleet_stock_snapshot`, so the new `reader/maintenance` pool pair
is proven to be emitted by a real Db method.
- Each fleet collection, reader proof included, runs under a relay-side
`tokio::time::timeout` (10s stock, 20s activity, both above their
server-side `statement_timeout`). A reader that stops answering expires
into the same failure mapping with
`buzz_usage_query_skipped_total{reason="timeout"}`, so it cannot stall
the leader's poller, hold the advisory lock past a liveness check, or
leave a frozen snapshot reporting available.
- Fleet telemetry reader checkouts (`ReaderOperation::Maintenance`) opt
into SQLx `close_on_drop` before their first post-acquire await, so an
abandoned collection's connection is closed within SQLx's 5s
`CLOSE_ON_DROP_TIMEOUT` instead of going through the unbounded
return-to-pool ping while it holds a slot in the reader pool that
serving reads share. Serving reads keep pooled reuse; the cost is one
reader reconnect per fleet refresh.
- On leader demotion, cached snapshots and availability are cleared, and
a fresh collection is forced after the lock is reacquired.
- Fleet-only mode passes no host map to the worker-snapshot storage
emitter. It therefore omits community attribution instead of reporting
every byte as `buzz_storage_unmapped_community_bytes`.
- Lifetime stored-message and active-channel totals are no longer
queried in fleet-only mode. Event throughput remains available through
the existing stored-event counter.
- No test-selector changes: main's `config::tests::` selector in `just
test-unit` and the `run-tests.sh unit` fallback already runs the new
replica-budget config test, and the new fleet schedule, availability,
and storage tests run through the existing bin `tests::` and
`storage_sweep::tests::` selectors.
- Added seeded PostgreSQL contract tests and operator docs covering
rollout, rollback, availability, the no-replica behavior, and leader
election.
No migrations, tables, indexes, or new persistent storage are added.
### What got simpler
The relay-side storage half of the original branch (totals-only S3
listing and `buzz_storage_community_breakdown_available`) was dropped.
Main moved S3 listing out of the relay in #7543 and #7845, so that code
was dead on arrival. The rework adds no storage code path; it only gates
attribution on the presence of a host map.
## Risk
Medium. The default changes which telemetry is emitted:
- In fleet-only mode, `buzz_communities_estimated` replaces the exact
`buzz_communities_total`.
- `buzz_total_messages` and `buzz_total_active_channels` are not
refreshed in fleet-only mode (also called out in the rollout steps of
`docs/usage-metrics.md`).
- Deployments without a verified reader (including local compose and
single-node self-hosts) skip database-derived fleet snapshots instead of
querying the writer, and report both families unavailable.
- Multi-pod dashboards must filter database and storage series to
`buzz_usage_poller_is_leader == 1`.
Setting `BUZZ_USAGE_METRICS_PER_COMMUNITY=all` restores the legacy
collector for temporary rollback.
## Verification
At head `1879ee3aedf4227942120a3b521c8d35fc7839fe` (tree `78477e74`;
merges main `da651ad6`; the PR's diff against main is byte-identical to
`1613823f`'s, minus the Justfile and `run-tests.sh` selector lines that
main's `config::tests::` selector now subsumes):
- On Blox: `cargo fmt --all -- --check` and `cargo clippy -p buzz-db -p
buzz-relay --all-targets -- -D warnings` pass. Relay bin `tests::`
34/34, `config::tests::` 55/55, `storage_sweep::tests::` 10/10 and
`store::usage::tests` pass, as do the four `fleet_` and lock PostgreSQL
tests that use scratch databases, including
`abandoned_fleet_collection_releases_its_reader_slot`.
- Against a freshly migrated PostgreSQL database:
`store::usage::postgres_tests::` 14/14,
`storage_sweep::postgres_tests::` and `buzz-db --test
observability_source` 14/14 pass. The new reader code passes main's
crate-wide typed-acquisition and serving-table policy checks.
- E2E at `1613823f` is reused for this head. It ran in an isolated relay
against a real PostgreSQL primary and streaming standby. In 2/2 runs a
stock read went dark mid-result; the deadline abandoned it at about 10s,
the reader pool's active and size counts dropped to zero about 1s later,
and a fresh activity read and the scheduled stock retry succeeded on new
backends while the dark session stayed open. The `all`-mode and
healthy-replica controls also passed. Main's merged changes are on the
writer path and in startup step timing, and touch none of the reader
checkout, the `close_on_drop` seam, the fleet poller or the deadlines.
At head `1613823fa8ef1a423bc1a499c57bc40d692fcd4f` (tree `12bc6d1b`;
test/docs-only on top of `57749418`), on a Blox workstation:
- `cargo fmt --all -- --check` and `cargo clippy -p buzz-db -p
buzz-relay --all-targets -- -D warnings` pass. Relay bin `tests::` 34/34
pass.
- New PostgreSQL test
`abandoned_fleet_collection_releases_its_reader_slot`: a one-slot reader
pool behind a loopback proxy that goes dark mid-session; the collection
is cut at a deadline, then a later acquire must succeed in under 8s on a
fresh connection. The dark call runs on a fresh `Db` with a cold Aurora
capability cache, so the first post-acquire await is the capability
probe. It passes in 3.4s. With `close_on_drop` removed, or moved below
the probe await, it fails with `PoolTimedOut` after the 15s acquire
timeout. The other three `fleet_` PostgreSQL tests that use scratch
databases pass.
- `each_fleet_family_is_abandoned_at_its_own_deadline` now has an outer
timeout: replacing the stock deadline with a bare `.await` fails it
(`Elapsed`) instead of hanging.
At head `fd194ec79d6c6796d6764564de225ad56485cfc7` (tree `79e94727`,
main merged at `932228936`), on a Blox workstation:
- `cargo fmt --all -- --check` and `cargo clippy -p buzz-relay -p
buzz-db --all-targets -- -D warnings` pass.
- Relay bin `tests::` nextest selection: 27/27 pass (plus
`store::usage::tests`), including
`fleet_refresh_outcomes_drive_availability_through_production_mapping`
and the new `stalled_fleet_collection_times_out_into_failed_refresh`
(paused time: two never-completing collections are abandoned at exactly
10s + 20s; both families read unavailable with cached values kept,
`reason="timeout"` skips, retry scheduled; the next tick runs and
recovers; no route decision is recorded for an abandoned attempt) and
`each_fleet_family_is_abandoned_at_its_own_deadline`. Compile-time
asserts keep each deadline above its `statement_timeout`.
- Falsifiability: replacing the stock deadline with a bare `.await`
makes the new test fail (the tick stalls); swapping the two deadlines
fails the per-family test; deleting `schedule.mark_stock_failure(now)`
from the `Ok(None)` arm fails the mapping test. Trees restored and
re-verified.
- At `970414db`: CI PostgreSQL Tests 809/809 including the
production-method matrix with `reader/maintenance`; Unit Tests ran
`finish_usage_read_records_every_attempt_in_route_decisions`.
- `just ci` / `just test` at `5b18914a` on Blox: not green for
environmental reasons (`mobile-check` could not reach pub.dev;
`_ensure-services` failed so service-backed suites did not run;
`buzz-acp git_bootstrap` fails identically on `origin/main`). PostgreSQL
and integration coverage comes from GitHub CI at this head.
E2E at the previous head `0a01e5f22` (isolated relay, real Postgres
primary plus streaming replica, worker storage snapshot, desktop
frontend in relay mode): with no reader, both families report
skipped/unavailable and route `skipped/disabled` (no writer queries);
fleet-only storage emits totals without `buzz_community_storage_*` or
unmapped bytes; `all` mode reports both availability gauges `1` with
per-community storage series; with a fresh replica, both fleet paths
route `replica/fresh` on `reader/maintenance` acquisitions; desktop
send/receive works. The cached success → due failure → recovery
transition was not reached live (hourly/daily cadence); it is covered by
the seam test above.
The original branch also recorded live verification at `aa3e7023`,
before this rework: reduced-idle cache re-emission, two-relay leader
takeover with demoted availability clearing, the no-reader skip with
60-second retry, `all` rollback restoring per-community series, and a
desktop smoke test. Those runs predate the availability and storage
changes and are not claimed for this head.
## Linear tracking
Related work: BUZZ-167.
---------
Signed-off-by: tornquist <tornquist@squareup.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>