mirror of
https://github.com/block/buzz.git
synced 2026-11-14 13:21:02 +00:00
359839f70b
## Why
On one deployment,
`buzz_db_advisory_lock_wait_seconds{lock_type="replacement"}` added up
to about 245,000 seconds of wait in 7 days. The metric cannot say which
coordinates wait. Replacement locks serialize writers of one `(kind,
pubkey[, d_tag | channel])` coordinate. Relay-owned lists (kinds 13535,
39000–39002) go through the same generic replacement paths as user
replaceables (kinds 0, 3, 10002, 30078, …). The 13534 membership list
takes its coordinate lock under `lock_type="membership"`. Before we
change how these lists are stored, we need to know which kinds the wait
comes from.
## What
- `buzz_db_advisory_lock_wait_seconds` and
`buzz_db_advisory_lock_acquisitions_total` get a `kind` label.
- Locks keyed by an event coordinate record that coordinate's kind,
through the new `observe_event_lock(lock_type, kind, …)`. This covers
all 4 replacement call sites (parameterized replace,
`take_replacement_lock` for addressable replace and group-state
snapshots, the kind:39002 member snapshot, and workflow deletion) and
the kind:13534 membership-list lock. The 13534 lock keeps
`lock_type="membership"`, so totals by `lock_type` do not change.
- All other locks record `kind="none"`, and their call sites do not
change. `observe_advisory_lock` has a `debug_assert` that rejects
`LockType::Replacement`, so a replacement lock cannot lose its kind by
mistake.
- `buzz_core::kind::registered_kind_label(kind)` gives the label value:
the decimal kind when it is in `ALL_KINDS`, otherwise `other`.
## Change to the #6700 label contract
#6700 says no database metric carries event kinds, and
`observability_source.rs` enforces this with a blanket ban on `"kind"
=>`. This PR makes one narrow exception: the two advisory-lock metrics.
- **Bounded:** the value set is the kind registry, not request data. Of
the 135 registered kinds, 39 are replaceable or addressable and can
reach a replacement lock. Any other kind a client sends becomes `other`.
Non-replacement locks are always `none`. New series in the worst case:
39 kinds × 4 outcomes, for the histogram and the counter.
- **Not sensitive:** no community, pubkey, coordinate, or d-tag is
attached. The kind of a replacement lock does not expose any of those.
- **The guard is narrowed, not removed:** `"kind" =>` must appear
exactly twice, inside the advisory-lock observer `observe_lock`, whose
kind must come from `registered_kind_label` inside `observe_event_lock`.
It is still forbidden everywhere else in `observability.rs` and in the
datastore tracing macro. The #7195 guard that forbids a literal
`"other"` in `observability.rs` is unchanged, because the fallback lives
in `buzz-core`.
## Dashboards and alerts
Both metrics gain a label, so existing selectors keep working:
`lock_type="replacement"` still matches every replacement series,
because Prometheus and Datadog selectors match on a subset of labels. A
consumer is affected only if it depends on the exact label set:
- Recording rules or panels that read a series without aggregating it
now get one series per kind. Aggregate with `sum by (lock_type, outcome)
(...)` to keep the old shape.
- Vector matching with `on(lock_type, outcome)` or `ignoring(...)` now
has more series on one side. Aggregate before the join.
To read the new attribution, group by `kind`, for example `sum by
(lock_type, kind) (rate(buzz_db_advisory_lock_wait_seconds_sum[5m]))`.
No consumer in this repository depends on the exact label set.
## Tests
- `buzz-core`: `registered_kind_label_bounds_unregistered_kinds` checks
that a registered kind passes through and that an unregistered or
negative kind becomes `other`.
- `buzz-db` recorder test: checks the replacement series for a
registered kind (`13534`) and an unregistered kind (`other`), and the
`kind="none"` series for a membership lock, on both metrics.
- `kindless_observer_rejects_replacement_locks`: the `debug_assert`
guard panics.
- `membership_list_lock_records_the_roster_kind` (Postgres): a 13534
list write records exactly one `membership` acquisition with
`kind="13534"`.
- `group_state_replacement_takes_its_lock_once` (Postgres): now counts
acquisitions only under the written kind. It expects `kind="39000"` for
the group-state snapshot path and `kind="0"` for
`replace_addressable_event`, so if `take_replacement_lock` drops or
swaps the kind, the count becomes 0 and the test fails.
- `observability_source.rs`: the narrowed guard described above.
Commands run:
- `cargo clippy --workspace --all-targets -- -D warnings`
- `cargo test -p buzz-core --lib`
- `cargo test -p buzz-db --lib`
- `cargo test -p buzz-db --test observability_source`
- `scripts/postgres-test-run.sh -p buzz-db --lib --tests` (the CI
PostgreSQL lane, run locally)
## Local relay check
The relay ran from this branch against an isolated database. `/metrics`
showed one series per locked kind after each write:
- `users set-profile`: `replacement`, `kind="0"`
- `users set-status`: `replacement`, `kind="30315"`
- `channels create`: `replacement`, `kind="39000"`, `"39001"`, `"39002"`
- `channels add-member`: `replacement`, `kind="39002"`
- `agents archive`: `replacement`, `kind="13535"`
- `moderation ban`: `replacement`, `kind="30622"`, plus `membership`,
`kind="none"`
- Invite mint and claim
(`e2e_relay::test_invite_mint_and_claim_admits_new_pubkey`):
`membership`, `kind="13534"`
- Background loops: `deletion` and `partition_maintenance`,
`kind="none"`
`buzz_db_advisory_lock_wait_seconds_count` matched
`buzz_db_advisory_lock_acquisitions_total` for every series. The author
also tested it against the local relay.
buzz-review-completed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>