jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-11-05 00:41:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(relay): partition audit observability follow-ups (#7994)

Browse Source 
> [!IMPORTANT]
> **The alert rules changed in c3a314e71.** If you reviewed the alert
guidance earlier, please look again.
> - **The two range-based rules now apply only to live pods.** A range
selector such as `[30m]` keeps reading a terminated pod's samples until
they leave the window. So `BuzzPartitionAuditNotRunning` could fire for
about 10 minutes after normal pod turnover, and
`BuzzPartitionAuditFailing` could keep firing for up to an hour after
the pod was gone. Each rule now adds `and <counter>`, which requires the
counter to be exported right now:
>   ```yaml
>   expr: |
> sum without (outcome) (increase(buzz_partition_audit_runs_total[30m]))
== 0
>       and sum without (outcome) (buzz_partition_audit_runs_total)
>   ```
>   ```yaml
>   expr: |
>     increase(buzz_partition_audit_failures_total[1h]) >= 2
>       and buzz_partition_audit_failures_total
>   ```
> - **The audit counters start at 0.** When the relay starts, it exports
every `buzz_partition_audit_runs_total{table, outcome}` series and
`buzz_partition_audit_failures_total` at 0, before the first audit.
`increase()` ignores a new series' first sample. Before this change, it
could miss the first periodic failure, or read a table's first
`degraded` run as a stopped loop. This change is also what makes the
`and <counter>` gate work: the counters now exist for a relay's whole
life.
>
> The doc's old claim that pod turnover can't leave an alert firing was
true only of the instant-selector stale rule. The doc now explains the
difference.

## Why

These are the observability follow-ups deferred from #6515 ("fix(db):
audit partition catalog before creation"):

- **Timestamp-age alerts go silent** ([review
comment](https://github.com/block/buzz/pull/6515#discussion_r4127212141)).
Partition gauges are set only when a table audits successfully, and the
exporter evicts idle gauges after 2,700 s by default. After sustained
failures, `buzz_partition_audit_last_success_timestamp_seconds`
disappears. A table that has never audited successfully never has it. An
alert on `time() - last_success` then has no input.
- **Startup fallback failures are not counted** ([review
comment](https://github.com/block/buzz/pull/6515#discussion_r4127212158)).
When the startup fallback audit fails, it logs an error but does not
increment `buzz_partition_audit_failures_total`. Periodic failures do.
- **Repaired environments are always `degraded`**.
`PartitionTableAudit::degraded()` counts every legacy leaf, even one
wholly in the past. The `bb-block-staging` audit of the repaired catalog
reports `degraded` only because of `events_p2026_07_08_legacy`
`[2026-07-01, 2026-09-01)`. With this rule, every repaired environment
stays `degraded`, so nobody can alert on the outcome.
- **Partition config tests can see a poisoned env** ([review
comment](https://github.com/block/buzz/pull/6515#discussion_r4137416297)).
Two partition config tests take only `ENV_MUTEX`. A NIP-FI test running
in the same process can make them see its deliberately invalid
environment.

Partition health is still diagnostic only, through
`/_status.partition_catalog` and metrics. Nothing here affects
`/_readiness`. Handling concurrent creation (#6515 item 4) is out of
scope and stays with the catch-all advancement work.

## What

- **Success timestamp survives failures.** Each failed table audit
refreshes `buzz_partition_audit_last_success_timestamp_seconds` with
`increment(0.0)`, which leaves the value unchanged. It still resets the
exporter's idle tracking, the same mechanism as
`refresh_legacy_active_gauge_recency`. Sustained failures therefore keep
the last success exported and its age rising. A table that has never
succeeded exports `0`. A failed audit still never makes the last success
look fresh.
- **Alert guidance.** New `docs/partition-catalog-monitoring.md` covers
the metrics, how long each series lives, and the audit outcomes. It
notes that `runs_total` has no `unsafe` outcome. It gives three alert
rules:
- timestamp age, which also covers never-successful tables and sustained
failures;
- `sum without (outcome)
(increase(buzz_partition_audit_runs_total[30m])) == 0` for a stopped
audit loop. Counters are never idle-evicted, and this rule fires before
the stopped loop's timestamp is evicted, so there is no false resolve;
  - an aggregate failure-counter rule.

Both range-based rules also require their counter to be exported right
now (`and <counter>`), so a terminated pod's leftover samples can't
raise or hold an alert.
- **Zero counter baselines.** `try_install` exports each
`buzz_partition_audit_runs_total{table, outcome}` series (`ok`,
`degraded`, and `error` for both tables) and
`buzz_partition_audit_failures_total` at 0 before the startup audit, the
same way the auth series already start at 0. A failure in the startup
audit itself can still happen before the first scrape. The doc says so;
the stale rule covers that case.
- **Failure counter.** `buzz_partition_audit_failures_total` now counts
failed explicit audit attempts: the startup fallback audit and each
periodic audit. Both paths go through one helper,
`counted_partition_audit`. A failed `ensure_future_partitions` call,
such as a DDL error that the fallback then recovers, is not counted by
itself. A single attempt never counts twice.
- **Historical legacy leaves.** A legacy leaf is historical when its
finite upper bound is at or before the current month start. It is still
reported under `children` as `legacy_leaf`, but it no longer degrades
the table. A legacy leaf that still covers the current month or any
lookahead month stays degraded. All other degradation causes are
unchanged. The verdict now lives in `PartitionAuditReport::outcome()`,
which `buzz-admin partition-audit` uses. The JSON outcome names and exit
codes are the same as before.
- **Test hygiene.** Both partition config tests now use `env_guards()`.

## How it was tested

- `cargo fmt --all -- --check` and `cargo clippy -p buzz-db -p
buzz-relay -p buzz-admin --all-targets -- -D warnings`.
- New unit tests:
- The repaired incident layout with an injected clock: `degraded` in
August 2026 and `ok` on 2026-09-01T00:00Z, when the upper bound equals
the current month start, and on 2026-09-15. The legacy leaf is still in
the serialized report. The `outcome` label on the run counter follows
the verdict.
- A table test of the legacy-leaf predicate: the upper bound before, at,
and one second after the month start; lookahead and beyond-horizon
leaves; historical leaves with trigger drift; anomalous and `DEFAULT`
children; no audited month.
- Outcome precedence (error, then unsafe, then degraded) and the
`buzz-admin` exit-code mapping.
- Startup: a failed fallback counts once, a successful fallback after a
DDL failure counts nothing, and a successful ensure never runs the
fallback. Each periodic failure counts once. Against an unreachable
database, startup counts one aggregate failure while each table still
records two `error` runs.
- On the production exporter, never-successful audits leave run counters
and a `0` success timestamp for both tables.
- On the production exporter, all six run-counter series and the failure
counter read 0 before any audit. After one failed audit, `error` reads 1
and `ok`/`degraded` stay at 0. Removing the initialization fails this
test.
- PostgreSQL lane (`scripts/postgres-test-run.sh`) with new or extended
tests:
- The repaired layout is not degraded in September. The same catalog is
degraded in August.
- With a 1 s idle timeout, a failed audit leaves the success timestamp
unchanged, and sustained failures keep it exported past the idle
timeout. When audits stop, the timestamp is evicted but the run counters
remain. The next success exports a fresh timestamp.
- Mutation check: removing the failure-path refresh fails both alert
fixtures.
- These fixtures check the emitted series. They do not evaluate the
rules themselves, including the `and <counter>` gate. `promtool` isn't
available locally or in CI, and the doc points to `promtool test rules`
for that.
- `scripts/test-postgres-test-discovery.sh`.
- `config::` and `nip_fi_config::` tests run together in one process
five times.

## Related

This PR conflicts textually with #7993 (catch-all advancement) in
`crates/buzz-db/src/store/partition.rs` and
`crates/buzz-relay/src/main.rs`. Whichever PR merges second will be
rebased onto the other.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ian Oberst
2026-10-13 13:57:54 -07:00
committed by GitHub
parent 5dc693aeaf
commit 85b7331784
11 changed files with 1156 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files

150
docs/partition-catalog-monitoring.md Normal file
View File

@@ -0,0 +1,150 @@
# Partition catalog monitoring
The relay audits the `events` and `delivery_log` partition catalogs at startup
and every `BUZZ_PARTITION_AUDIT_INTERVAL_SECS` (default 900 seconds). Results are
diagnostic only. They appear in `/_status.partition_catalog` and in the metrics
below, and they never change `/_readiness` or `/_liveness`. Alert on them; do not
gate traffic on them.
## Outcomes
`buzz-admin partition-audit` prints one outcome for the whole catalog.
| Outcome | Meaning | `buzz-admin` exit code |
|---|---|---|
| `ok` | Every managed table covers now and needs no operator attention. | 0 |
| `degraded` | Every table covers now, but one needs attention. | 0 |
| `unsafe` | A table cannot route a row timestamped now. | 2 |
| `error` | At least one table could not be audited. | 5 |
The relay's `buzz_partition_audit_runs_total{outcome}` is per table and is only
`ok`, `degraded`, or `error`. There is no `unsafe` outcome there: a table that
cannot route a row timestamped now is `degraded` and has
`buzz_partition_serving_safe` 0.
A table is `degraded` when it has a partition-key mismatch, a `DEFAULT` or
anomalous child, trigger drift, an occupied catch-all or `DEFAULT` leaf, or a
current or lookahead month not covered by a bounded leaf. A bounded legacy leaf
with a non-canonical name degrades the table while it covers the current month or
later. Once its upper bound is at or before the start of the current month, it is
historical. It remains in the per-child report as `legacy_leaf` but no longer
degrades the table.
## Metrics
Per-table gauges are set only by a successful audit of that table:
`buzz_partition_audit_last_success_timestamp_seconds`,
`buzz_partition_serving_safe`, `buzz_partition_uncovered_months`,
`buzz_partition_catch_all_covered_months`, `buzz_partition_default_covered_months`,
`buzz_partition_anomalous_children`, `buzz_partition_catch_all_nonempty`,
`buzz_partition_default_nonempty`, `buzz_partition_trigger_parity_missing`, and
`buzz_partition_trigger_parity_extra`. A failed audit leaves them unchanged, so a
failure never makes the last success look fresh.
The exporter removes a gauge that has not been updated within its idle timeout.
The default is 2,700 seconds: three times the larger of the usage-metrics and
partition-audit intervals. A failed audit of a table refreshes that table's
`buzz_partition_audit_last_success_timestamp_seconds` without changing its
value, so sustained failures keep the last success exported and its age keeps
rising. A table that has never been audited successfully exports `0`. The
timestamp disappears only when audits stop running for longer than the idle
timeout; the other per-table gauges can be evicted during sustained failures.
Counters are not idle-evicted while the process runs. The relay exports every
run outcome for both managed tables, and the failure counter, at 0 when its
metrics exporter starts, so `increase()` counts the first failure and the first
run with a new outcome:
- `buzz_partition_audit_runs_total{table, outcome}` increments once per table
per audit. The outcome is `ok`, `degraded`, or `error`, including the opening
audit of each partition maintenance pass.
- `buzz_partition_audit_failures_total` counts failed explicit audit attempts:
the fallback audit that runs when a maintenance pass fails, and each periodic
audit when maintenance does not run. A maintenance failure is not counted by
itself. If the fallback audit then fails, that attempt counts once.
- `buzz_partition_create_attempts_total{table, outcome}` counts per-month
decisions in each maintenance pass: `created`, `error`, or `skipped_covered`
for a month that was already covered. Maintenance runs at startup, and on
every periodic audit when `BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED` is on, so
`skipped_covered` then grows on every pass in steady state.
## Alerts
Use all three rules below. Aggregate with `without (...)`, not `by (...)`, so
cluster, account, environment, namespace, and pod labels stay in the alert.
When a pod terminates, its scrape target disappears and its series go stale.
Instant selectors drop a stale series at once, but range selectors such as
`[30m]` keep reading its earlier samples until they leave the window. Each
range-based rule therefore also requires the counter to be exported now, which
it is for the whole life of a relay, so a terminated pod cannot keep or start an
alert.
`BuzzPartitionAuditNotRunning` also requires the run counter to have existed one
window ago. After a successful startup audit, the first periodic audit runs a
full interval later, so a new pod's counter stays flat until then and would
otherwise read as a stopped loop.
```yaml
# The last success is older than two default audit intervals. This also fires
# for a table that has never audited successfully, whose timestamp is 0, and
# keeps firing through sustained failures. Scale 1800 with
# BUZZ_PARTITION_AUDIT_INTERVAL_SECS.
- alert: BuzzPartitionAuditStale
expr: time() - buzz_partition_audit_last_success_timestamp_seconds > 1800
for: 10m
# The relay is running but its audit loop has stopped. Use a window of about
# two audit intervals, and keep the window plus `for` below the gauge idle
# timeout (2,700 s by default), so this fires before the stopped loop's
# timestamp is evicted and the stale rule resolves. Use the same duration for
# the window and the offset. A loop that stops in a pod's first window is
# caught once the pod is a window old, at about 35 minutes.
- alert: BuzzPartitionAuditNotRunning
expr: |
sum without (outcome) (increase(buzz_partition_audit_runs_total[30m])) == 0
and sum without (outcome) (buzz_partition_audit_runs_total)
and sum without (outcome) (buzz_partition_audit_runs_total offset 30m)
for: 5m
# Explicit audit attempts keep failing, even if a later attempt succeeds.
- alert: BuzzPartitionAuditFailing
expr: |
increase(buzz_partition_audit_failures_total[1h]) >= 2
and buzz_partition_audit_failures_total
```
The counters exist from relay start, and the success timestamp exists after the
first audit attempt, whether or not it succeeded. The startup audit runs within
seconds of the zero baselines, usually before the first scrape, so
`BuzzPartitionAuditFailing` can miss a failure in that one attempt; if it does
not recover, `BuzzPartitionAuditStale` fires. The run counters are exported
before the relay connects to the database, so a live relay that is stuck before
its first audit shows flat zero counters, which `BuzzPartitionAuditNotRunning`
catches.
`degraded` asks for operator attention; it does not mean rows cannot be
written. Historical legacy leaves no longer count, so a repaired catalog
returns to `ok` once the current month passes the repaired range and every
current and lookahead month is a bounded monthly partition. Months still covered
by a catch-all stay `degraded` until advancement moves the catch-all. Coverage
problems show directly in `buzz_partition_serving_safe` and
`buzz_partition_uncovered_months`.
The tests in `crates/buzz-relay/src/metrics.rs` (`partition_alert_tests`) check
that the production exporter and audit emit the series these rules read: zero
counter baselines before the first audit, a `0` timestamp before any success,
the last success kept through failures past the idle timeout, and run counters
that outlive an evicted timestamp. They do not
evaluate the rules, their `for` durations, or label matching.
`partition-catalog-monitoring.rules.test.yml` evaluates the rules in
`partition-catalog-monitoring.rules.yml`, including their `for` durations,
against a fresh pod, a stopped loop, a terminated pod, a table that never
succeeded, and repeated failures. CI does not run it. Keep the rules file
identical to the block above, and run it after changing either:
```bash
promtool test rules docs/partition-catalog-monitoring.rules.test.yml
```
Then validate label matching against your own alerting configuration.

122
docs/partition-catalog-monitoring.rules.test.yml Normal file
View File

@@ -0,0 +1,122 @@
# Run with: promtool test rules docs/partition-catalog-monitoring.rules.test.yml
#
# Series are sampled every minute and use the default 900 s audit interval. The
# startup audit finishes before the first scrape, so run counters start at 1.
rule_files:
- partition-catalog-monitoring.rules.yml
evaluation_interval: 1m
tests:
# A healthy new pod audits at startup and then every 15 minutes. Its counter
# is flat for the first interval, which must not read as a stopped loop.
- interval: 1m
input_series:
- series: 'buzz_partition_audit_runs_total{pod="fresh",table="events",outcome="ok"}'
values: '1x14 2x14 3x14 4x14 5x14'
- series: 'buzz_partition_audit_runs_total{pod="fresh",table="events",outcome="degraded"}'
values: '0x74'
- series: 'buzz_partition_audit_last_success_timestamp_seconds{pod="fresh",table="events"}'
values: '0x14 900x14 1800x14 2700x14 3600x14'
alert_rule_test:
- eval_time: 10m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 14m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 29m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 70m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 70m
alertname: BuzzPartitionAuditStale
exp_alerts: []
# The audit loop stops after the 15-minute audit while the pod keeps running.
# Both rules fire before the timestamp's 2,700 s idle eviction at 60 minutes.
- interval: 1m
input_series:
- series: 'buzz_partition_audit_runs_total{pod="stopped",table="events",outcome="ok"}'
values: '1x14 2x60'
- series: 'buzz_partition_audit_last_success_timestamp_seconds{pod="stopped",table="events"}'
values: '0x14 900x45'
alert_rule_test:
- eval_time: 40m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 52m
alertname: BuzzPartitionAuditNotRunning
exp_alerts:
- exp_labels: {pod: stopped, table: events}
- eval_time: 57m
alertname: BuzzPartitionAuditStale
exp_alerts:
- exp_labels: {pod: stopped, table: events}
# The audit loop stops right after the startup audit. It is caught once the
# pod is one window old.
- interval: 1m
input_series:
- series: 'buzz_partition_audit_runs_total{pod="early",table="events",outcome="ok"}'
values: '1x80'
alert_rule_test:
- eval_time: 33m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 36m
alertname: BuzzPartitionAuditNotRunning
exp_alerts:
- exp_labels: {pod: early, table: events}
# A pod terminates at 24 minutes. Range selectors still read its earlier
# samples, long enough to complete `for: 5m` at 50 minutes, but it must not
# start an alert.
- interval: 1m
input_series:
- series: 'buzz_partition_audit_runs_total{pod="gone",table="events",outcome="ok"}'
values: '1x14 2x9 stale'
- series: 'buzz_partition_audit_failures_total{pod="gone"}'
values: '0x10 1 2 3x7 stale'
alert_rule_test:
- eval_time: 45m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 52m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 60m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 30m
alertname: BuzzPartitionAuditFailing
exp_alerts: []
# A table that has never audited successfully exports a 0 timestamp.
- interval: 1m
input_series:
- series: 'buzz_partition_audit_last_success_timestamp_seconds{pod="never",table="events"}'
values: '0x60'
alert_rule_test:
- eval_time: 35m
alertname: BuzzPartitionAuditStale
exp_alerts: []
- eval_time: 45m
alertname: BuzzPartitionAuditStale
exp_alerts:
- exp_labels: {pod: never, table: events}
# One failure does not alert; a second within an hour does.
- interval: 1m
input_series:
- series: 'buzz_partition_audit_failures_total{pod="once"}'
values: '0x10 1x50'
- series: 'buzz_partition_audit_failures_total{pod="twice"}'
values: '0x10 1x10 2x40'
alert_rule_test:
- eval_time: 40m
alertname: BuzzPartitionAuditFailing
exp_alerts:
- exp_labels: {pod: twice}

20
docs/partition-catalog-monitoring.rules.yml Normal file
View File

@@ -0,0 +1,20 @@
# Alert rules from docs/partition-catalog-monitoring.md. Keep them identical to
# the doc; partition-catalog-monitoring.rules.test.yml tests them.
groups:
- name: buzz-partition-catalog
rules:
- alert: BuzzPartitionAuditStale
expr: time() - buzz_partition_audit_last_success_timestamp_seconds > 1800
for: 10m
- alert: BuzzPartitionAuditNotRunning
expr: |
sum without (outcome) (increase(buzz_partition_audit_runs_total[30m])) == 0
and sum without (outcome) (buzz_partition_audit_runs_total)
and sum without (outcome) (buzz_partition_audit_runs_total offset 30m)
for: 5m
- alert: BuzzPartitionAuditFailing
expr: |
increase(buzz_partition_audit_failures_total[1h]) >= 2
and buzz_partition_audit_failures_total