jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-10-27 12:40:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
8ceca8ac46ad1a76e037380236f90a9fce29e16b
buzz/docs/partition-catalog-monitoring.rules.test.yml
Ian Oberst 85b7331784 fix(relay): partition audit observability follow-ups (#7994) 
> [!IMPORTANT]
> **The alert rules changed in c3a314e71.** If you reviewed the alert
guidance earlier, please look again.
> - **The two range-based rules now apply only to live pods.** A range
selector such as `[30m]` keeps reading a terminated pod's samples until
they leave the window. So `BuzzPartitionAuditNotRunning` could fire for
about 10 minutes after normal pod turnover, and
`BuzzPartitionAuditFailing` could keep firing for up to an hour after
the pod was gone. Each rule now adds `and <counter>`, which requires the
counter to be exported right now:
>   ```yaml
>   expr: |
> sum without (outcome) (increase(buzz_partition_audit_runs_total[30m]))
== 0
>       and sum without (outcome) (buzz_partition_audit_runs_total)
>   ```
>   ```yaml
>   expr: |
>     increase(buzz_partition_audit_failures_total[1h]) >= 2
>       and buzz_partition_audit_failures_total
>   ```
> - **The audit counters start at 0.** When the relay starts, it exports
every `buzz_partition_audit_runs_total{table, outcome}` series and
`buzz_partition_audit_failures_total` at 0, before the first audit.
`increase()` ignores a new series' first sample. Before this change, it
could miss the first periodic failure, or read a table's first
`degraded` run as a stopped loop. This change is also what makes the
`and <counter>` gate work: the counters now exist for a relay's whole
life.
>
> The doc's old claim that pod turnover can't leave an alert firing was
true only of the instant-selector stale rule. The doc now explains the
difference.

## Why

These are the observability follow-ups deferred from #6515 ("fix(db):
audit partition catalog before creation"):

- **Timestamp-age alerts go silent** ([review
comment](https://github.com/block/buzz/pull/6515#discussion_r4127212141)).
Partition gauges are set only when a table audits successfully, and the
exporter evicts idle gauges after 2,700 s by default. After sustained
failures, `buzz_partition_audit_last_success_timestamp_seconds`
disappears. A table that has never audited successfully never has it. An
alert on `time() - last_success` then has no input.
- **Startup fallback failures are not counted** ([review
comment](https://github.com/block/buzz/pull/6515#discussion_r4127212158)).
When the startup fallback audit fails, it logs an error but does not
increment `buzz_partition_audit_failures_total`. Periodic failures do.
- **Repaired environments are always `degraded`**.
`PartitionTableAudit::degraded()` counts every legacy leaf, even one
wholly in the past. The `bb-block-staging` audit of the repaired catalog
reports `degraded` only because of `events_p2026_07_08_legacy`
`[2026-07-01, 2026-09-01)`. With this rule, every repaired environment
stays `degraded`, so nobody can alert on the outcome.
- **Partition config tests can see a poisoned env** ([review
comment](https://github.com/block/buzz/pull/6515#discussion_r4137416297)).
Two partition config tests take only `ENV_MUTEX`. A NIP-FI test running
in the same process can make them see its deliberately invalid
environment.

Partition health is still diagnostic only, through
`/_status.partition_catalog` and metrics. Nothing here affects
`/_readiness`. Handling concurrent creation (#6515 item 4) is out of
scope and stays with the catch-all advancement work.

## What

- **Success timestamp survives failures.** Each failed table audit
refreshes `buzz_partition_audit_last_success_timestamp_seconds` with
`increment(0.0)`, which leaves the value unchanged. It still resets the
exporter's idle tracking, the same mechanism as
`refresh_legacy_active_gauge_recency`. Sustained failures therefore keep
the last success exported and its age rising. A table that has never
succeeded exports `0`. A failed audit still never makes the last success
look fresh.
- **Alert guidance.** New `docs/partition-catalog-monitoring.md` covers
the metrics, how long each series lives, and the audit outcomes. It
notes that `runs_total` has no `unsafe` outcome. It gives three alert
rules:
- timestamp age, which also covers never-successful tables and sustained
failures;
- `sum without (outcome)
(increase(buzz_partition_audit_runs_total[30m])) == 0` for a stopped
audit loop. Counters are never idle-evicted, and this rule fires before
the stopped loop's timestamp is evicted, so there is no false resolve;
  - an aggregate failure-counter rule.

Both range-based rules also require their counter to be exported right
now (`and <counter>`), so a terminated pod's leftover samples can't
raise or hold an alert.
- **Zero counter baselines.** `try_install` exports each
`buzz_partition_audit_runs_total{table, outcome}` series (`ok`,
`degraded`, and `error` for both tables) and
`buzz_partition_audit_failures_total` at 0 before the startup audit, the
same way the auth series already start at 0. A failure in the startup
audit itself can still happen before the first scrape. The doc says so;
the stale rule covers that case.
- **Failure counter.** `buzz_partition_audit_failures_total` now counts
failed explicit audit attempts: the startup fallback audit and each
periodic audit. Both paths go through one helper,
`counted_partition_audit`. A failed `ensure_future_partitions` call,
such as a DDL error that the fallback then recovers, is not counted by
itself. A single attempt never counts twice.
- **Historical legacy leaves.** A legacy leaf is historical when its
finite upper bound is at or before the current month start. It is still
reported under `children` as `legacy_leaf`, but it no longer degrades
the table. A legacy leaf that still covers the current month or any
lookahead month stays degraded. All other degradation causes are
unchanged. The verdict now lives in `PartitionAuditReport::outcome()`,
which `buzz-admin partition-audit` uses. The JSON outcome names and exit
codes are the same as before.
- **Test hygiene.** Both partition config tests now use `env_guards()`.

## How it was tested

- `cargo fmt --all -- --check` and `cargo clippy -p buzz-db -p
buzz-relay -p buzz-admin --all-targets -- -D warnings`.
- New unit tests:
- The repaired incident layout with an injected clock: `degraded` in
August 2026 and `ok` on 2026-09-01T00:00Z, when the upper bound equals
the current month start, and on 2026-09-15. The legacy leaf is still in
the serialized report. The `outcome` label on the run counter follows
the verdict.
- A table test of the legacy-leaf predicate: the upper bound before, at,
and one second after the month start; lookahead and beyond-horizon
leaves; historical leaves with trigger drift; anomalous and `DEFAULT`
children; no audited month.
- Outcome precedence (error, then unsafe, then degraded) and the
`buzz-admin` exit-code mapping.
- Startup: a failed fallback counts once, a successful fallback after a
DDL failure counts nothing, and a successful ensure never runs the
fallback. Each periodic failure counts once. Against an unreachable
database, startup counts one aggregate failure while each table still
records two `error` runs.
- On the production exporter, never-successful audits leave run counters
and a `0` success timestamp for both tables.
- On the production exporter, all six run-counter series and the failure
counter read 0 before any audit. After one failed audit, `error` reads 1
and `ok`/`degraded` stay at 0. Removing the initialization fails this
test.
- PostgreSQL lane (`scripts/postgres-test-run.sh`) with new or extended
tests:
- The repaired layout is not degraded in September. The same catalog is
degraded in August.
- With a 1 s idle timeout, a failed audit leaves the success timestamp
unchanged, and sustained failures keep it exported past the idle
timeout. When audits stop, the timestamp is evicted but the run counters
remain. The next success exports a fresh timestamp.
- Mutation check: removing the failure-path refresh fails both alert
fixtures.
- These fixtures check the emitted series. They do not evaluate the
rules themselves, including the `and <counter>` gate. `promtool` isn't
available locally or in CI, and the doc points to `promtool test rules`
for that.
- `scripts/test-postgres-test-discovery.sh`.
- `config::` and `nip_fi_config::` tests run together in one process
five times.

## Related

This PR conflicts textually with #7993 (catch-all advancement) in
`crates/buzz-db/src/store/partition.rs` and
`crates/buzz-relay/src/main.rs`. Whichever PR merges second will be
rebased onto the other.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 13:57:54 -07:00

123 lines
4.4 KiB
YAML
Raw Blame History

# Run with: promtool test rules docs/partition-catalog-monitoring.rules.test.yml
#
# Series are sampled every minute and use the default 900 s audit interval. The
# startup audit finishes before the first scrape, so run counters start at 1.
rule_files:
- partition-catalog-monitoring.rules.yml
evaluation_interval: 1m
tests:
# A healthy new pod audits at startup and then every 15 minutes. Its counter
# is flat for the first interval, which must not read as a stopped loop.
- interval: 1m
input_series:
- series: 'buzz_partition_audit_runs_total{pod="fresh",table="events",outcome="ok"}'
values: '1x14 2x14 3x14 4x14 5x14'
- series: 'buzz_partition_audit_runs_total{pod="fresh",table="events",outcome="degraded"}'
values: '0x74'
- series: 'buzz_partition_audit_last_success_timestamp_seconds{pod="fresh",table="events"}'
values: '0x14 900x14 1800x14 2700x14 3600x14'
alert_rule_test:
- eval_time: 10m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 14m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 29m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 70m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 70m
alertname: BuzzPartitionAuditStale
exp_alerts: []
# The audit loop stops after the 15-minute audit while the pod keeps running.
# Both rules fire before the timestamp's 2,700 s idle eviction at 60 minutes.
- interval: 1m
input_series:
- series: 'buzz_partition_audit_runs_total{pod="stopped",table="events",outcome="ok"}'
values: '1x14 2x60'
- series: 'buzz_partition_audit_last_success_timestamp_seconds{pod="stopped",table="events"}'
values: '0x14 900x45'
alert_rule_test:
- eval_time: 40m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 52m
alertname: BuzzPartitionAuditNotRunning
exp_alerts:
- exp_labels: {pod: stopped, table: events}
- eval_time: 57m
alertname: BuzzPartitionAuditStale
exp_alerts:
- exp_labels: {pod: stopped, table: events}
# The audit loop stops right after the startup audit. It is caught once the
# pod is one window old.
- interval: 1m
input_series:
- series: 'buzz_partition_audit_runs_total{pod="early",table="events",outcome="ok"}'
values: '1x80'
alert_rule_test:
- eval_time: 33m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 36m
alertname: BuzzPartitionAuditNotRunning
exp_alerts:
- exp_labels: {pod: early, table: events}
# A pod terminates at 24 minutes. Range selectors still read its earlier
# samples, long enough to complete `for: 5m` at 50 minutes, but it must not
# start an alert.
- interval: 1m
input_series:
- series: 'buzz_partition_audit_runs_total{pod="gone",table="events",outcome="ok"}'
values: '1x14 2x9 stale'
- series: 'buzz_partition_audit_failures_total{pod="gone"}'
values: '0x10 1 2 3x7 stale'
alert_rule_test:
- eval_time: 45m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 52m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 60m
alertname: BuzzPartitionAuditNotRunning
exp_alerts: []
- eval_time: 30m
alertname: BuzzPartitionAuditFailing
exp_alerts: []
# A table that has never audited successfully exports a 0 timestamp.
- interval: 1m
input_series:
- series: 'buzz_partition_audit_last_success_timestamp_seconds{pod="never",table="events"}'
values: '0x60'
alert_rule_test:
- eval_time: 35m
alertname: BuzzPartitionAuditStale
exp_alerts: []
- eval_time: 45m
alertname: BuzzPartitionAuditStale
exp_alerts:
- exp_labels: {pod: never, table: events}
# One failure does not alert; a second within an hour does.
- interval: 1m
input_series:
- series: 'buzz_partition_audit_failures_total{pod="once"}'
values: '0x10 1x50'
- series: 'buzz_partition_audit_failures_total{pod="twice"}'
values: '0x10 1x10 2x40'
alert_rule_test:
- eval_time: 40m
alertname: BuzzPartitionAuditFailing
exp_alerts:
- exp_labels: {pod: twice}
Reference in New Issue View Git Blame Copy Permalink