mirror of
https://github.com/block/buzz.git
synced 2026-10-27 12:40:30 +00:00
14a752fb24
## Why
The partition manager checks partition existence by name only and treats
Postgres 42P17 "would overlap partition" errors as success. On databases
where a catch-all partition (`events_p_future` /
`delivery_log_p_future`) covers future months, the monthly creation path
silently does nothing while real coverage erodes toward a write outage.
There is currently no visibility into partition catalog health and no
signal before writes start failing.
## What
- **Read-only catalog audit** in `buzz-db` (`partition.rs`): classifies
the live partition catalog for `events` and `delivery_log` via
`pg_inherits` and bound parsing — coverage bounds, catch-all detection,
anomalous children, and trigger parity checked recursively down to
routable leaf partitions.
- **Creation gated on the audit verdict**: the 42P17-to-success swallow
is removed; monthly CREATE runs only for months the audit proves
uncovered, a canonical-name mismatch is an error, and per-month outcome
capture replaces first-error abort. New
`BUZZ_PARTITION_MANAGER_CREATE_ENABLED` kill switch (default `true`).
- **Prometheus metrics** per table: `buzz_partition_serving_safe`,
`buzz_partition_uncovered_months` (audit window is now through +3
months), `buzz_partition_catch_all_covered_months`,
`buzz_partition_anomalous_children`,
`buzz_partition_trigger_parity_missing`/`_extra`,
`buzz_partition_audit_duration_seconds`,
`buzz_partition_audit_runs_total{outcome}`,
`buzz_partition_audit_failures_total`,
`buzz_partition_create_attempts_total`, and
`buzz_partition_audit_last_success_timestamp_seconds` for staleness
alerting.
- **Cached diagnostics**: `/_status.partition_catalog` reports partition
serving-safety, audit age, and freshness. `/_readiness` stays
process-local, consistent with #7341; missing, unsafe, or stale audits
do not remove pods from service.
- **Periodic audit task** in the relay, plus config surface for the
audit interval and the monthly-create kill switch. Catch-alls are
detected from catalog bounds rather than configured names.
- Doc correction: the mesh heartbeat comment no longer claims readiness
gating (it never was gated; behavior unchanged).
## Behavior changes & risk
- No DETACH/DROP/ATTACH and no schema migrations. The only DDL is the
pre-existing monthly CREATE path, now strictly narrower
(audit-proven-uncovered months only) and behind a kill switch.
- Partition audit health does not gate readiness. Until the first
completed audit, read-only retries back off from 5 seconds to a
60-second cap. Any completed audit, including unsafe coverage, restores
the normal interval.
- Partition-overlap errors now surface as errors instead of being
swallowed as success.
- Rollback: revert the commit, or set
`BUZZ_PARTITION_MANAGER_CREATE_ENABLED=false` to disable the creation
path while keeping the audit and metrics.
## Validation
- New Postgres-backed regression tests for catalog classification and
trigger parity (disabled child trigger, `ENABLE ALWAYS` parent, disabled
nested leaf clone in a multi-level layout, healthy-catalog zero-parity
checks). Each new regression was verified to fail against the previous
logic before the fix.
- Local affected-target validation passed against a disposable
current-schema database, with the known mesh forwarded-echo and tracing
callsite failures excluded. This is not a claim that the full suite
passes.
Follow-up work (separate PRs): bounded catch-all advance (empty-only
golden path; non-empty reattach only behind an explicit operator flag,
default off), fresh-install convergence, and proactive coverage-runway
alerting.
---
## Update — 2026-08-21
- Fixed nested-leaf trigger parity auditing so child-only row triggers
are checked on every routable descendant leaf and reported as
`{leaf}:{trigger}` under the owning root child.
- Added a Postgres regression test for a child-only trigger on a nested
leaf; mutation-reverting to immediate-child-only extra-trigger scanning
makes it fail.
Generated with Codex
Signed-off-by: coder 0 <d97ebdbb198c7237c94f84ea8bb8a73583ea067407eebd0062abbb3962527fb1@buzz.block.builderlab.xyz>
Signed-off-by: Ian Oberst <ioberst@squareup.com>
Signed-off-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: coder 0 <d97ebdbb198c7237c94f84ea8bb8a73583ea067407eebd0062abbb3962527fb1@buzz.block.builderlab.xyz>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>