mirror of
https://github.com/block/buzz.git
synced 2026-11-07 09:52:03 +00:00
38795ada87
## Summary
A failed replica fence probe no longer drops the proofs already
retained, so one failed probe no longer sends every read to the writer.
**Why.** `run_probe` called `fence.close()` on any probe error, which
cleared the ring. Until the next successful probe 500 ms later,
`fence.newest()` was `None` and every routed read went to the writer
with `reason="uninitialized"`. Under load this showed up as a steady
share of `bridge_query` reads labelled `uninitialized` on every pod,
while `heartbeat_age` stayed small. The module already documents that a
retained entry's proof never decays: a failed probe proves nothing new,
but it doesn't make an earlier proof wrong. So clearing the ring only
cost reader capacity.
**What changes.** A failed probe logs and records nothing. The retained
entries stand. A probe that keeps failing still closes the fence through
the existing 30 s staleness gate (`FENCE_STALENESS`), and
`heartbeat_age` remains the liveness signal. Regression handling is
unchanged: an epoch change or a same-epoch token regression still clears
the ring in `record()`. `close()` has no production caller left and
becomes a test hook.
**Likely trigger.** The masked-activity check
(`ProbeError::MaskedActivity`) counts a client backend whose state is
`active` with a `NULL` `xact_start` as masked. A backend that has just
started a statement can briefly show as `active` before its `xact_start`
is set, so busy writers hit this transiently. Under pgbench on Postgres
17, between 0.3% and 1.3% of scans saw such a backend. This PR doesn't
change that classification. See the debt below.
No protocol, schema, metric or client change.
### Testing
- `replica_fence::postgres_tests::failed_probe_keeps_retained_proofs`
(new, no database needed, so it runs in the unit lane like its
neighbours): records an entry, runs a probe tick against a closed writer
pool (the probe fails), then asserts `newest()` and `resolve()` still
return the entry, and that a rewound token recorded under the same epoch
after the outage is a `TokenRegression`. Mutation-checked on Blox:
putting `fence.close()` back on the error path fails it, including when
only the regression assertion is left.
- On Blox: the full `buzz-db` Postgres lane, the `buzz-db` unit tests,
`cargo fmt`, clippy for `buzz-db` and `buzz-relay`, and a `buzz-relay`
build.
### Rollout
There's no flag. Rollback is a revert. After deploy, watch:
- `buzz_db_route_decision{decision="writer",reason="uninitialized"}`
should drop to about 0 once a pod has booted, and the
`decision="reader"` share should rise.
- `buzz_db_replica_heartbeat_age_seconds` should stay small. Staleness
is now the only thing that closes the fence on probe failure, so this is
the liveness signal.
- `reason="stale"` may rise a little. `newest()` returns `None` for an
empty ring and for one older than the staleness gate, and both are
labelled `uninitialized`. Bounded reads can also miss the budget for one
tick when a probe beat is missed and fall back as `stale`. Some of
today's `uninitialized` share is expected to move to `stale` rather than
disappear.
## What got simpler
`ReplicaFence::close()` leaves the production API. A probe error now has
one effect (log and keep proving from what's retained) instead of a
second path that discards valid state. The probe loop is split into a
one-tick function that the test calls directly.
Identified, out of scope:
- `buzz_db` warnings, including this probe's failure warning, are
dropped by the relay's default log filter (`buzz_relay=info`), so
operators can't see why a probe fails.
- `MaskedActivity` misclassifies the transient
`active`-without-`xact_start` state. Treating it as not-open looks
sound, because such a backend's transaction starts after the scan, but
it touches the proof and deserves its own review.
- The probe's exclusive advisory lock waits behind channel writers'
shared locks, so a slow writer delays the probe.
- Probe failures have no metric. With retained proofs a failed probe no
longer shows up as `uninitialized` routes, and its warning is filtered
out of the logs (above), so intermittent failures are now invisible. A
probe-failure counter would fix that.
- An epoch rotation that fails still leaves an opening for a restore
that keeps the epoch. `TokenRegression` clears the ring, so if the
rotation `UPDATE` then errors, the next sample stores the rewound token
under the old epoch against an empty ring. This PR doesn't change that
path. A writer-side high-water token kept outside the ring would close
it.
- `newest()` gives the same `uninitialized` label to an empty ring and a
stale one, which hides a dead probe in route metrics.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>