mirror of
https://github.com/block/buzz.git
synced 2026-11-10 11:21:05 +00:00
5b1698ec24
## Problem
The replica-fence probe (`replica_fence::run_probe`) commits a heartbeat
token every 500 ms. It borrowed its connection from the **serving writer
pool**, so under load each beat queued behind request traffic.
In a production deployment during a reconnect/replay storm:
-
`buzz_db_pool_acquire_duration_seconds{pool_role="writer",operation="maintenance"}`
went from ≤50 ms max to 2.7–3.0 s, which is the writer pool's acquire
timeout.
- `buzz_db_replica_heartbeat_age_seconds` went from about 0.47 s to
1.25–3.02 s, in the same 10 s buckets.
- Bounded reads then failed the freshness check
(`buzz_db_route_decision{decision="writer",reason="stale"}` went from
0–6 to 1.3k–6.4k per 10 s), so read traffic spilled onto the writer
exactly when it was saturated. Replica lag stayed at tens of
milliseconds the whole time.
- No `replica fence probe failed` warning fired, because each beat
eventually succeeded, just late. The probe had no span or metric of its
own (`maintenance` is shared by 15+ call sites), so this was only
visible indirectly.
## Change
- **Dedicated pool** (`replica_fence::connect_probe_pool`):
- lazy, one connection, `application_name=buzz-replica-fence-probe`;
- same credentials and `max_lifetime` as the writer, so it follows
failover;
- modelled on the push producer's pool.
- It doesn't run the writer pool's `after_connect`: the probe never
writes `events`, so no floor-guard arming is needed. Instead it installs
its own session timeouts:
- `lock_timeout` 1 s. While the probe waits for the exclusive floor
lock, new channel writes queue behind it, so failing a beat fast is
cheaper than holding the queue for the serving writer's 5 s budget.
Retained proofs stand.
- `statement_timeout` and `idle_in_transaction_session_timeout` 2 s. The
idle timeout reaps a stalled client that still holds the exclusive lock.
- A 5 s client-side deadline bounds the whole probe.
- Cost: one extra writer connection per relay process.
- The `Db::connect_writer_pool` doc now names this pool and the
push-enqueue pool as deliberate exceptions to its "every writer pool
uses this constructor" rule, and says why.
- **Spans** (via `datastore_span`, which also records
`buzz_db_operation_duration_seconds{operation=…}`):
- `replica_fence_probe` carries `acquire_ms`, `lock_ms`, `scan_ms`,
`heartbeat_ms`, `commit_ms`, `beat_gap_ms`, `token` and `error_kind`.
- Child spans `replica_fence_probe_acquire` and
`replica_fence_probe_lock` cover the two waits.
- **Logs:**
- Failure: `warn` with `error_kind` (`acquire_timeout`, `lock_timeout`,
`statement_timeout`, `deadline`, `masked_activity`, …),
`consecutive_failures`, `heartbeat_age_ms` and per-phase timings.
- Recovery after failures: `info`.
- **Reader spill metric:** new `buzz_db_reader_spill_total{path,
reason}`. It counts only the routed reads the replica could have served
but the writer did. `buzz_db_route_decision{decision="writer"}` already
existed but merges three different causes under `reason="stale"` and
also counts configuration (`disabled`). Reasons:
| Reason | Meaning |
|---|---|
| `heartbeat_stale` | The newest beat is older than the bounded-read
budget. The probe is late, not the replica. |
| `heartbeat_missing` | No live proof: before the first beat, or nothing
within `FENCE_STALENESS`. |
| `fence_wall_behind` | The read's upper bound is newer than the fence
wall. |
| `replica_lag` | The session proved only an older entry than the read
needs. |
| `reader_acquire_timeout` | Couldn't check out a reader connection. |
| `reader_proof_timeout`, `reader_token_behind`,
`reader_validation_error` | The reader couldn't prove the fence. |
`buzz_db_route_decision` keeps its labels and values unchanged for
existing dashboards. History reads rejected on a saturated reader stay
`decision="rejected"` and aren't counted as spills.
- **Heartbeat interval versus budget:**
- The problem: at 500 ms beats, a 1000 ms `BUZZ_REPLICA_READ_MAX_AGE_MS`
tolerates no missed beat. Just before each commit the newest entry is
already about one interval old, so a single late or failed beat puts
every bounded read over budget.
- The default beat interval stays at **500 ms**. Every beat takes the
deployment-wide exclusive floor lock that channel writes queue behind,
so beating faster costs more lock drains. The budget is what should
absorb a slow beat.
- New `min_read_budget(interval)` = interval + `PROBE_LOCK_TIMEOUT` (1
s) + 200 ms, which is **1700 ms** at the default. That covers one beat
that waits out the lock timeout, plus commit latency. `Db::new` warns at
startup when the configured budget is below it. A deployment running a
1000 ms budget will see this warning until the budget is raised.
- The interval is now configurable with
`BUZZ_REPLICA_FENCE_PROBE_INTERVAL_MS`. Unset means the 500 ms default.
Values outside 300–10000 ms fail startup. The 300 ms floor keeps the
120-entry proof ring covering more than the 30 s staleness gate, and the
ring assertion checks against that floor.
The proof ordering is unchanged: exclusive lock, then `S`, then activity
scan, then token commit last. `committed_at` is still captured before
the `UPDATE` round trip.
## Tests
- New `fence_probe_beats_while_serving_writer_pool_is_exhausted`: goes
through `Db::new` + `spawn_fence_probe`, holds the serving pool's only
connection, and requires two fresh beats and exactly one probe session.
It fails on `main`'s wiring, which I checked by temporarily reverting
the `spawn_fence_probe` change.
- New `cluster_global_probe_pool_bounds_the_floor_lock_wait`: a held
shared floor lock makes the probe fail with `lock_timeout` after about 1
s, the retained proof survives, and the next beat succeeds after
release.
- New `reader_spills_are_counted_by_cause`:
- fresh heartbeat → no spill;
- late heartbeat → `heartbeat_stale`, with the legacy `stale` still
emitted;
- past the staleness gate → `heartbeat_missing`;
- routing disabled → no spill.
- Unit tests cover the log throttle and the interval/budget helpers. The
epoch-rotation test now runs through the one-connection probe pool.
- The `observability_source` source-policy test now asserts the probe
never takes a serving-writer `Maintenance` checkout and is spawned
through `connect_probe_pool`.
- Existing replica-fence and `spawn_fence_probe` tests pass unchanged,
apart from signature updates.
## Not done / follow-ups
- Locally: `buzz-db` unit tests and clippy (`-D warnings`) pass, and 38
replica-fence and routing Postgres tests pass on a fresh database. I
didn't run the full Postgres suite, because it needs the nextest
per-test-database wrapper, so CI's Postgres lane is the gate there.
- No local relay run yet. This needs a reader-configured relay, then
pool exhaustion to see the spans and logs end to end.
- The deployed `BUZZ_REPLICA_READ_MAX_AGE_MS` lives in deployment
config. Check it against the new startup warning.
- The `PROBE_DEADLINE` path has no test. It's only reachable through a
stall that no server-side timeout catches.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Jemiah Westerman <jemiah@squareup.com>
Co-authored-by: Son of Anton <8e0ef2ab02d69a7ac089ef5eca90d1a3fbbdc546f6322dc34d0c4f3deecea7f4@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>