mirror of
https://github.com/block/buzz.git
synced 2026-11-06 17:31:19 +00:00
47be2e2d3f
## Problem
A routed read that misses the reader's 150ms acquire budget
(`reader_acquire_timeout`) falls back to the writer, whatever the read
is. Under a burst of subscription-history reads (REQ history, channel
and thread windows, feeds, search hydration, COUNT), the reader pool
fills, and every miss moves a heavy history query onto the writer. There
it competes with the authentication, tenant-resolution and moderation
lookups that only the writer serves, and their latency climbs with it.
Nothing bounds how many history reads a pod sends to the reader at once,
so the reader queue grows without limit as well.
Separately, the compatibility families
`buzz_db_pool_acquire_wait_seconds` and
`buzz_db_pool_acquisitions_total` have no `operation` label, so a
breakdown by operation reports every sample as unlabelled.
## Changes
1. **History reads no longer spill onto the writer when the reader is
busy** (`Db::route_read`). `route_read` now returns
`Result<RouteDecision>`. For `ReaderOperation::SubscriptionHistory`, a
`reader_acquire_timeout` on a saturated reader pool rejects the read
with `sqlx::Error::PoolTimedOut`. Saturated means every connection the
pool may hold exists and none is idle (`Db::reader_saturated`). The
check is needed because sqlx also reports a reader that refuses dials as
`PoolTimedOut`, and that reader is down, not busy. The relay already
answers that with its generic retryable `CLOSED` `error: database
error`, so there is no new client contract. A reader that is wrong
rather than busy (unreachable, validation failure, proof timeout, stale
token) and prechecks that never reach the reader still fall back to the
writer. Authorization reads (`is_relay_member`) keep the writer fallback
unchanged.
2. **Per-pod cap on reader-served history reads**
(`runtime/history_reads.rs`). A FIFO `tokio::sync::Semaphore` admits at
most the reader pool size less one fifth (at least one) history reads at
once. For example, the default pool of 20 admits 16. The remaining
connections are normally free for authorization reads. A read waits at
most 1s for a permit and is otherwise rejected the same way. The permit
is owned by the reader transaction (`ReaderTx`) and released when the
transaction is dropped; routed reads drop it before re-running a failed
replica read on the writer. That holds whether the transaction is a
per-call routed read or a `ReadSession` the relay keeps for follow-up
queries. The cap bounds history transactions, not connections: sqlx
returns a dropped transaction's connection asynchronously through the
reader release check. Released connections can therefore briefly occupy
the free slots, for one round trip normally and at most the 5s release
timeout when sockets stall. In that window, authorization reads fall
back to the writer as before, and history reads on the saturated pool
are rejected, not spilled.
3. **Rejections are observable**:
`buzz_db_route_decision{decision="rejected",reason="history_read_cap"|"reader_acquire_timeout"}`.
4. **Legacy pool families gain `operation`.** The population is
unchanged, and summing over `operation` gives the original series.
5. **HTTP reads answer 503 `database unavailable` when the database
can't be reached** (`api::db_read_error`). This covers a pool timeout,
closed pool or socket failure on `POST /query`, `/count` and every
bridge `/query` branch that serves a history read (channel window, the
three feeds, thread replies), including a shed history read. These used
to return a generic 500. The thread-window path already answered 503 for
pool timeouts; this applies the same mapping to the shared helper.
Pool sizes are unchanged on purpose. A larger pool adds lock contention
on the writer rather than relieving it.
Docs: `deploy/charts/buzz/README.md` (operator contract) and
`PLANS/REPLICA_FULL_READ_ROUTING_DESIGN.md` (fail-closed exceptions).
## What got simpler
The concept count is roughly flat. Instead of a caller-side permit, the
permit lives in the one type every replica transaction already flows
through (`RouteDecision::Replica` / `ReadSessionInner::Replica`). The 11
routed call sites change only by a `?`, and the compiler enforces that
change: a call site cannot silently fall through to the writer on a
rejection. No new error variant, CLOSED reason or env var.
## Tests
-
`postgres_tests::saturated_reader_spends_one_acquire_budget_and_rejects_history_reads`
(rewrites the former
`routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold`).
A size-1 reader is held saturated with a cold Aurora cache. A history
read (`count_events_routed`) is rejected with `PoolTimedOut` within one
acquire budget, records `rejected/reader_acquire_timeout`, and releases
its permit. An authorization read on the same reader still falls back as
`writer/reader_acquire_timeout`.
- `postgres_tests::match_nothing_reads_take_no_connection` (unchanged):
its control still requires a history read against an unreachable reader
to fall back to the writer. It caught the first cut of this change,
which rejected that read.
-
`postgres_tests::history_permit_follows_the_read_session_and_a_full_gate_rejects`:
a held replica `ReadSession` keeps its permit. With the gate full, the
next history read waits the full permit budget and is then rejected as
`rejected/history_read_cap`, with no writer route recorded. Dropping the
session frees the permit and the next read routes to the replica.
- `history_reads::tests`: a cap sizing table, plus bounded-wait
rejection and reopen under paused time.
- `observability` label tests now require `operation` on both legacy
families.
-
`api::db_read_error_tests::unavailable_database_maps_to_retryable_503`:
pool timeout, closed pool and I/O map to 503 `database unavailable`.
Other errors stay 500.
-
`postgres_tests::shed_history_read_maps_to_503_on_every_bridge_branch`:
with the reader saturated, `/query` channel-window, mentions,
needs_action, activity and thread-reply requests each answer 503
`database unavailable` (each serves 200 beforehand).
- Mutation-checked on Blox: re-enabling the writer spill, forcing the
saturation check either way, skipping the gate, dropping the permit
early and removing the legacy label each fail a test.
## Residual
- During a reader failover whose old sockets stall, both slots sit in
the release check, so the pool reads as saturated. History reads are
shed as retryable for up to the release bound (5s) instead of loading
the writer.
`stalled_reader_sessions_release_their_slots_for_new_connections` now
retries through that window and still requires recovery onto the reader.
- A reader that accepts no TCP at all can fill the pool with in-flight
dials and look saturated. Its history reads are then rejected rather
than moved to the writer, which is the same outcome as a full gate.
- Shedding relies on clients backing off on a retryable `CLOSED` (RPR
#9). WebSocket clients already treat `error: database error` as
retryable.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>