mirror of
https://github.com/block/buzz.git
synced 2026-11-06 09:21:45 +00:00
74abe2f0fe
## Summary Newest-first `events` page reads are now split at the start of the previous UTC month, so Postgres can prune the older partitions whenever the newest ones fill the page. Results are unchanged. **Why.** `events` is range-partitioned by month, and every partition carries the full index set. A page read with no `created_at` lower bound can't prune, so Postgres plans and locks every partition and every partition index: about 200 relations per read with 19 partitions. A backend has 16 fast-path lock slots, and every lock past those goes through the shared lock table. On a production-shaped load rig at pool 200, active backends spent 36% of their time waiting on `LWLock:LockManager` and 1.6% on disk. **What changes.** `query_events_on` (REQ history, bridge `/query`, mentions, rosters, notices) and `get_channel_window_on` (channel windows) now read in up to two passes on the same connection, through one helper (`store/recent_pass.rs`): 1. **Recent pass:** the same read plus `created_at >= <floor>`, where the floor is the start of the previous UTC month. When it returns a full page, that page is the answer, because every row at or above the floor sorts before every row below it. 2. **Older pass,** only when the recent pass comes up short: `created_at < <floor>`, limited to the rows still missing, appended after the recent rows. The ranges don't overlap, so no row is read twice and the page is exact. For channel windows the limit includes the +1 probe row, so `has_more` stays exact. **One snapshot, one deadline.** The two passes run in one `REPEATABLE READ, READ ONLY` transaction (`ReadTransaction`), so both read the same snapshot. Under `READ COMMITTED`, a replacement committing between the passes (a replaceable event, an artifact head) could move the live row from below the floor to above it, and it would land in neither pass. Inside a transaction the passes run in a savepoint instead; the only callers that read inside a transaction are routed replica reads, which are already `REPEATABLE READ`. Each pass after the first runs under whatever is left of the 20 s read deadline, so a split read never gets more time than an unsplit one. Single-statement reads (COUNT, unsplit pages) keep their old path, now through the same type. Channel windows now take the read deadline too. On a routed reader that is the reader's session deadline. On the writer they used to run under the writer's own session timeout (2 minutes), and now get the 20 s read deadline, like every other event read there. The floor is a SQL literal, not a bind parameter, so Postgres prunes when it plans the statement, cached generic plans included. It changes once a month, so the statement cache holds one statement per pass and shape. `#p` reads put the same bound on `event_mentions.event_created_at`. These reads are not split: - Reads whose own bounds already prune (`since` at or after the floor) or fall entirely before the floor (`until` before it). - `OFFSET` reads, since the offset spans both sides of the floor. - Lookups that match a fixed set of rows wherever they sit: `ids`, `#e`, `#d`, artifact `d` tags. - Fetch-everything reads (`max_limit`, the COUNT fallback). - Reads limited to replaceable or addressable kinds. Those are latest-state rows, often old and few, so a recent pass would rarely fill the page. No protocol, schema, index or client change. **Not covered:** COUNT and the per-row aux `#e` reads, neither of which has a page to fill. The recent pass also has no upper bound, because rows dated in the future must stay visible. So partitions ahead of the current month (the manager keeps 6) and the catch-all are still planned. ## Evidence A production-shaped load rig (6,600 desktop clients connecting cold at 100/s, against a 19-partition `events` table), comparing `main` with this branch. Each row is one arm, so expect run-to-run noise of about ±10%. | Arm | Postgres exec time (top statements) | Errors | Relation locks outside fast path (avg) | |---|---|---|---| | Pool 200 + channel-first index, `main` | 13,217 s | 4.75% | 9,481 | | Pool 200 + channel-first index, this PR | 10,931 s | 3.68% | 4,988 | | Default pool, no extra index, `main` | 6,705 s | 5.96% | — | | Default pool, no extra index, this PR | 6,145 s | 5.15% | — | | Default pool + index, `main` (repeat) | 1,070 s | 0.99% | — | | Default pool + index, this PR (repeat) | 1,013 s | 0.93% | — | Pages that fill above the floor stop planning old partitions, and that halves the locks taken outside the fast path. `LWLock:LockManager` stays at about 41% of active backend samples at pool 200, though. The dominant remaining cost there is the cold unread repair: a multi-channel read with no `since` and a page that never fills. That read needs its own fix (a lookback bound or a better plan) and is out of scope here. An earlier version re-ran the whole read after a short recent pass. That doubled the work on non-filling reads: errors went from 4.75% to 11.88% at pool 200. This version reads only the missing older rows. ## Tests - `recent_pass::tests`: floor arithmetic (including the year wrap), the predicate literals, an eligibility table, and an in-memory exactness test that drives the passes at every limit and pins which passes run with which limits. - `e_tag_filter_shape_tests::split_passes_bound_created_at_with_a_literal` / `recent_pass_skips_lookups`: SQL shape (both the `e.` and `m.` sides for `#p`, no bound on unsplit reads or COUNT) and the exclusions, including offset. - `postgres_tests::split_page_reads_match_the_unsplit_read` (buzz-db event): rows on both sides of the floor. A full recent page is answered alone; short pages at limits 4, 5 and 6 add exactly the missing older rows; offset reads across the floor are not split; a cursor below the floor works. It reads `pg_prepared_statements` to show which passes ran. With another session locking a partition only the older pass plans (and a short `lock_timeout`), a full recent page still answers and a short one returns `Err` rather than a partial page. - `postgres_tests::channel_window_split_matches_the_unsplit_read` (buzz-db thread): the same checks for channel windows, including `has_more` when the recent pass has no probe row, and an `Err` when the older pass fails. - `split_read_passes_share_one_snapshot` (event, both deadline modes) and `channel_window_passes_share_one_snapshot` (thread): another session holds a lock that only the older pass needs, so the read stops between its passes. A replacement then commits that deletes a row below the floor and makes its replacement live above it. The page must still be the first pass's snapshot: the replaced row is present and its replacement is absent. - `read_statements_share_one_deadline`: per-read, session and inside-a-transaction reads. The second statement runs under less than the full deadline, and finishing the read restores the connection's own timeout. - `channel_window_passes_share_one_deadline` (thread): through the production writer entry. The recent pass waits 8 s on a lock and the older pass is blocked for good. The window must be cancelled within the one shared deadline, not a full deadline after its older pass starts. - `channel_scoped_mention_reads_share_one_prepared_statement` now expects one recent and one older statement, shared by every channel count. Mutation-checked. Each of these mutants fails a new Postgres test: dropping the older pass, giving the older pass the full limit, removing the split, splitting offset reads, dropping the window's probe row from the split limit, removing the window's split, swallowing an older-pass error in either path, opening the split read `READ COMMITTED` (events and windows), opening it with a plain `BEGIN`, giving every pass the full deadline, and running a window with no read deadline. ## What got simpler Nothing got simpler overall. This adds one small module (`store/recent_pass.rs`) shared by both read paths. The single-statement deadline helper is now a thin wrapper over `ReadTransaction`, so the begin, savepoint and tighten logic lives in one place. The alternative, a `since` in every client plus a new index per read shape, means more code in more places and does nothing for app versions already shipped. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>