mirror of
https://github.com/block/buzz.git
synced 2026-11-10 19:32:00 +00:00
22d96d94ef
> **The pin is a temporary workaround for this week's public release.
The SQLx fix is still unmerged.**
>
> 1. We considered a maintained fork, but repository permissions blocked
it.
> 2. We will move to an official SQLx release once it includes the
required fixes and is verified with Buzz. Revisit the fork if needed.
## Bug
https://linear.app/squareup/issue/BUZZ-656/relay-writer-pool-connections-stranded-after-a-burst-bb-public-2026-11
A short database interruption can leave the relay unable to handle
database requests after traffic returns to normal. All connection slots
can remain occupied even though no request is using them and PostgreSQL
is idle. New requests then time out waiting for a slot.
## Cause
SQLx, our database library, returns connections to the pool in a
background task. If a socket stays open but stops answering, the final
health check has no timeout. That task keeps its slot, and
idle-connection cleanup cannot reclaim it
([#4349](https://github.com/transact-rs/sqlx/issues/4349)).
## Fix
Pin SQLx to
[`c3735abf`](c3735abf03)
from [PR #4407](https://github.com/transact-rs/sqlx/pull/4407). After
five seconds, it stops a stalled return operation, discards the
connection, and frees the slot so new requests can open a replacement.
Preserve the existing [transaction rollback
(#4394)](https://github.com/transact-rs/sqlx/pull/4394) and [SQLite
compatibility (#4402)](https://github.com/transact-rs/sqlx/pull/4402)
fixes. Give the reader's own check four seconds so it can record its
timeout before the outer deadline.
## Critical risk
**This fix depends on a SQLx patch that has not been merged.** Its
five-second limit affects every pool using this revision, so it can also
discard slow but healthy connections and cause more reconnections. The
local reproduction proves recovery from the injected failure; it does
not establish production behavior.
**Prevent accidental replacement:** require SQLx `=0.9.0`, disable
Renovate updates for all seven crates, and fail required CI if the
approved revision, lockfile source, or update restriction changes. These
guards prevent version drift; they do not remove the runtime risk of the
patch. An upgrade must explicitly update the policy after verifying all
required fixes.
## Before and after evidence
With real PostgreSQL 17.6 on macOS arm64, a proxy silenced 100 existing
connections while allowing new connections. All borrowers dropped their
connections; PostgreSQL showed 100 idle sessions.
| Observation | Before | After |
| --- | --- | --- |
| Pool slots still occupied | 100 after 10 seconds | 0 after 5.008
seconds |
| New work | Next checkout timed out | All 100 replacement connections
served `SELECT 1` |
<details>
<summary>Captured before and after output</summary>
**Before**
```text
REPRO burst: 100 dropped borrowers, all release pings held: size=100 idle=0 active=100
REPRO burst DB sessions: [("idle", "ClientRead", 100)]
REPRO burst after 10.000680583s: size=100 idle=0 active=100
REPRO burst: later checkout timed out despite PostgreSQL being healthy
```
**After**
```text
REPRO burst: 100 dropped borrowers, all release pings held: size=100 idle=0 active=100
REPRO burst DB sessions: [("idle", "ClientRead", 100)]
REPRO burst after 5.008456917s: size=0 idle=0 active=0
REPRO burst: all 100 replacement slots serve SELECT 1 while old sessions stay silent
```
Before evidence used Buzz `96af488d968a1ae9d961c2132bea27822735f8da`
with SQLx `f1e94ec3`. After evidence used Buzz
`4bfe60197e8c390a8701053228cd50c5391f37d3` with the SQLx pin committed
in `f253760085e24358728fa4f4fef059ccf892157f`. These are local
fault-injection results: they establish the stranded-slot mechanism and
recovery, not what originally interrupted production database traffic.
</details>
---------
Signed-off-by: Helen Tian <htian@squareup.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>