jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-11-10 11:21:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
e8a4bc15917e61dc0b8066e24c026dc6f3e7ce36
buzz/.github/workflows/ci.yml
Helen Tian 22d96d94ef [buzz] Recover database pool capacity after stalled connections (#9267) 
> **The pin is a temporary workaround for this week's public release.
The SQLx fix is still unmerged.**
>
> 1. We considered a maintained fork, but repository permissions blocked
it.
> 2. We will move to an official SQLx release once it includes the
required fixes and is verified with Buzz. Revisit the fork if needed.

## Bug


https://linear.app/squareup/issue/BUZZ-656/relay-writer-pool-connections-stranded-after-a-burst-bb-public-2026-11

A short database interruption can leave the relay unable to handle
database requests after traffic returns to normal. All connection slots
can remain occupied even though no request is using them and PostgreSQL
is idle. New requests then time out waiting for a slot.

## Cause

SQLx, our database library, returns connections to the pool in a
background task. If a socket stays open but stops answering, the final
health check has no timeout. That task keeps its slot, and
idle-connection cleanup cannot reclaim it
([#4349](https://github.com/transact-rs/sqlx/issues/4349)).

## Fix

Pin SQLx to
[`c3735abf`](c3735abf03)
from [PR #4407](https://github.com/transact-rs/sqlx/pull/4407). After
five seconds, it stops a stalled return operation, discards the
connection, and frees the slot so new requests can open a replacement.

Preserve the existing [transaction rollback
(#4394)](https://github.com/transact-rs/sqlx/pull/4394) and [SQLite
compatibility (#4402)](https://github.com/transact-rs/sqlx/pull/4402)
fixes. Give the reader's own check four seconds so it can record its
timeout before the outer deadline.

## Critical risk

**This fix depends on a SQLx patch that has not been merged.** Its
five-second limit affects every pool using this revision, so it can also
discard slow but healthy connections and cause more reconnections. The
local reproduction proves recovery from the injected failure; it does
not establish production behavior.

**Prevent accidental replacement:** require SQLx `=0.9.0`, disable
Renovate updates for all seven crates, and fail required CI if the
approved revision, lockfile source, or update restriction changes. These
guards prevent version drift; they do not remove the runtime risk of the
patch. An upgrade must explicitly update the policy after verifying all
required fixes.

## Before and after evidence

With real PostgreSQL 17.6 on macOS arm64, a proxy silenced 100 existing
connections while allowing new connections. All borrowers dropped their
connections; PostgreSQL showed 100 idle sessions.

| Observation | Before | After |
| --- | --- | --- |
| Pool slots still occupied | 100 after 10 seconds | 0 after 5.008
seconds |
| New work | Next checkout timed out | All 100 replacement connections
served `SELECT 1` |

<details>
<summary>Captured before and after output</summary>

**Before**

```text
REPRO burst: 100 dropped borrowers, all release pings held: size=100 idle=0 active=100
REPRO burst DB sessions: [("idle", "ClientRead", 100)]
REPRO burst after 10.000680583s: size=100 idle=0 active=100
REPRO burst: later checkout timed out despite PostgreSQL being healthy
```

**After**

```text
REPRO burst: 100 dropped borrowers, all release pings held: size=100 idle=0 active=100
REPRO burst DB sessions: [("idle", "ClientRead", 100)]
REPRO burst after 5.008456917s: size=0 idle=0 active=0
REPRO burst: all 100 replacement slots serve SELECT 1 while old sessions stay silent
```

Before evidence used Buzz `96af488d968a1ae9d961c2132bea27822735f8da`
with SQLx `f1e94ec3`. After evidence used Buzz
`4bfe60197e8c390a8701053228cd50c5391f37d3` with the SQLx pin committed
in `f253760085e24358728fa4f4fef059ccf892157f`. These are local
fault-injection results: they establish the stranded-slot mechanism and
recovery, not what originally interrupted production database traffic.

</details>

---------

Signed-off-by: Helen Tian <htian@squareup.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-11-09 17:40:31 +11:00

469 lines
20 KiB
YAML
Raw Blame History

name: CI
on:
push:
branches: [main, release]
pull_request:
concurrency:
group: ci-${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.sha }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
env:
CARGO_TERM_COLOR: always
BUZZ_TEST_POSTGRES_PASSWORD: buzz_dev
PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/.cache/ms-playwright
jobs:
changes:
name: Detect Changed Paths
runs-on: ubuntu-latest
# The steps take about 75 seconds, but GitHub can finish the job minutes
# after the last step. A 2-minute limit cancelled finished runs.
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
rust: ${{ steps.selection.outputs.rust }}
desktop: ${{ steps.selection.outputs.desktop }}
desktop-rust: ${{ steps.selection.outputs.desktop-rust }}
web: ${{ steps.selection.outputs.web }}
mobile: ${{ steps.selection.outputs.mobile }}
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 2
- uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
# The first call of each Hermit tool downloads it. A stalled Node download
# hung the selection step until this job's cap cancelled every gate. One
# invocation fetches both tools, so its worst case (under 6 minutes)
# still leaves the rest of the job inside the 10-minute cap.
- name: Fetch Hermit tools
run: scripts/ci-hermit-retry.sh sh -c './bin/node --version && ./bin/just --version'
- uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3
id: filter
with:
# PR files come from GitHub's PR diff. A tokenless comparison uses the
# checked-out merge commit and can include newer, unrelated base changes.
token: ${{ github.token }}
# Root and docs/ Markdown do not select runtime suites. Markdown under
# runtime directories remains covered by the directory filters below.
filters: |
rust:
- 'crates/**'
- 'migrations/**'
- 'schema/**'
- 'Cargo.toml'
- 'Cargo.lock'
- 'rust-toolchain.toml'
- '.config/nextest.toml'
- 'scripts/postgres-test-*.sh'
- 'scripts/reconcile-schema-after-pgschema.sql'
- 'scripts/repair-events-pkey.sql'
- 'scripts/start-relay-for-tests.sh'
- 'bin/pgschema'
- 'bin/.pgschema-*.pkg'
- 'scripts/check-postgres-test-discovery.py'
- 'scripts/test-postgres-test-discovery.sh'
- 'scripts/test-postgres-test-wrapper.sh'
- 'deny.toml'
- '.github/workflows/ci.yml'
- '.github/workflows/_ci-*.yml'
- 'docker-compose.yml'
- 'docker-compose.ci.yml'
- 'scripts/run-tests.sh'
- 'scripts/model-capabilities.json'
- 'scripts/normative-corpus.json'
- 'scripts/ci-apt-retry.sh'
- 'scripts/ci-hermit-retry.sh'
- 'Justfile'
desktop:
- '.github/workflows/ci.yml'
- '.github/workflows/_ci-desktop.yml'
- 'scripts/ci-apt-retry.sh'
- 'Justfile'
- 'scripts/generate-macos-icon.py'
- 'scripts/macos-icon.sh'
- 'bin/uv'
- 'bin/.uv-*.pkg'
- 'scripts/model-capabilities.json'
- 'scripts/normative-corpus.json'
- 'desktop/**'
- 'pnpm-lock.yaml'
- 'scripts/flake-check.mjs'
- 'scripts/flake-check.test.mjs'
- '.github/workflows/_ci-flake.yml'
desktop-rust:
- 'desktop/src-tauri/**'
web:
- 'web/**'
- 'pnpm-lock.yaml'
mobile:
- 'mobile/**'
- 'Justfile'
- 'scripts/test-mobile-gateway-recipes.sh'
- 'scripts/mobile-release.sh'
- 'scripts/mobile-release-source.sh'
- 'scripts/mobile-worktree-overrides.sh'
- 'scripts/mobile-worktree-clean.sh'
- 'scripts/release-rulesets.sh'
- 'scripts/test-mobile-release-contract.sh'
- 'scripts/test-mobile-worktree-overrides.sh'
- 'scripts/ci-hermit-retry.sh'
- '.github/workflows/ci.yml'
- '.github/workflows/_ci-*.yml'
- name: Select runtime suites conservatively at the PR file-list limit
id: selection
env:
FILTER_OUTPUTS: ${{ toJSON(steps.filter.outputs) }}
run: node scripts/ci-runtime-selection.mjs
- name: Validate PostgreSQL test discovery
if: github.event_name == 'push' || steps.selection.outputs.rust == 'true'
run: |
scripts/test-postgres-test-discovery.sh
scripts/test-postgres-test-wrapper.sh
- name: Release workflow source contract
run: scripts/test-release-ref-contract.sh
- name: Relay image eligibility contract
run: scripts/test-relay-image-eligibility-workflow.sh
- name: Desktop release candidate contract
run: scripts/test-desktop-release-candidate.sh
- name: OSS desktop promotion contract
run: |
scripts/test-oss-desktop-promotion.sh
scripts/test-oss-desktop-promotion-behavior.sh
- name: Mobile release contract
run: scripts/test-mobile-release-contract.sh
- name: Desktop instance environment contract
run: scripts/test-desktop-instance-detection.sh
- name: Mobile worktree identity contract
run: scripts/test-mobile-worktree-overrides.sh
- name: Codex security review contract
run: just security-review-check
- name: Rust cache contract
run: |
scripts/test-rust-cache-contract.sh
scripts/test-rust-cache-contract-regressions.sh
- name: SQLx dependency pin
run: |
python3 scripts/check-sqlx-pin.py
python3 scripts/test-sqlx-pin.py
- name: CI required-context isolation contract
run: scripts/test-ci-required-context-isolation.sh
- name: CI path selection regression tests
run: node --test scripts/ci-selection.test.mjs scripts/ci-required-gates.test.mjs scripts/ci-apt-retry.test.mjs scripts/ci-hermit-retry.test.mjs
- name: File size policy
run: just file-size-check
dead-token-guard:
name: Dead Token Reference Guard
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Check for dead API token references in client code
run: |
# Fail if dead API token patterns reappear in desktop, mobile, docs, or config.
# Relay crates are excluded — they still use token auth internally.
PATTERNS='TokenScope|MintTokenResponse|hasApiToken|spr_tok_'
PATHS='desktop/src/ desktop/tests/ mobile/test/ mobile/lib/ .env.example'
EXCLUDES='--exclude-dir=node_modules --exclude-dir=.dart_tool'
if grep -rn $EXCLUDES -E "$PATTERNS" $PATHS 2>/dev/null; then
echo "::error::Dead API token references found in client code. See above."
exit 1
fi
echo "No dead token references found."
rust:
name: Rust
needs: [changes]
if: github.event_name == 'push' || needs.changes.outputs.rust == 'true' || needs.changes.outputs.desktop-rust == 'true'
uses: ./.github/workflows/_ci-rust.yml
with:
rust: ${{ needs.changes.outputs.rust == 'true' }}
desktop_rust: ${{ needs.changes.outputs.desktop-rust == 'true' }}
lane: required
rust-cross-compile-domain:
name: Rust Cross-Compile
needs: [changes]
if: github.event_name == 'push' || needs.changes.outputs.rust == 'true'
uses: ./.github/workflows/_ci-rust.yml
with:
rust: ${{ needs.changes.outputs.rust == 'true' }}
desktop_rust: ${{ needs.changes.outputs.desktop-rust == 'true' }}
lane: cross-compile
desktop-domain:
name: Desktop Domain
needs: [changes]
if: github.event_name == 'push' || needs.changes.outputs.desktop == 'true' || needs.changes.outputs.desktop-rust == 'true' || needs.changes.outputs.rust == 'true'
uses: ./.github/workflows/_ci-desktop.yml
with:
rust: ${{ needs.changes.outputs.rust == 'true' }}
desktop: ${{ needs.changes.outputs.desktop == 'true' }}
desktop_rust: ${{ needs.changes.outputs.desktop-rust == 'true' }}
relay-artifacts-domain:
name: Relay Artifact Producer
needs: [changes]
if: github.event_name == 'push' || needs.changes.outputs.desktop == 'true' || needs.changes.outputs.desktop-rust == 'true' || needs.changes.outputs.rust == 'true'
uses: ./.github/workflows/_ci-relay.yml
with:
rust: ${{ needs.changes.outputs.rust == 'true' }}
desktop: ${{ needs.changes.outputs.desktop == 'true' }}
desktop_rust: ${{ needs.changes.outputs.desktop-rust == 'true' }}
lane: artifacts
postgres-domain:
name: PostgreSQL Domain
needs: [changes, relay-artifacts-domain]
if: github.event_name == 'push' || needs.changes.outputs.rust == 'true'
uses: ./.github/workflows/_ci-relay.yml
with:
rust: ${{ needs.changes.outputs.rust == 'true' }}
desktop: ${{ needs.changes.outputs.desktop == 'true' }}
desktop_rust: ${{ needs.changes.outputs.desktop-rust == 'true' }}
lane: postgres
desktop-macos-domain:
name: Desktop macOS Domain
needs: [changes]
if: github.event_name == 'push' || needs.changes.outputs.desktop == 'true' || needs.changes.outputs.desktop-rust == 'true' || needs.changes.outputs.rust == 'true'
uses: ./.github/workflows/_ci-desktop-macos.yml
with:
rust: ${{ needs.changes.outputs.rust == 'true' }}
desktop: ${{ needs.changes.outputs.desktop == 'true' }}
desktop_rust: ${{ needs.changes.outputs.desktop-rust == 'true' }}
relay-domain:
name: Relay and PostgreSQL
needs: [changes, relay-artifacts-domain]
if: github.event_name == 'push' || needs.changes.outputs.desktop == 'true' || needs.changes.outputs.desktop-rust == 'true' || needs.changes.outputs.rust == 'true'
uses: ./.github/workflows/_ci-relay.yml
with:
rust: ${{ needs.changes.outputs.rust == 'true' }}
desktop: ${{ needs.changes.outputs.desktop == 'true' }}
desktop_rust: ${{ needs.changes.outputs.desktop-rust == 'true' }}
lane: required
clients:
name: Clients
needs: [changes]
if: github.event_name == 'push' || needs.changes.outputs.web == 'true' || needs.changes.outputs.mobile == 'true'
uses: ./.github/workflows/_ci-clients.yml
with:
web: ${{ needs.changes.outputs.web == 'true' }}
mobile: ${{ needs.changes.outputs.mobile == 'true' }}
lane: required
mobile-swift-domain:
name: Mobile Swift Domain
needs: [changes]
if: needs.changes.outputs.mobile == 'true'
uses: ./.github/workflows/_ci-clients.yml
with:
web: ${{ needs.changes.outputs.web == 'true' }}
mobile: ${{ needs.changes.outputs.mobile == 'true' }}
lane: mobile-swift
# Advisory flake check: repeats the tests a pull request changes. Its lanes
# skip when no test changed and none of the required gates below reads them;
# their own statuses and step summaries carry the result.
flake-check:
name: Flake Check
if: github.event_name == 'pull_request'
uses: ./.github/workflows/_ci-flake.yml
security-domain:
name: Security Domain
needs: [changes]
if: github.event_name == 'push' || needs.changes.outputs.rust == 'true'
uses: ./.github/workflows/_ci-security.yml
with:
rust: ${{ needs.changes.outputs.rust == 'true' }}
rust-lint:
name: Rust Lint
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.rust == 'true' || needs.changes.outputs.desktop-rust == 'true')))
needs: [changes, rust]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Rust Lint result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.rust.outputs.rust_lint_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
unit-tests:
name: Unit Tests
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.rust == 'true')))
needs: [changes, rust]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Unit Tests result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.rust.outputs.unit_tests_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
windows-rust:
name: Windows Rust (x86_64-pc-windows-msvc)
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.rust == 'true' || needs.changes.outputs.desktop-rust == 'true')))
needs: [changes, rust]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Windows Rust result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.rust.outputs.windows_rust_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
desktop:
name: Desktop
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.desktop == 'true' || needs.changes.outputs.desktop-rust == 'true' || needs.changes.outputs.rust == 'true')))
needs: [changes, desktop-domain]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Desktop result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.desktop-domain.outputs.desktop_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
desktop-build-macos:
name: Desktop Build (macOS)
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.desktop == 'true' || needs.changes.outputs.desktop-rust == 'true' || needs.changes.outputs.rust == 'true')))
needs: [changes, desktop-macos-domain]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Desktop Build (macOS) result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.desktop-macos-domain.outputs.desktop_macos_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
desktop-e2e-relay:
name: Desktop E2E Relay
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.desktop == 'true' || needs.changes.outputs.desktop-rust == 'true' || needs.changes.outputs.rust == 'true')))
needs: [changes, relay-artifacts-domain]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Desktop E2E Relay result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.relay-artifacts-domain.outputs.desktop_e2e_relay_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
desktop-e2e-integration:
name: Desktop E2E Integration
if: always() && (needs.changes.result != 'success' || (needs.relay-artifacts-domain.result == 'success' && (github.event_name == 'push' || needs.changes.outputs.desktop == 'true' || needs.changes.outputs.desktop-rust == 'true' || needs.changes.outputs.rust == 'true')))
needs: [changes, relay-artifacts-domain, relay-domain]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Desktop E2E Integration result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.relay-domain.outputs.desktop_e2e_integration_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
backend-integration:
name: Backend Integration (relay e2e)
if: always() && (needs.changes.result != 'success' || (needs.relay-artifacts-domain.result == 'success' && (github.event_name == 'push' || needs.changes.outputs.rust == 'true')))
needs: [changes, relay-artifacts-domain, relay-domain]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Backend Integration result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.relay-domain.outputs.backend_integration_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
postgres-tests:
name: PostgreSQL Tests
if: always() && (needs.changes.result != 'success' || (needs.relay-artifacts-domain.result == 'success' && (github.event_name == 'push' || needs.changes.outputs.rust == 'true')))
needs: [changes, relay-artifacts-domain, postgres-domain]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check PostgreSQL Tests result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.postgres-domain.outputs.postgres_tests_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
relay-e2e:
name: Relay E2E
if: always() && (needs.changes.result != 'success' || (needs.relay-artifacts-domain.result == 'success' && (github.event_name == 'push' || needs.changes.outputs.rust == 'true')))
needs: [changes, relay-artifacts-domain, relay-domain]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Relay E2E result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.relay-domain.outputs.relay_e2e_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
web:
name: Web
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.web == 'true')))
needs: [changes, clients]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Web result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.clients.outputs.web_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
mobile:
name: Mobile
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.mobile == 'true')))
needs: [changes, clients]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Mobile result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.clients.outputs.mobile_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
security:
name: Security
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.rust == 'true')))
needs: [changes, security-domain]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Security result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.security-domain.outputs.security_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
Reference in New Issue View Git Blame Copy Permalink