Files
buzz/.github/workflows/ci.yml
Matt Toohey eda26b9423 ci: repeat changed tests ten times on pull requests (#8613)
This ports the advisory flake check from block/buzz-app
(https://github.com/block/buzz-app/pull/890) to this repository: pull
requests repeat the tests they change ten times, with retries off, so a
test that passes CI by luck is caught before it merges.

## What it does

`flake-select` diffs the PR merge commit against its first parent
(`HEAD^1..HEAD`, `fetch-depth: 2`) and maps changed hunks to tests per
runner: a hunk inside a test body selects that test; a change anywhere
else in a test file, or a new file, selects the whole file. One lane per
runner then repeats only that selection 10× with retries disabled and
writes a test × iteration table to its step summary. Lanes skip when
nothing was selected, run only on `pull_request`, and are not read by
any required gate in `ci.yml` — their own statuses carry the result.

## Per runner

| Runner | Selection granularity | Repeat mechanism |
|---|---|---|
| Rust (cargo-nextest) | `crates/<crate>/<path>.rs::<module::test>` from
`#[test]`-style attribute spans (comments and literals blanked first). A
hunk outside any test body selects the file's tests. `#[ignore]`d and
Postgres / `external_infra` tests are excluded. | One `cargo nextest run
--profile flake` per package per iteration, with target flags derived
from the path (`src/**` → `--lib --bins`, `tests/<stem>.rs` → `--test
<stem>`), a filterset of exact `binary_id & test` pairs, and JUnit
parsed per iteration. New `[profile.flake]`: `retries = 0`, `fail-fast =
false`. |
| node:test | Whole changed `desktop/src/**/*.test.mjs`,
`*.jsdom-test.mjs`, `desktop/scripts/*.test.mjs` | 10× loop, each file
alone, with the flag sets from `desktop/package.json` `test` |
| Playwright `smoke` | `file:line` journeys from `test()`/`it()` spans
(TypeScript AST) intersected with `playwright test --list
--project=smoke`, deduplicated | Native `--repeat-each=10 --retries=0`,
sharded `min(10, ceil(journeys / 5))` ways; `playwright-summary`
tabulates the JSON report |
| Flutter | Whole changed `mobile/test/**/*_test.dart` | 10× `flutter
test --file-reporter=json` loop with the mobile lane's `--dart-define` |

A runner whose selection exceeds 64 KiB of entries, or more than 250
Playwright journeys, is skipped with a note rather than overrunning its
timeout.

## Differences from the buzz-app version

- Lanes live in a reusable `_ci-flake.yml` called from `ci.yml`,
matching this repo's `_ci-*.yml` convention, instead of inline jobs.
- Playwright specs are parsed with the desktop app's own `typescript`
package rather than `rolldown/parseAst`.
- No Playwright `integration` project lane: it needs the relay artifact
chain. Integration-only specs list nothing for `smoke` and the selection
stays empty.
- Playwright shards are capped at 10, the same bound as the regular
smoke lane.
- Rust entries carry the file and in-file module path, and the filterset
is anchored to exact `(binary, test)` pairs, so a same-named test in
another module is not repeated.
- nextest target flags are derived from the changed file's path, so a
`buzz-relay` change does not compile every test target of the crate.
- A journey-count cap (250) skips the Playwright lane for selections ten
repetitions could not finish within its 45-minute timeout.

## Heads-up

- The Unit Tests `rust-cache` moves from `key: sherpa-cache-v1` to
`shared-key: unit-tests-sherpa-cache-v1` so the flake lane can restore
it read-only. This changes the cache key once: the first Unit Tests run
after merge builds cold.
- This PR itself changes no file the selector recognises
(`scripts/flake-check.test.mjs` is a root script, run by the Desktop
lane), so its own Flake Check lanes are expected to skip.

## Verification

- `node --test scripts/flake-check.test.mjs` — all unit tests pass (also
wired into the Desktop lane).
- `actionlint` on the changed workflows; `biome check` on the two
scripts.
- `scripts/test-rust-cache-contract.sh` and its regression harness pass
with the new shared key.
- Synthetic-diff selection runs for each runner, plus `select` against
this PR's own merge-base (empty, as expected).
- 2-iteration repeats of a real selection on all four runners locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-29 14:34:20 +11:00

462 lines
19 KiB
YAML

name: CI
on:
push:
branches: [main, release]
pull_request:
concurrency:
group: ci-${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.sha }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
env:
CARGO_TERM_COLOR: always
BUZZ_TEST_POSTGRES_PASSWORD: buzz_dev
PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/.cache/ms-playwright
jobs:
changes:
name: Detect Changed Paths
runs-on: ubuntu-latest
# The steps take about 75 seconds, but GitHub can finish the job minutes
# after the last step. A 2-minute limit cancelled finished runs.
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
rust: ${{ steps.selection.outputs.rust }}
desktop: ${{ steps.selection.outputs.desktop }}
desktop-rust: ${{ steps.selection.outputs.desktop-rust }}
web: ${{ steps.selection.outputs.web }}
mobile: ${{ steps.selection.outputs.mobile }}
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 2
- uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
- uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3
id: filter
with:
# PR files come from GitHub's PR diff. A tokenless comparison uses the
# checked-out merge commit and can include newer, unrelated base changes.
token: ${{ github.token }}
# Root and docs/ Markdown do not select runtime suites. Markdown under
# runtime directories remains covered by the directory filters below.
filters: |
rust:
- 'crates/**'
- 'migrations/**'
- 'schema/**'
- 'Cargo.toml'
- 'Cargo.lock'
- 'rust-toolchain.toml'
- '.config/nextest.toml'
- 'scripts/postgres-test-*.sh'
- 'scripts/reconcile-schema-after-pgschema.sql'
- 'scripts/repair-events-pkey.sql'
- 'scripts/start-relay-for-tests.sh'
- 'bin/pgschema'
- 'bin/.pgschema-*.pkg'
- 'scripts/check-postgres-test-discovery.py'
- 'scripts/test-postgres-test-discovery.sh'
- 'scripts/test-postgres-test-wrapper.sh'
- 'deny.toml'
- '.github/workflows/ci.yml'
- '.github/workflows/_ci-*.yml'
- 'docker-compose.yml'
- 'docker-compose.ci.yml'
- 'scripts/run-tests.sh'
- 'scripts/model-capabilities.json'
- 'scripts/normative-corpus.json'
- 'scripts/ci-apt-retry.sh'
- 'Justfile'
desktop:
- '.github/workflows/ci.yml'
- '.github/workflows/_ci-desktop.yml'
- 'scripts/ci-apt-retry.sh'
- 'Justfile'
- 'scripts/generate-macos-icon.py'
- 'scripts/macos-icon.sh'
- 'bin/uv'
- 'bin/.uv-*.pkg'
- 'scripts/model-capabilities.json'
- 'scripts/normative-corpus.json'
- 'desktop/**'
- 'pnpm-lock.yaml'
- 'scripts/flake-check.mjs'
- 'scripts/flake-check.test.mjs'
- '.github/workflows/_ci-flake.yml'
desktop-rust:
- 'desktop/src-tauri/**'
web:
- 'web/**'
- 'pnpm-lock.yaml'
mobile:
- 'mobile/**'
- 'Justfile'
- 'scripts/test-mobile-gateway-recipes.sh'
- 'scripts/mobile-release.sh'
- 'scripts/mobile-release-source.sh'
- 'scripts/mobile-worktree-overrides.sh'
- 'scripts/mobile-worktree-clean.sh'
- 'scripts/publish-mobile-release-candidate.sh'
- 'scripts/release-rulesets.sh'
- 'scripts/test-mobile-release-contract.sh'
- 'scripts/test-mobile-release-candidate-publisher.sh'
- 'scripts/test-mobile-worktree-overrides.sh'
- '.github/workflows/mobile-release-candidate.yml'
- '.github/workflows/ci.yml'
- '.github/workflows/_ci-*.yml'
- name: Select runtime suites conservatively at the PR file-list limit
id: selection
env:
FILTER_OUTPUTS: ${{ toJSON(steps.filter.outputs) }}
run: node scripts/ci-runtime-selection.mjs
- name: Validate PostgreSQL test discovery
if: github.event_name == 'push' || steps.selection.outputs.rust == 'true'
run: |
scripts/test-postgres-test-discovery.sh
scripts/test-postgres-test-wrapper.sh
- name: Release workflow source contract
run: scripts/test-release-ref-contract.sh
- name: Relay image eligibility contract
run: scripts/test-relay-image-eligibility-workflow.sh
- name: Desktop release candidate contract
run: scripts/test-desktop-release-candidate.sh
- name: OSS desktop promotion contract
run: |
scripts/test-oss-desktop-promotion.sh
scripts/test-oss-desktop-promotion-behavior.sh
- name: Mobile release contract
run: |
scripts/test-mobile-release-contract.sh
scripts/test-mobile-release-candidate-publisher.sh
- name: Desktop instance environment contract
run: scripts/test-desktop-instance-detection.sh
- name: Mobile worktree identity contract
run: scripts/test-mobile-worktree-overrides.sh
- name: Codex security review contract
run: just security-review-check
- name: Rust cache contract
run: |
scripts/test-rust-cache-contract.sh
scripts/test-rust-cache-contract-regressions.sh
- name: CI required-context isolation contract
run: scripts/test-ci-required-context-isolation.sh
- name: CI path selection regression tests
run: node --test scripts/ci-selection.test.mjs scripts/ci-required-gates.test.mjs scripts/ci-apt-retry.test.mjs
- name: File size policy
run: just file-size-check
dead-token-guard:
name: Dead Token Reference Guard
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Check for dead API token references in client code
run: |
# Fail if dead API token patterns reappear in desktop, mobile, docs, or config.
# Relay crates are excluded — they still use token auth internally.
PATTERNS='TokenScope|MintTokenResponse|hasApiToken|spr_tok_'
PATHS='desktop/src/ desktop/tests/ mobile/test/ mobile/lib/ .env.example'
EXCLUDES='--exclude-dir=node_modules --exclude-dir=.dart_tool'
if grep -rn $EXCLUDES -E "$PATTERNS" $PATHS 2>/dev/null; then
echo "::error::Dead API token references found in client code. See above."
exit 1
fi
echo "No dead token references found."
rust:
name: Rust
needs: [changes]
if: github.event_name == 'push' || needs.changes.outputs.rust == 'true' || needs.changes.outputs.desktop-rust == 'true'
uses: ./.github/workflows/_ci-rust.yml
with:
rust: ${{ needs.changes.outputs.rust == 'true' }}
desktop_rust: ${{ needs.changes.outputs.desktop-rust == 'true' }}
lane: required
rust-cross-compile-domain:
name: Rust Cross-Compile
needs: [changes]
if: github.event_name == 'push' || needs.changes.outputs.rust == 'true'
uses: ./.github/workflows/_ci-rust.yml
with:
rust: ${{ needs.changes.outputs.rust == 'true' }}
desktop_rust: ${{ needs.changes.outputs.desktop-rust == 'true' }}
lane: cross-compile
desktop-domain:
name: Desktop Domain
needs: [changes]
if: github.event_name == 'push' || needs.changes.outputs.desktop == 'true' || needs.changes.outputs.desktop-rust == 'true' || needs.changes.outputs.rust == 'true'
uses: ./.github/workflows/_ci-desktop.yml
with:
rust: ${{ needs.changes.outputs.rust == 'true' }}
desktop: ${{ needs.changes.outputs.desktop == 'true' }}
desktop_rust: ${{ needs.changes.outputs.desktop-rust == 'true' }}
relay-artifacts-domain:
name: Relay Artifact Producer
needs: [changes]
if: github.event_name == 'push' || needs.changes.outputs.desktop == 'true' || needs.changes.outputs.desktop-rust == 'true' || needs.changes.outputs.rust == 'true'
uses: ./.github/workflows/_ci-relay.yml
with:
rust: ${{ needs.changes.outputs.rust == 'true' }}
desktop: ${{ needs.changes.outputs.desktop == 'true' }}
desktop_rust: ${{ needs.changes.outputs.desktop-rust == 'true' }}
lane: artifacts
postgres-domain:
name: PostgreSQL Domain
needs: [changes, relay-artifacts-domain]
if: github.event_name == 'push' || needs.changes.outputs.rust == 'true'
uses: ./.github/workflows/_ci-relay.yml
with:
rust: ${{ needs.changes.outputs.rust == 'true' }}
desktop: ${{ needs.changes.outputs.desktop == 'true' }}
desktop_rust: ${{ needs.changes.outputs.desktop-rust == 'true' }}
lane: postgres
desktop-macos-domain:
name: Desktop macOS Domain
needs: [changes]
if: github.event_name == 'push' || needs.changes.outputs.desktop == 'true' || needs.changes.outputs.desktop-rust == 'true' || needs.changes.outputs.rust == 'true'
uses: ./.github/workflows/_ci-desktop-macos.yml
with:
rust: ${{ needs.changes.outputs.rust == 'true' }}
desktop: ${{ needs.changes.outputs.desktop == 'true' }}
desktop_rust: ${{ needs.changes.outputs.desktop-rust == 'true' }}
relay-domain:
name: Relay and PostgreSQL
needs: [changes, relay-artifacts-domain]
if: github.event_name == 'push' || needs.changes.outputs.desktop == 'true' || needs.changes.outputs.desktop-rust == 'true' || needs.changes.outputs.rust == 'true'
uses: ./.github/workflows/_ci-relay.yml
with:
rust: ${{ needs.changes.outputs.rust == 'true' }}
desktop: ${{ needs.changes.outputs.desktop == 'true' }}
desktop_rust: ${{ needs.changes.outputs.desktop-rust == 'true' }}
lane: required
clients:
name: Clients
needs: [changes]
if: github.event_name == 'push' || needs.changes.outputs.web == 'true' || needs.changes.outputs.mobile == 'true'
uses: ./.github/workflows/_ci-clients.yml
with:
web: ${{ needs.changes.outputs.web == 'true' }}
mobile: ${{ needs.changes.outputs.mobile == 'true' }}
lane: required
mobile-swift-domain:
name: Mobile Swift Domain
needs: [changes]
if: needs.changes.outputs.mobile == 'true'
uses: ./.github/workflows/_ci-clients.yml
with:
web: ${{ needs.changes.outputs.web == 'true' }}
mobile: ${{ needs.changes.outputs.mobile == 'true' }}
lane: mobile-swift
# Advisory flake check: repeats the tests a pull request changes. Its lanes
# skip when no test changed and none of the required gates below reads them;
# their own statuses and step summaries carry the result.
flake-check:
name: Flake Check
if: github.event_name == 'pull_request'
uses: ./.github/workflows/_ci-flake.yml
security-domain:
name: Security Domain
needs: [changes]
if: github.event_name == 'push' || needs.changes.outputs.rust == 'true'
uses: ./.github/workflows/_ci-security.yml
with:
rust: ${{ needs.changes.outputs.rust == 'true' }}
rust-lint:
name: Rust Lint
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.rust == 'true' || needs.changes.outputs.desktop-rust == 'true')))
needs: [changes, rust]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Rust Lint result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.rust.outputs.rust_lint_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
unit-tests:
name: Unit Tests
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.rust == 'true')))
needs: [changes, rust]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Unit Tests result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.rust.outputs.unit_tests_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
windows-rust:
name: Windows Rust (x86_64-pc-windows-msvc)
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.rust == 'true' || needs.changes.outputs.desktop-rust == 'true')))
needs: [changes, rust]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Windows Rust result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.rust.outputs.windows_rust_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
desktop:
name: Desktop
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.desktop == 'true' || needs.changes.outputs.desktop-rust == 'true' || needs.changes.outputs.rust == 'true')))
needs: [changes, desktop-domain]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Desktop result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.desktop-domain.outputs.desktop_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
desktop-build-macos:
name: Desktop Build (macOS)
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.desktop == 'true' || needs.changes.outputs.desktop-rust == 'true' || needs.changes.outputs.rust == 'true')))
needs: [changes, desktop-macos-domain]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Desktop Build (macOS) result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.desktop-macos-domain.outputs.desktop_macos_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
desktop-e2e-relay:
name: Desktop E2E Relay
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.desktop == 'true' || needs.changes.outputs.desktop-rust == 'true' || needs.changes.outputs.rust == 'true')))
needs: [changes, relay-artifacts-domain]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Desktop E2E Relay result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.relay-artifacts-domain.outputs.desktop_e2e_relay_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
desktop-e2e-integration:
name: Desktop E2E Integration
if: always() && (needs.changes.result != 'success' || (needs.relay-artifacts-domain.result == 'success' && (github.event_name == 'push' || needs.changes.outputs.desktop == 'true' || needs.changes.outputs.desktop-rust == 'true' || needs.changes.outputs.rust == 'true')))
needs: [changes, relay-artifacts-domain, relay-domain]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Desktop E2E Integration result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.relay-domain.outputs.desktop_e2e_integration_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
backend-integration:
name: Backend Integration (relay e2e)
if: always() && (needs.changes.result != 'success' || (needs.relay-artifacts-domain.result == 'success' && (github.event_name == 'push' || needs.changes.outputs.rust == 'true')))
needs: [changes, relay-artifacts-domain, relay-domain]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Backend Integration result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.relay-domain.outputs.backend_integration_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
postgres-tests:
name: PostgreSQL Tests
if: always() && (needs.changes.result != 'success' || (needs.relay-artifacts-domain.result == 'success' && (github.event_name == 'push' || needs.changes.outputs.rust == 'true')))
needs: [changes, relay-artifacts-domain, postgres-domain]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check PostgreSQL Tests result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.postgres-domain.outputs.postgres_tests_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
relay-e2e:
name: Relay E2E
if: always() && (needs.changes.result != 'success' || (needs.relay-artifacts-domain.result == 'success' && (github.event_name == 'push' || needs.changes.outputs.rust == 'true')))
needs: [changes, relay-artifacts-domain, relay-domain]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Relay E2E result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.relay-domain.outputs.relay_e2e_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
web:
name: Web
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.web == 'true')))
needs: [changes, clients]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Web result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.clients.outputs.web_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
mobile:
name: Mobile
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.mobile == 'true')))
needs: [changes, clients]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Mobile result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.clients.outputs.mobile_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success
security:
name: Security
if: always() && (needs.changes.result != 'success' || ((github.event_name == 'push' || needs.changes.outputs.rust == 'true')))
needs: [changes, security-domain]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Check Security result
env:
SELECTION_RESULT: ${{ needs.changes.result }}
RESULT: ${{ needs.security-domain.outputs.security_result }}
run: test "$SELECTION_RESULT" = success && test "$RESULT" = success