mirror of
https://github.com/block/buzz.git
synced 2026-10-30 22:11:36 +00:00
eda26b9423
This ports the advisory flake check from block/buzz-app (https://github.com/block/buzz-app/pull/890) to this repository: pull requests repeat the tests they change ten times, with retries off, so a test that passes CI by luck is caught before it merges. ## What it does `flake-select` diffs the PR merge commit against its first parent (`HEAD^1..HEAD`, `fetch-depth: 2`) and maps changed hunks to tests per runner: a hunk inside a test body selects that test; a change anywhere else in a test file, or a new file, selects the whole file. One lane per runner then repeats only that selection 10× with retries disabled and writes a test × iteration table to its step summary. Lanes skip when nothing was selected, run only on `pull_request`, and are not read by any required gate in `ci.yml` — their own statuses carry the result. ## Per runner | Runner | Selection granularity | Repeat mechanism | |---|---|---| | Rust (cargo-nextest) | `crates/<crate>/<path>.rs::<module::test>` from `#[test]`-style attribute spans (comments and literals blanked first). A hunk outside any test body selects the file's tests. `#[ignore]`d and Postgres / `external_infra` tests are excluded. | One `cargo nextest run --profile flake` per package per iteration, with target flags derived from the path (`src/**` → `--lib --bins`, `tests/<stem>.rs` → `--test <stem>`), a filterset of exact `binary_id & test` pairs, and JUnit parsed per iteration. New `[profile.flake]`: `retries = 0`, `fail-fast = false`. | | node:test | Whole changed `desktop/src/**/*.test.mjs`, `*.jsdom-test.mjs`, `desktop/scripts/*.test.mjs` | 10× loop, each file alone, with the flag sets from `desktop/package.json` `test` | | Playwright `smoke` | `file:line` journeys from `test()`/`it()` spans (TypeScript AST) intersected with `playwright test --list --project=smoke`, deduplicated | Native `--repeat-each=10 --retries=0`, sharded `min(10, ceil(journeys / 5))` ways; `playwright-summary` tabulates the JSON report | | Flutter | Whole changed `mobile/test/**/*_test.dart` | 10× `flutter test --file-reporter=json` loop with the mobile lane's `--dart-define` | A runner whose selection exceeds 64 KiB of entries, or more than 250 Playwright journeys, is skipped with a note rather than overrunning its timeout. ## Differences from the buzz-app version - Lanes live in a reusable `_ci-flake.yml` called from `ci.yml`, matching this repo's `_ci-*.yml` convention, instead of inline jobs. - Playwright specs are parsed with the desktop app's own `typescript` package rather than `rolldown/parseAst`. - No Playwright `integration` project lane: it needs the relay artifact chain. Integration-only specs list nothing for `smoke` and the selection stays empty. - Playwright shards are capped at 10, the same bound as the regular smoke lane. - Rust entries carry the file and in-file module path, and the filterset is anchored to exact `(binary, test)` pairs, so a same-named test in another module is not repeated. - nextest target flags are derived from the changed file's path, so a `buzz-relay` change does not compile every test target of the crate. - A journey-count cap (250) skips the Playwright lane for selections ten repetitions could not finish within its 45-minute timeout. ## Heads-up - The Unit Tests `rust-cache` moves from `key: sherpa-cache-v1` to `shared-key: unit-tests-sherpa-cache-v1` so the flake lane can restore it read-only. This changes the cache key once: the first Unit Tests run after merge builds cold. - This PR itself changes no file the selector recognises (`scripts/flake-check.test.mjs` is a root script, run by the Desktop lane), so its own Flake Check lanes are expected to skip. ## Verification - `node --test scripts/flake-check.test.mjs` — all unit tests pass (also wired into the Desktop lane). - `actionlint` on the changed workflows; `biome check` on the two scripts. - `scripts/test-rust-cache-contract.sh` and its regression harness pass with the new shared key. - Synthetic-diff selection runs for each runner, plus `select` against this PR's own merge-base (empty, as expected). - 2-iteration repeats of a real selection on all four runners locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
177 lines
4.6 KiB
Bash
Executable File
177 lines
4.6 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
|
|
root=$(cd "$(dirname "$0")/.." && pwd)
|
|
known_good='e18b497796c12c097a38f9edb9d0641fb99eee32'
|
|
tmp=$(mktemp -d)
|
|
trap 'rm -rf "$tmp"' EXIT
|
|
mkdir -p "$tmp/.github/workflows"
|
|
cp "$root"/.github/workflows/*.yml "$tmp/.github/workflows/"
|
|
for workflow in "$root"/.github/workflows/*.yaml; do
|
|
[[ -e "$workflow" ]] || continue
|
|
cp "$workflow" "$tmp/.github/workflows/"
|
|
done
|
|
cp "$root/renovate.json" "$tmp/renovate.json"
|
|
|
|
run_contract() {
|
|
BUZZ_RUST_CACHE_CONTRACT_ROOT="$tmp" "$root/scripts/test-rust-cache-contract.sh"
|
|
}
|
|
|
|
expect_failure() {
|
|
local expected=$1
|
|
local output
|
|
if output=$(run_contract 2>&1); then
|
|
echo "expected rust cache contract failure containing: $expected" >&2
|
|
exit 1
|
|
fi
|
|
if [[ "$output" != *"$expected"* ]]; then
|
|
printf 'unexpected rust cache contract error:\n%s\n' "$output" >&2
|
|
exit 1
|
|
fi
|
|
}
|
|
|
|
write_cache_workflow() {
|
|
cat > "$tmp/.github/workflows/new-cache-user.yaml"
|
|
}
|
|
|
|
run_contract >/dev/null
|
|
|
|
write_cache_workflow <<'YAML'
|
|
name: Bad digest
|
|
on: workflow_dispatch
|
|
jobs:
|
|
cache:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6
|
|
YAML
|
|
expect_failure 'restored rust-cache v2.9.2'
|
|
|
|
write_cache_workflow <<'YAML'
|
|
name: Mutable tag
|
|
on: workflow_dispatch
|
|
jobs:
|
|
cache:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: Swatinem/rust-cache@v2
|
|
YAML
|
|
expect_failure 'rust-cache must stay on the v2.9.1 digest'
|
|
|
|
write_cache_workflow <<'YAML'
|
|
name: Expression ref
|
|
on: workflow_dispatch
|
|
jobs:
|
|
cache:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: Swatinem/rust-cache@${{ matrix.cache-ref }}
|
|
YAML
|
|
expect_failure 'rust-cache must stay on the v2.9.1 digest'
|
|
|
|
write_cache_workflow <<'YAML'
|
|
name: Spaced key
|
|
on: workflow_dispatch
|
|
jobs:
|
|
cache:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses : Swatinem/rust-cache@v2
|
|
YAML
|
|
expect_failure 'rust-cache must stay on the v2.9.1 digest'
|
|
|
|
write_cache_workflow <<'YAML'
|
|
name: Flow mapping
|
|
on: workflow_dispatch
|
|
jobs:
|
|
cache:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- { uses: Swatinem/rust-cache@v2 }
|
|
YAML
|
|
expect_failure 'rust-cache must stay on the v2.9.1 digest'
|
|
|
|
write_cache_workflow <<'YAML'
|
|
name: Quoted key
|
|
on: workflow_dispatch
|
|
jobs:
|
|
cache:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- "uses": Swatinem/rust-cache@v2
|
|
YAML
|
|
expect_failure 'rust-cache must stay on the v2.9.1 digest'
|
|
rm "$tmp/.github/workflows/new-cache-user.yaml"
|
|
|
|
mkdir -p "$tmp/.github/actions/cache"
|
|
cat > "$tmp/.github/actions/cache/action.yml" <<'YAML'
|
|
name: Unsafe cache wrapper
|
|
runs:
|
|
using: composite
|
|
steps:
|
|
- { uses: Swatinem/rust-cache@v2 }
|
|
YAML
|
|
write_cache_workflow <<'YAML'
|
|
name: Local cache wrapper
|
|
on: workflow_dispatch
|
|
jobs:
|
|
cache:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: ./.github/actions/cache
|
|
YAML
|
|
expect_failure 'rust-cache must stay on the v2.9.1 digest'
|
|
|
|
cat > "$tmp/.github/actions/cache/action.yml" <<YAML
|
|
name: Safe preferred cache wrapper
|
|
runs:
|
|
using: composite
|
|
steps:
|
|
- uses: Swatinem/rust-cache@$known_good
|
|
YAML
|
|
cat > "$tmp/.github/actions/cache/action.yaml" <<'YAML'
|
|
name: Unreachable stale cache wrapper
|
|
runs:
|
|
using: composite
|
|
steps:
|
|
- uses: Swatinem/rust-cache@v2
|
|
YAML
|
|
run_contract >/dev/null
|
|
rm -rf "$tmp/.github/actions" "$tmp/.github/workflows/new-cache-user.yaml"
|
|
|
|
cp "$root/.github/workflows/_ci-flake.yml" "$tmp/flake.orig"
|
|
python3 - "$tmp/.github/workflows/_ci-flake.yml" <<'PY'
|
|
import pathlib
|
|
import sys
|
|
|
|
path = pathlib.Path(sys.argv[1])
|
|
workflow = path.read_text()
|
|
needle = " shared-key: unit-tests-sherpa-cache-v1\n save-if: false\n"
|
|
if workflow.count(needle) != 1:
|
|
raise SystemExit("expected one flake-rust cache restore key")
|
|
path.write_text(workflow.replace(needle, " shared-key: unit-tests\n save-if: false\n"))
|
|
PY
|
|
expect_failure 'flake-rust rust-cache action must restore with.shared-key unit-tests-sherpa-cache-v1'
|
|
mv "$tmp/flake.orig" "$tmp/.github/workflows/_ci-flake.yml"
|
|
|
|
python3 - "$tmp/.github/workflows/_ci-rust.yml" <<'PY'
|
|
import pathlib
|
|
import sys
|
|
|
|
path = pathlib.Path(sys.argv[1])
|
|
workflow = path.read_text()
|
|
needle = " shared-key: unit-tests-sherpa-cache-v1\n save-if:"
|
|
replacement = " save-if:"
|
|
if workflow.count(needle) != 1:
|
|
raise SystemExit("expected one Unit Tests cache generation key")
|
|
workflow = workflow.replace(needle, replacement)
|
|
job = " unit-tests:\n"
|
|
if workflow.count(job) != 1:
|
|
raise SystemExit("expected one Unit Tests job")
|
|
workflow = workflow.replace(job, job + " env:\n shared-key: unit-tests-sherpa-cache-v1\n", 1)
|
|
path.write_text(workflow)
|
|
PY
|
|
expect_failure 'rust-cache action must keep with.shared-key set to unit-tests-sherpa-cache-v1'
|
|
|
|
echo "rust cache contract regressions passed"
|