Address Jude's integrated review of f02a4235a on #8613.
The required Desktop JS gate was red: scripts/flake-check.test.mjs spawns
the real selector, which writes its summary to $GITHUB_STEP_SUMMARY and
its outputs to $GITHUB_OUTPUT when those are set, as they are under
Actions, so the regression test read an empty stdout there. selectAppending
now strips both from the child's environment when it asserts on stdout,
and its `github` mode sets them to temp files and reads those instead; the
Tauri selection is asserted both ways, equal, with nothing on stdout in
the routed mode. The suite passes with and without the two variables set.
A tests/<stem>.rs root with no test function of its own was treated as a
helper: buzz-acp's pool_lifecycle_state compiles src/pool_lifecycle.rs in
by `#[path]`, and a change to it selected git_bootstrap, run_task and
stdio_contract and never pool_lifecycle_state. rustTargetRoot now tells a
binary's root (tests/<stem>.rs, tests/<dir>/main.rs) from a module of one;
rustModules resolves the file modules a file declares by cargo's
conventions (`#[path]` relative to the declaring file, else x.rs or
x/mod.rs under the module's directory, with comments and strings blanked);
rustIncludedTests collects their tests transitively under their module
path, the names nextest lists for the binary. selectRust takes these as
`included`: a whole selection of a root carries them, a hunk inside one of
the root's own tests does not, and a root with none of its own selects
them as shared code changed. A root with no tests at all gets a row and
selects nothing; only a non-root tests/** file without tests takes the
helper branch, and the helper importers' whole-binary expansion uses the
same test set.
Regression tests go through the real select entry point: a comment
appended to pool_lifecycle_state.rs selects exactly the nine
pool_lifecycle::tests::* entries under that file (and fails if the
included tests are dropped), and rustGroups builds `-p buzz-acp --test
pool_lifecycle_state` with those exact names. Verified live: `cargo nextest
list` with the produced filterset matches all nine in
buzz-acp::pool_lifecycle_state, and `flake-check.mjs rust --iterations=2`
over the selection ran green end to end.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Matt Toohey <contact@matttoohey.com>
Address the two High findings of review 5659898932 on #8613: a changed
Rust test under desktop/src-tauri, or a changed helper module under a
crate's tests/, produced an empty flake selection and every repeat lane
skipped, a false "nothing selected" green.
Desktop Tauri Rust is now a runner of its own rather than an unclassified
path. rustFile places a Rust file in its workspace (the root crates, or
desktop/src-tauri and its crates/*), and classify, cratePackage,
rustTarget, rustModulePath and rustGroups read that instead of assuming
crates/<crate>/. The new flake-tauri lane mirrors the Desktop Tauri core
lane's setup (its apt packages, sidecar stubs and desktop-core rust-cache,
restored read-only) and runs the same script against
desktop/src-tauri/Cargo.toml. nextest reads its configuration from the
workspace root, so that workspace gets its own .config/nextest.toml with
the flake profile, and the Postgres exclusion applies only in the root
workspace: nextest rejects a binary(/^postgres_/) predicate where no such
binary exists, which the Tauri workspace's first live run showed.
Supporting the surface for real was straightforward, since the workspace
is an ordinary cargo workspace that nextest lists and runs by manifest
path, so no out-of-scope declaration was needed.
A tests/** file without test functions is a helper that integration
binaries compile in with `mod common;`. select now finds the package's
integration targets that include it (rustIncludes reads `mod <first>;`
declarations with comments and strings blanked, `#[path]` attributes, and
a tests/<dir>/main.rs owning its directory; a helper no target names
falls back to every target) and selects each of those binaries whole, so
the ignored and infrastructure exclusions still apply, with a row naming
the importers. A helper and its importer changed together select each
test once. A src/** file without test functions still selects nothing; a
comment beside the row records that this is deliberate, since the regular
lanes run what depends on it.
Regression tests go through the real select entry point over a synthetic
head tree (HEAD with a comment appended, as the reviewer probed): a
changed desktop/src-tauri/tests/rodio_mixer_diagnostic.rs yields its
three tests under tauri=, and a comment-only change to
crates/buzz-agent/tests/common/mod.rs yields rust= entries in exactly
anthropic_continuation, fake_llm, permission_boundary and regressions,
the four files declaring `mod common;`. Verified live: cargo nextest list
with the produced -p/--test flags and filterset matches all 85 entries,
and `flake-check.mjs tauri --iterations=2` on the Tauri selection ran
green end to end with its JUnit report under
desktop/src-tauri/target/nextest.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Matt Toohey <contact@matttoohey.com>
Address the Codex security review and the zizmor code-scanning alerts
(386-390) on #8613.
repeatRust merged each package's cargo target flags through a Set of
argv tokens, so two integration-test files in one package collapsed
`--test codec --test connection` into `--test codec connection`.
nextest reads the trailing `connection` as a test-name filter, lists
nothing, and the lane took the "nothing to repeat" branch and reported
success without running a test (verified against buzz-audio-client:
the collapsed argv lists no tests and exits 0). rustTarget now returns
one flag per entry and the new rustGroups keys each whole flag, so a
`--test <name>` is emitted once per name while `--lib`/`--bins` stay
single; the regression test asserts the exact argv for two integration
files and for a src/ plus tests/ file in the same package, through the
seam repeatRust uses. The `-p`, `-E` and filterset arguments were
re-read for the same class of bug and pass single values.
In _ci-flake.yml, actions/upload-artifact 043fb46d is v7.0.1 (the
`v7` tag has since moved to v7.0.2), so its four comments now say so,
as the other workflows do. rui314/setup-mold 7e4f20ad is a commit the
`v1` tag no longer points at (339e96b4, mold 3.0.0), so the flake
workflow pins the current `v1`; _ci-rust.yml, _ci-relay.yml and
_ci-desktop.yml carry the same stale pin and are left for a separate
change.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Matt Toohey <contact@matttoohey.com>
Address the review of the flake check port (99b21e4fb).
An integration-only Playwright selection no longer fails flake-select:
Playwright exits 1 with a lone "No tests found" error when the file
filters match nothing in the smoke project, which listPlaywright now
reads as an empty selection, as buzz-app's listBrowser does. Listed
journeys are deduplicated, since a looped test() declaration is listed
once per iteration, and a selection over 250 journeys skips the
Playwright lane with a note: the smoke lane runs ~150 journey-runs per
shard in ~11 minutes, so 250 journey-runs per shard across ten shards
is what fits the 45-minute timeout after setup and the E2E build.
Rust entries now carry the file and the test's in-file module path
(`crates/.../auth.rs::tests::name`). repeatRust runs one cargo
invocation per package with target flags derived from the path
(src/** builds --lib --bins, tests/<stem>.rs builds --test <stem>), so
a buzz-relay change no longer compiles every test target of the crate.
It lists candidates by in-file path, prefers the test whose full name
the file's path predicts, and falls back to every suffix match only
when nothing matches (a `#[path]` module), so a same-named test in
another module is no longer repeated for a change it never saw.
The rust-cache bust knob moves into the shared key
(`unit-tests-sherpa-cache-v1`) in both the Unit Tests and flake-rust
jobs, since rust-cache ignores `key` once shared-key is set; the
contract script checks both and the flake restore's save-if, and the
regression harness covers both. flake-select installs desktop
dependencies only when a spec changed. Changed .rs files without test
functions get a summary row, test.fail() declarations count as tests,
a captured nextest 0.9.136 list/JUnit pair pins the classname contract,
and CONTRIBUTING lists what the lanes deliberately do not repeat.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Matt Toohey <contact@matttoohey.com>
Add an advisory flake check that selects the tests a pull request
changes and repeats each one ten times with retries off, so a test
that passes CI by luck is caught before it merges. This mirrors
block/buzz-app#890 (and its sharding follow-up), adapted to the
runners this repository actually has.
scripts/flake-check.mjs `select` diffs HEAD^1..HEAD and maps hunks to
tests per runner: Rust files to `<package>::<fn>` entries by parsing
`#[test]`-style attributes (shared code changes select the file's
tests; `#[ignore]`d and Postgres tests are excluded), desktop
Playwright specs to `file:line` smoke journeys via `playwright --list`
(non-smoke and unregistered specs are excluded), and desktop node:test
and Flutter test files whole. Deleted and renamed-away files drop out
of the selection, and any runner whose selection exceeds 64 KiB is
dropped with a note so GitHub outputs stay within limits.
The reusable _ci-flake.yml adds `flake-select` plus one repeat lane per
runner (`flake-rust`, `flake-node`, `flake-playwright` sharded by
journey count, `flake-dart`). Each lane skips when its selection is
empty, runs only on pull_request, and is not read by any required
gate; its status and step summary carry the result. A `flake` nextest
profile disables retries and emits JUnit so the Rust lane can report
per-test, per-iteration results. The Unit Tests rust-cache gains a
`shared-key` so the flake lane can restore it read-only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Matt Toohey <contact@matttoohey.com>
