From 3d4017b15ab55daa459fc8a970a4397f636d832a Mon Sep 17 00:00:00 2001 From: Ben Brandt Date: Fri, 2 Oct 2026 06:15:00 -0700 Subject: [PATCH] test: keep the suite off the real Claude CLI (#1223) Fixes an intermittent failure of "replays a subagent from its own transcript file", and with it a broader leak: the unit suite was running the real Claude CLI hundreds of times per run. ## Cause The agent runs `claude auth status --json` in the background, without awaiting it, on every `initialize` and prompt. The scenario suite stubs it (`CLAUDE_CODE_EXECUTABLE=/usr/bin/false`), but other files didn't. Logging the probe's executable over one `vitest run` showed about 670 runs of the real CLI: | File | Real CLI runs | |---|---| | `acp-agent.test.ts` | 375 | | `create-session-options.test.ts` | 14 | | `incomplete-tools.test.ts` | 12 | | `session-titles.test.ts` | 8 | | `file-change-audit-integration.test.ts` | 5 | Each run uses the machine's own Claude config, unless a test points `CLAUDE_CONFIG_DIR` elsewhere. That flaky test points `CLAUDE_CONFIG_DIR` at a temp dir. The CLI it started then wrote `.claude.json`, `.claude.json.lock` and `backups/` into that dir about a second later, after the test body had finished. Under full-suite load, that overlapped the `rm` in the test's `finally`, which failed with `ENOTEMPTY`. Its assertions always passed. ## Change `vitest.config.ts` sets `CLAUDE_CODE_EXECUTABLE=/usr/bin/false` in `test.env`: - It's the override `claudeCliPath()` already honors, and the value the scenario suite already uses, so production code doesn't change. - `test.env` is the base environment, so it survives the `vi.unstubAllEnvs()` that three files call. - Runs that exercise the real CLI keep it: with `RUN_INTEGRATION_TESTS` (`npm run test:integration`) or `RUN_LIVE_CAPABILITY_AUDIT=true`, the variable is left unset. Tests that mock `execFile` or set their own executable are unaffected. ## Evidence - After the change, the same logging shows 0 runs of the real CLI: 688 probes run `/usr/bin/false`, and 108 hit files that already mock `execFile`. - The flaky test's temp dir now holds only the test's own `projects/` after the test, with no `.claude.json`. - Before the change, it failed in about 2 of 6 full runs on my machine. After, 8 of 8 full runs passed. - A full run takes about 4.8 s instead of 8.4 s. - The variable is `/usr/bin/false` in a default run and unset with either opt-in flag. - `npm run build`, `npm run check`, `npm run test:run`. --- vitest.config.ts | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/vitest.config.ts b/vitest.config.ts index 0f8d1d9d..fd29ac41 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -1,10 +1,19 @@ import { defineConfig } from "vitest/config"; +// Runs that exercise the real Claude CLI opt in with these variables. +const usesRealCli = + Boolean(process.env.RUN_INTEGRATION_TESTS) || process.env.RUN_LIVE_CAPABILITY_AUDIT === "true"; + export default defineConfig({ test: { watch: false, globals: true, environment: "node", + // The agent runs `claude auth status` in the background on every + // initialize and prompt. Pointing it at a CLI that exits at once keeps + // tests off the machine's real CLI and Claude config, and stops a run + // from outliving its test (it wrote into temp config dirs being deleted). + env: usesRealCli ? {} : { CLAUDE_CODE_EXECUTABLE: "/usr/bin/false" }, include: [ "src/**/*.{test,spec}.{js,mjs,cjs,ts,mts,cts,jsx,tsx}", // Release tooling lives in scripts/ rather than src/ so it stays outside tsc's