From ac6aed1eafd02d4624d4e712ab6652ebea7307d7 Mon Sep 17 00:00:00 2001 From: Michael Neale Date: Mon, 27 Jul 2026 17:28:56 +1000 Subject: [PATCH] docs(buzz-agent): rewrite HANDTEST.md for the swap The old version described a parallel `buzz-agent-core` crate and a `just agent-core` recipe, neither of which exists any more -- the goose-backed loop IS buzz-agent now. There is nothing special to run: `just dev` and `just goose` already build and use the swapped crate. If you see a difference, that is the bug. Keeps the seven human-only checks (persona arrival, _Stop veto, streaming feel, cancel leaving no stuck spinner, steering, model picker, hook-tool hygiene) and the known gaps. Signed-off-by: Michael Neale --- crates/buzz-agent/HANDTEST.md | 150 +++++++++++++--------------------- 1 file changed, 56 insertions(+), 94 deletions(-) diff --git a/crates/buzz-agent/HANDTEST.md b/crates/buzz-agent/HANDTEST.md index b24f722313..8661798b20 100644 --- a/crates/buzz-agent/HANDTEST.md +++ b/crates/buzz-agent/HANDTEST.md @@ -1,129 +1,91 @@ -# Hand-testing `buzz-agent-core` +# Hand-testing `buzz-agent` -The premise: this binary **is** `buzz-agent`. Same name, same ACP contract, -goose underneath. So hand-testing is a *swap*, not a new setup — you run the -normal buzz-acp flow and repoint one environment variable. +This is a **swap**, not a new thing. `buzz-agent` keeps its name, its binary, +its ACP contract and its place in the workspace — only the internals changed +from a hand-written agent loop to the `goose` crate used as a Rust library. -`buzz-acp` derives harness identity from the command's **basename** -(`normalize_agent_command_identity`, `buzz-acp/src/config.rs:600-615`) and -already has a `"buzz-agent"` arm meaning "no extra args" -(`default_agent_args`, `:617-624`). The binary is emitted as `buzz-agent` -precisely so buzz-acp cannot tell the difference. - -## 0. Build +So there is nothing special to run. The normal flow *is* the test: ```bash -. ./bin/activate-hermit -cargo build --release -p buzz-acp -p buzz-dev-mcp -p buzz-cli -cargo build --release --manifest-path buzz-agent-core/Cargo.toml +just dev # the whole app +# or +just relay # terminal 1 +just goose # terminal 2 — agent against that relay ``` -Takes a few minutes the first time (goose pulls ~700 crates into its own -lockfile). +Both already build and use the swapped `buzz-agent`. If you see a difference, +that is the bug. -## 1. The swap +## What to look at -```bash -just relay # terminal 1 +Ordered by risk. The automated suite (41 tests) covers each of these at the +stdio layer; this list is the part only a human can judge. -# terminal 2 -just agent-core # new: goose-backed buzz-agent -just goose # old: for A/B comparison -``` - -`just agent-core` is `just goose` with two lines changed: - -```diff -- BUZZ_ACP_AGENT_COMMAND=goose -- BUZZ_ACP_AGENT_ARGS=acp -+ BUZZ_ACP_AGENT_COMMAND=/buzz-agent-core/target/release/buzz-agent -+ BUZZ_ACP_MCP_COMMAND=/target/release/buzz-dev-mcp -``` - -Everything else — relay URL, keys, agent count, heartbeat — is identical. - -## 2. What to actually check - -Ordered by how likely it is to be broken and how badly it would matter. -The automated suite covers each of these at the stdio layer; this is about -the parts only a human can see. - -### A. Persona actually arrives — the whole reason for embedding +### 1. Persona actually arrives ``` @fizz who are you? ``` -Should answer *as Fizz*, and should know the `buzz` CLI exists. If it answers -as generic goose, `systemPrompt` was dropped — the exact failure that makes -plain-ACP embedding impossible (goose's own ACP server never reads -`systemPrompt`; both PRs that would have wired it, buzz#1290 and goose#9971, -are closed unmerged). +Should answer *as Fizz* and know the `buzz` CLI exists. A generic goose answer +means the system prompt was dropped — the exact failure that makes plain-ACP +embedding impossible, and the reason this uses the library API instead. -### B. `_Stop` veto — buzz-agent's most load-bearing behaviour +### 2. `_Stop` veto ``` @fizz make a todo list with 3 items, then stop immediately without doing them ``` -It must **refuse to stop** while items are open, and keep working. Capped at 3 -vetoes, so it will eventually end regardless. Watch for -`_Stop hook vetoed end of turn` in the buzz-acp log. +Must refuse to stop while items are open. Capped at 3 vetoes, so it ends +eventually regardless. Look for `_Stop hook vetoed end of turn` in the log. -### C. Streaming feel +### 3. Streaming feel -buzz-agent emitted one chunk per round; goose streams token-by-token. The -automated tests prove the relay isn't write-amplified (chunks are coalesced by -identity key, flushed at 500ms, paced at 167ms/90-per-minute), but **only a -human can tell you whether it feels better or worse in the desktop app.** -This is the single most likely source of "something feels off". +The old loop emitted one chunk per round; goose streams token-by-token. Tests +prove the relay is not write-amplified (chunks are coalesced by identity key, +flushed at 500ms, paced at 167ms/90-per-minute), but **only a human can say +whether it feels better or worse in the desktop app.** This is the most likely +source of "something is off". -### D. Cancel mid-tool +### 4. Cancel mid-tool -Ask for something long (`count slowly to 100 with a shell sleep`), then hit -stop. Check: the turn ends promptly, and **no tool call is left spinning** in -the UI. Cancellation is a cooperative drain (5s budget) — dropping the stream -instead leaves the MCP child running and the spinner stuck forever. That bug -existed and is fixed; this is the visual confirmation. +Ask for something long (`count slowly to 100 with a shell sleep`), then stop. +The turn should end promptly with **no tool call left spinning**. Cancellation +is a cooperative drain with a 5s budget — dropping the stream instead would +leave the MCP child running and the spinner stuck forever. That bug existed and +is fixed; this is the visual confirmation. -### E. Steering +### 5. Steering Send a second message while the agent is working. It should be absorbed into -the running turn, *not* cancel-and-restart it. If the desktop visibly restarts -the turn, `activeRunId` is at the wrong nesting depth — it must be at -`params.update._meta.goose.activeRunId`, with `_meta` **inside** `update` -(`buzz-acp/src/acp.rs:1607-1613`). Wrong depth degrades silently to -cancel+re-prompt with no error anywhere. +the running turn, not cancel-and-restart it. -### F. Model picker +### 6. Model picker -Open the model picker in the desktop app. It should list models, not just the -current one. Absent catalog is degraded UX, never a session failure. +Should list models, not just the current one. An absent catalog is degraded UX, +never a session failure. -### G. Tool hygiene — known deviation +### 7. Tool hygiene — known deviation -The model can now *see* `_Stop` and `_PostCompact` (goose's allowlist gates -advertising and dispatch together, so hiding them would break the veto). A -system-prompt extension tells it not to call them. Watch for the agent calling -`_Stop` on its own — if that happens, the guidance text needs strengthening. +The model can now *see* `_Stop` and `_PostCompact`: goose's allowlist gates +advertising and dispatch through the same cache, so hiding them would also make +them undispatchable and break the veto. A system-prompt extension tells the +model to leave them alone. Watch for it calling `_Stop` itself — if that +happens the guidance needs strengthening. -## 3. A/B against the old agent +## Comparing against the old behaviour -The old binary still exists, so run both against the same relay and compare: +The old loop is gone, so A/B means checking out `main` in a second worktree and +running that relay side by side. -```bash -just goose # old, terminal 2 -just agent-core # new, terminal 3 -``` +## Known gaps -Two agents, two identities, same channel. Ask both the same thing. - -## 4. Known not-done - -* Never run against a **real** provider — all automated coverage uses a fake - SSE server. Databricks OAuth in particular is entirely goose's code path now - and completely unexercised. -* Never run inside the **desktop app** — only via buzz-acp on the CLI. -* `buzz-agent` (old) is not deleted, nothing is wired into packaging or the - harness catalog. -* Binary is +22.7 MiB raw / +6.0 MiB gzip vs the old one. +* **Never run against a real provider.** All automated coverage uses a fake SSE + server. Databricks *chat* now goes through goose entirely and is unexercised + — though Databricks *model discovery* still uses our own code, moved to + `buzz-model-catalog` (the desktop cannot link goose: native `sqlite3` + collision with its own rusqlite). +* Binary is roughly +22.7 MiB raw / +6.0 MiB gzip. +* Nothing changed in packaging or the harness catalog — same sidecar name, so + nothing needed to.