jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-11-01 23:11:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(buzz-agent): rewrite HANDTEST.md for the swap

Browse Source 
The old version described a parallel `buzz-agent-core` crate and a `just
agent-core` recipe, neither of which exists any more -- the goose-backed loop
IS buzz-agent now.

There is nothing special to run: `just dev` and `just goose` already build and
use the swapped crate. If you see a difference, that is the bug.

Keeps the seven human-only checks (persona arrival, _Stop veto, streaming feel,
cancel leaving no stuck spinner, steering, model picker, hook-tool hygiene) and
the known gaps.

Signed-off-by: Michael Neale <michael.neale@gmail.com>
This commit is contained in:
Michael Neale
2026-07-27 17:28:56 +10:00
parent 1f6b75a7dc
commit ac6aed1eaf
1 changed files with 56 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files

150
crates/buzz-agent/HANDTEST.md
View File

@@ -1,129 +1,91 @@
# Hand-testing `buzz-agent-core`
# Hand-testing `buzz-agent`
The premise: this binary **is** `buzz-agent`. Same name, same ACP contract,
goose underneath. So hand-testing is a *swap*, not a new setup — you run the
normal buzz-acp flow and repoint one environment variable.
This is a **swap**, not a new thing. `buzz-agent` keeps its name, its binary,
its ACP contract and its place in the workspace — only the internals changed
from a hand-written agent loop to the `goose` crate used as a Rust library.
`buzz-acp` derives harness identity from the command's **basename**
(`normalize_agent_command_identity`, `buzz-acp/src/config.rs:600-615`) and
already has a `"buzz-agent"` arm meaning "no extra args"
(`default_agent_args`, `:617-624`). The binary is emitted as `buzz-agent`
precisely so buzz-acp cannot tell the difference.
## 0. Build
So there is nothing special to run. The normal flow *is* the test:
```bash
. ./bin/activate-hermit
cargo build --release -p buzz-acp -p buzz-dev-mcp -p buzz-cli
cargo build --release --manifest-path buzz-agent-core/Cargo.toml
just dev # the whole app
# or
just relay # terminal 1
just goose # terminal 2 — agent against that relay
```
Takes a few minutes the first time (goose pulls ~700 crates into its own
lockfile).
Both already build and use the swapped `buzz-agent`. If you see a difference,
that is the bug.
## 1. The swap
## What to look at
```bash
just relay # terminal 1
Ordered by risk. The automated suite (41 tests) covers each of these at the
stdio layer; this list is the part only a human can judge.
# terminal 2
just agent-core # new: goose-backed buzz-agent
just goose # old: for A/B comparison
```
`just agent-core` is `just goose` with two lines changed:
```diff
- BUZZ_ACP_AGENT_COMMAND=goose
- BUZZ_ACP_AGENT_ARGS=acp
+ BUZZ_ACP_AGENT_COMMAND=<repo>/buzz-agent-core/target/release/buzz-agent
+ BUZZ_ACP_MCP_COMMAND=<repo>/target/release/buzz-dev-mcp
```
Everything else — relay URL, keys, agent count, heartbeat — is identical.
## 2. What to actually check
Ordered by how likely it is to be broken and how badly it would matter.
The automated suite covers each of these at the stdio layer; this is about
the parts only a human can see.
### A. Persona actually arrives — the whole reason for embedding
### 1. Persona actually arrives
```
@fizz who are you?
```
Should answer *as Fizz*, and should know the `buzz` CLI exists. If it answers
as generic goose, `systemPrompt` was dropped — the exact failure that makes
plain-ACP embedding impossible (goose's own ACP server never reads
`systemPrompt`; both PRs that would have wired it, buzz#1290 and goose#9971,
are closed unmerged).
Should answer *as Fizz* and know the `buzz` CLI exists. A generic goose answer
means the system prompt was dropped — the exact failure that makes plain-ACP
embedding impossible, and the reason this uses the library API instead.
### B. `_Stop` veto — buzz-agent's most load-bearing behaviour
### 2. `_Stop` veto
```
@fizz make a todo list with 3 items, then stop immediately without doing them
```
It must **refuse to stop** while items are open, and keep working. Capped at 3
vetoes, so it will eventually end regardless. Watch for
`_Stop hook vetoed end of turn` in the buzz-acp log.
Must refuse to stop while items are open. Capped at 3 vetoes, so it ends
eventually regardless. Look for `_Stop hook vetoed end of turn` in the log.
### C. Streaming feel
### 3. Streaming feel
buzz-agent emitted one chunk per round; goose streams token-by-token. The
automated tests prove the relay isn't write-amplified (chunks are coalesced by
identity key, flushed at 500ms, paced at 167ms/90-per-minute), but **only a
human can tell you whether it feels better or worse in the desktop app.**
This is the single most likely source of "something feels off".
The old loop emitted one chunk per round; goose streams token-by-token. Tests
prove the relay is not write-amplified (chunks are coalesced by identity key,
flushed at 500ms, paced at 167ms/90-per-minute), but **only a human can say
whether it feels better or worse in the desktop app.** This is the most likely
source of "something is off".
### D. Cancel mid-tool
### 4. Cancel mid-tool
Ask for something long (`count slowly to 100 with a shell sleep`), then hit
stop. Check: the turn ends promptly, and **no tool call is left spinning** in
the UI. Cancellation is a cooperative drain (5s budget) — dropping the stream
instead leaves the MCP child running and the spinner stuck forever. That bug
existed and is fixed; this is the visual confirmation.
Ask for something long (`count slowly to 100 with a shell sleep`), then stop.
The turn should end promptly with **no tool call left spinning**. Cancellation
is a cooperative drain with a 5s budget — dropping the stream instead would
leave the MCP child running and the spinner stuck forever. That bug existed and
is fixed; this is the visual confirmation.
### E. Steering
### 5. Steering
Send a second message while the agent is working. It should be absorbed into
the running turn, *not* cancel-and-restart it. If the desktop visibly restarts
the turn, `activeRunId` is at the wrong nesting depth — it must be at
`params.update._meta.goose.activeRunId`, with `_meta` **inside** `update`
(`buzz-acp/src/acp.rs:1607-1613`). Wrong depth degrades silently to
cancel+re-prompt with no error anywhere.
the running turn, not cancel-and-restart it.
### F. Model picker
### 6. Model picker
Open the model picker in the desktop app. It should list models, not just the
current one. Absent catalog is degraded UX, never a session failure.
Should list models, not just the current one. An absent catalog is degraded UX,
never a session failure.
### G. Tool hygiene — known deviation
### 7. Tool hygiene — known deviation
The model can now *see* `_Stop` and `_PostCompact` (goose's allowlist gates
advertising and dispatch together, so hiding them would break the veto). A
system-prompt extension tells it not to call them. Watch for the agent calling
`_Stop` on its own — if that happens, the guidance text needs strengthening.
The model can now *see* `_Stop` and `_PostCompact`: goose's allowlist gates
advertising and dispatch through the same cache, so hiding them would also make
them undispatchable and break the veto. A system-prompt extension tells the
model to leave them alone. Watch for it calling `_Stop` itself — if that
happens the guidance needs strengthening.
## 3. A/B against the old agent
## Comparing against the old behaviour
The old binary still exists, so run both against the same relay and compare:
The old loop is gone, so A/B means checking out `main` in a second worktree and
running that relay side by side.
```bash
just goose # old, terminal 2
just agent-core # new, terminal 3
```
## Known gaps
Two agents, two identities, same channel. Ask both the same thing.
## 4. Known not-done
* Never run against a **real** provider — all automated coverage uses a fake
SSE server. Databricks OAuth in particular is entirely goose's code path now
and completely unexercised.
* Never run inside the **desktop app** — only via buzz-acp on the CLI.
* `buzz-agent` (old) is not deleted, nothing is wired into packaging or the
harness catalog.
* Binary is +22.7 MiB raw / +6.0 MiB gzip vs the old one.
* **Never run against a real provider.** All automated coverage uses a fake SSE
server. Databricks *chat* now goes through goose entirely and is unexercised
— though Databricks *model discovery* still uses our own code, moved to
`buzz-model-catalog` (the desktop cannot link goose: native `sqlite3`
collision with its own rusqlite).
* Binary is roughly +22.7 MiB raw / +6.0 MiB gzip.
* Nothing changed in packaging or the harness catalog — same sidecar name, so
nothing needed to.