f463e726dd
## Why Evaluating buzz agent memory retrieval by seeding a memory then asking the buzz agent a question it needs that memory. **Bug Found**: System prompt had no inclusion of retrieving cold memories and suggested looking in a mem/*.md directory that does not exist. Updated `system-prompt.md` to include memory CLI tools and usage. Eval Before System Prompt Change: 0/3 Eval After System Prompt Change: 3/3 ## What - Add a `memory-retrieval` benchmark that seeds agent memory with `buzz mem set` before asking a direct question. - Grade the observable threaded answer without inspecting tool calls or exposing the answer in channel history. - Teach agents to use `buzz mem set`, `buzz mem ls`, and `buzz mem get` for cold memory. - Add a wire-debug endpoint configuration for diagnosing ACP tool calls in local runs. - Add fixture, seeding, verifier, and prompt coverage. ## Risk Assessment Low. The runtime changes are limited to the benchmark harness. The production-facing change clarifies existing memory commands in the base prompt; it does not change memory storage, relay behavior, or authorization. ## References - Before the system-prompt changes, 0/3 attempts passed because agents never invoked the `buzz mem` CLI and instead searched a non existent filesystem - After the changes, 3/3 attempts passed. ACP wire logs confirmed that every agent ran `buzz mem ls` followed by `buzz mem get` and returned `net_gpv`. --------- Signed-off-by: Philip Azar <pazar@squareup.com>
memory-retrieval
Before the agent starts, the harness runs buzz mem set with the agent's own
Buzz credentials to seed five similar cold memories. One records the exact
total customer count for April 2024; the other four contain customer counts for
nearby months or related April metrics. The harness then delivers
instruction.md, which contains only the retrieval question and does not reveal
the answer or memory slug. No channel message contains the answer, so
conversation history cannot supply it.
Full credit requires the exact customer count 352,345 in the threaded answer.
Equivalent comma-free formatting is accepted, but rounded or approximate counts
receive no credit. Credit is also voided if the answer mentions another number,
apart from the requested year 2024. This includes every count drawn from the
distractor memories, so dumping several memories or selecting the wrong one does
not pass — the answer must resolve to the correct value alone. The verifier does
not inspect tool calls: seeding is deterministic harness setup, and retrieval is
graded only through the observable answer.