mirror of
https://github.com/block/buzz.git
synced 2026-10-30 05:51:46 +00:00
f463e726dd
## Why Evaluating buzz agent memory retrieval by seeding a memory then asking the buzz agent a question it needs that memory. **Bug Found**: System prompt had no inclusion of retrieving cold memories and suggested looking in a mem/*.md directory that does not exist. Updated `system-prompt.md` to include memory CLI tools and usage. Eval Before System Prompt Change: 0/3 Eval After System Prompt Change: 3/3 ## What - Add a `memory-retrieval` benchmark that seeds agent memory with `buzz mem set` before asking a direct question. - Grade the observable threaded answer without inspecting tool calls or exposing the answer in channel history. - Teach agents to use `buzz mem set`, `buzz mem ls`, and `buzz mem get` for cold memory. - Add a wire-debug endpoint configuration for diagnosing ACP tool calls in local runs. - Add fixture, seeding, verifier, and prompt coverage. ## Risk Assessment Low. The runtime changes are limited to the benchmark harness. The production-facing change clarifies existing memory commands in the base prompt; it does not change memory storage, relay behavior, or authorization. ## References - Before the system-prompt changes, 0/3 attempts passed because agents never invoked the `buzz mem` CLI and instead searched a non existent filesystem - After the changes, 3/3 attempts passed. ACP wire logs confirmed that every agent ran `buzz mem ls` followed by `buzz mem get` and returned `net_gpv`. --------- Signed-off-by: Philip Azar <pazar@squareup.com>
19 lines
1.1 KiB
Markdown
19 lines
1.1 KiB
Markdown
# memory-retrieval
|
|
|
|
Before the agent starts, the harness runs `buzz mem set` with the agent's own
|
|
Buzz credentials to seed five similar cold memories. One records the exact
|
|
total customer count for April 2024; the other four contain customer counts for
|
|
nearby months or related April metrics. The harness then delivers
|
|
`instruction.md`, which contains only the retrieval question and does not reveal
|
|
the answer or memory slug. No channel message contains the answer, so
|
|
conversation history cannot supply it.
|
|
|
|
Full credit requires the exact customer count `352,345` in the threaded answer.
|
|
Equivalent comma-free formatting is accepted, but rounded or approximate counts
|
|
receive no credit. Credit is also voided if the answer mentions another number,
|
|
apart from the requested year `2024`. This includes every count drawn from the
|
|
distractor memories, so dumping several memories or selecting the wrong one does
|
|
not pass — the answer must resolve to the correct value alone. The verifier does
|
|
not inspect tool calls: seeding is deterministic harness setup, and retrieval is
|
|
graded only through the observable answer.
|