Commit Graph

2 Commits

Author SHA1 Message Date
Phil Azar
f463e726dd fix: retrieving cold memories; add regression task (#6950)
## Why

Evaluating buzz agent memory retrieval by seeding a memory then asking
the buzz agent a question it needs that memory.

**Bug Found**: System prompt had no inclusion of retrieving cold
memories and suggested looking in a mem/*.md directory that does not
exist. Updated `system-prompt.md` to include memory CLI tools and usage.

Eval Before System Prompt Change: 0/3 
Eval After System Prompt Change: 3/3 

## What

- Add a `memory-retrieval` benchmark that seeds agent memory with `buzz
mem set` before asking a direct question.
- Grade the observable threaded answer without inspecting tool calls or
exposing the answer in channel history.
- Teach agents to use `buzz mem set`, `buzz mem ls`, and `buzz mem get`
for cold memory.
- Add a wire-debug endpoint configuration for diagnosing ACP tool calls
in local runs.
- Add fixture, seeding, verifier, and prompt coverage.

## Risk Assessment

Low. The runtime changes are limited to the benchmark harness. The
production-facing change clarifies existing memory commands in the base
prompt; it does not change memory storage, relay behavior, or
authorization.

## References

- Before the system-prompt changes, 0/3 attempts passed because agents
never invoked the `buzz mem` CLI and instead searched a non existent
filesystem
- After the changes, 3/3 attempts passed. ACP wire logs confirmed that
every agent ran `buzz mem ls` followed by `buzz mem get` and returned
`net_gpv`.

---------

Signed-off-by: Philip Azar <pazar@squareup.com>
2026-08-31 17:26:24 +00:00
Tyler
f437e43d09 benchmarks: Harbor Terminal-Bench harness for Buzz agent-team orchestration (harbor-buzz-orchestra) (#1504)
Signed-off-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
2026-07-08 11:32:11 -04:00