## Why
Evaluating buzz agent memory retrieval by seeding a memory then asking
the buzz agent a question it needs that memory.
**Bug Found**: System prompt had no inclusion of retrieving cold
memories and suggested looking in a mem/*.md directory that does not
exist. Updated `system-prompt.md` to include memory CLI tools and usage.
Eval Before System Prompt Change: 0/3
Eval After System Prompt Change: 3/3
## What
- Add a `memory-retrieval` benchmark that seeds agent memory with `buzz
mem set` before asking a direct question.
- Grade the observable threaded answer without inspecting tool calls or
exposing the answer in channel history.
- Teach agents to use `buzz mem set`, `buzz mem ls`, and `buzz mem get`
for cold memory.
- Add a wire-debug endpoint configuration for diagnosing ACP tool calls
in local runs.
- Add fixture, seeding, verifier, and prompt coverage.
## Risk Assessment
Low. The runtime changes are limited to the benchmark harness. The
production-facing change clarifies existing memory commands in the base
prompt; it does not change memory storage, relay behavior, or
authorization.
## References
- Before the system-prompt changes, 0/3 attempts passed because agents
never invoked the `buzz mem` CLI and instead searched a non existent
filesystem
- After the changes, 3/3 attempts passed. ACP wire logs confirmed that
every agent ran `buzz mem ls` followed by `buzz mem get` and returned
`net_gpv`.
---------
Signed-off-by: Philip Azar <pazar@squareup.com>
## Why
Terminal-Bench can show task correctness while missing Buzz-specific
collaboration failures, such as posting a final answer outside the
user's thread. Add a small Buzz-native suite that scores relay behavior
directly.
## What
- Add Harbor tasks for direct thread replies, event-level callback
mentions to a three-word user identity, targeted reads of explicitly
named paths outside the workspace, and exact temporary-channel
membership against a seeded directory of 50 users and 10 bots
- Export sanitized post-agent relay evidence and extend solo
runtime/provisioning for deterministic verification
- Expose channel TTL in CLI search output and cover positive and
negative transcript/state fixtures
## Risk Assessment
Low to medium — this primarily adds benchmark and testbed machinery. The
CLI change is an additive `ttl_seconds` field in channel-search JSON;
relay behavior is unchanged.
## References
- Motivated by the Buzz-specific behavior observed while benchmarking
https://github.com/block/buzz/pull/6128
- The named-path regression case covers the issue being fixed by
https://github.com/block/buzz/pull/6261
- Local live Sonnet runs: `reply-to-thread`, `user-mention`, and
`create-channel-invite-users` each scored reward 1.0 with zero
exceptions
- The current branch prompt scored 3/5 on a concurrent
`read-named-path-outside-workspace` batch: one explicit
workspace-boundary refusal and one missing final response
Generated with Codex
---------
Signed-off-by: Salman Mohammed <smohammed@squareup.com>
