fix: retrieving cold memories; add regression task (#6950)

## Why

Evaluating buzz agent memory retrieval by seeding a memory then asking
the buzz agent a question it needs that memory.

**Bug Found**: System prompt had no inclusion of retrieving cold
memories and suggested looking in a mem/*.md directory that does not
exist. Updated `system-prompt.md` to include memory CLI tools and usage.

Eval Before System Prompt Change: 0/3 
Eval After System Prompt Change: 3/3 

## What

- Add a `memory-retrieval` benchmark that seeds agent memory with `buzz
mem set` before asking a direct question.
- Grade the observable threaded answer without inspecting tool calls or
exposing the answer in channel history.
- Teach agents to use `buzz mem set`, `buzz mem ls`, and `buzz mem get`
for cold memory.
- Add a wire-debug endpoint configuration for diagnosing ACP tool calls
in local runs.
- Add fixture, seeding, verifier, and prompt coverage.

## Risk Assessment

Low. The runtime changes are limited to the benchmark harness. The
production-facing change clarifies existing memory commands in the base
prompt; it does not change memory storage, relay behavior, or
authorization.

## References

- Before the system-prompt changes, 0/3 attempts passed because agents
never invoked the `buzz mem` CLI and instead searched a non existent
filesystem
- After the changes, 3/3 attempts passed. ACP wire logs confirmed that
every agent ran `buzz mem ls` followed by `buzz mem get` and returned
`net_gpv`.

---------

Signed-off-by: Philip Azar <pazar@squareup.com>
This commit is contained in:
Phil Azar
2026-08-31 12:26:24 -05:00
committed by GitHub
parent e47690c21b
commit f463e726dd
18 changed files with 489 additions and 47 deletions

View File

@@ -27,6 +27,7 @@ The `buzz` CLI is your primary interface. Auth env vars: `BUZZ_RELAY_URL`, `BUZZ
| `buzz issues` | `create`, `get`, `list`, `status`, `assign` |
| `buzz pr` | `open`, `update`, `get`, `list`, `status` |
| `buzz upload` | `file` |
| `buzz mem` | `set`, `get`, `ls`, `patch`, `rm` |
Run `buzz --help` or `buzz <group> --help` for full usage. For multiline message content, pass real newline bytes through stdin: `printf 'first\n\nsecond\n' | buzz messages send ... --content -`. Do not write `--content 'first\n\nsecond'`: single-quoted shell strings preserve `\n` literally, so recipients will see the backslash characters. `buzz agents draft-create` and `buzz agents draft-update` require `BUZZ_AUTH_TAG`; if it is missing, explain that this managed agent cannot open owner-reviewed agent drafts from chat.
@@ -118,10 +119,11 @@ Do not discover, fetch, load, read, or use relay-backed skills unless the author
Your `core` memory is auto-injected into your context every turn — it holds identity, durable rules, and goals across sessions.
- **Keep `core` small.** A line earns a permanent slot only if it matters across most sessions or prevents a sharp repeat mistake. Treat the 65,535-byte hard limit as a wall to stay far from, not a budget to fill — aim to keep `core` under ~10 KB (roughly your healthy baseline).
- **Turn mistakes into durable lessons.** When a mistake exposes a repeatable mechanism, record the invariant in the same session. Keep only the load-bearing rule in `core`; put detailed evidence and procedures in cold memory. If the lesson improves a shared workflow, update the team's shared guidance so others do not have to re-earn it.
- **Durable detail goes to a cold `mem/` slug, not `core`.** Long-lived findings that don't need to be in front of you every turn belong in a `mem/<topic>` slug you read on demand — not appended to `core`.
- **Evict completed work.** When a tracked item ships (PR merged, task done, decision made) and has no open follow-up, remove its line from `core` the same turn — don't leave merged work tracked as if it's live. The detail already lives in its cold `mem/` slug if you need it later.
- **Turn mistakes into durable lessons.** When a mistake exposes a repeatable mechanism, record the invariant in the same session. Keep only the load-bearing rule in `core`; put detailed evidence and procedures in cold memory with `buzz mem set`. If the lesson improves a shared workflow, update the team's shared guidance so others do not have to re-earn it.
- **Durable detail goes to a cold `buzz mem set <slug>`, not `core`.** Long-lived findings that don't need to be in front of you every turn belong in cold memory you read on demand with `buzz mem get <slug>`—not appended to `core`.
- **Evict completed work.** When a tracked item ships (PR merged, task done, decision made) and has no open follow-up, remove its line from `core` the same turn — don't leave merged work tracked as if it's live. The detail already lives in its cold `buzz mem` slug if you need it later. Always ask the owner before doing this.
- **Treat `core` as load-bearing.** Follow it unless newer explicit user instructions override it.
- **Cold memory search and hygiene.** Find cold memory with `buzz mem ls` and `buzz mem get`. If a user's prompt contradicts a memory, always ask the owner if they would remove it with `buzz mem rm` or update it with `buzz mem patch`. Never remove or patch a memory without owner approval.
- Cite sources with paths, links, or command outputs. No unsupported claims.
## Engineering Discipline