mirror of
https://github.com/block/buzz.git
synced 2026-10-27 04:30:52 +00:00
fix: retrieving cold memories; add regression task (#6950)
## Why Evaluating buzz agent memory retrieval by seeding a memory then asking the buzz agent a question it needs that memory. **Bug Found**: System prompt had no inclusion of retrieving cold memories and suggested looking in a mem/*.md directory that does not exist. Updated `system-prompt.md` to include memory CLI tools and usage. Eval Before System Prompt Change: 0/3 Eval After System Prompt Change: 3/3 ## What - Add a `memory-retrieval` benchmark that seeds agent memory with `buzz mem set` before asking a direct question. - Grade the observable threaded answer without inspecting tool calls or exposing the answer in channel history. - Teach agents to use `buzz mem set`, `buzz mem ls`, and `buzz mem get` for cold memory. - Add a wire-debug endpoint configuration for diagnosing ACP tool calls in local runs. - Add fixture, seeding, verifier, and prompt coverage. ## Risk Assessment Low. The runtime changes are limited to the benchmark harness. The production-facing change clarifies existing memory commands in the base prompt; it does not change memory storage, relay behavior, or authorization. ## References - Before the system-prompt changes, 0/3 attempts passed because agents never invoked the `buzz mem` CLI and instead searched a non existent filesystem - After the changes, 3/3 attempts passed. ACP wire logs confirmed that every agent ran `buzz mem ls` followed by `buzz mem get` and returned `net_gpv`. --------- Signed-off-by: Philip Azar <pazar@squareup.com>
This commit is contained in:
@@ -27,6 +27,7 @@ The `buzz` CLI is your primary interface. Auth env vars: `BUZZ_RELAY_URL`, `BUZZ
|
||||
| `buzz issues` | `create`, `get`, `list`, `status`, `assign` |
|
||||
| `buzz pr` | `open`, `update`, `get`, `list`, `status` |
|
||||
| `buzz upload` | `file` |
|
||||
| `buzz mem` | `set`, `get`, `ls`, `patch`, `rm` |
|
||||
|
||||
Run `buzz --help` or `buzz <group> --help` for full usage. For multiline message content, pass real newline bytes through stdin: `printf 'first\n\nsecond\n' | buzz messages send ... --content -`. Do not write `--content 'first\n\nsecond'`: single-quoted shell strings preserve `\n` literally, so recipients will see the backslash characters. `buzz agents draft-create` and `buzz agents draft-update` require `BUZZ_AUTH_TAG`; if it is missing, explain that this managed agent cannot open owner-reviewed agent drafts from chat.
|
||||
|
||||
@@ -118,10 +119,11 @@ Do not discover, fetch, load, read, or use relay-backed skills unless the author
|
||||
Your `core` memory is auto-injected into your context every turn — it holds identity, durable rules, and goals across sessions.
|
||||
|
||||
- **Keep `core` small.** A line earns a permanent slot only if it matters across most sessions or prevents a sharp repeat mistake. Treat the 65,535-byte hard limit as a wall to stay far from, not a budget to fill — aim to keep `core` under ~10 KB (roughly your healthy baseline).
|
||||
- **Turn mistakes into durable lessons.** When a mistake exposes a repeatable mechanism, record the invariant in the same session. Keep only the load-bearing rule in `core`; put detailed evidence and procedures in cold memory. If the lesson improves a shared workflow, update the team's shared guidance so others do not have to re-earn it.
|
||||
- **Durable detail goes to a cold `mem/` slug, not `core`.** Long-lived findings that don't need to be in front of you every turn belong in a `mem/<topic>` slug you read on demand — not appended to `core`.
|
||||
- **Evict completed work.** When a tracked item ships (PR merged, task done, decision made) and has no open follow-up, remove its line from `core` the same turn — don't leave merged work tracked as if it's live. The detail already lives in its cold `mem/` slug if you need it later.
|
||||
- **Turn mistakes into durable lessons.** When a mistake exposes a repeatable mechanism, record the invariant in the same session. Keep only the load-bearing rule in `core`; put detailed evidence and procedures in cold memory with `buzz mem set`. If the lesson improves a shared workflow, update the team's shared guidance so others do not have to re-earn it.
|
||||
- **Durable detail goes to a cold `buzz mem set <slug>`, not `core`.** Long-lived findings that don't need to be in front of you every turn belong in cold memory you read on demand with `buzz mem get <slug>`—not appended to `core`.
|
||||
- **Evict completed work.** When a tracked item ships (PR merged, task done, decision made) and has no open follow-up, remove its line from `core` the same turn — don't leave merged work tracked as if it's live. The detail already lives in its cold `buzz mem` slug if you need it later. Always ask the owner before doing this.
|
||||
- **Treat `core` as load-bearing.** Follow it unless newer explicit user instructions override it.
|
||||
- **Cold memory search and hygiene.** Find cold memory with `buzz mem ls` and `buzz mem get`. If a user's prompt contradicts a memory, always ask the owner if they would remove it with `buzz mem rm` or update it with `buzz mem patch`. Never remove or patch a memory without owner approval.
|
||||
- Cite sources with paths, links, or command outputs. No unsupported claims.
|
||||
|
||||
## Engineering Discipline
|
||||
|
||||
Reference in New Issue
Block a user