mirror of
https://github.com/block/buzz.git
synced 2026-10-27 04:30:52 +00:00
f463e726dd
## Why Evaluating buzz agent memory retrieval by seeding a memory then asking the buzz agent a question it needs that memory. **Bug Found**: System prompt had no inclusion of retrieving cold memories and suggested looking in a mem/*.md directory that does not exist. Updated `system-prompt.md` to include memory CLI tools and usage. Eval Before System Prompt Change: 0/3 Eval After System Prompt Change: 3/3 ## What - Add a `memory-retrieval` benchmark that seeds agent memory with `buzz mem set` before asking a direct question. - Grade the observable threaded answer without inspecting tool calls or exposing the answer in channel history. - Teach agents to use `buzz mem set`, `buzz mem ls`, and `buzz mem get` for cold memory. - Add a wire-debug endpoint configuration for diagnosing ACP tool calls in local runs. - Add fixture, seeding, verifier, and prompt coverage. ## Risk Assessment Low. The runtime changes are limited to the benchmark harness. The production-facing change clarifies existing memory commands in the base prompt; it does not change memory storage, relay behavior, or authorization. ## References - Before the system-prompt changes, 0/3 attempts passed because agents never invoked the `buzz mem` CLI and instead searched a non existent filesystem - After the changes, 3/3 attempts passed. ACP wire logs confirmed that every agent ran `buzz mem ls` followed by `buzz mem get` and returned `net_gpv`. --------- Signed-off-by: Philip Azar <pazar@squareup.com>
77 lines
4.3 KiB
Markdown
77 lines
4.3 KiB
Markdown
# buzz-dataset
|
|
|
|
Harbor tasks that score **Buzz product behavior**, not just task correctness.
|
|
Each task poses an ordinary-looking question; what is graded is how the agent
|
|
answers it through Buzz — where the reply lands, who it notifies, what it was
|
|
willing to read.
|
|
|
|
| Task | Layer | Behavior under test |
|
|
| --- | --- | --- |
|
|
| [`reply-to-thread`](reply-to-thread) | Regression | Answers in the user's thread instead of as a new top-level message |
|
|
| [`user-mention`](user-mention) | Regression | Hands the turn back with an event-level `p`-tag mention of the requesting human |
|
|
| [`read-named-path-outside-workspace`](read-named-path-outside-workspace) | Regression | Reads a path the user named explicitly instead of refusing it as out of bounds |
|
|
| [`create-channel-invite-users`](create-channel-invite-users) | Workflow | Creates a channel with the exact shape, TTL, and membership asked for |
|
|
| [`multiline-message`](multiline-message) | Regression | Preserves real newlines and blank-line structure through the CLI publish path |
|
|
| [`narrative-agent-names`](narrative-agent-names) | Regression | Names agents in narrative without waking them through `p` tags |
|
|
| [`interleaved-agent-reports`](interleaved-agent-reports) | Workflow | Retains and synthesizes every report in a batch of agent messages |
|
|
| [`cross-thread-requests`](cross-thread-requests) | Workflow | Keeps simultaneous top-level requests isolated and replies to both exact threads |
|
|
| [`ambiguous-user-mention`](ambiguous-user-mention) | Workflow | Resolves duplicate display names and notifies only the intended pubkey |
|
|
| [`memory-retrieval`](memory-retrieval) | Regression | Answers from harness-seeded cold memory without the value appearing in channel history |
|
|
|
|
For `reply-to-thread` and `user-mention` the graded behavior is **deliberately
|
|
absent from `instruction.md`** — it has to come from `buzz-acp`'s production
|
|
base prompt. Read a task's own `README.md` before editing its instruction or
|
|
verifier.
|
|
|
|
## Evaluation layers
|
|
|
|
Every task declares `metadata.evaluation_layer` in `task.toml`:
|
|
|
|
| Layer | Question | Default trials | Typical cadence |
|
|
| --- | --- | ---: | --- |
|
|
| Regression | Did Buzz preserve a known product contract? | k=1 | Targeted PR, nightly, or pre-release |
|
|
| Workflow | How capable is the agent at realistic Buzz work? | k=3 | Nightly or weekly on a fixed condition |
|
|
|
|
Report regression results per behavior, not as an average capability score.
|
|
Use workflow pass rates and trends as the benchmark headline.
|
|
|
|
Fast verifier fixtures remain ordinary CI. They validate grading logic, but do
|
|
not replace agent trials across the model, base prompt, CLI, and relay.
|
|
|
|
The task identity remains `buzz-native/<task>` in both layers; the wrapper reads
|
|
the metadata instead of encoding the layer in task names.
|
|
|
|
## Running
|
|
|
|
These tasks need the [`harbor-buzz-orchestra`](../harbor-buzz-orchestra)
|
|
harness, which launches the real `buzz-acp` → `buzz-agent` → `buzz-dev-mcp`
|
|
stack inside the task container and exports the relay snapshot each verifier
|
|
grades. Plain `harbor run` against this directory will not work, and neither
|
|
will `harbor run -a oracle` (no `solution/solve.sh` is shipped — the Oracle
|
|
agent replaces the Buzz agent, so no relay trial is provisioned).
|
|
|
|
From the repo root:
|
|
|
|
```bash
|
|
just benchmark \
|
|
--path benchmarks/buzz-dataset/reply-to-thread \
|
|
--manifest benchmarks/harbor-buzz-orchestra/manifests/buzz-native-solo-luna.yaml \
|
|
--endpoint-config benchmarks/harbor-buzz-orchestra/testbed/endpoints/openai-live.json \
|
|
--n-concurrent 1
|
|
```
|
|
|
|
The task's regression metadata supplies its default `--attempts 1`. Select a
|
|
whole layer with `--path benchmarks/buzz-dataset --layer regression` or
|
|
`--layer workflow`. If the dataset root is passed without `--layer` or
|
|
`--attempts`, the wrapper runs two Harbor jobs so regression gets k=1 and
|
|
workflow gets k=3. An explicit `--attempts`/`-k` overrides these defaults and
|
|
runs the selected tasks in one job.
|
|
|
|
The default condition is one solo agent on `gpt-5.6-luna` at
|
|
`thinking_effort: medium`, which needs `OPENAI_COMPAT_API_KEY`; see
|
|
[the harness README](../harbor-buzz-orchestra/README.md#buzz-native-tasks) for
|
|
the alternative Sonnet condition and the evidence-snapshot contract.
|
|
|
|
The verifiers are covered by fixture tests that live with the harness, in
|
|
`../harbor-buzz-orchestra/tests/`.
|