f463e726dd
## Why Evaluating buzz agent memory retrieval by seeding a memory then asking the buzz agent a question it needs that memory. **Bug Found**: System prompt had no inclusion of retrieving cold memories and suggested looking in a mem/*.md directory that does not exist. Updated `system-prompt.md` to include memory CLI tools and usage. Eval Before System Prompt Change: 0/3 Eval After System Prompt Change: 3/3 ## What - Add a `memory-retrieval` benchmark that seeds agent memory with `buzz mem set` before asking a direct question. - Grade the observable threaded answer without inspecting tool calls or exposing the answer in channel history. - Teach agents to use `buzz mem set`, `buzz mem ls`, and `buzz mem get` for cold memory. - Add a wire-debug endpoint configuration for diagnosing ACP tool calls in local runs. - Add fixture, seeding, verifier, and prompt coverage. ## Risk Assessment Low. The runtime changes are limited to the benchmark harness. The production-facing change clarifies existing memory commands in the base prompt; it does not change memory storage, relay behavior, or authorization. ## References - Before the system-prompt changes, 0/3 attempts passed because agents never invoked the `buzz mem` CLI and instead searched a non existent filesystem - After the changes, 3/3 attempts passed. ACP wire logs confirmed that every agent ran `buzz mem ls` followed by `buzz mem get` and returned `net_gpv`. --------- Signed-off-by: Philip Azar <pazar@squareup.com>
buzz-dataset
Harbor tasks that score Buzz product behavior, not just task correctness. Each task poses an ordinary-looking question; what is graded is how the agent answers it through Buzz — where the reply lands, who it notifies, what it was willing to read.
| Task | Layer | Behavior under test |
|---|---|---|
reply-to-thread |
Regression | Answers in the user's thread instead of as a new top-level message |
user-mention |
Regression | Hands the turn back with an event-level p-tag mention of the requesting human |
read-named-path-outside-workspace |
Regression | Reads a path the user named explicitly instead of refusing it as out of bounds |
create-channel-invite-users |
Workflow | Creates a channel with the exact shape, TTL, and membership asked for |
multiline-message |
Regression | Preserves real newlines and blank-line structure through the CLI publish path |
narrative-agent-names |
Regression | Names agents in narrative without waking them through p tags |
interleaved-agent-reports |
Workflow | Retains and synthesizes every report in a batch of agent messages |
cross-thread-requests |
Workflow | Keeps simultaneous top-level requests isolated and replies to both exact threads |
ambiguous-user-mention |
Workflow | Resolves duplicate display names and notifies only the intended pubkey |
memory-retrieval |
Regression | Answers from harness-seeded cold memory without the value appearing in channel history |
For reply-to-thread and user-mention the graded behavior is deliberately
absent from instruction.md — it has to come from buzz-acp's production
base prompt. Read a task's own README.md before editing its instruction or
verifier.
Evaluation layers
Every task declares metadata.evaluation_layer in task.toml:
| Layer | Question | Default trials | Typical cadence |
|---|---|---|---|
| Regression | Did Buzz preserve a known product contract? | k=1 | Targeted PR, nightly, or pre-release |
| Workflow | How capable is the agent at realistic Buzz work? | k=3 | Nightly or weekly on a fixed condition |
Report regression results per behavior, not as an average capability score. Use workflow pass rates and trends as the benchmark headline.
Fast verifier fixtures remain ordinary CI. They validate grading logic, but do not replace agent trials across the model, base prompt, CLI, and relay.
The task identity remains buzz-native/<task> in both layers; the wrapper reads
the metadata instead of encoding the layer in task names.
Running
These tasks need the harbor-buzz-orchestra
harness, which launches the real buzz-acp → buzz-agent → buzz-dev-mcp
stack inside the task container and exports the relay snapshot each verifier
grades. Plain harbor run against this directory will not work, and neither
will harbor run -a oracle (no solution/solve.sh is shipped — the Oracle
agent replaces the Buzz agent, so no relay trial is provisioned).
From the repo root:
just benchmark \
--path benchmarks/buzz-dataset/reply-to-thread \
--manifest benchmarks/harbor-buzz-orchestra/manifests/buzz-native-solo-luna.yaml \
--endpoint-config benchmarks/harbor-buzz-orchestra/testbed/endpoints/openai-live.json \
--n-concurrent 1
The task's regression metadata supplies its default --attempts 1. Select a
whole layer with --path benchmarks/buzz-dataset --layer regression or
--layer workflow. If the dataset root is passed without --layer or
--attempts, the wrapper runs two Harbor jobs so regression gets k=1 and
workflow gets k=3. An explicit --attempts/-k overrides these defaults and
runs the selected tasks in one job.
The default condition is one solo agent on gpt-5.6-luna at
thinking_effort: medium, which needs OPENAI_COMPAT_API_KEY; see
the harness README for
the alternative Sonnet condition and the evidence-snapshot contract.
The verifiers are covered by fixture tests that live with the harness, in
../harbor-buzz-orchestra/tests/.