b56a52ca11
## Why Terminal task correctness alone does not guard Buzz-native collaboration behavior such as exact reply routing, non-waking narrative names, batched agent synthesis, thread isolation, and ambiguous identity targeting. ## What - Add five medium/hard Harbor tasks for multiline delivery, narrative agent names, interleaved agent reports, cross-thread requests, and ambiguous user mentions - Add signed scripted-event fixtures, duplicate-display-name profiles, and public-only evidence export support - Add positive and adversarial verifier fixtures covering dropped inputs, extra posts, incorrect routing, and unintended mentions ## Risk Assessment Low — changes are limited to benchmark tooling and datasets; production behavior is exercised but not modified. ## References - Grounded in behavior reported in `buzz-community` and public issues #5787, #5176, #5839, #4942, #4072, #4303, and #6257 - Live Claude Sonnet 4.6 smoke run: 2/5 passed; the three failures exposed top-level instead of threaded delivery, empty narrative output, and silent ambiguous-identity completion - Live GPT-5.6 Luna run (`buzz-native-solo-luna.yaml`): 3/5 passed. Passed `ambiguous-user-mention`, `cross-thread-requests`, and `interleaved-agent-reports`. Failed `multiline-message` and `narrative-agent-names` because both omitted the event-level user mention despite otherwise-correct content and threading. <img width="1176" height="805" alt="Screenshot 2026-08-21 at 11 11 17 AM" src="https://github.com/user-attachments/assets/fc259ec9-892c-4436-b022-1bf85415617d" /> Signed-off-by: Salman Mohammed <smohammed@squareup.com>
2.9 KiB
buzz-dataset
Harbor tasks that score Buzz product behavior, not just task correctness. Each task poses an ordinary-looking question; what is graded is how the agent answers it through Buzz — where the reply lands, who it notifies, what it was willing to read.
| Task | Behavior under test |
|---|---|
reply-to-thread |
Answers in the user's thread instead of as a new top-level message |
user-mention |
Hands the turn back with an event-level p-tag mention of the requesting human |
read-named-path-outside-workspace |
Reads a path the user named explicitly instead of refusing it as out of bounds |
create-channel-invite-users |
Creates a channel with the exact shape, TTL, and membership asked for |
multiline-message |
Preserves real newlines and blank-line structure through the CLI publish path |
narrative-agent-names |
Names agents in narrative without waking them through p tags |
interleaved-agent-reports |
Retains and synthesizes every report in a batch of agent messages |
cross-thread-requests |
Keeps simultaneous top-level requests isolated and replies to both exact threads |
ambiguous-user-mention |
Resolves duplicate display names and notifies only the intended pubkey |
For reply-to-thread and user-mention the graded behavior is deliberately
absent from instruction.md — it has to come from buzz-acp's production
base prompt. Read a task's own README.md before editing its instruction or
verifier.
Running
These tasks need the harbor-buzz-orchestra
harness, which launches the real buzz-acp → buzz-agent → buzz-dev-mcp
stack inside the task container and exports the relay snapshot each verifier
grades. Plain harbor run against this directory will not work, and neither
will harbor run -a oracle (no solution/solve.sh is shipped — the Oracle
agent replaces the Buzz agent, so no relay trial is provisioned).
From the repo root:
just benchmark \
--path benchmarks/buzz-dataset/reply-to-thread \
--attempts 1 \
--manifest benchmarks/harbor-buzz-orchestra/manifests/buzz-native-solo-luna.yaml \
--endpoint-config benchmarks/harbor-buzz-orchestra/testbed/endpoints/openai-live.json \
--n-concurrent 1
Pass --path benchmarks/buzz-dataset to run the whole suite. The default
condition is one solo agent on gpt-5.6-luna at thinking_effort: medium,
which needs OPENAI_COMPAT_API_KEY; see
the harness README for
the alternative Sonnet condition and the evidence-snapshot contract.
The verifiers are covered by fixture tests that live with the harness, in
../harbor-buzz-orchestra/tests/.