a9640c7cc4
## Why Terminal-Bench can show task correctness while missing Buzz-specific collaboration failures, such as posting a final answer outside the user's thread. Add a small Buzz-native suite that scores relay behavior directly. ## What - Add Harbor tasks for direct thread replies, event-level callback mentions to a three-word user identity, targeted reads of explicitly named paths outside the workspace, and exact temporary-channel membership against a seeded directory of 50 users and 10 bots - Export sanitized post-agent relay evidence and extend solo runtime/provisioning for deterministic verification - Expose channel TTL in CLI search output and cover positive and negative transcript/state fixtures ## Risk Assessment Low to medium — this primarily adds benchmark and testbed machinery. The CLI change is an additive `ttl_seconds` field in channel-search JSON; relay behavior is unchanged. ## References - Motivated by the Buzz-specific behavior observed while benchmarking https://github.com/block/buzz/pull/6128 - The named-path regression case covers the issue being fixed by https://github.com/block/buzz/pull/6261 - Local live Sonnet runs: `reply-to-thread`, `user-mention`, and `create-channel-invite-users` each scored reward 1.0 with zero exceptions - The current branch prompt scored 3/5 on a concurrent `read-named-path-outside-workspace` batch: one explicit workspace-boundary refusal and one missing final response Generated with Codex --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com>
buzz-dataset
Harbor tasks that score Buzz product behavior, not just task correctness. Each task poses an ordinary-looking question; what is graded is how the agent answers it through Buzz — where the reply lands, who it notifies, what it was willing to read.
| Task | Behavior under test |
|---|---|
reply-to-thread |
Answers in the user's thread instead of as a new top-level message |
user-mention |
Hands the turn back with an event-level p-tag mention of the requesting human |
read-named-path-outside-workspace |
Reads a path the user named explicitly instead of refusing it as out of bounds |
create-channel-invite-users |
Creates a channel with the exact shape, TTL, and membership asked for |
For reply-to-thread and user-mention the graded behavior is deliberately
absent from instruction.md — it has to come from buzz-acp's production
base prompt. Read a task's own README.md before editing its instruction or
verifier.
Running
These tasks need the harbor-buzz-orchestra
harness, which launches the real buzz-acp → buzz-agent → buzz-dev-mcp
stack inside the task container and exports the relay snapshot each verifier
grades. Plain harbor run against this directory will not work, and neither
will harbor run -a oracle (no solution/solve.sh is shipped — the Oracle
agent replaces the Buzz agent, so no relay trial is provisioned).
From the repo root:
just benchmark \
--path benchmarks/buzz-dataset/reply-to-thread \
--attempts 1 \
--manifest benchmarks/harbor-buzz-orchestra/manifests/buzz-native-solo-luna.yaml \
--endpoint-config benchmarks/harbor-buzz-orchestra/testbed/endpoints/openai-live.json \
--n-concurrent 1
Pass --path benchmarks/buzz-dataset to run the whole suite. The default
condition is one solo agent on gpt-5.6-luna at thinking_effort: medium,
which needs OPENAI_COMPAT_API_KEY; see
the harness README for
the alternative Sonnet condition and the evidence-snapshot contract.
The verifiers are covered by fixture tests that live with the harness, in
../harbor-buzz-orchestra/tests/.