a9640c7cc4
## Why Terminal-Bench can show task correctness while missing Buzz-specific collaboration failures, such as posting a final answer outside the user's thread. Add a small Buzz-native suite that scores relay behavior directly. ## What - Add Harbor tasks for direct thread replies, event-level callback mentions to a three-word user identity, targeted reads of explicitly named paths outside the workspace, and exact temporary-channel membership against a seeded directory of 50 users and 10 bots - Export sanitized post-agent relay evidence and extend solo runtime/provisioning for deterministic verification - Expose channel TTL in CLI search output and cover positive and negative transcript/state fixtures ## Risk Assessment Low to medium — this primarily adds benchmark and testbed machinery. The CLI change is an additive `ttl_seconds` field in channel-search JSON; relay behavior is unchanged. ## References - Motivated by the Buzz-specific behavior observed while benchmarking https://github.com/block/buzz/pull/6128 - The named-path regression case covers the issue being fixed by https://github.com/block/buzz/pull/6261 - Local live Sonnet runs: `reply-to-thread`, `user-mention`, and `create-channel-invite-users` each scored reward 1.0 with zero exceptions - The current branch prompt scored 3/5 on a concurrent `read-named-path-outside-workspace` batch: one explicit workspace-boundary refusal and one missing final response Generated with Codex --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com>
user-mention
What the agent does
Answers a one-line licensing calculation (instruction.md). The arithmetic is incidental — this task measures whether the agent hands the turn back with an event-level mention of the requesting human, so the user gets a real Buzz notification instead of a message they have to notice.
The instruction deliberately says nothing about mentioning anyone. The mention is the behavior under test and must come from
buzz-acp's production base prompt. Do not add "mention the user" to the instruction.
The trial user for this task is provisioned with the stable three-word display
name John Vincent Doe (task_fixtures.USER_MENTION_DISPLAY_NAME), which
forces the agent to resolve a multi-word identity to a pubkey rather than
guessing a single-token handle.
Environment
python:3.12-slim-bookworm, no extra packages: the agent never runs in this
container's shell. BuzzOrchestraAgent launches the real buzz-acp /
buzz-agent stack against a dedicated relay. Agent timeout 300s.
Verifier
Reads the post-agent /logs/artifacts/buzz-evidence.json snapshot. Every
dimension is programmatic; reward is the conjunction of all of them.
| Dimension | Type | Measures |
|---|---|---|
evidence_complete |
programmatic | Snapshot is v1, untruncated, names this task, and resolves exactly one orchestrator, one user, and a candidate reply. Harness health, not agent skill |
three_word_user |
programmatic | The provisioner seeded the three-word display name. Fixture self-check |
expected_author |
programmatic | The scored message was published by the orchestrator |
same_channel |
programmatic | Reply carries the trial channel's h tag |
user_p_tagged |
programmatic | Reply carries a p tag for the user's pubkey — the behavior under test. Presentation-only @text does not count |
answer_correct |
programmatic | Annual total 5,328 (12 licenses × $37 × 12 months), ±1 |
Layout
user-mention/
├── instruction.md # Prompt posted to the agent as the trial user
├── task.toml # Metadata, timeouts, 1 CPU / 1 GiB environment
├── environment/Dockerfile # Bare python image; the relay stack is uploaded
└── tests/
├── test.sh # Runs verify.py against the evidence snapshot
└── verify.py # Deterministic scorer (see table above)
Running
just benchmark \
--path benchmarks/buzz-dataset/user-mention \
--attempts 1 \
--manifest benchmarks/harbor-buzz-orchestra/manifests/buzz-native-solo-luna.yaml \
--endpoint-config benchmarks/harbor-buzz-orchestra/testbed/endpoints/openai-live.json \
--n-concurrent 1
harbor run -a oracle does not work here, and no solution/solve.sh is
shipped: the Oracle agent replaces BuzzOrchestraAgent, so no relay trial is
provisioned and no evidence snapshot is exported. The verifier is covered
instead by fixture tests in
../harbor-buzz-orchestra/tests/test_user_mention_verifier.py.