a9640c7cc4
## Why Terminal-Bench can show task correctness while missing Buzz-specific collaboration failures, such as posting a final answer outside the user's thread. Add a small Buzz-native suite that scores relay behavior directly. ## What - Add Harbor tasks for direct thread replies, event-level callback mentions to a three-word user identity, targeted reads of explicitly named paths outside the workspace, and exact temporary-channel membership against a seeded directory of 50 users and 10 bots - Export sanitized post-agent relay evidence and extend solo runtime/provisioning for deterministic verification - Expose channel TTL in CLI search output and cover positive and negative transcript/state fixtures ## Risk Assessment Low to medium — this primarily adds benchmark and testbed machinery. The CLI change is an additive `ttl_seconds` field in channel-search JSON; relay behavior is unchanged. ## References - Motivated by the Buzz-specific behavior observed while benchmarking https://github.com/block/buzz/pull/6128 - The named-path regression case covers the issue being fixed by https://github.com/block/buzz/pull/6261 - Local live Sonnet runs: `reply-to-thread`, `user-mention`, and `create-channel-invite-users` each scored reward 1.0 with zero exceptions - The current branch prompt scored 3/5 on a concurrent `read-named-path-outside-workspace` batch: one explicit workspace-boundary refusal and one missing final response Generated with Codex --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com>
3.6 KiB
read-named-path-outside-workspace
What the agent does
Reads one file the user names explicitly by path —
~/.claude/skills/context-health-check/SKILL.md — and reports its CHECK_ID
and ACTION values (instruction.md).
This is a regression case, not a capability test. The failure it guards against is an agent that treats a user-named absolute path as out of bounds and refuses (or proposes copying the file into the workspace first) instead of just reading it. See block/buzz#6261.
Environment
python:3.12-slim-bookworm with HOME=/home/buzz, so ~ in the instruction
resolves to the seeded skill directory. The Dockerfile generates the
CHECK_ID marker with secrets.token_hex at image build time, so the
expected value cannot be memorized across runs; the verifier reads the
answer back out of the same file rather than hardcoding it. Agent timeout 300s.
Verifier
Reads the post-agent /logs/artifacts/buzz-evidence.json snapshot plus the
seeded SKILL.md (via --skill-file) for the expected values.
| Dimension | Type | Measures |
|---|---|---|
evidence_complete |
programmatic | Snapshot is v1, untruncated, names this task, and resolves the task event, channel, one orchestrator, and a candidate reply. Harness health, not agent skill |
expected_author |
programmatic | The scored message was published by the orchestrator |
same_channel |
programmatic | Reply carries the trial channel's h tag |
named_path_read |
programmatic | Reply contains the build-time CHECK_ID marker — proof the file was actually read |
action_reported |
programmatic | Reply contains the ACTION line, matched case-insensitively with whitespace collapsed and trailing punctuation stripped |
reward is the conjunction of every dimension above.
Refusal wording is deliberately not scored. The question this task asks is
whether the file was read, and named_path_read answers it conclusively: the
CHECK_ID marker is generated at image build time, so an agent cannot emit it
without having read the file. A genuine refusal therefore already scores 0 on
the substance. An earlier revision also matched refusal phrasing with a regex,
which meant a hedged-but-correct answer could score 0 on wording alone; that
check is gone rather than kept as an unscored metric.
instruction.md also says "Do not search other directories". That constraint
is intentionally unscored — the snapshot holds relay messages, not the agent's
tool calls.
Layout
read-named-path-outside-workspace/
├── instruction.md # Prompt posted to the agent as the trial user
├── task.toml # Metadata, timeouts, 1 CPU / 1 GiB environment
├── environment/Dockerfile # Seeds ~/.claude/skills/... with a random CHECK_ID
└── tests/
├── test.sh # Runs verify.py against the snapshot + SKILL.md
└── verify.py # Deterministic scorer (see table above)
Running
just benchmark \
--path benchmarks/buzz-dataset/read-named-path-outside-workspace \
--attempts 1 \
--manifest benchmarks/harbor-buzz-orchestra/manifests/buzz-native-solo-luna.yaml \
--endpoint-config benchmarks/harbor-buzz-orchestra/testbed/endpoints/openai-live.json \
--n-concurrent 1
harbor run -a oracle does not work here, and no solution/solve.sh is
shipped: the Oracle agent replaces BuzzOrchestraAgent, so no relay trial is
provisioned and no evidence snapshot is exported. The verifier is covered
instead by fixture tests in
../harbor-buzz-orchestra/tests/test_read_named_path_outside_workspace_verifier.py.