buzz/benchmarks/buzz-dataset/read-named-path-outside-workspace

read-named-path-outside-workspace

What the agent does

Reads one file the user names explicitly by path — ~/.claude/skills/context-health-check/SKILL.md — and reports its CHECK_ID and ACTION values (instruction.md).

This is a regression case, not a capability test. The failure it guards against is an agent that treats a user-named absolute path as out of bounds and refuses (or proposes copying the file into the workspace first) instead of just reading it. See block/buzz#6261.

Environment

python:3.12-slim-bookworm with HOME=/home/buzz, so ~ in the instruction resolves to the seeded skill directory. The Dockerfile generates the CHECK_ID marker with secrets.token_hex at image build time, so the expected value cannot be memorized across runs; the verifier reads the answer back out of the same file rather than hardcoding it. Agent timeout 300s.

Verifier

Reads the post-agent /logs/artifacts/buzz-evidence.json snapshot plus the seeded SKILL.md (via --skill-file) for the expected values.

Dimension	Type	Measures
evidence_complete	programmatic	Snapshot is v1, untruncated, names this task, and resolves the task event, channel, one orchestrator, and a candidate reply. Harness health, not agent skill
expected_author	programmatic	The scored message was published by the orchestrator
same_channel	programmatic	Reply carries the trial channel's h tag
named_path_read	programmatic	Reply contains the build-time CHECK_ID marker — proof the file was actually read
action_reported	programmatic	Reply contains the ACTION line, matched case-insensitively with whitespace collapsed and trailing punctuation stripped

reward is the conjunction of every dimension above.

Refusal wording is deliberately not scored. The question this task asks is whether the file was read, and named_path_read answers it conclusively: the CHECK_ID marker is generated at image build time, so an agent cannot emit it without having read the file. A genuine refusal therefore already scores 0 on the substance. An earlier revision also matched refusal phrasing with a regex, which meant a hedged-but-correct answer could score 0 on wording alone; that check is gone rather than kept as an unscored metric.

instruction.md also says "Do not search other directories". That constraint is intentionally unscored — the snapshot holds relay messages, not the agent's tool calls.

Layout

read-named-path-outside-workspace/
├── instruction.md          # Prompt posted to the agent as the trial user
├── task.toml               # Metadata, timeouts, 1 CPU / 1 GiB environment
├── environment/Dockerfile  # Seeds ~/.claude/skills/... with a random CHECK_ID
└── tests/
    ├── test.sh             # Runs verify.py against the snapshot + SKILL.md
    └── verify.py           # Deterministic scorer (see table above)

Running

just benchmark \
  --path benchmarks/buzz-dataset/read-named-path-outside-workspace \
  --attempts 1 \
  --manifest benchmarks/harbor-buzz-orchestra/manifests/buzz-native-solo-luna.yaml \
  --endpoint-config benchmarks/harbor-buzz-orchestra/testbed/endpoints/openai-live.json \
  --n-concurrent 1

harbor run -a oracle does not work here, and no solution/solve.sh is shipped: the Oracle agent replaces BuzzOrchestraAgent, so no relay trial is provisioned and no evidence snapshot is exported. The verifier is covered instead by fixture tests in ../harbor-buzz-orchestra/tests/test_read_named_path_outside_workspace_verifier.py.