mirror of
https://github.com/block/buzz.git
synced 2026-10-30 05:51:46 +00:00
a9640c7cc4
## Why Terminal-Bench can show task correctness while missing Buzz-specific collaboration failures, such as posting a final answer outside the user's thread. Add a small Buzz-native suite that scores relay behavior directly. ## What - Add Harbor tasks for direct thread replies, event-level callback mentions to a three-word user identity, targeted reads of explicitly named paths outside the workspace, and exact temporary-channel membership against a seeded directory of 50 users and 10 bots - Export sanitized post-agent relay evidence and extend solo runtime/provisioning for deterministic verification - Expose channel TTL in CLI search output and cover positive and negative transcript/state fixtures ## Risk Assessment Low to medium — this primarily adds benchmark and testbed machinery. The CLI change is an additive `ttl_seconds` field in channel-search JSON; relay behavior is unchanged. ## References - Motivated by the Buzz-specific behavior observed while benchmarking https://github.com/block/buzz/pull/6128 - The named-path regression case covers the issue being fixed by https://github.com/block/buzz/pull/6261 - Local live Sonnet runs: `reply-to-thread`, `user-mention`, and `create-channel-invite-users` each scored reward 1.0 with zero exceptions - The current branch prompt scored 3/5 on a concurrent `read-named-path-outside-workspace` batch: one explicit workspace-boundary refusal and one missing final response Generated with Codex --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com>
72 lines
3.3 KiB
Markdown
72 lines
3.3 KiB
Markdown
# reply-to-thread
|
|
|
|
## What the agent does
|
|
|
|
Answers a six-month financial projection posted by the trial user
|
|
([instruction.md](instruction.md)). The arithmetic is incidental — this task
|
|
measures **where** the answer lands: in the user's thread, not as a new
|
|
top-level channel message.
|
|
|
|
> **The instruction deliberately says nothing about threading.** Threading is
|
|
> the behavior under test, and it must come from `buzz-acp`'s production base
|
|
> prompt rather than from the task prompt. Do not "fix" the instruction by
|
|
> telling the agent to reply in-thread — that would make the task measure
|
|
> instruction-following instead of product behavior.
|
|
|
|
## Environment
|
|
|
|
`python:3.12-slim-bookworm`, no extra packages: the agent never runs in this
|
|
container's shell. `BuzzOrchestraAgent` launches the real `buzz-acp` /
|
|
`buzz-agent` stack against a dedicated relay, and the agent works entirely
|
|
through Buzz. Agent timeout 300s; the manifest's `trial_budget` is the
|
|
effective clock.
|
|
|
|
## Verifier
|
|
|
|
Reads the post-agent `/logs/artifacts/buzz-evidence.json` snapshot (written by
|
|
`BuzzContainerRuntime._collect_evidence` after the agent stops, so the agent
|
|
cannot influence it). Every dimension is programmatic; `reward` is the
|
|
conjunction of all of them.
|
|
|
|
| Dimension | Type | Measures |
|
|
| --- | --- | --- |
|
|
| `evidence_complete` | programmatic | Snapshot is v1, untruncated, has one orchestrator, and resolves the task event and a candidate reply. Harness health, not agent skill — a 0 here means investigate the run |
|
|
| `expected_author` | programmatic | The scored message was published by the orchestrator |
|
|
| `same_channel` | programmatic | Reply carries the trial channel's `h` tag |
|
|
| `reply_to_thread` | programmatic | Reply carries `["e", <task event>, "", "reply"]` — the behavior under test |
|
|
| `answer_correct` | programmatic | Month-6 revenue (160,811), month-6 expenses (84,462), and cumulative profit (374,470), each ±1 and each on a line naming it |
|
|
|
|
`answer_correct` requires the label and the value on the same line so a
|
|
work-showing table with a wrong stated answer cannot pass on its intermediate
|
|
rows. `instruction.md` asks for that formatting explicitly.
|
|
|
|
## Layout
|
|
|
|
```
|
|
reply-to-thread/
|
|
├── instruction.md # Prompt posted to the agent as the trial user
|
|
├── task.toml # Metadata, timeouts, 1 CPU / 1 GiB environment
|
|
├── environment/Dockerfile # Bare python image; the relay stack is uploaded
|
|
└── tests/
|
|
├── test.sh # Runs verify.py against the evidence snapshot
|
|
└── verify.py # Deterministic scorer (see table above)
|
|
```
|
|
|
|
## Running
|
|
|
|
```bash
|
|
just benchmark \
|
|
--path benchmarks/buzz-dataset/reply-to-thread \
|
|
--attempts 1 \
|
|
--manifest benchmarks/harbor-buzz-orchestra/manifests/buzz-native-solo-luna.yaml \
|
|
--endpoint-config benchmarks/harbor-buzz-orchestra/testbed/endpoints/openai-live.json \
|
|
--n-concurrent 1
|
|
```
|
|
|
|
`harbor run -a oracle` does **not** work here, and no `solution/solve.sh` is
|
|
shipped: the Oracle agent replaces `BuzzOrchestraAgent`, so no relay trial is
|
|
provisioned and no evidence snapshot is exported. The verifier is covered
|
|
instead by positive and negative fixture tests in
|
|
`../harbor-buzz-orchestra/tests/test_reply_to_thread_verifier.py` (run from
|
|
`benchmarks/harbor-buzz-orchestra`: `uv run --extra dev pytest -q`).
|