mirror of
https://github.com/block/buzz.git
synced 2026-10-30 05:51:46 +00:00
a9640c7cc4
## Why Terminal-Bench can show task correctness while missing Buzz-specific collaboration failures, such as posting a final answer outside the user's thread. Add a small Buzz-native suite that scores relay behavior directly. ## What - Add Harbor tasks for direct thread replies, event-level callback mentions to a three-word user identity, targeted reads of explicitly named paths outside the workspace, and exact temporary-channel membership against a seeded directory of 50 users and 10 bots - Export sanitized post-agent relay evidence and extend solo runtime/provisioning for deterministic verification - Expose channel TTL in CLI search output and cover positive and negative transcript/state fixtures ## Risk Assessment Low to medium — this primarily adds benchmark and testbed machinery. The CLI change is an additive `ttl_seconds` field in channel-search JSON; relay behavior is unchanged. ## References - Motivated by the Buzz-specific behavior observed while benchmarking https://github.com/block/buzz/pull/6128 - The named-path regression case covers the issue being fixed by https://github.com/block/buzz/pull/6261 - Local live Sonnet runs: `reply-to-thread`, `user-mention`, and `create-channel-invite-users` each scored reward 1.0 with zero exceptions - The current branch prompt scored 3/5 on a concurrent `read-named-path-outside-workspace` batch: one explicit workspace-boundary refusal and one missing final response Generated with Codex --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com>
67 lines
3.0 KiB
Markdown
67 lines
3.0 KiB
Markdown
# user-mention
|
||
|
||
## What the agent does
|
||
|
||
Answers a one-line licensing calculation ([instruction.md](instruction.md)).
|
||
The arithmetic is incidental — this task measures whether the agent hands the
|
||
turn back with an **event-level mention** of the requesting human, so the user
|
||
gets a real Buzz notification instead of a message they have to notice.
|
||
|
||
> **The instruction deliberately says nothing about mentioning anyone.** The
|
||
> mention is the behavior under test and must come from `buzz-acp`'s production
|
||
> base prompt. Do not add "mention the user" to the instruction.
|
||
|
||
The trial user for this task is provisioned with the stable three-word display
|
||
name `John Vincent Doe` (`task_fixtures.USER_MENTION_DISPLAY_NAME`), which
|
||
forces the agent to resolve a multi-word identity to a pubkey rather than
|
||
guessing a single-token handle.
|
||
|
||
## Environment
|
||
|
||
`python:3.12-slim-bookworm`, no extra packages: the agent never runs in this
|
||
container's shell. `BuzzOrchestraAgent` launches the real `buzz-acp` /
|
||
`buzz-agent` stack against a dedicated relay. Agent timeout 300s.
|
||
|
||
## Verifier
|
||
|
||
Reads the post-agent `/logs/artifacts/buzz-evidence.json` snapshot. Every
|
||
dimension is programmatic; `reward` is the conjunction of all of them.
|
||
|
||
| Dimension | Type | Measures |
|
||
| --- | --- | --- |
|
||
| `evidence_complete` | programmatic | Snapshot is v1, untruncated, names this task, and resolves exactly one orchestrator, one user, and a candidate reply. Harness health, not agent skill |
|
||
| `three_word_user` | programmatic | The provisioner seeded the three-word display name. Fixture self-check |
|
||
| `expected_author` | programmatic | The scored message was published by the orchestrator |
|
||
| `same_channel` | programmatic | Reply carries the trial channel's `h` tag |
|
||
| `user_p_tagged` | programmatic | Reply carries a `p` tag for the user's pubkey — the behavior under test. Presentation-only `@text` does not count |
|
||
| `answer_correct` | programmatic | Annual total 5,328 (12 licenses × $37 × 12 months), ±1 |
|
||
|
||
## Layout
|
||
|
||
```
|
||
user-mention/
|
||
├── instruction.md # Prompt posted to the agent as the trial user
|
||
├── task.toml # Metadata, timeouts, 1 CPU / 1 GiB environment
|
||
├── environment/Dockerfile # Bare python image; the relay stack is uploaded
|
||
└── tests/
|
||
├── test.sh # Runs verify.py against the evidence snapshot
|
||
└── verify.py # Deterministic scorer (see table above)
|
||
```
|
||
|
||
## Running
|
||
|
||
```bash
|
||
just benchmark \
|
||
--path benchmarks/buzz-dataset/user-mention \
|
||
--attempts 1 \
|
||
--manifest benchmarks/harbor-buzz-orchestra/manifests/buzz-native-solo-luna.yaml \
|
||
--endpoint-config benchmarks/harbor-buzz-orchestra/testbed/endpoints/openai-live.json \
|
||
--n-concurrent 1
|
||
```
|
||
|
||
`harbor run -a oracle` does **not** work here, and no `solution/solve.sh` is
|
||
shipped: the Oracle agent replaces `BuzzOrchestraAgent`, so no relay trial is
|
||
provisioned and no evidence snapshot is exported. The verifier is covered
|
||
instead by fixture tests in
|
||
`../harbor-buzz-orchestra/tests/test_user_mention_verifier.py`.
|