mirror of
https://github.com/block/buzz.git
synced 2026-10-27 12:40:30 +00:00
a9640c7cc4
## Why Terminal-Bench can show task correctness while missing Buzz-specific collaboration failures, such as posting a final answer outside the user's thread. Add a small Buzz-native suite that scores relay behavior directly. ## What - Add Harbor tasks for direct thread replies, event-level callback mentions to a three-word user identity, targeted reads of explicitly named paths outside the workspace, and exact temporary-channel membership against a seeded directory of 50 users and 10 bots - Export sanitized post-agent relay evidence and extend solo runtime/provisioning for deterministic verification - Expose channel TTL in CLI search output and cover positive and negative transcript/state fixtures ## Risk Assessment Low to medium — this primarily adds benchmark and testbed machinery. The CLI change is an additive `ttl_seconds` field in channel-search JSON; relay behavior is unchanged. ## References - Motivated by the Buzz-specific behavior observed while benchmarking https://github.com/block/buzz/pull/6128 - The named-path regression case covers the issue being fixed by https://github.com/block/buzz/pull/6261 - Local live Sonnet runs: `reply-to-thread`, `user-mention`, and `create-channel-invite-users` each scored reward 1.0 with zero exceptions - The current branch prompt scored 3/5 on a concurrent `read-named-path-outside-workspace` batch: one explicit workspace-boundary refusal and one missing final response Generated with Codex --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com>
33 lines
1.1 KiB
JSON
33 lines
1.1 KiB
JSON
{
|
|
"channel_id": "6a178caa-07de-4594-9296-1f130b3f32e2",
|
|
"message_count": 2,
|
|
"truncated": false,
|
|
"messages": [
|
|
{
|
|
"id": "585eeddbd9c1384696a615faeafdca5c00ff806276c8c6c66b654e7bbb40e167",
|
|
"author": "user",
|
|
"pubkey": "779c3f730c638f67cc06c7e7d55d31720c85c9eb74cec2050d0ac5fdabaafea8",
|
|
"content": "@solo-1 Complete the requested task.",
|
|
"created_at": 1786995079,
|
|
"kind": 9,
|
|
"tags": [
|
|
["h", "6a178caa-07de-4594-9296-1f130b3f32e2"],
|
|
["p", "ed8ce3ee42988114b5940d9dd0023d576649e06c47bc018ef74933b2472e4854"]
|
|
]
|
|
},
|
|
{
|
|
"id": "220c6ac96cf28a995cc14ff8b72d5ec7d968c0a859eeaa0d2ec354670def60e5",
|
|
"author": "solo-1",
|
|
"pubkey": "ed8ce3ee42988114b5940d9dd0023d576649e06c47bc018ef74933b2472e4854",
|
|
"content": "DONE: Completed the requested task.",
|
|
"created_at": 1786995113,
|
|
"kind": 9,
|
|
"tags": [
|
|
["h", "6a178caa-07de-4594-9296-1f130b3f32e2"],
|
|
["e", "585eeddbd9c1384696a615faeafdca5c00ff806276c8c6c66b654e7bbb40e167", "", "reply"],
|
|
["p", "779c3f730c638f67cc06c7e7d55d31720c85c9eb74cec2050d0ac5fdabaafea8"]
|
|
]
|
|
}
|
|
]
|
|
}
|