Commit Graph

2 Commits

Author SHA1 Message Date
Salman Mohammed
a9640c7cc4 Add Buzz-native collaboration benchmarks (#6264)
## Why
Terminal-Bench can show task correctness while missing Buzz-specific
collaboration failures, such as posting a final answer outside the
user's thread. Add a small Buzz-native suite that scores relay behavior
directly.

## What
- Add Harbor tasks for direct thread replies, event-level callback
mentions to a three-word user identity, targeted reads of explicitly
named paths outside the workspace, and exact temporary-channel
membership against a seeded directory of 50 users and 10 bots
- Export sanitized post-agent relay evidence and extend solo
runtime/provisioning for deterministic verification
- Expose channel TTL in CLI search output and cover positive and
negative transcript/state fixtures

## Risk Assessment
Low to medium — this primarily adds benchmark and testbed machinery. The
CLI change is an additive `ttl_seconds` field in channel-search JSON;
relay behavior is unchanged.

## References
- Motivated by the Buzz-specific behavior observed while benchmarking
https://github.com/block/buzz/pull/6128
- The named-path regression case covers the issue being fixed by
https://github.com/block/buzz/pull/6261
- Local live Sonnet runs: `reply-to-thread`, `user-mention`, and
`create-channel-invite-users` each scored reward 1.0 with zero
exceptions
- The current branch prompt scored 3/5 on a concurrent
`read-named-path-outside-workspace` batch: one explicit
workspace-boundary refusal and one missing final response

Generated with Codex

---------

Signed-off-by: Salman Mohammed <smohammed@squareup.com>
2026-08-20 10:14:39 -04:00
Tyler
f437e43d09 benchmarks: Harbor Terminal-Bench harness for Buzz agent-team orchestration (harbor-buzz-orchestra) (#1504)
Signed-off-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
2026-07-08 11:32:11 -04:00