jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-10-28 21:11:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
db514b153a8cb17631d6e80b91c95fdc45deb147
buzz/benchmarks/buzz-dataset/interleaved-agent-reports
History
Salman Mohammed b56a52ca11 test(benchmarks): expand Buzz-native dataset (#6448) 
## Why
Terminal task correctness alone does not guard Buzz-native collaboration
behavior such as exact reply routing, non-waking narrative names,
batched agent synthesis, thread isolation, and ambiguous identity
targeting.

## What
- Add five medium/hard Harbor tasks for multiline delivery, narrative
agent names, interleaved agent reports, cross-thread requests, and
ambiguous user mentions
- Add signed scripted-event fixtures, duplicate-display-name profiles,
and public-only evidence export support
- Add positive and adversarial verifier fixtures covering dropped
inputs, extra posts, incorrect routing, and unintended mentions

## Risk Assessment
Low — changes are limited to benchmark tooling and datasets; production
behavior is exercised but not modified.

## References
- Grounded in behavior reported in `buzz-community` and public issues
#5787, #5176, #5839, #4942, #4072, #4303, and #6257
- Live Claude Sonnet 4.6 smoke run: 2/5 passed; the three failures
exposed top-level instead of threaded delivery, empty narrative output,
and silent ambiguous-identity completion
- Live GPT-5.6 Luna run (`buzz-native-solo-luna.yaml`): 3/5 passed.
Passed `ambiguous-user-mention`, `cross-thread-requests`, and
`interleaved-agent-reports`. Failed `multiline-message` and
`narrative-agent-names` because both omitted the event-level user
mention despite otherwise-correct content and threading.



<img width="1176" height="805" alt="Screenshot 2026-08-21 at 11 11
17 AM"
src="https://github.com/user-attachments/assets/fc259ec9-892c-4436-b022-1bf85415617d"
/>

Signed-off-by: Salman Mohammed <smohammed@squareup.com>
2026-08-21 12:00:04 -04:00
..
environment
test(benchmarks): expand Buzz-native dataset (#6448)
2026-08-21 12:00:04 -04:00
tests
test(benchmarks): expand Buzz-native dataset (#6448)
2026-08-21 12:00:04 -04:00
instruction.md
test(benchmarks): expand Buzz-native dataset (#6448)
2026-08-21 12:00:04 -04:00
README.md
test(benchmarks): expand Buzz-native dataset (#6448)
2026-08-21 12:00:04 -04:00
task.toml
test(benchmarks): expand Buzz-native dataset (#6448)
2026-08-21 12:00:04 -04:00

README.md

interleaved-agent-reports

Three signed bot identities publish independent reports immediately after the human request. The solo agent must retain every input, calculate 87, notify the human exactly once, and avoid waking the reporters again. The case exercises the batching/steering family reported in block/buzz#5839 and block/buzz#4942.