jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-10-28 21:11:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
codex/fix-agent-auth-error
buzz/benchmarks/buzz-dataset/multiline-message/README.md
Salman Mohammed b56a52ca11 test(benchmarks): expand Buzz-native dataset (#6448) 
## Why
Terminal task correctness alone does not guard Buzz-native collaboration
behavior such as exact reply routing, non-waking narrative names,
batched agent synthesis, thread isolation, and ambiguous identity
targeting.

## What
- Add five medium/hard Harbor tasks for multiline delivery, narrative
agent names, interleaved agent reports, cross-thread requests, and
ambiguous user mentions
- Add signed scripted-event fixtures, duplicate-display-name profiles,
and public-only evidence export support
- Add positive and adversarial verifier fixtures covering dropped
inputs, extra posts, incorrect routing, and unintended mentions

## Risk Assessment
Low — changes are limited to benchmark tooling and datasets; production
behavior is exercised but not modified.

## References
- Grounded in behavior reported in `buzz-community` and public issues
#5787, #5176, #5839, #4942, #4072, #4303, and #6257
- Live Claude Sonnet 4.6 smoke run: 2/5 passed; the three failures
exposed top-level instead of threaded delivery, empty narrative output,
and silent ambiguous-identity completion
- Live GPT-5.6 Luna run (`buzz-native-solo-luna.yaml`): 3/5 passed.
Passed `ambiguous-user-mention`, `cross-thread-requests`, and
`interleaved-agent-reports`. Failed `multiline-message` and
`narrative-agent-names` because both omitted the event-level user
mention despite otherwise-correct content and threading.



<img width="1176" height="805" alt="Screenshot 2026-08-21 at 11 11
17 AM"
src="https://github.com/user-attachments/assets/fc259ec9-892c-4436-b022-1bf85415617d"
/>

Signed-off-by: Salman Mohammed <smohammed@squareup.com>
2026-08-21 12:00:04 -04:00

501 B
Raw Permalink Blame History

multiline-message

The agent sends a short release update whose blank lines and bullet boundaries must survive the buzz messages send shell call as real newline bytes. The verifier also checks the normal reply anchor and callback mention. This guards the first-newline truncation failure described in block/buzz#5787.

Run with the command in the parent README, replacing the task path with benchmarks/buzz-dataset/multiline-message.