jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-10-30 14:01:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
db514b153a8cb17631d6e80b91c95fdc45deb147
buzz/benchmarks/buzz-dataset/ambiguous-user-mention
History
Salman Mohammed b56a52ca11 test(benchmarks): expand Buzz-native dataset (#6448) 
## Why
Terminal task correctness alone does not guard Buzz-native collaboration
behavior such as exact reply routing, non-waking narrative names,
batched agent synthesis, thread isolation, and ambiguous identity
targeting.

## What
- Add five medium/hard Harbor tasks for multiline delivery, narrative
agent names, interleaved agent reports, cross-thread requests, and
ambiguous user mentions
- Add signed scripted-event fixtures, duplicate-display-name profiles,
and public-only evidence export support
- Add positive and adversarial verifier fixtures covering dropped
inputs, extra posts, incorrect routing, and unintended mentions

## Risk Assessment
Low — changes are limited to benchmark tooling and datasets; production
behavior is exercised but not modified.

## References
- Grounded in behavior reported in `buzz-community` and public issues
#5787, #5176, #5839, #4942, #4072, #4303, and #6257
- Live Claude Sonnet 4.6 smoke run: 2/5 passed; the three failures
exposed top-level instead of threaded delivery, empty narrative output,
and silent ambiguous-identity completion
- Live GPT-5.6 Luna run (`buzz-native-solo-luna.yaml`): 3/5 passed.
Passed `ambiguous-user-mention`, `cross-thread-requests`, and
`interleaved-agent-reports`. Failed `multiline-message` and
`narrative-agent-names` because both omitted the event-level user
mention despite otherwise-correct content and threading.



<img width="1176" height="805" alt="Screenshot 2026-08-21 at 11 11
17 AM"
src="https://github.com/user-attachments/assets/fc259ec9-892c-4436-b022-1bf85415617d"
/>

Signed-off-by: Salman Mohammed <smohammed@squareup.com>
2026-08-21 12:00:04 -04:00
..
environment
test(benchmarks): expand Buzz-native dataset (#6448)
2026-08-21 12:00:04 -04:00
tests
test(benchmarks): expand Buzz-native dataset (#6448)
2026-08-21 12:00:04 -04:00
instruction.md
test(benchmarks): expand Buzz-native dataset (#6448)
2026-08-21 12:00:04 -04:00
README.md
test(benchmarks): expand Buzz-native dataset (#6448)
2026-08-21 12:00:04 -04:00
task.toml
test(benchmarks): expand Buzz-native dataset (#6448)
2026-08-21 12:00:04 -04:00

README.md

ambiguous-user-mention

The channel contains two real identities with the exact same three-word display name. Their profile about fields carry different routing codes. The agent must discover the intended pubkey, notify it exactly once, never notify the twin, and separately callback the requester. This guards the silent ambiguity family reported in block/buzz#4303 and block/buzz#6257.