jerry/buzz

mirror of https://github.com/block/buzz.git synced 2026-10-27 12:40:30 +00:00

Files

Salman Mohammed 79df468255 Add Buzz benchmark evaluation layers (#6823 )

## Summary

- classify every Buzz-native benchmark task as `regression` or
`workflow`
- add metadata-backed `--layer` selection with k=1 regression and k=3
workflow defaults, while preserving explicit `--attempts`/`-k`
- split an unqualified mixed Buzz dataset run into two Harbor jobs so
each layer gets the correct default
- document the layer contract and cover metadata, selection, defaults,
overrides, invalid input, and command construction

---------

Signed-off-by: Salman Mohammed <smohammed@squareup.com>

2026-08-26 17:41:56 -04:00

environment

test(benchmarks): expand Buzz-native dataset (#6448 )

2026-08-21 12:00:04 -04:00

tests

test(benchmarks): expand Buzz-native dataset (#6448 )

2026-08-21 12:00:04 -04:00

instruction.md

test(benchmarks): expand Buzz-native dataset (#6448 )

2026-08-21 12:00:04 -04:00

README.md

test(benchmarks): expand Buzz-native dataset (#6448 )

2026-08-21 12:00:04 -04:00

task.toml

Add Buzz benchmark evaluation layers (#6823 )

2026-08-26 17:41:56 -04:00

README.md

multiline-message

The agent sends a short release update whose blank lines and bullet boundaries must survive the buzz messages send shell call as real newline bytes. The verifier also checks the normal reply anchor and callback mention. This guards the first-newline truncation failure described in block/buzz#5787.

Run with the command in the parent README, replacing the task path with benchmarks/buzz-dataset/multiline-message.