mirror of
https://github.com/block/buzz.git
synced 2026-10-27 12:40:30 +00:00
a9640c7cc4
## Why Terminal-Bench can show task correctness while missing Buzz-specific collaboration failures, such as posting a final answer outside the user's thread. Add a small Buzz-native suite that scores relay behavior directly. ## What - Add Harbor tasks for direct thread replies, event-level callback mentions to a three-word user identity, targeted reads of explicitly named paths outside the workspace, and exact temporary-channel membership against a seeded directory of 50 users and 10 bots - Export sanitized post-agent relay evidence and extend solo runtime/provisioning for deterministic verification - Expose channel TTL in CLI search output and cover positive and negative transcript/state fixtures ## Risk Assessment Low to medium — this primarily adds benchmark and testbed machinery. The CLI change is an additive `ttl_seconds` field in channel-search JSON; relay behavior is unchanged. ## References - Motivated by the Buzz-specific behavior observed while benchmarking https://github.com/block/buzz/pull/6128 - The named-path regression case covers the issue being fixed by https://github.com/block/buzz/pull/6261 - Local live Sonnet runs: `reply-to-thread`, `user-mention`, and `create-channel-invite-users` each scored reward 1.0 with zero exceptions - The current branch prompt scored 3/5 on a concurrent `read-named-path-outside-workspace` batch: one explicit workspace-boundary refusal and one missing final response Generated with Codex --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com>
45 lines
1.4 KiB
YAML
45 lines
1.4 KiB
YAML
name: Harbor Buzz Orchestra
|
|
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
paths:
|
|
- "benchmarks/harbor-buzz-orchestra/**"
|
|
- "benchmarks/buzz-dataset/**"
|
|
- ".github/workflows/benchmark-harbor.yml"
|
|
pull_request:
|
|
paths:
|
|
- "benchmarks/harbor-buzz-orchestra/**"
|
|
- "benchmarks/buzz-dataset/**"
|
|
- ".github/workflows/benchmark-harbor.yml"
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
test:
|
|
name: Python tests and lint
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 15
|
|
steps:
|
|
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
|
|
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
|
|
with:
|
|
python-version: "3.12"
|
|
cache: pip
|
|
- name: Test adapter
|
|
working-directory: benchmarks/harbor-buzz-orchestra
|
|
run: |
|
|
python -m pip install --disable-pip-version-check -e ".[dev]"
|
|
pytest -q
|
|
ruff check .
|
|
# The task verifiers live in the sibling benchmarks/buzz-dataset, so
|
|
# they need the harness config passed explicitly to stay linted.
|
|
ruff check --config pyproject.toml ../buzz-dataset
|
|
- name: Test provisioner
|
|
working-directory: benchmarks/harbor-buzz-orchestra/testbed
|
|
run: |
|
|
python -m pip install --disable-pip-version-check -e ".[dev]"
|
|
pytest -q
|
|
ruff check .
|