## Why
Terminal-Bench can show task correctness while missing Buzz-specific
collaboration failures, such as posting a final answer outside the
user's thread. Add a small Buzz-native suite that scores relay behavior
directly.
## What
- Add Harbor tasks for direct thread replies, event-level callback
mentions to a three-word user identity, targeted reads of explicitly
named paths outside the workspace, and exact temporary-channel
membership against a seeded directory of 50 users and 10 bots
- Export sanitized post-agent relay evidence and extend solo
runtime/provisioning for deterministic verification
- Expose channel TTL in CLI search output and cover positive and
negative transcript/state fixtures
## Risk Assessment
Low to medium — this primarily adds benchmark and testbed machinery. The
CLI change is an additive `ttl_seconds` field in channel-search JSON;
relay behavior is unchanged.
## References
- Motivated by the Buzz-specific behavior observed while benchmarking
https://github.com/block/buzz/pull/6128
- The named-path regression case covers the issue being fixed by
https://github.com/block/buzz/pull/6261
- Local live Sonnet runs: `reply-to-thread`, `user-mention`, and
`create-channel-invite-users` each scored reward 1.0 with zero
exceptions
- The current branch prompt scored 3/5 on a concurrent
`read-named-path-outside-workspace` batch: one explicit
workspace-boundary refusal and one missing final response
Generated with Codex
---------
Signed-off-by: Salman Mohammed <smohammed@squareup.com>
Search migrated to Postgres FTS (commit f8bbe6efc).
The Typesense container was removed from compose.yml and the Helm chart,
but the cleanup missed two template/config files:
- `deploy/compose/.env.example`: `TYPESENSE_API_KEY` and
`TYPESENSE_PORT` are dead — no typesense service exists in compose.yml
and the relay binary no longer reads `TYPESENSE_API_KEY`. The
`CHANGE_ME_RANDOM_API_KEY` placeholder was never consumed, so removing
it also unbreaks the sed loop in the blog draft (one fewer no-op secret
to generate).
- `benchmarks/harbor-buzz-orchestra/scripts/benchmark.py`: generates a
typesense_api_key in state and writes `TYPESENSE_API_KEY` to the .env
file it creates.
- *Editing this file caused the
https://github.com/block/buzz/blob/main/.github/workflows/benchmark-harbor.yml
linter ci checks to run, which seemingly haven't run before, so I needed
fix the lint issues to pass this.*
---------
Signed-off-by: Kalvin Chau <kalvin@block.xyz>
Co-authored-by: npub1c4alndp82zyt9veaklm5d965quss79vlhk9awv7qu5erwhmf42qqlvc25c <c57bf9b4275088b2b33db7f746975407210f159fbd8bd733c0e532375f69aa80@buzz.block.builderlab.xyz>
