a9640c7cc4
## Why Terminal-Bench can show task correctness while missing Buzz-specific collaboration failures, such as posting a final answer outside the user's thread. Add a small Buzz-native suite that scores relay behavior directly. ## What - Add Harbor tasks for direct thread replies, event-level callback mentions to a three-word user identity, targeted reads of explicitly named paths outside the workspace, and exact temporary-channel membership against a seeded directory of 50 users and 10 bots - Export sanitized post-agent relay evidence and extend solo runtime/provisioning for deterministic verification - Expose channel TTL in CLI search output and cover positive and negative transcript/state fixtures ## Risk Assessment Low to medium — this primarily adds benchmark and testbed machinery. The CLI change is an additive `ttl_seconds` field in channel-search JSON; relay behavior is unchanged. ## References - Motivated by the Buzz-specific behavior observed while benchmarking https://github.com/block/buzz/pull/6128 - The named-path regression case covers the issue being fixed by https://github.com/block/buzz/pull/6261 - Local live Sonnet runs: `reply-to-thread`, `user-mention`, and `create-channel-invite-users` each scored reward 1.0 with zero exceptions - The current branch prompt scored 3/5 on a concurrent `read-named-path-outside-workspace` batch: one explicit workspace-boundary refusal and one missing final response Generated with Codex --------- Signed-off-by: Salman Mohammed <smohammed@squareup.com>
Endpoint launch configs
Deployment-time mapping from manifest endpoint names to
EndpointLaunchConfig (provider / api_key_env / env), passed to
harbor run as --agent-kwarg endpoint_config=<path>. This is deployment
config, deliberately OUTSIDE the immutable condition manifest — the manifest
endpoint string remains the join key.
Every key in these files must be a manifest endpoint name; the loader treats all entries as endpoint configs (no comment keys).
m1-local.json
M1 wiring proof: both placeholder endpoints resolve to one local llama-server
(OpenAI-compatible, http://127.0.0.1:8091/v1, no cloud keys).
buzz-agent env contract (crates/buzz-agent/src/config.rs, pinned at the M1
binary SHA): provider=openai reads OPENAI_COMPAT_API_KEY +
OPENAI_COMPAT_BASE_URL; the runtime sets BUZZ_AGENT_MODEL from the
manifest endpoint name, which overrides OPENAI_COMPAT_MODEL — llama-server
ignores the model name, so the placeholder value is harmless there.
llama-server needs no real key; the provisioner's per-endpoint
llm_api_keys map supplies a dummy value.
The Databricks pilot config is the same file shape with real serving endpoint hosts/keys.