## Why
The Buzz-native runtime could stop a solo agent after one turn even when
a scripted follow-up event was queued or already running. That could
turn a harness timing race into a benchmark failure.
Follow-up to #6448.
## What
- Treat solo tasks with scripted events separately from ordinary
one-turn tasks.
- Wait until turn counts and authored message IDs remain idle and
unchanged for several polls before stopping the agent.
- Remove the message-count shortcut that could stop an active turn.
- Add focused regressions for a delayed follow-up turn and an
already-running turn with a `DONE:` message.
## Testing
- `pytest -q tests/test_container_runtime.py -k "solo_turn_end or
scripted_events"` — 3 passed
- `ruff check` on the three touched Python files
- `ruff format --check` on the three touched Python files
- Local `buzz-native-solo-luna.yaml` Harbor run, one attempt per
affected task:
- `cross-thread-requests` — reward 1.0; all 6 task-specific checks
passed
- `interleaved-agent-reports` — reward 1.0; all 7 task-specific checks
passed
- 2 completed trials, 0 exceptions, 56 seconds total
## Risk
Low. This only changes completion detection for solo benchmark tasks
with scripted events. Ordinary solo tasks retain immediate completion
after their first turn ends.
---
**Update Aug 21, 13:07 EDT:** Replaced the fixed settle delay with exact
delivery receipts after [review
feedback](https://github.com/block/buzz/pull/6487#discussion_r3832050452).
- ACP now records the event IDs delivered by each completed turn; the
runtime also recognizes existing successful-steer receipts.
- Scripted trials stop only after every expected event ID is
acknowledged and no turn is active. Missing receipts wait for the trial
budget instead of producing partial evidence.
- TDD regression delayed delivery beyond the old five-poll boundary and
failed before the implementation; the receipt parser and Rust receipt
format are pinned independently.
- Focused verification: 31 container-runtime tests passed, 6 ACP
delivery tests passed, plus Ruff, rustfmt, and Clippy.
- Receipt-gated Luna run: 2 completed trials and 0 exceptions.
`interleaved-agent-reports` scored 1.0. `cross-thread-requests`
completed both calculations and thread isolation but Luna omitted the
user mention on ALPHA, so that model-output dimension scored 0.
- Risk remains low: the ACP production change adds delivery receipt
logging and restores a missing turn-end log on a
completed-before-control race; it does not change queue dispatch
behavior.
Generated with Codex
---------
Signed-off-by: Salman Mohammed <smohammed@squareup.com>
## Why
Terminal task correctness alone does not guard Buzz-native collaboration
behavior such as exact reply routing, non-waking narrative names,
batched agent synthesis, thread isolation, and ambiguous identity
targeting.
## What
- Add five medium/hard Harbor tasks for multiline delivery, narrative
agent names, interleaved agent reports, cross-thread requests, and
ambiguous user mentions
- Add signed scripted-event fixtures, duplicate-display-name profiles,
and public-only evidence export support
- Add positive and adversarial verifier fixtures covering dropped
inputs, extra posts, incorrect routing, and unintended mentions
## Risk Assessment
Low — changes are limited to benchmark tooling and datasets; production
behavior is exercised but not modified.
## References
- Grounded in behavior reported in `buzz-community` and public issues
#5787, #5176, #5839, #4942, #4072, #4303, and #6257
- Live Claude Sonnet 4.6 smoke run: 2/5 passed; the three failures
exposed top-level instead of threaded delivery, empty narrative output,
and silent ambiguous-identity completion
- Live GPT-5.6 Luna run (`buzz-native-solo-luna.yaml`): 3/5 passed.
Passed `ambiguous-user-mention`, `cross-thread-requests`, and
`interleaved-agent-reports`. Failed `multiline-message` and
`narrative-agent-names` because both omitted the event-level user
mention despite otherwise-correct content and threading.
<img width="1176" height="805" alt="Screenshot 2026-08-21 at 11 11
17 AM"
src="https://github.com/user-attachments/assets/fc259ec9-892c-4436-b022-1bf85415617d"
/>
Signed-off-by: Salman Mohammed <smohammed@squareup.com>
## Why
Terminal-Bench can show task correctness while missing Buzz-specific
collaboration failures, such as posting a final answer outside the
user's thread. Add a small Buzz-native suite that scores relay behavior
directly.
## What
- Add Harbor tasks for direct thread replies, event-level callback
mentions to a three-word user identity, targeted reads of explicitly
named paths outside the workspace, and exact temporary-channel
membership against a seeded directory of 50 users and 10 bots
- Export sanitized post-agent relay evidence and extend solo
runtime/provisioning for deterministic verification
- Expose channel TTL in CLI search output and cover positive and
negative transcript/state fixtures
## Risk Assessment
Low to medium — this primarily adds benchmark and testbed machinery. The
CLI change is an additive `ttl_seconds` field in channel-search JSON;
relay behavior is unchanged.
## References
- Motivated by the Buzz-specific behavior observed while benchmarking
https://github.com/block/buzz/pull/6128
- The named-path regression case covers the issue being fixed by
https://github.com/block/buzz/pull/6261
- Local live Sonnet runs: `reply-to-thread`, `user-mention`, and
`create-channel-invite-users` each scored reward 1.0 with zero
exceptions
- The current branch prompt scored 3/5 on a concurrent
`read-named-path-outside-workspace` batch: one explicit
workspace-boundary refusal and one missing final response
Generated with Codex
---------
Signed-off-by: Salman Mohammed <smohammed@squareup.com>
## Problem
Two failure modes from the `tb21-glm52-crusoe-1` benchmark run (GLM-5.2
solo, TB2.1) wedged or killed 13 of 89 trials without the model being at
fault:
1. **Conversation poisoning on text-only endpoints.** Crusoe's
serverless `crusoeai/GLM-5.2-NVFP4` rejects any request whose history
contains an image with `400: ... is not a multimodal model`. The
recovery machinery for exactly this case already exists —
`AgentError::UnsupportedImageInput` → `replace_unsupported_images()`
strips the image blocks, marks the tool result as an error, substitutes
a text placeholder, and continues the turn. But classification only
matched OpenRouter's 404 body (`no endpoints found that support image
input`) and was only consulted on the 404 arms. The Crusoe 400 fell
through to terminal `AgentError::Llm`: the image stayed in history,
every subsequent call failed identically, buzz-acp rode its 10-retry
ladder (~40 min), and the trial idled to budget death. Measured blast
radius: **8 trials wedged, 12.7h aggregate idle-after-poison.**
2. **Bounded agent rounds in benchmark trials.** The harness default
`DEFAULT_MAX_AGENT_ROUNDS = 32` ended solo trials mid-work when turns
rotated (thinking-heavy models hit max_tokens rotation fast; 4 trials
died this way). Benchmark trials already have a wall-clock budget as the
real limit — the round cap only converts recoverable rotation into trial
death.
## Fix
- `is_unsupported_image_input_error()` also matches the verbatim `is not
a multimodal model` body. Matcher stays deliberately tight (same
doctrine as `is_context_length_error`): misclassifying a generic 400 as
recoverable would mutate history for an error that removing images
cannot fix.
- Both status ladders — shared `post()` and `openrouter_post()` —
consult it on their 400 arms and return the typed
`UnsupportedImageInput` (OpenAI-compatible providers report this as 400;
a BYOK/passthrough upstream can surface the provider's own 400 through
OpenRouter).
- Harness `DEFAULT_MAX_AGENT_ROUNDS` → `0` (unbounded —
`BUZZ_AGENT_MAX_ROUNDS=0` is the agent config's documented unbounded
value). Per-agent `budget.max_calls` in manifests still overrides.
## Acceptance
- A 400 with the image-rejection body reaches the existing image-strip
recovery path instead of wedging the session — asserted through
`complete()` (covers the return path into the convergence mapper) and at
the `openrouter_post` terminal, both proving single-attempt (a
deterministic capability rejection must never be retried).
- Ordinary 400s stay terminal `AgentError::Llm` (existing negative tests
unchanged).
- Benchmark trials run unbounded rounds by default; python tests updated
for 0-is-legal with a negative arm at -1.
## Verification
- `cargo test -p buzz-agent`: 427 + 18 + 20 + 15 + 8 + 1 + 48 passed, 0
failed (full package, 3 consecutive clean runs)
- `cargo clippy -p buzz-agent --all-targets`, `cargo fmt --check`: clean
- `uv run --extra dev pytest tests/` in harbor-buzz-orchestra: 35 passed
- Pre-push hooks (full workspace rust-tests + desktop-tauri-checks)
green on rustc 1.95.0 at head b0438602
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
## Problem
The harness posts each trial's task via `buzz messages send`, relying on
`@<orchestrator-id>` name resolution. Task text is untrusted payload:
when it contains @-tokens of its own, the CLI's mention resolver tries
to resolve them as channel members, fails, and refuses to send — killing
the trial with `RuntimeLaunchError` before the agent ever saw the task.
Live occurrence: TB 2.1's `large-scale-text-editing` task embeds Vim
macros (`:%normal! @a`). In the tb21-solo-1 run the trial died at
launch:
```
RuntimeLaunchError: buzz messages send ... exited 1:
{"error":"user_error","message":"mention '@a' does not match a current channel member; retry with --mention <pubkey>"}
```
Any TB task whose statement contains @-syntax is silently zeroed this
way.
## Fix
Pass the orchestrator's pubkey as an explicit `--mention` when posting
the task. The CLI demotes unresolved @-tokens in the text to
presentation-only when any explicit identity is supplied, so delivery
still targets exactly the orchestrator and every @-token in the task
statement becomes inert. The harness already holds the orchestrator's
`AgentCredential` (it writes that pubkey into the worker roster tables),
so no persistence is needed — fresh key per trial, fresh `--mention` per
trial.
Verified both halves against a live relay: a fenced `@a` without
`--mention` still hard-fails (the resolver is not markdown-aware); the
same content with an explicit `--mention` sends clean with
`mention_pubkeys` containing only the target.
## Testing
- `benchmarks/harbor-buzz-orchestra`: full pytest suite — 35 passed (34
baseline + new `test_send_mentions_by_pubkey_so_task_text_stays_inert`),
ruff clean. Run against `origin/main` 769ac70b with exactly this patch
applied.
- `testbed`: full pytest suite — 23 passed, 1 skipped; ruff clean.
## Acceptance
A task statement containing arbitrary @-tokens (Vim registers, emails,
decorators) launches and delivers to the orchestrator instead of dying
in `_send`.
Originating Buzz thread:
`buzz://message?channel=c3252dd2-0142-4e01-88c7-a2183c3960a5&id=74a65a0990fd2197882b66b5ea2707169d4a3dbd2020d1610c45150fb99f140b`
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Search migrated to Postgres FTS (commit f8bbe6efc).
The Typesense container was removed from compose.yml and the Helm chart,
but the cleanup missed two template/config files:
- `deploy/compose/.env.example`: `TYPESENSE_API_KEY` and
`TYPESENSE_PORT` are dead — no typesense service exists in compose.yml
and the relay binary no longer reads `TYPESENSE_API_KEY`. The
`CHANGE_ME_RANDOM_API_KEY` placeholder was never consumed, so removing
it also unbreaks the sed loop in the blog draft (one fewer no-op secret
to generate).
- `benchmarks/harbor-buzz-orchestra/scripts/benchmark.py`: generates a
typesense_api_key in state and writes `TYPESENSE_API_KEY` to the .env
file it creates.
- *Editing this file caused the
https://github.com/block/buzz/blob/main/.github/workflows/benchmark-harbor.yml
linter ci checks to run, which seemingly haven't run before, so I needed
fix the lint issues to pass this.*
---------
Signed-off-by: Kalvin Chau <kalvin@block.xyz>
Co-authored-by: npub1c4alndp82zyt9veaklm5d965quss79vlhk9awv7qu5erwhmf42qqlvc25c <c57bf9b4275088b2b33db7f746975407210f159fbd8bd733c0e532375f69aa80@buzz.block.builderlab.xyz>
