Amendments from the tb21-twins-1 fail triage (all 24 zero-reward trials):
- Delegation paragraph replaced: four wake triggers (parallel exploration,
rival hypotheses, test-as-I-build, independent verification) with the
burden inverted — an idle Mie is a decision to defend. Outstanding Mie
lane doubles as Mia's heartbeat (fixes the dead-turn trial).
- Todo paragraph: task event id written at minute 0 + mandatory fetch-by-id
reread of the full task text as the last item before DONE: (fixes
grading-against-memory failures).
- Deliverable-first clause: rough deliverable at the exact task path ASAP
(12 of 24 fails were timeouts with nothing at the required path; 4 other
timeouts scored 1.0 purely on disk artifacts).
- Mention-mechanics clause: silent lane => treat wake as unrouted, recheck
exact @name, resend once (filter-js-from-html lost its verify lane to a
missing @ prefix).
mie.md unchanged. Manifest mia sha256 re-pinned; loader validates.
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Two-seat twin roster smoked green on hello-world (reward 1.0) and now
running the full TB2.1 89-task bench as tb21-twins-1. Mia holds the conn
(orchestrator seat — the harness watches that pubkey for DONE:), Mie is
the delegable worker twin. Both personas are the Meli solo prompt plus
twin team sections, byte-identical to each other except the identity
line and the conn/worker paragraph.
Contract: thread-of-record comms (--reply-to the task post), both keep
the todo tool loaded, Mia never ends her turn while the task is
incomplete (sleep-30 poll loop; requires unbounded agent rounds from
#5318), Mie ends every turn with a report mentioning Mia, only Mia sends
DONE:. Model: deepseek-v4-flash via OpenRouter pinned cloudflare/fp8 —
the existing openrouter-cloudflare.json endpoint config, unchanged.
Manifest persona sha256 pins match the committed files.
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Commits the tb21 Meli-solo condition that until now lived only as
untracked files on the runner box, plus the take-2 variant Tyler asked
for (thread 74a65a09, event f7457e0b):
- tb-meli-solo.yaml + personas/bench/meli-solo.md: the exact manifest
and persona (sha256 de633019...) that produced run tb21-solo-1 on the
gmicloud/fp8 pin. Recorded for provenance.
- tb-meli-solo-baseten.yaml: identical cell, condition
tb-meli-solo-baseten, priced at the baseten/fp8 listed rates
(prompt $0.13/M, completion $0.26/M — OpenRouter endpoints API,
2026-08-07).
- testbed/endpoints/openrouter-baseten.json: the provider pin,
OPENROUTER_PROVIDER_ORDER=baseten/fp8. A separate endpoint file so
openrouter-live.json (which run 1's provenance points at) stays
byte-identical.
Run 2 executes on the post-merge harness (9cafe8f1, PR #5145), which
carries #4896 (vision-input recovery) and #5136 (explicit-mention task
post) that run 1 predated.
Originating Buzz thread:
buzz://message?channel=c3252dd2-0142-4e01-88c7-a2183c3960a5&id=74a65a0990fd2197882b66b5ea2707169d4a3dbd2020d1610c45150fb99f140b
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
`_settle_usage` waited for the agent to report its tokens before teardown
killed it, but only on the success path. The timeout path fell straight
through to the kill, on the stated reasoning that "a turn that never
completed has no usage to flush".
That was true while buzz-agent reported once per turn. It is not true now
that it reports after every provider round, and it was always the wrong
path to skip: under `continue_until_timeout` every phase but the last ends
here, so the skip applied to essentially the whole run. It is the harness
half of the 97%-zero-rows / 5.9x-undercount measured on the OpenRouter
LHTB-46 cell; the agent half is the per-round reporting.
Moving the call into the `finally` covers both paths and keeps the ordering
that matters — settle, then kill. It stays cheap: it returns on the first
poll once a usage line exists, which after the first round it does, so only
a phase that never completed a single round can spend the full budget, and
that phase has nothing to report anyway.
Depends on the per-round reporting in buzz-agent to be worth much on its
own.
Signed-off-by: Atish Patel <atish@squareup.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Three cells, all at reasoning effort `high`, all on pinned OpenRouter
upstreams:
tb-solo-kimi-k3-high TB 2.1 solo, moonshotai/kimi-k3
tb-solo-deepseek-v4-flash-high TB 2.1 solo, deepseek-v4-flash-0731
lhtb-or-kimi-lead-2deepseek-high LHTB-46 team: kimi lead, deepseek
scout, deepseek worker
The first cells in the study on open-weight models and on a route other
than direct-OpenAI or Databricks. The solo pair spans a 22x input price
ratio ($0.133 vs $3.00/Mtok), making it the open-weight analogue of the
luna -> sol ladder; the team cell is the open-weight restaging of
tb-gt-sol-luna-terra-high, which is currently the study's only clean
long-horizon team win.
NO MEDIUM-EFFORT CELLS. The original plan had a high/medium pair per
model. Neither model supports it: both advertise supported_efforts
["max","high","low"], and while OpenRouter accepts `medium` without
error, measured reasoning length on deepseek did not separate low from
medium from high (medians 1097 / 1102 / 1067, within-level spread wider
than between-level). A cell labelled `medium` would name a condition the
provider does not implement, so the effort axis is dropped rather than
faked.
Effort is still pinned on all three cells even though `high` matches
deepseek's provider default -- an unpinned cell records no effort in its
manifest, so a later provider-side default change would silently redefine
the condition without moving the hash. kimi defaults to `max`, so there
`high` is a deliberate step down.
Prices are the PINNED endpoints' own rates, not the model ids' headline
rates -- gmicloud/fp8 bills $0.133/$0.266 where the model id advertises
$0.14/$0.28, and two of kimi's ten upstreams bill 1.5x the row here. The
pin is what makes these rows the correct ones, so dropping it would
require restating cost as well as re-establishing the condition.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Atish Patel <atish@squareup.com>
An OpenRouter model id is a family of upstream deployments, not one
deployment. `deepseek-v4-flash-0731` is served by nine upstreams spanning
fp4 to fp8 quantization, 262K to 1M context and a 1.6x price spread;
`kimi-k3` by ten spanning 1.5x. Unpinned, consecutive requests land on
different ones, so a benchmark cell measures a mixture whose composition
moves with provider load rather than a fixed condition.
It also decides whether prompt caching happens at all. Measured today
against a fixed 21.8K-token prefix, three calls each:
deepseek, unpinned cached 0, 0, 21760 (GMICloud, Cloudflare x2)
deepseek gmicloud/fp8 cached 21760 x3 $0.00059/call
deepseek siliconflow/fp8 cached 0 x3 $0.00306/call
kimi, unpinned cached 0, 0, 20480 (Fireworks, Moonshot x2)
kimi moonshotai/mxfp4 cached 20480 x3 $0.00715 vs $0.06245
That is a 5-9x swing in input cost for identical work, decided by
routing luck. The `/endpoints` metadata is no guide: it advertises
`supports_implicit_caching: false` for every endpoint measured caching
above, and `true` only for one that is not routable on this account.
So: `OPENROUTER_PROVIDER_ORDER` (comma-separated, accepting both the bare
`moonshotai` slug and the `gmicloud/fp8` slug/quantization form) sets
`provider.order`. It is strictly opt-in -- unset leaves the body byte
identical, which is what keeps the existing "no body shape adds a provider
routing filter" assertions honest.
Paired with `allow_fallbacks: false`, because `order` alone is only a
preference: OpenRouter still serves from elsewhere when the named upstream
is busy, which is exactly the silent mid-run condition change this exists
to prevent. A pin that quietly falls back is not a pin. This is not
`require_parameters` -- that filter screens on advertised parameters and
404s otherwise-valid model ids, which is why the reasoning block nearby
stays away from it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Atish Patel <atish@squareup.com>
Brings the benchmark harness (accounting, container runtime, manifests,
personas, summarize/leaderboard tooling) forward onto main. main does not
touch benchmarks/, so this is the branch tree verbatim.
Signed-off-by: Atish Patel <atish@squareup.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
## Problem
Two failure modes from the `tb21-glm52-crusoe-1` benchmark run (GLM-5.2
solo, TB2.1) wedged or killed 13 of 89 trials without the model being at
fault:
1. **Conversation poisoning on text-only endpoints.** Crusoe's
serverless `crusoeai/GLM-5.2-NVFP4` rejects any request whose history
contains an image with `400: ... is not a multimodal model`. The
recovery machinery for exactly this case already exists —
`AgentError::UnsupportedImageInput` → `replace_unsupported_images()`
strips the image blocks, marks the tool result as an error, substitutes
a text placeholder, and continues the turn. But classification only
matched OpenRouter's 404 body (`no endpoints found that support image
input`) and was only consulted on the 404 arms. The Crusoe 400 fell
through to terminal `AgentError::Llm`: the image stayed in history,
every subsequent call failed identically, buzz-acp rode its 10-retry
ladder (~40 min), and the trial idled to budget death. Measured blast
radius: **8 trials wedged, 12.7h aggregate idle-after-poison.**
2. **Bounded agent rounds in benchmark trials.** The harness default
`DEFAULT_MAX_AGENT_ROUNDS = 32` ended solo trials mid-work when turns
rotated (thinking-heavy models hit max_tokens rotation fast; 4 trials
died this way). Benchmark trials already have a wall-clock budget as the
real limit — the round cap only converts recoverable rotation into trial
death.
## Fix
- `is_unsupported_image_input_error()` also matches the verbatim `is not
a multimodal model` body. Matcher stays deliberately tight (same
doctrine as `is_context_length_error`): misclassifying a generic 400 as
recoverable would mutate history for an error that removing images
cannot fix.
- Both status ladders — shared `post()` and `openrouter_post()` —
consult it on their 400 arms and return the typed
`UnsupportedImageInput` (OpenAI-compatible providers report this as 400;
a BYOK/passthrough upstream can surface the provider's own 400 through
OpenRouter).
- Harness `DEFAULT_MAX_AGENT_ROUNDS` → `0` (unbounded —
`BUZZ_AGENT_MAX_ROUNDS=0` is the agent config's documented unbounded
value). Per-agent `budget.max_calls` in manifests still overrides.
## Acceptance
- A 400 with the image-rejection body reaches the existing image-strip
recovery path instead of wedging the session — asserted through
`complete()` (covers the return path into the convergence mapper) and at
the `openrouter_post` terminal, both proving single-attempt (a
deterministic capability rejection must never be retried).
- Ordinary 400s stay terminal `AgentError::Llm` (existing negative tests
unchanged).
- Benchmark trials run unbounded rounds by default; python tests updated
for 0-is-legal with a negative arm at -1.
## Verification
- `cargo test -p buzz-agent`: 427 + 18 + 20 + 15 + 8 + 1 + 48 passed, 0
failed (full package, 3 consecutive clean runs)
- `cargo clippy -p buzz-agent --all-targets`, `cargo fmt --check`: clean
- `uv run --extra dev pytest tests/` in harbor-buzz-orchestra: 35 passed
- Pre-push hooks (full workspace rust-tests + desktop-tauri-checks)
green on rustc 1.95.0 at head b0438602
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
## Problem
The harness posts each trial's task via `buzz messages send`, relying on
`@<orchestrator-id>` name resolution. Task text is untrusted payload:
when it contains @-tokens of its own, the CLI's mention resolver tries
to resolve them as channel members, fails, and refuses to send — killing
the trial with `RuntimeLaunchError` before the agent ever saw the task.
Live occurrence: TB 2.1's `large-scale-text-editing` task embeds Vim
macros (`:%normal! @a`). In the tb21-solo-1 run the trial died at
launch:
```
RuntimeLaunchError: buzz messages send ... exited 1:
{"error":"user_error","message":"mention '@a' does not match a current channel member; retry with --mention <pubkey>"}
```
Any TB task whose statement contains @-syntax is silently zeroed this
way.
## Fix
Pass the orchestrator's pubkey as an explicit `--mention` when posting
the task. The CLI demotes unresolved @-tokens in the text to
presentation-only when any explicit identity is supplied, so delivery
still targets exactly the orchestrator and every @-token in the task
statement becomes inert. The harness already holds the orchestrator's
`AgentCredential` (it writes that pubkey into the worker roster tables),
so no persistence is needed — fresh key per trial, fresh `--mention` per
trial.
Verified both halves against a live relay: a fenced `@a` without
`--mention` still hard-fails (the resolver is not markdown-aware); the
same content with an explicit `--mention` sends clean with
`mention_pubkeys` containing only the target.
## Testing
- `benchmarks/harbor-buzz-orchestra`: full pytest suite — 35 passed (34
baseline + new `test_send_mentions_by_pubkey_so_task_text_stays_inert`),
ruff clean. Run against `origin/main` 769ac70b with exactly this patch
applied.
- `testbed`: full pytest suite — 23 passed, 1 skipped; ruff clean.
## Acceptance
A task statement containing arbitrary @-tokens (Vim registers, emails,
decorators) launches and delivers to the orchestrator instead of dying
in `_send`.
Originating Buzz thread:
`buzz://message?channel=c3252dd2-0142-4e01-88c7-a2183c3960a5&id=74a65a0990fd2197882b66b5ea2707169d4a3dbd2020d1610c45150fb99f140b`
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Search migrated to Postgres FTS (commit f8bbe6efc).
The Typesense container was removed from compose.yml and the Helm chart,
but the cleanup missed two template/config files:
- `deploy/compose/.env.example`: `TYPESENSE_API_KEY` and
`TYPESENSE_PORT` are dead — no typesense service exists in compose.yml
and the relay binary no longer reads `TYPESENSE_API_KEY`. The
`CHANGE_ME_RANDOM_API_KEY` placeholder was never consumed, so removing
it also unbreaks the sed loop in the blog draft (one fewer no-op secret
to generate).
- `benchmarks/harbor-buzz-orchestra/scripts/benchmark.py`: generates a
typesense_api_key in state and writes `TYPESENSE_API_KEY` to the .env
file it creates.
- *Editing this file caused the
https://github.com/block/buzz/blob/main/.github/workflows/benchmark-harbor.yml
linter ci checks to run, which seemingly haven't run before, so I needed
fix the lint issues to pass this.*
---------
Signed-off-by: Kalvin Chau <kalvin@block.xyz>
Co-authored-by: npub1c4alndp82zyt9veaklm5d965quss79vlhk9awv7qu5erwhmf42qqlvc25c <c57bf9b4275088b2b33db7f746975407210f159fbd8bd733c0e532375f69aa80@buzz.block.builderlab.xyz>
