mirror of
https://github.com/block/buzz.git
synced 2026-11-01 23:11:58 +00:00
ss-dev-01/transcript-presentation
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Atish Patel
|
c405ad1d4b |
feat(agent): fix Anthropic prompt caching with Databricks (+ MCP proxy/TLS passthrough) (#3463)
> On 8 tasks matched by name across the two runs, cost fell $8.36 → $1.77 (4.71×) and wall-clock 12,423 s → 1,085 s (11.45×). ## Summary Two independent, self-contained fixes to `buzz-agent`/`buzz-acp`, split out of the benchmark branch so they can land while the harness work continues: 1. **Request and surface Anthropic prompt caching.** buzz never sent a `cache_control` breakpoint, so on the Databricks Anthropic route `cache_read_input_tokens` was **structurally always 0** and the ~10× cache-read discount was never claimed. This teaches `anthropic_body()` to mark the cacheable prefix, and plumbs the cache split end-to-end so accounting can price it. 2. **Pass proxy + TLS-trust env into MCP tool subprocesses**, so agent tools on a proxy-only host stop reporting a live network as offline. ## Why the caching gap matters The Anthropic Messages API does **not** cache unless the request carries a `cache_control` breakpoint, and the Databricks AI Gateway — a third-party proxy in front of the model, in the same category as Bedrock/Vertex — does **not** auto-cache (only the first-party Anthropic API and Claude-on-AWS do zero-config caching). So every request was billed cold. Measured live against the Databricks gateway (`databricks-claude-opus-5`, 2026-07-28), the same call with and without a single `cache_control` marker: | Run | `input_tokens` | `cache_creation` | `cache_read` | latency | |---|---|---|---|---| | No `cache_control`, two byte-identical calls | 121,625 | 0 | **0** | ~9.3 s | | With one marker — cold (write) | 4 | 121,625 | 0 | 9.3 s | | With one marker — warm (read) | 4 | 0 | **121,625** | **4.5 s** | One marker moved 121,625 tokens from full-price input to a 0.1× cache read and roughly halved latency (a clean, isolated ~2.07× prefill speedup on this single-threaded microbenchmark). The gateway honours `cache_control`; buzz simply never sent it. At fleet scale this was a real budget item. Across matched Terminal-Bench solo sweeps (89 tasks, `-n 20`, before the fix), the two OpenAI-route models independently landed at ~86–87% cache reads — the expected shape for an agentic loop, where system + tools + append-only history repeat every turn — while the Anthropic route returned a hard 0% on every receipt: | Condition | Route | Input tokens | Cache reads | Cost | Cost if uncached | Discount | |---|---|---|---|---|---|---| | luna (`gpt-5-6`) | OpenAI | 20,320,818 | **17.7M (87.0%)** | $6.96 | $22.87 | **3.28×** | | sol (`gpt-5-6`) | OpenAI | 22,312,290 | **19.2M (85.9%)** | $37.07 | $123.35 | **3.33×** | | opus (`claude-opus-5`) | Anthropic | 12,459,822 | **0 (0.0%)** | $81.31 | $81.31 | **1.00×** | Applying luna's measured 87% read rate to the opus token counts at list prices (`input $5/M`, `cached_input $0.5/M`, `output $25/M`) puts the opus run at **~$32.53 vs the $81.31 actually paid — a ~60% overspend on those 49 trials (~$89 on a full sweep)**. That is an upper bound (it prices every cached token at the 0.1× read rate and ignores the 1.25× write premium), and the opus discount is structurally smaller than luna/sol's because opus emits ~3.5× more uncacheable output per trial, which sets a floor on what caching can recover. There is also a plausible **second-order effect**: Databricks appears to meter its per-minute rate limit on *uncached* input tokens, so the missing cache also cost rate-limit headroom — the opus endpoint lost 63% of its trials to fatal 429s while running alone at one-third of a GPT endpoint's raw throughput. This is a hypothesis, not a proven mechanism (the only zero-cache condition is also the only Anthropic endpoint), but it is the reading that explains the throttling with one rule instead of two. ## Post-fix results (provisional — first trials of an in-flight re-run) On 8 tasks matched by name across the two runs, cost fell **$8.36 → $1.77 (4.71×)** and wall-clock **12,423 s → 1,085 s (11.45×)**. | Metric | before (`4a955a858`) | after (`3bef1f6a`) | |---|---|---| | Cache reads as % of input | **0.0%** | **78.7%** (still climbing toward the ~86% steady state) | | `cost_usd_no_cache_discount / cost_usd` | **1.00×** | **2.18×** (tracking the projected ~2.5×) | | Trials with a fatal 429 (same `-n 20`) | **63%** | **15–19%** | To be clear about attribution: **~2× of that is the clean prefill saving from caching itself**; the rest is second-order — cached requests burn far less rate-limit budget, so they stall less and redo less destroyed work. The 11.45× is a system-level result specific to this throttled workspace, not a caching benchmark. Quality held (7/8 solved in each run). A controlled low-`-n` A/B (neither arm hitting a 429), which the `BUZZ_AGENT_PROMPT_CACHING` opt-out exists to enable, is still owed before this becomes a published claim. ## What changed ### 1. Request caching (`llm.rs`, `config.rs`) `anthropic_body()` emits ephemeral `cache_control` breakpoints, gated by `BUZZ_AGENT_PROMPT_CACHING` (**default on**, `=0` to opt out): - **Static prefix** — marker on the `system` block. Prefix order is `tools → system → messages`, so this single marker caches **tools + system** together. Byte-identical on every turn of a run, and survives a context handoff (system/tools come from cfg/mcp, not `self.history`). - **Rolling tail + leapfrog** — marker on the last block of the last **two** messages. The append-only history re-reads the prior turn's prefix from cache; marking two messages (not one) keeps consecutive breakpoints inside Anthropic's **20-block lookback window** even as tool parallelism rises, avoiding a silent full-price miss. An empty system prompt stays a bare string (Anthropic rejects empty text blocks), and below-threshold prefixes are silently not cached, so the flag is safe on by default. ### 2. Surface the cache split end-to-end — the plumbing (`types.rs`, `llm.rs`, `agent.rs`, `lib.rs`, `usage.rs`, `acp.rs`) This is the part that makes gaps like the one above **visible** instead of silent. A consumer that prices all of `input_tokens` at the full rate can't tell a route that's caching from one that isn't — the total looks right either way. So: - `LlmResponse` gains `cached_input_tokens` (a **subset** of `input_tokens`, never an addition); `parse_anthropic` / `parse_openai` / `parse_responses` each populate it. - A `usage_first()` helper reads the cache count wherever a provider hides it — flat `cache_read_input_tokens` (Anthropic), `prompt_tokens_details.cached_tokens` (OpenAI chat), `input_tokens_details.cached_tokens` (Responses) — taking the **first present value, never a sum**. Reading only flat keys is exactly why the OpenAI route's nested `cached_tokens` had *also* been going unclaimed: `prompt_tokens` is already inclusive, so the total looked correct while the discount silently went unreported. - The per-turn/per-session accumulators and the goose `usage_update` payload now carry `accumulatedCachedInputTokens`; `buzz-acp` deserializes it (`serde` default `0` for goose, which doesn't send it) and logs `cached=<n>`. ### 3. Fix a Databricks MLflow-route double-count (`llm.rs`) The Databricks MLflow route reports the flat Anthropic-spelled `cache_read_input_tokens` *alongside* an already-inclusive `prompt_tokens`, so the old code summed them and nearly doubled the count — inflating both the context-budget gate and cost. `openai_chat_input_tokens()` now reads `prompt_tokens` alone. Verified on a live `databricks-glm-5-2` response where `prompt_tokens + completion == total` proves inclusivity. (Anthropic's native route genuinely *excludes* the cache fields and is still summed — the two never collide, because `claude*` models route to the Anthropic path.) ### 4. Proxy + TLS-trust passthrough into MCP tools (`mcp.rs`) — independent fix `buzz-agent` `env_clear()`s each MCP child, and the allowlist carried no proxy/TLS vars. On a proxy-only host that doesn't degrade the tools, it **blinds** them: apt, curl, pip, git connect directly, the egress firewall resets the socket, and the agent reports "Connection reset by peer" — indistinguishable from a genuinely offline task. Adds both spellings of `HTTP(S)_PROXY`/`NO_PROXY`/`ALL_PROXY` (curl/git read lowercase; Go/Python read uppercase; libcurl ignores uppercase `HTTP_PROXY`) plus `SSL_CERT_FILE`/`SSL_CERT_DIR` for TLS-terminating proxies that present their own CA. ## Testing - `cargo fmt --all -- --check`, `cargo clippy -p buzz-agent -p buzz-acp --all-targets -- -D warnings` — clean. - `cargo test -p buzz-agent -p buzz-acp` — **all green** (632 + 299 lib tests plus integration suites, 0 failures). New tests cover: the three breakpoints and the disabled/empty-system/single-message edge cases; nested-vs-flat cache parsing for all three routes; the Databricks inclusive-`prompt_tokens` fix; wire deserialization of `accumulatedCachedInputTokens`; and the proxy/TLS passthrough allowlist. - Pre-push lefthook suite green (branch-skew, rust-tests, test, desktop-check/test/tauri). ## Relationship to the benchmark branch These are the non-`benchmarks/` changes from `benchmark/harness-accounting-and-solo`, lifted onto a clean base off `main` so they can merge independently. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Atish Patel <atish@squareup.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Will Pfleger
|
18eef633d8 |
feat(git): use agent display name as git author name (#3040)
Agent commits were authored by a raw 63-character npub, which makes `git log`, `git blame`, and GitHub's author column effectively unreadable. This uses the agent's display name for `user.name` instead, while leaving the pubkey where it does real work. ## What changes `build_git_env` in `crates/buzz-dev-mcp/src/shim.rs` now reads `BUZZ_ACP_DISPLAY_NAME`, sanitizes it, and uses the result as `user.name`. When the variable is absent or unusable it falls back to `info.npub` — byte-identical to today's behavior. `user.email`, `user.signingkey`, and the whole credential/signing block are untouched. The pubkey is what NIP-98 auth, NIP-GS signing, and contributor matching key on, and it stays in the email verbatim. `crates/buzz-acp/src/lib.rs` forwards the variable into the dev-mcp server's declared env, mirroring the existing `BUZZ_AUTH_TAG` block. It reads `std::env::var` directly rather than going through `Config`, so the variable is picked up whenever the process has it. `crates/buzz-agent/src/mcp.rs` adds one `PASSTHROUGH_ENV` entry so ACP clients that spawn `buzz-agent` without declaring the variable on the wire still propagate it. ## Why a dedicated variable `BUZZ_ACP_DISPLAY_NAME` is its own contract rather than a reuse of the ACP session title. Commits outlive sessions: a session title is per-session UI chrome and may be composed downstream into `Agent · #channel`, and if that composed form ever reached the env var, git attribution would change silently with no test able to catch it. Git identity gets a variable whose contract is "bare agent display name, never channel-qualified." Nothing writes it yet — a one-line Desktop write lands as a follow-up. Until then `std::env::var` returns `Err`, the npub fallback fires, and behavior is byte-for-byte current `main`. ## Sanitizing Strip control characters, Unicode format characters, and angle brackets; collapse whitespace runs, trim, cap at 80 characters (by `chars()`, so a multi-byte name is never split mid-UTF-8). Angle brackets go because git drops them silently rather than erroring: `Duncan <evil@x.com>` renders as `Duncan evil@x.com <hex@relay>`. It forges nothing, but it reads as though it might. The empty result also has to cover more than literal emptiness. git's `ident.c` treats a set of characters as "crud" — stripped from both ends, and fatal when a name is *nothing but* those characters: ``` $ git -c user.name=';;' commit -m t fatal: name consists only of disallowed characters: ;; ``` Verified against git 2.54.0 by committing with each ASCII byte 32..=126 as the entire `user.name`: exactly space, `"`, `'`, `,`, `:`, `;`, `<`, `>`, `\` abort, plus all control characters (the predicate is `c <= 32`). `.` is not crud in this version, despite older lore. Names that merely *contain* crud are fine — `O'Brien` and `Smith, Jr.` both commit cleanly — so the check is "at least one non-crud character survives," not "no crud present." Without it, a display name of `;;` or `""` would abort every commit that agent makes. ## Unicode format characters `char::is_control` covers only category `Cc`. Category `Cf` — zero-width spaces and joiners, bidi embedding and override marks, invisible math operators, tag characters — is neither control, nor whitespace, nor git crud, so those characters survived every one of the checks above. A display name of nothing but U+200B ZERO WIDTH SPACE therefore satisfied "at least one non-crud character survives" and git accepted the commit with a visually blank author: ``` # pre-fix, BUZZ_ACP_DISPLAY_NAME set to two U+200B $ git log -1 --format='%an' | xxd -p e2808be2808b0a ``` Embedded marks were the other half: a trailing U+202E RIGHT-TO-LEFT OVERRIDE reorders everything after it, so a stored author line renders as something other than what it stores — the same confusion class the angle-bracket filtering exists to prevent. `is_unicode_format` rejects the whole `Cf` category rather than the known-bad marks, because the boundary that matters is "invisible or reorders text", not "the codepoint someone thought of". The 21 ranges come from the UCD's `DerivedGeneralCategory.txt` (17.0.0), cross-checked against Python's `unicodedata` (16.0.0); both yield exactly the same set. They are inlined as a `matches!` rather than pulling in a Unicode-tables crate for one predicate, and a test asserts both endpoints of every range plus the codepoints immediately outside them — including U+2065, which sits inside the U+2060 block but is unassigned rather than `Cf`. Filtering happens inside the existing per-word filter, so a format-only name collapses to empty and falls out through the same `None` → npub path as a crud-only name. No new fallback logic. And because filtering precedes truncation, invisible padding cannot eat the 80-character budget. ## NUL is handled one layer up An interior NUL is a sibling constraint that cannot be fixed here: it makes `Command::env` fail the entire spawn before this code runs, so it has to die at the writer. #3028 establishes that pattern for the session title in `resolve_session_title` via `filter(|c| !c.is_control())`, and the Desktop follow-up that writes `BUZZ_ACP_DISPLAY_NAME` inherits it. The shim sanitizer is a second line of defense for values that arrive from somewhere other than Desktop. ## Verified end to end Driving the real `buzz-dev-mcp` binary over stdio MCP and committing inside its shimmed environment: ``` # BUZZ_ACP_DISPLAY_NAME="Duncan Idaho" Duncan Idaho <dcfd242e...0f95@buzz.block.builderlab.xyz> verify_exit=0 # BUZZ_ACP_DISPLAY_NAME unset npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e...0f95@buzz.block.builderlab.xyz> verify_exit=0 # BUZZ_ACP_DISPLAY_NAME=";;" (crud-only; would otherwise be fatal) npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e...0f95@buzz.block.builderlab.xyz> verify_exit=0 # BUZZ_ACP_DISPLAY_NAME=U+200B U+200B (format-only; would otherwise be blank) npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e...0f95@buzz.block.builderlab.xyz> verify_exit=0 # BUZZ_ACP_DISPLAY_NAME="Duncan" + U+202E (bidi override stripped) Duncan <dcfd242e...0f95@buzz.block.builderlab.xyz> verify_exit=0 # BUZZ_ACP_DISPLAY_NAME="Dun" + U+200B + "can" (zero-width removed, word not split) Duncan <dcfd242e...0f95@buzz.block.builderlab.xyz> verify_exit=0 ``` Signature verification passes in every case — the signing identity is unchanged. `Related: #3028` — it establishes the Desktop-side env plumbing this builds beside; the one-line Desktop follow-up that writes `BUZZ_ACP_DISPLAY_NAME` alongside the session title ships after it merges. Not a dependency: with the variable absent, `std::env::var` returns `Err` and the npub fallback keeps current behavior exactly. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> |
||
|
Will Pfleger
|
5afa16157a |
fix(desktop): suppress Windows console flashes and reject WSL bash alias (#2587)
Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> |
||
|
morgmart
|
54e174b5e1 | Chat-first agent creation via portable buzz CLI drafts (#1878) | ||
|
Will Pfleger
|
020ac7f405 |
fix(desktop): resolve Git Bash for Windows shell tool via PATH/git/registry fallback (#1821)
Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@sprout-oss.stage.blox.sqprod.co> |
||
|
Will Pfleger
|
4201311821 |
fix: make managed-agent spawn and teardown portable to Windows (#1097)
Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@sprout-oss.stage.blox.sqprod.co> |
||
|
tlongwell-block
|
84f499cb6e |
fix(buzz-agent): cap tool-result text at 50 KiB with middle elision (#952)
Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co> |
||
|
Will Pfleger
|
d99ad131f1 |
refactor: rename sprout backend to buzz (#958)
Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <wpfleger@block.xyz> Signed-off-by: Will Pfleger <wpfleger@squareup.com> Signed-off-by: Will Pfleger <wpfleger96@gmail.com> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub1fgdl5qqnh3k3f2xkqrvt7cujalhm623x4s7fdjdj5yrtp5fzjl9qrjpucw <4a1bfa0013bc6d14a8d600d8bf6392efefbd2a26ac3c96c9b2a106b0d12297ca@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub16v54tttfqacx9ycvc3k0ut0npj564ahcuajzy6qjvh57ntmsf4uq4806j2 <d32955ad69077062930cc46cfe2df30ca9aaf6f8e76422681265e9e9af704d78@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Will Pfleger <wpfleger96@gmail.com> |