mirror of
https://github.com/block/buzz.git
synced 2026-10-27 20:41:57 +00:00
4b0744d7f3161d9f045dd67a73e014e4440c2b47
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tyler
|
068a83b097 |
feat(huddle): cut voice-turn time-to-first-audio from ~1.0 s to ~0.35 s (env-gated latency levers) (#5671)
## Overview
**Category:** feat (env-gated experiment + one exact always-on
optimization)
**Problem:** Speech-end -> first TTS audio through the desktop huddle
pipeline measures **924–1087 ms** on an M4 Max with a 0 ms LLM leg.
Voice turns feel sluggish no matter how fast the agent replies. Baseline
breakdown: ~300 ms hardcoded VAD silence flush + ~150–250 ms Parakeet
decode + ~380–550 ms TTS synthesis before the first player append.
**Outcome:** With all levers enabled, e2e time-to-first-audio measures
**347–384 ms** (307–357 ms on a longer utterance) on the same hardware,
harness, and production pipelines. Defaults preserve production behavior
everywhere except one deterministic, bit-exact cache win.
## What's in here
### Levers (all default-off, env-gated)
| Lever | Env | Effect (measured medians, short utterance) |
|---|---|---|
| Speculative Parakeet decode | `BUZZ_STT_SPECULATIVE=1` | STT leg ->
~max(flush, decode) |
| Streaming TTS synthesis | `BUZZ_TTS_STREAMING=1`,
`BUZZ_TTS_EMIT_FRAMES` | first audio 380–550 -> 211–320 ms (emit=12,
bit-exact) |
| ONNX intra-op threads | `BUZZ_STT_THREADS`, `BUZZ_TTS_THREADS` | TTS
first audio 211–320 -> 129–180 ms (4 threads) |
- **Speculative decode** starts the Parakeet decode at the *first*
silent VAD frame, overlapping it with the flush window. Resumed speech
invalidates the result (voiced-frame-count check); held silence emits it
instantly at the flush boundary.
- **Streaming TTS**: new `synth_chunk_streaming` (buzz-voice)
interleaves the Flow LM frame loop with incremental *stateful* Mimi
decoding, emitting PCM deltas to the player via the existing
`PlaybackChunkAudio` decoration. At `emit_frames=12` (the decoder's
native chunk) streamed audio is **bit-identical** to the batch path —
verified by the ignored test
`incremental_stateful_decode_matches_batch_decode` (max|diff|=0).
Smaller deltas are faster but diverge (~23 dB SNR; decoder intra-chunk
lookahead), hence the default of 12.
> **Removed after live testing:** the `BUZZ_STT_FLUSH_MS` flush-window
override. Lowering the silence window below natural mid-sentence pauses
(the fast-path recipe said 150 ms) split single spoken sentences into
multiple messages and confused the listening agents. The window is a
turn-taking quality knob, not a latency lever — it is now fixed at the
production 300 ms value.
### Push-to-talk grouping fix (always-on)
A held push-to-talk shortcut is an explicit "I am not done talking"
signal, so silence never ends the utterance while it is held — even when
the microphone is also manually open. The utterance flushes on shortcut
release (existing transmit-edge flush); a manually open mic with the
shortcut up keeps normal VAD pause flushing. Gate is the pure
`vad_flush_allowed` function with a unit-test truth table.
### Always-on (exact): voice-conditioning cache
Phase profiling (`BUZZ_TTS_PHASE_LOG=1`) showed a fixed ~160 ms
`condition_voice` Flow-LM pass on *every* chunk, re-deriving the same
post-conditioning state for the same reference voice. The state is now
snapshotted after first computation and restored per chunk (dtype-tagged
tensor copies, keyed identically to the existing `cached_voice`).
Deterministic — same tensors in, same tensors out. The default path's
TTS leg drops from 380–550 ms to 225–355 ms with no configuration.
### Bench harness
`huddle::latency_bench` (`#[cfg(test)]` + `#[ignore]`) drives the real
`SttPipeline` and `TtsPipeline`, feeding a 48 kHz WAV in real-time 100
ms batches (AudioWorklet cadence) with a configurable fake LLM in place
of the relay leg, timing speech-end -> transcript -> speak() -> first
accepted player append.
```
BUZZ_STT_SPECULATIVE=1 BUZZ_TTS_STREAMING=1 \
BUZZ_TTS_THREADS=4 BUZZ_STT_THREADS=2 \
BUZZ_BENCH_WAV=<48k f32 mono wav> \
cargo test --release -p buzz-desktop --lib huddle::latency_bench -- --ignored --nocapture
```
## Tradeoffs to weigh before promoting any lever to a default
- **Speculative decode**: the speculative buffer has ~1 silent tail
frame vs ~19; observed one CTC wobble ("fail" vs "failed") in 24 turns.
Mitigation if productionized: zero-pad the speculative buffer to match
the flush-path shape.
- **Threads**: defaults stay 1 pending the min-spec (4-core Intel) A/B
flagged in the existing `STT_NUM_THREADS` comment.
- **Streaming at emit<12** is NOT the same waveform — don't ship below
12 without an ear pass.
## Validation
- Full desktop lib suite: **2408 passed / 0 failed** at this head
(`18fab2e1c`).
- buzz-voice suite green; bit-exactness test passes against the
production batch decode.
- Defaults-only bench rerun stays in the baseline family everywhere
except the exact conditioning-cache win (stt 525–532, tts 225–355).
- `cargo clippy --workspace --all-targets -- -D warnings` + fmt clean
(pre-push hook battery green).
Measurement notes with per-lever logs: Eva's workspace,
`RESEARCH/HUDDLE_E2E_LATENCY_OPTIMIZATION_2026_08_12.md` +
`RESEARCH/HUDDLE_E2E_STT_FAKELLM_TTS_BASELINE_2026_08_12.md`.
## Suggested promotion order
1. Conditioning cache (in this PR, always-on, exact).
2. Streaming TTS at emit=12: bit-exact audio, biggest UX win — needs the
env-gate removed + barge-in soak + an ear pass on a real huddle.
3. Speculative decode with silence padding: near-free ~100–150 ms.
4. Threads: after min-spec A/B.
---------
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Signed-off-by: Max <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz>
Signed-off-by: Mari <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@buzz.block.builderlab.xyz>
Signed-off-by: tlongwell-block <tlongwell@block.xyz>
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Co-authored-by: Max <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz>
Co-authored-by: Mari <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@buzz.block.builderlab.xyz>
Co-authored-by: tlongwell-block <tlongwell@block.xyz>
|
||
|
klopez4212
|
0571f5455b |
Polish glass Huddle tray behavior (#5590)
## Summary - inset the in-app Huddle tray with four rounded corners and even 8px spacing when Glass background is enabled - keep the popped-out Huddle dock full-width - hide and suppress Glass background on Linux ## Why The in-app tray reused the opaque backing needed by non-glass windows, which covered the native vibrancy around it. Linux does not support this window treatment. ## Testing - `pnpm -C desktop build:e2e` - focused Appearance and Huddle Playwright smoke tests - pre-push desktop checks, typecheck, and 4,666 unit tests --------- Signed-off-by: kenny lopez <klopez4212@gmail.com> Signed-off-by: Princess Donut <b238ea756dee4d98afa5883fc7f1de61eeabe65bf700e3a5a5a80db5e42e2c2b@buzz.block.builderlab.xyz> Co-authored-by: Princess Donut <b238ea756dee4d98afa5883fc7f1de61eeabe65bf700e3a5a5a80db5e42e2c2b@buzz.block.builderlab.xyz> |
||
|
klopez4212
|
ce3cf3cd25 |
Polish Huddle voice controls (#4694)
## Summary - add a visible Stop control for interrupting agent speech - make push-to-talk available by default while preserving manual mute controls - refine agent management, muted audio states, drawer layering, and return navigation - suppress duplicate notification sounds for Huddle messages ## Why Huddles could trap users behind long agent speech, hide useful agent controls, and leave temporary Huddle state visible after the call. The drawer also regressed when the terminal substrate began painting behind the rounded app surface. ## Validation - `just desktop-ci` - focused Huddle Playwright coverage for the drawer, speech interruption, agent picker, and leave navigation --------- Signed-off-by: kenny lopez <klopez4212@gmail.com> Signed-off-by: Wes <wesbillman@users.noreply.github.com> Co-authored-by: Wes <wesbillman@users.noreply.github.com> Co-authored-by: Carl <c7ebe626f000404285d3686e1dc74cc07cc60a9754a150041ba132e14bd3e2ec@buzz.block.builderlab.xyz> |
||
|
Wes
|
cb4a73e17d |
Dock Buzz Term within channel workspace (#4724)
## Summary - replace Buzz Term's full-app takeover with a resizable bottom dock inside the channel content surface - add a discoverable channel-header button plus hide and maximize/restore controls - create PTYs lazily and keep separate, persistent terminal workspaces per channel - capture immutable channel/thread context on every terminal session ## Multiple-channel behavior The dock is a single surface, but its tabs are partitioned by channel. Switching channels swaps to that channel's sessions without terminating background PTYs; returning restores them. New tabs capture the currently visible channel/thread context. ## Verification At commit `7ca087f8e08c80528387684364a65bf4ccd6315f`: - `pnpm --dir desktop typecheck` - `pnpm --dir desktop test` — 4,129 passed - pre-push repository hooks — desktop check/test, Tauri checks, terminal Rust suites all passed --------- Signed-off-by: Wes <wesbillman@users.noreply.github.com> Signed-off-by: kenny lopez <klopez4212@gmail.com> Co-authored-by: Carl <c7ebe626f000404285d3686e1dc74cc07cc60a9754a150041ba132e14bd3e2ec@buzz.block.builderlab.xyz> Co-authored-by: kenny lopez <klopez4212@gmail.com> |
||
|
Tyler
|
631b05c883 |
feat: ship Buzz Term (#4347)
## Summary - ship **Buzz Term** end to end: the terminal engine/runtime, mounted desktop substrate, and user-visible naming - add Quinn's tape-deck-inspired banner: a beveled chassis filled by the `buzz term` wordmark, surrounded by a complete-hex field - derive the wordmark's three-stop sweep from each theme's terminal palette so primary, secondary, and accent roles remain visibly distinct across all 62 shipped themes, including light themes - paint the banner once on its own pointer-transparent canvas; PTY rendering beneath it remains unchanged ## Banner behavior - uses the renderer's shared `8.4 × 17` cell metrics and production aspect ratio `2.0238` - regenerates only for viewport/theme changes; palette switches repaint correctly while the banner is visible - dismisses on non-empty output from the active terminal session; empty output and inactive sessions do not dismiss it - fails closed below **70 columns** rather than squeezing or clipping the wordmark - adds **8 lines** to `terminalRenderer.ts` for shared cell metrics and **zero lines inside `paint()`** ## Screenshots | Buzz (light) | Buzz Dark | |---|---| |  |  | | Kanagawa Lotus (light) | Red | |---|---| |  |  | Additional production-aspect finals: [Vesper](https://buzz.block.builderlab.xyz/media/9ca6514b63f8cfb2107a85ca46f16a940c0883848e6fbc718e411af94aa13100.png), [Min Dark](https://buzz.block.builderlab.xyz/media/f67bd2970e5d64ffb07b1ae78ab58c847e6ebc23e7e7a48e067eb024dba64ec8.png), and [Dark Plus](https://buzz.block.builderlab.xyz/media/290fee08924f37d064abc687ecf3e9526ab05b87e8e56d610f23048949793dbe.png). The screenshot harness was checked against the shipped painter at this exact head: all **2,541 draw calls** matched on color, glyph, x, and y; four deliberate divergence controls fired. ## Verification at `98ebc8f9048bd5f0ceb7e843b67874d642f0b7fd` - desktop tests: **3,946 / 3,946** - TypeScript: clean - checks: pass (two pre-existing informational `useTemplate` notices only) - integration/e2e: PASS (independent exact-SHA lane; artifacts recorded in the originating Buzz thread) - artifact/dead-path sweep: clean - redteam G1–G7: PASS - all six named banner emitter-deletion mutants die - independent handwritten five-row full-wordmark fixture kills Quinn's seven-mutant battery, including a one-pixel glyph change - real `112 × 46` canvas-rect dismissal tests separately cover active non-empty, active empty, and inactive non-empty output - layer-drop and zero-draw painter mutants die; z-order and pointer-events verified - CI's `tsc && vite build` includes all three banner modules - performance at DPR 2 (worst-case measured envelope): - one-time content paint: **~0.7–0.8 ms**, paid only when the banner is built or its palette changes - busy compositor, CSS `1277 × 697`, backing `2554 × 1394`: **470–497 µs/frame** for the full banner (**2.82–2.98%** of a 60 Hz frame) - busy compositor, CSS `1920 × 1080`, backing `3840 × 2160`: **1,139–1,212 µs/frame** (**6.83–7.27%**) - empty, one-glyph, and full-banner controls converge: compositor cost follows backing-layer area and DPR rather than painted-cell count - in the actual idle welcome state, cost is below both vsync-clamped rigs' resolution; it is not claimed as zero - **Pane cross-rig spread: resolved at matched loop rate.** Two independent rigs initially differed 2.3× (58–68 vs 136 µs/Mpx of backing store; pane, CSS 1277×697 / backing 2554×1394, DPR 2). The cause of *that* spread is rAF loop rate: the higher figure came from a free-running loop at ~1600fps. Throttled to ~200–236fps, both rigs read 58–68 µs/Mpx (1.25–1.44% of a 60Hz frame). The busy-composite figures quoted above remain the **unthrottled worst case** and are conservative by ~2.3× at the pane. Not established: the mechanism and sign of free-running distortion (one rig under-charges ~15%, the other over-charges 2.3×), and the 1080p figure has not been re-measured throttled. - the layer paints only on generation/theme/resize and dismisses on first non-empty active-session output, so the measurable busy cost is a short-lived worst case rather than a persistent PTY paint-path tax ## Follow-ups in this PR These are intentionally subsequent commits after the certified static-banner head, not claims about `98ebc8f90`: 1. close the compositor metrology: remeasure the 1080p point throttled and characterize the opposite-sign free-running rAF distortion, with each measurement regime stated 2. add Tyler's animated honeycomb color waves, gated by `prefers-reduced-motion`, a full 62-theme phase-sweep contrast check, and DPR-2 per-tick performance certification 3. land the already-proven mounted theme-switch regression probe from `RESEARCH/BUZZ_TERM_G3A_PROBE/` 4. bound the slow/hang-shaped G1-c mutant `waitFor` 5. optionally trim the generator to its ink bounding box, reducing the minimum viewport from 70 to 62 columns --------- Signed-off-by: tlongwell-block <109685178+tlongwell-block@users.noreply.github.com> Signed-off-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz> Signed-off-by: npub1jh9wn95s0472h86ahapupaf7m6kx4v9sx2n0atj2hltcfer8k06s5n3pyf <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@buzz.block.builderlab.xyz> Signed-off-by: npub17jjz49l9jjmhhk7cac63j8yt9z555n9cw8vk7v5jz4vzw4ppld5qgj57cc <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: Dawn (sprout agent) <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@buzz.block.builderlab.xyz> Co-authored-by: npub1cc3ha7z055mu0rwwu7806t2wt8mj3pvu0uv5mfp2c50dahaqhczshdalg6 <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@buzz.block.builderlab.xyz> Co-authored-by: npub17jjz49l9jjmhhk7cac63j8yt9z555n9cw8vk7v5jz4vzw4ppld5qgj57cc <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@buzz.block.builderlab.xyz> Co-authored-by: npub1jh9wn95s0472h86ahapupaf7m6kx4v9sx2n0atj2hltcfer8k06s5n3pyf <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@buzz.block.builderlab.xyz> Co-authored-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz> Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> |
||
|
klopez4212
|
b29c8cdaa4 |
feat(desktop): redesign the Huddle experience (#4281)
## Summary - open Huddles in a focused companion window with a clean handoff back to the in-app drawer and backing channel - redesign the participant film strip, sidebar control, transcript surface, and themed shell treatment - preserve microphone and device control across windows, start agent voice on the first reply, and show agent speaking activity in the film strip - give each agent a distinct session voice, beginning with the configured default, plus compact per-agent text-to-speech and voice controls - enroll only agents explicitly mentioned or deliberately added through an agent panel into the live Huddle roster - keep temporary Huddle channels out of the sidebar unless the user explicitly brings one into the main app - remove Huddle-only avatar policy badges and filter short silence or noise segments before speech-to-text posts ## Why The previous flow exposed the temporary channel as product UI, obscured who was present or speaking, and split transcript and audio state between the main and companion windows. This keeps backing channels as implementation details unless a user explicitly brings a Huddle into the app, while sharing the live conversation and audio lifecycle across both surfaces. Agent participants now join only after an explicit invitation, distinct voices make multi-agent Huddles easier to follow, and short microphone noise no longer becomes stray transcript messages. ## Validation - `pnpm check` - `pnpm build:e2e` - `pnpm exec playwright test tests/e2e/huddle-transcription.spec.ts --project=smoke` (13 passed) - Huddle sidebar visibility unit coverage (4 passed) - focused managed-agent and persona-mention E2E coverage (2 passed) - `pnpm test` (3,910 passed) - `cargo clippy --manifest-path desktop/src-tauri/Cargo.toml --all-targets -- -D warnings` - `cargo test --manifest-path desktop/src-tauri/Cargo.toml` (2,093 passed, 14 ignored; 3 diagnostics passed) --------- Signed-off-by: kenny lopez <klopez4212@gmail.com> Signed-off-by: Wes <wesbillman@users.noreply.github.com> Co-authored-by: Wes <wesbillman@users.noreply.github.com> Co-authored-by: Carl <c7ebe626f000404285d3686e1dc74cc07cc60a9754a150041ba132e14bd3e2ec@buzz.block.builderlab.xyz> |
