2 Commits

Author	SHA1	Message	Date
Tyler	068a83b097	feat(huddle): cut voice-turn time-to-first-audio from ~1.0 s to ~0.35 s (env-gated latency levers) (#5671) ... ## Overview **Category:** feat (env-gated experiment + one exact always-on optimization) **Problem:** Speech-end -> first TTS audio through the desktop huddle pipeline measures **924–1087 ms** on an M4 Max with a 0 ms LLM leg. Voice turns feel sluggish no matter how fast the agent replies. Baseline breakdown: ~300 ms hardcoded VAD silence flush + ~150–250 ms Parakeet decode + ~380–550 ms TTS synthesis before the first player append. **Outcome:** With all levers enabled, e2e time-to-first-audio measures **347–384 ms** (307–357 ms on a longer utterance) on the same hardware, harness, and production pipelines. Defaults preserve production behavior everywhere except one deterministic, bit-exact cache win. ## What's in here ### Levers (all default-off, env-gated) | Lever | Env | Effect (measured medians, short utterance) | |---|---|---| | Speculative Parakeet decode | `BUZZ_STT_SPECULATIVE=1` | STT leg -> ~max(flush, decode) | | Streaming TTS synthesis | `BUZZ_TTS_STREAMING=1`, `BUZZ_TTS_EMIT_FRAMES` | first audio 380–550 -> 211–320 ms (emit=12, bit-exact) | | ONNX intra-op threads | `BUZZ_STT_THREADS`, `BUZZ_TTS_THREADS` | TTS first audio 211–320 -> 129–180 ms (4 threads) | - **Speculative decode** starts the Parakeet decode at the *first* silent VAD frame, overlapping it with the flush window. Resumed speech invalidates the result (voiced-frame-count check); held silence emits it instantly at the flush boundary. - **Streaming TTS**: new `synth_chunk_streaming` (buzz-voice) interleaves the Flow LM frame loop with incremental *stateful* Mimi decoding, emitting PCM deltas to the player via the existing `PlaybackChunkAudio` decoration. At `emit_frames=12` (the decoder's native chunk) streamed audio is **bit-identical** to the batch path — verified by the ignored test `incremental_stateful_decode_matches_batch_decode` (max|diff|=0). Smaller deltas are faster but diverge (~23 dB SNR; decoder intra-chunk lookahead), hence the default of 12. > **Removed after live testing:** the `BUZZ_STT_FLUSH_MS` flush-window override. Lowering the silence window below natural mid-sentence pauses (the fast-path recipe said 150 ms) split single spoken sentences into multiple messages and confused the listening agents. The window is a turn-taking quality knob, not a latency lever — it is now fixed at the production 300 ms value. ### Push-to-talk grouping fix (always-on) A held push-to-talk shortcut is an explicit "I am not done talking" signal, so silence never ends the utterance while it is held — even when the microphone is also manually open. The utterance flushes on shortcut release (existing transmit-edge flush); a manually open mic with the shortcut up keeps normal VAD pause flushing. Gate is the pure `vad_flush_allowed` function with a unit-test truth table. ### Always-on (exact): voice-conditioning cache Phase profiling (`BUZZ_TTS_PHASE_LOG=1`) showed a fixed ~160 ms `condition_voice` Flow-LM pass on *every* chunk, re-deriving the same post-conditioning state for the same reference voice. The state is now snapshotted after first computation and restored per chunk (dtype-tagged tensor copies, keyed identically to the existing `cached_voice`). Deterministic — same tensors in, same tensors out. The default path's TTS leg drops from 380–550 ms to 225–355 ms with no configuration. ### Bench harness `huddle::latency_bench` (`#[cfg(test)]` + `#[ignore]`) drives the real `SttPipeline` and `TtsPipeline`, feeding a 48 kHz WAV in real-time 100 ms batches (AudioWorklet cadence) with a configurable fake LLM in place of the relay leg, timing speech-end -> transcript -> speak() -> first accepted player append. ``` BUZZ_STT_SPECULATIVE=1 BUZZ_TTS_STREAMING=1 \ BUZZ_TTS_THREADS=4 BUZZ_STT_THREADS=2 \ BUZZ_BENCH_WAV=<48k f32 mono wav> \ cargo test --release -p buzz-desktop --lib huddle::latency_bench -- --ignored --nocapture ``` ## Tradeoffs to weigh before promoting any lever to a default - **Speculative decode**: the speculative buffer has ~1 silent tail frame vs ~19; observed one CTC wobble ("fail" vs "failed") in 24 turns. Mitigation if productionized: zero-pad the speculative buffer to match the flush-path shape. - **Threads**: defaults stay 1 pending the min-spec (4-core Intel) A/B flagged in the existing `STT_NUM_THREADS` comment. - **Streaming at emit<12** is NOT the same waveform — don't ship below 12 without an ear pass. ## Validation - Full desktop lib suite: **2408 passed / 0 failed** at this head (`18fab2e1c`). - buzz-voice suite green; bit-exactness test passes against the production batch decode. - Defaults-only bench rerun stays in the baseline family everywhere except the exact conditioning-cache win (stt 525–532, tts 225–355). - `cargo clippy --workspace --all-targets -- -D warnings` + fmt clean (pre-push hook battery green). Measurement notes with per-lever logs: Eva's workspace, `RESEARCH/HUDDLE_E2E_LATENCY_OPTIMIZATION_2026_08_12.md` + `RESEARCH/HUDDLE_E2E_STT_FAKELLM_TTS_BASELINE_2026_08_12.md`. ## Suggested promotion order 1. Conditioning cache (in this PR, always-on, exact). 2. Streaming TTS at emit=12: bit-exact audio, biggest UX win — needs the env-gate removed + barge-in soak + an ear pass on a real huddle. 3. Speculative decode with silence padding: near-free ~100–150 ms. 4. Threads: after min-spec A/B. --------- Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz> Signed-off-by: Max <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz> Signed-off-by: Mari <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@buzz.block.builderlab.xyz> Signed-off-by: tlongwell-block <tlongwell@block.xyz> Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz> Co-authored-by: Max <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz> Co-authored-by: Mari <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@buzz.block.builderlab.xyz> Co-authored-by: tlongwell-block <tlongwell@block.xyz>	2026-08-13 19:42:54 -04:00
klopez4212	ce3cf3cd25	Polish Huddle voice controls (#4694) ... ## Summary - add a visible Stop control for interrupting agent speech - make push-to-talk available by default while preserving manual mute controls - refine agent management, muted audio states, drawer layering, and return navigation - suppress duplicate notification sounds for Huddle messages ## Why Huddles could trap users behind long agent speech, hide useful agent controls, and leave temporary Huddle state visible after the call. The drawer also regressed when the terminal substrate began painting behind the rounded app surface. ## Validation - `just desktop-ci` - focused Huddle Playwright coverage for the drawer, speech interruption, agent picker, and leave navigation --------- Signed-off-by: kenny lopez <klopez4212@gmail.com> Signed-off-by: Wes <wesbillman@users.noreply.github.com> Co-authored-by: Wes <wesbillman@users.noreply.github.com> Co-authored-by: Carl <c7ebe626f000404285d3686e1dc74cc07cc60a9754a150041ba132e14bd3e2ec@buzz.block.builderlab.xyz>	2026-08-04 13:07:50 -07:00