Files
buzz/desktop/src/app/AppHuddleShell.tsx
Tyler 068a83b097 feat(huddle): cut voice-turn time-to-first-audio from ~1.0 s to ~0.35 s (env-gated latency levers) (#5671)
## Overview

**Category:** feat (env-gated experiment + one exact always-on
optimization)
**Problem:** Speech-end -> first TTS audio through the desktop huddle
pipeline measures **924–1087 ms** on an M4 Max with a 0 ms LLM leg.
Voice turns feel sluggish no matter how fast the agent replies. Baseline
breakdown: ~300 ms hardcoded VAD silence flush + ~150–250 ms Parakeet
decode + ~380–550 ms TTS synthesis before the first player append.
**Outcome:** With all levers enabled, e2e time-to-first-audio measures
**347–384 ms** (307–357 ms on a longer utterance) on the same hardware,
harness, and production pipelines. Defaults preserve production behavior
everywhere except one deterministic, bit-exact cache win.

## What's in here

### Levers (all default-off, env-gated)

| Lever | Env | Effect (measured medians, short utterance) |
|---|---|---|
| Speculative Parakeet decode | `BUZZ_STT_SPECULATIVE=1` | STT leg ->
~max(flush, decode) |
| Streaming TTS synthesis | `BUZZ_TTS_STREAMING=1`,
`BUZZ_TTS_EMIT_FRAMES` | first audio 380–550 -> 211–320 ms (emit=12,
bit-exact) |
| ONNX intra-op threads | `BUZZ_STT_THREADS`, `BUZZ_TTS_THREADS` | TTS
first audio 211–320 -> 129–180 ms (4 threads) |

- **Speculative decode** starts the Parakeet decode at the *first*
silent VAD frame, overlapping it with the flush window. Resumed speech
invalidates the result (voiced-frame-count check); held silence emits it
instantly at the flush boundary.
- **Streaming TTS**: new `synth_chunk_streaming` (buzz-voice)
interleaves the Flow LM frame loop with incremental *stateful* Mimi
decoding, emitting PCM deltas to the player via the existing
`PlaybackChunkAudio` decoration. At `emit_frames=12` (the decoder's
native chunk) streamed audio is **bit-identical** to the batch path —
verified by the ignored test
`incremental_stateful_decode_matches_batch_decode` (max|diff|=0).
Smaller deltas are faster but diverge (~23 dB SNR; decoder intra-chunk
lookahead), hence the default of 12.

> **Removed after live testing:** the `BUZZ_STT_FLUSH_MS` flush-window
override. Lowering the silence window below natural mid-sentence pauses
(the fast-path recipe said 150 ms) split single spoken sentences into
multiple messages and confused the listening agents. The window is a
turn-taking quality knob, not a latency lever — it is now fixed at the
production 300 ms value.

### Push-to-talk grouping fix (always-on)

A held push-to-talk shortcut is an explicit "I am not done talking"
signal, so silence never ends the utterance while it is held — even when
the microphone is also manually open. The utterance flushes on shortcut
release (existing transmit-edge flush); a manually open mic with the
shortcut up keeps normal VAD pause flushing. Gate is the pure
`vad_flush_allowed` function with a unit-test truth table.

### Always-on (exact): voice-conditioning cache

Phase profiling (`BUZZ_TTS_PHASE_LOG=1`) showed a fixed ~160 ms
`condition_voice` Flow-LM pass on *every* chunk, re-deriving the same
post-conditioning state for the same reference voice. The state is now
snapshotted after first computation and restored per chunk (dtype-tagged
tensor copies, keyed identically to the existing `cached_voice`).
Deterministic — same tensors in, same tensors out. The default path's
TTS leg drops from 380–550 ms to 225–355 ms with no configuration.

### Bench harness

`huddle::latency_bench` (`#[cfg(test)]` + `#[ignore]`) drives the real
`SttPipeline` and `TtsPipeline`, feeding a 48 kHz WAV in real-time 100
ms batches (AudioWorklet cadence) with a configurable fake LLM in place
of the relay leg, timing speech-end -> transcript -> speak() -> first
accepted player append.

```
BUZZ_STT_SPECULATIVE=1 BUZZ_TTS_STREAMING=1 \
BUZZ_TTS_THREADS=4 BUZZ_STT_THREADS=2 \
BUZZ_BENCH_WAV=<48k f32 mono wav> \
cargo test --release -p buzz-desktop --lib huddle::latency_bench -- --ignored --nocapture
```

## Tradeoffs to weigh before promoting any lever to a default

- **Speculative decode**: the speculative buffer has ~1 silent tail
frame vs ~19; observed one CTC wobble ("fail" vs "failed") in 24 turns.
Mitigation if productionized: zero-pad the speculative buffer to match
the flush-path shape.
- **Threads**: defaults stay 1 pending the min-spec (4-core Intel) A/B
flagged in the existing `STT_NUM_THREADS` comment.
- **Streaming at emit<12** is NOT the same waveform — don't ship below
12 without an ear pass.

## Validation

- Full desktop lib suite: **2408 passed / 0 failed** at this head
(`18fab2e1c`).
- buzz-voice suite green; bit-exactness test passes against the
production batch decode.
- Defaults-only bench rerun stays in the baseline family everywhere
except the exact conditioning-cache win (stt 525–532, tts 225–355).
- `cargo clippy --workspace --all-targets -- -D warnings` + fmt clean
(pre-push hook battery green).

Measurement notes with per-lever logs: Eva's workspace,
`RESEARCH/HUDDLE_E2E_LATENCY_OPTIMIZATION_2026_08_12.md` +
`RESEARCH/HUDDLE_E2E_STT_FAKELLM_TTS_BASELINE_2026_08_12.md`.

## Suggested promotion order

1. Conditioning cache (in this PR, always-on, exact).
2. Streaming TTS at emit=12: bit-exact audio, biggest UX win — needs the
env-gate removed + barge-in soak + an ear pass on a real huddle.
3. Speculative decode with silence padding: near-free ~100–150 ms.
4. Threads: after min-spec A/B.

---------

Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Signed-off-by: Max <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz>
Signed-off-by: Mari <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@buzz.block.builderlab.xyz>
Signed-off-by: tlongwell-block <tlongwell@block.xyz>
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Co-authored-by: Max <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz>
Co-authored-by: Mari <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@buzz.block.builderlab.xyz>
Co-authored-by: tlongwell-block <tlongwell@block.xyz>
2026-08-13 19:42:54 -04:00

109 lines
3.5 KiB
TypeScript

import * as React from "react";
import { AppHuddleBar } from "@/app/AppHuddleBar";
import * as BuzzTheme from "@/app/BuzzThemeSurfaces";
import { HuddleProvider, useHuddle } from "@/features/huddle";
import { HUDDLE_SHORTCUT_EVENT } from "@/shared/lib/keyboard-shortcuts";
import { RemindMeLaterProvider } from "@/features/reminders/ui/RemindMeLaterProvider";
import { cn } from "@/shared/lib/cn";
type AppHuddleShellProps = {
children: React.ReactNode;
currentPubkey?: string;
isCompanionOpen: boolean;
isDrawerOpen: boolean;
isRoom: boolean;
onCompanionOpen: () => void;
onHuddleStartPendingChange: (pending: boolean) => void;
onHuddleStarted: (ephemeralChannelId: string) => void | Promise<void>;
onShowHuddleInMainApp: (ephemeralChannelId: string) => void;
onViewHuddleChannel: (ephemeralChannelId: string) => void;
onVisibilityChange: (visible: boolean) => void;
};
type HuddleShortcutHandlerProps = {
children: React.ReactNode;
};
function HuddleShortcutHandler({ children }: HuddleShortcutHandlerProps) {
const { activeEphemeralChannelId, leaveHuddle } = useHuddle();
React.useEffect(() => {
if (!activeEphemeralChannelId) return;
function handleHuddleShortcut() {
void leaveHuddle();
}
window.addEventListener(HUDDLE_SHORTCUT_EVENT, handleHuddleShortcut);
return () =>
window.removeEventListener(HUDDLE_SHORTCUT_EVENT, handleHuddleShortcut);
}, [activeEphemeralChannelId, leaveHuddle]);
return children;
}
export function AppHuddleShell({
children,
currentPubkey,
isCompanionOpen,
isDrawerOpen,
isRoom,
onCompanionOpen,
onHuddleStartPendingChange,
onHuddleStarted,
onShowHuddleInMainApp,
onViewHuddleChannel,
onVisibilityChange,
}: AppHuddleShellProps) {
return (
<HuddleProvider
ownsAudioSession={!isRoom}
onHuddleStartPendingChange={
isRoom ? undefined : onHuddleStartPendingChange
}
onHuddleStarted={isRoom ? undefined : onHuddleStarted}
onShowHuddleInMainApp={isRoom ? undefined : onShowHuddleInMainApp}
onViewHuddleChannel={isRoom ? undefined : onViewHuddleChannel}
>
<HuddleShortcutHandler>
<RemindMeLaterProvider pubkey={currentPubkey}>
<div
className="buzz-huddle-shell relative h-dvh overflow-hidden overscroll-none"
data-huddle-open={isDrawerOpen}
data-huddle-window={isRoom}
>
<div
aria-hidden="true"
className={cn(
"buzz-huddle-drawer-backdrop",
isDrawerOpen && "buzz-huddle-drawer-backdrop-open",
)}
/>
<div
className={cn(
"buzz-huddle-app-surface z-10 flex min-h-0 flex-row overflow-hidden bg-background",
isDrawerOpen &&
(isRoom
? "buzz-huddle-app-surface-room-open"
: "buzz-huddle-app-surface-open"),
)}
>
<BuzzTheme.GradientLayer />
{children}
</div>
{isRoom || !isCompanionOpen ? (
<div className="buzz-huddle-drawer-slot absolute inset-x-0 bottom-0 z-[2] h-(--buzz-huddle-drawer-height)">
<AppHuddleBar
mode={isRoom ? "room" : "main"}
onOpenHuddleWindow={isRoom ? undefined : onCompanionOpen}
onVisibilityChange={onVisibilityChange}
/>
</div>
) : null}
</div>
</RemindMeLaterProvider>
</HuddleShortcutHandler>
</HuddleProvider>
);
}