mirror of
https://github.com/block/buzz.git
synced 2026-10-31 22:41:12 +00:00
## Overview
**Category:** feat (env-gated experiment + one exact always-on
optimization)
**Problem:** Speech-end -> first TTS audio through the desktop huddle
pipeline measures **924–1087 ms** on an M4 Max with a 0 ms LLM leg.
Voice turns feel sluggish no matter how fast the agent replies. Baseline
breakdown: ~300 ms hardcoded VAD silence flush + ~150–250 ms Parakeet
decode + ~380–550 ms TTS synthesis before the first player append.
**Outcome:** With all levers enabled, e2e time-to-first-audio measures
**347–384 ms** (307–357 ms on a longer utterance) on the same hardware,
harness, and production pipelines. Defaults preserve production behavior
everywhere except one deterministic, bit-exact cache win.
## What's in here
### Levers (all default-off, env-gated)
| Lever | Env | Effect (measured medians, short utterance) |
|---|---|---|
| Speculative Parakeet decode | `BUZZ_STT_SPECULATIVE=1` | STT leg ->
~max(flush, decode) |
| Streaming TTS synthesis | `BUZZ_TTS_STREAMING=1`,
`BUZZ_TTS_EMIT_FRAMES` | first audio 380–550 -> 211–320 ms (emit=12,
bit-exact) |
| ONNX intra-op threads | `BUZZ_STT_THREADS`, `BUZZ_TTS_THREADS` | TTS
first audio 211–320 -> 129–180 ms (4 threads) |
- **Speculative decode** starts the Parakeet decode at the *first*
silent VAD frame, overlapping it with the flush window. Resumed speech
invalidates the result (voiced-frame-count check); held silence emits it
instantly at the flush boundary.
- **Streaming TTS**: new `synth_chunk_streaming` (buzz-voice)
interleaves the Flow LM frame loop with incremental *stateful* Mimi
decoding, emitting PCM deltas to the player via the existing
`PlaybackChunkAudio` decoration. At `emit_frames=12` (the decoder's
native chunk) streamed audio is **bit-identical** to the batch path —
verified by the ignored test
`incremental_stateful_decode_matches_batch_decode` (max|diff|=0).
Smaller deltas are faster but diverge (~23 dB SNR; decoder intra-chunk
lookahead), hence the default of 12.
> **Removed after live testing:** the `BUZZ_STT_FLUSH_MS` flush-window
override. Lowering the silence window below natural mid-sentence pauses
(the fast-path recipe said 150 ms) split single spoken sentences into
multiple messages and confused the listening agents. The window is a
turn-taking quality knob, not a latency lever — it is now fixed at the
production 300 ms value.
### Push-to-talk grouping fix (always-on)
A held push-to-talk shortcut is an explicit "I am not done talking"
signal, so silence never ends the utterance while it is held — even when
the microphone is also manually open. The utterance flushes on shortcut
release (existing transmit-edge flush); a manually open mic with the
shortcut up keeps normal VAD pause flushing. Gate is the pure
`vad_flush_allowed` function with a unit-test truth table.
### Always-on (exact): voice-conditioning cache
Phase profiling (`BUZZ_TTS_PHASE_LOG=1`) showed a fixed ~160 ms
`condition_voice` Flow-LM pass on *every* chunk, re-deriving the same
post-conditioning state for the same reference voice. The state is now
snapshotted after first computation and restored per chunk (dtype-tagged
tensor copies, keyed identically to the existing `cached_voice`).
Deterministic — same tensors in, same tensors out. The default path's
TTS leg drops from 380–550 ms to 225–355 ms with no configuration.
### Bench harness
`huddle::latency_bench` (`#[cfg(test)]` + `#[ignore]`) drives the real
`SttPipeline` and `TtsPipeline`, feeding a 48 kHz WAV in real-time 100
ms batches (AudioWorklet cadence) with a configurable fake LLM in place
of the relay leg, timing speech-end -> transcript -> speak() -> first
accepted player append.
```
BUZZ_STT_SPECULATIVE=1 BUZZ_TTS_STREAMING=1 \
BUZZ_TTS_THREADS=4 BUZZ_STT_THREADS=2 \
BUZZ_BENCH_WAV=<48k f32 mono wav> \
cargo test --release -p buzz-desktop --lib huddle::latency_bench -- --ignored --nocapture
```
## Tradeoffs to weigh before promoting any lever to a default
- **Speculative decode**: the speculative buffer has ~1 silent tail
frame vs ~19; observed one CTC wobble ("fail" vs "failed") in 24 turns.
Mitigation if productionized: zero-pad the speculative buffer to match
the flush-path shape.
- **Threads**: defaults stay 1 pending the min-spec (4-core Intel) A/B
flagged in the existing `STT_NUM_THREADS` comment.
- **Streaming at emit<12** is NOT the same waveform — don't ship below
12 without an ear pass.
## Validation
- Full desktop lib suite: **2408 passed / 0 failed** at this head
(`18fab2e1c`).
- buzz-voice suite green; bit-exactness test passes against the
production batch decode.
- Defaults-only bench rerun stays in the baseline family everywhere
except the exact conditioning-cache win (stt 525–532, tts 225–355).
- `cargo clippy --workspace --all-targets -- -D warnings` + fmt clean
(pre-push hook battery green).
Measurement notes with per-lever logs: Eva's workspace,
`RESEARCH/HUDDLE_E2E_LATENCY_OPTIMIZATION_2026_08_12.md` +
`RESEARCH/HUDDLE_E2E_STT_FAKELLM_TTS_BASELINE_2026_08_12.md`.
## Suggested promotion order
1. Conditioning cache (in this PR, always-on, exact).
2. Streaming TTS at emit=12: bit-exact audio, biggest UX win — needs the
env-gate removed + barge-in soak + an ear pass on a real huddle.
3. Speculative decode with silence padding: near-free ~100–150 ms.
4. Threads: after min-spec A/B.
---------
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Signed-off-by: Max <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz>
Signed-off-by: Mari <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@buzz.block.builderlab.xyz>
Signed-off-by: tlongwell-block <tlongwell@block.xyz>
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Co-authored-by: Max <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz>
Co-authored-by: Mari <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@buzz.block.builderlab.xyz>
Co-authored-by: tlongwell-block <tlongwell@block.xyz>
217 lines
7.4 KiB
Rust
217 lines
7.4 KiB
Rust
//! Small Huddle controls that mutate an active session.
|
|
|
|
use std::sync::{atomic::Ordering, Arc};
|
|
|
|
use tauri::State;
|
|
use uuid::Uuid;
|
|
|
|
use crate::{app_state::AppState, events, relay::submit_event};
|
|
|
|
use super::pipeline::start_auto_enabled_transcription;
|
|
use super::relay_api::MAX_HUDDLE_AGENTS;
|
|
use super::{agents, relay_api::validate_pubkey_hex, HuddlePhase};
|
|
|
|
/// Update the clickable microphone control independently from the PTT shortcut.
|
|
#[tauri::command]
|
|
pub fn set_huddle_manual_mic_unmuted(
|
|
enabled: bool,
|
|
state: State<'_, AppState>,
|
|
) -> Result<(), String> {
|
|
let huddle = state.huddle()?;
|
|
if !matches!(huddle.phase, HuddlePhase::Connected | HuddlePhase::Active) {
|
|
return Err("no active huddle".to_string());
|
|
}
|
|
huddle.manual_mic_unmuted.store(enabled, Ordering::Release);
|
|
Ok(())
|
|
}
|
|
|
|
/// Immediately interrupt the agent utterance that is currently speaking.
|
|
#[tauri::command]
|
|
pub fn interrupt_huddle_speech(
|
|
agent_pubkey: String,
|
|
state: State<'_, AppState>,
|
|
) -> Result<(), String> {
|
|
validate_pubkey_hex(&agent_pubkey)?;
|
|
let tts_pipeline = {
|
|
let huddle = state.huddle()?;
|
|
if !matches!(huddle.phase, HuddlePhase::Connected | HuddlePhase::Active) {
|
|
return Err("no active huddle".to_string());
|
|
}
|
|
huddle.tts_pipeline.as_ref().map(Arc::clone)
|
|
};
|
|
if let Some(tts_pipeline) = tts_pipeline {
|
|
tts_pipeline.cancel_active_speaker(&agent_pubkey);
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
/// Remove an agent from the active huddle without removing its parent-channel
|
|
/// membership. Keeping the parent membership intact means it remains available
|
|
/// to rejoin this huddle from the agent picker.
|
|
#[tauri::command]
|
|
pub async fn remove_agent_from_huddle(
|
|
agent_pubkey: String,
|
|
state: State<'_, AppState>,
|
|
) -> Result<(), String> {
|
|
validate_pubkey_hex(&agent_pubkey)?;
|
|
|
|
let (ephemeral_channel_id, huddle_generation) = {
|
|
let huddle = state.huddle()?;
|
|
if !matches!(huddle.phase, HuddlePhase::Connected | HuddlePhase::Active) {
|
|
return Err("no active huddle".to_string());
|
|
}
|
|
|
|
let is_huddle_agent = huddle
|
|
.agent_pubkeys
|
|
.lock()
|
|
.unwrap_or_else(|error| error.into_inner())
|
|
.iter()
|
|
.any(|pubkey| pubkey.eq_ignore_ascii_case(&agent_pubkey));
|
|
if !is_huddle_agent {
|
|
return Err("agent is not in this huddle".to_string());
|
|
}
|
|
|
|
(
|
|
huddle
|
|
.ephemeral_channel_id
|
|
.clone()
|
|
.ok_or("no ephemeral channel")?,
|
|
huddle.huddle_generation,
|
|
)
|
|
};
|
|
|
|
let ephemeral_channel_uuid =
|
|
Uuid::parse_str(&ephemeral_channel_id).map_err(|error| error.to_string())?;
|
|
submit_event(
|
|
events::build_remove_member(ephemeral_channel_uuid, &agent_pubkey)?,
|
|
&state,
|
|
)
|
|
.await?;
|
|
|
|
let (roster_changed, tts_pipeline) = {
|
|
let mut huddle = state.huddle()?;
|
|
if !huddle.is_current_huddle(&ephemeral_channel_id, huddle_generation) {
|
|
(false, None)
|
|
} else {
|
|
let mut agent_pubkeys = huddle
|
|
.agent_pubkeys
|
|
.lock()
|
|
.unwrap_or_else(|error| error.into_inner());
|
|
let initial_count = agent_pubkeys.len();
|
|
agent_pubkeys.retain(|pubkey| !pubkey.eq_ignore_ascii_case(&agent_pubkey));
|
|
let changed = agent_pubkeys.len() != initial_count;
|
|
drop(agent_pubkeys);
|
|
|
|
if changed {
|
|
huddle
|
|
.participants
|
|
.retain(|pubkey| !pubkey.eq_ignore_ascii_case(&agent_pubkey));
|
|
if let Some(settings_pubkey) = huddle
|
|
.agent_voice_settings
|
|
.keys()
|
|
.find(|pubkey| pubkey.eq_ignore_ascii_case(&agent_pubkey))
|
|
.cloned()
|
|
{
|
|
huddle.agent_voice_settings.remove(&settings_pubkey);
|
|
}
|
|
}
|
|
let tts_pipeline = changed
|
|
.then_some(huddle.tts_pipeline.as_ref())
|
|
.flatten()
|
|
.map(Arc::clone);
|
|
(changed, tts_pipeline)
|
|
}
|
|
};
|
|
|
|
if let Some(tts_pipeline) = tts_pipeline {
|
|
tts_pipeline.cancel_speaker(&agent_pubkey);
|
|
}
|
|
if roster_changed {
|
|
state.emit_huddle_state_changed();
|
|
}
|
|
|
|
Ok(())
|
|
}
|
|
|
|
/// Add an agent to the active huddle.
|
|
///
|
|
/// Steps:
|
|
/// 1. Validates the huddle is in the Connected or Active phase.
|
|
/// 2. Adds the agent to both the ephemeral and parent channels (kind:9000).
|
|
/// 3. Only appends the agent pubkey to `agent_pubkeys` if the ephemeral add
|
|
/// succeeded — failed adds (policy rejection) are NOT p-tagged.
|
|
///
|
|
/// Returns a structured `AgentAddResult` so the frontend can surface
|
|
/// parent-channel errors without treating them as hard failures.
|
|
///
|
|
/// The running ACP process for this agent auto-subscribes when it receives
|
|
/// the kind:9000 membership notification — no separate process spawn needed.
|
|
#[tauri::command]
|
|
pub async fn add_agent_to_huddle(
|
|
agent_pubkey: String,
|
|
state: State<'_, AppState>,
|
|
) -> Result<agents::AgentAddResult, String> {
|
|
validate_pubkey_hex(&agent_pubkey)?;
|
|
|
|
let (eph_id, parent_id, huddle_generation) = {
|
|
let hs = state.huddle()?;
|
|
if !matches!(hs.phase, HuddlePhase::Connected | HuddlePhase::Active) {
|
|
return Err("no active huddle".to_string());
|
|
}
|
|
|
|
// Enforce agent cap on incremental adds too.
|
|
let current_agent_count = hs
|
|
.agent_pubkeys
|
|
.lock()
|
|
.unwrap_or_else(|e| e.into_inner())
|
|
.len();
|
|
if current_agent_count >= MAX_HUDDLE_AGENTS {
|
|
return Err(format!(
|
|
"agent limit reached: {} (max {})",
|
|
current_agent_count, MAX_HUDDLE_AGENTS
|
|
));
|
|
}
|
|
|
|
let eph = hs
|
|
.ephemeral_channel_id
|
|
.clone()
|
|
.ok_or("no ephemeral channel")?;
|
|
let parent = hs.parent_channel_id.clone().ok_or("no parent channel")?;
|
|
(eph, parent, hs.huddle_generation)
|
|
};
|
|
|
|
let eph_uuid = Uuid::parse_str(&eph_id).map_err(|e| e.to_string())?;
|
|
let parent_uuid = Uuid::parse_str(&parent_id).map_err(|e| e.to_string())?;
|
|
|
|
// Returns Err only if the ephemeral add fails — parent failure is in the result.
|
|
let result = agents::add_agent_to_huddle(eph_uuid, parent_uuid, &agent_pubkey, &state).await?;
|
|
|
|
// Ephemeral add succeeded — register it only if this is still the huddle
|
|
// that initiated the relay operation.
|
|
let transcription_auto_enabled = {
|
|
let mut hs = state.huddle()?;
|
|
if !hs.is_current_huddle(&eph_id, huddle_generation) {
|
|
return Ok(result);
|
|
}
|
|
let mut pubkeys = hs.agent_pubkeys.lock().unwrap_or_else(|e| e.into_inner());
|
|
if !pubkeys.contains(&agent_pubkey) {
|
|
pubkeys.push(agent_pubkey.clone());
|
|
}
|
|
drop(pubkeys);
|
|
if !hs.participants.contains(&agent_pubkey) {
|
|
hs.participants.push(agent_pubkey.clone());
|
|
}
|
|
hs.maybe_auto_enable_transcription_for_agents()
|
|
};
|
|
|
|
// No guidelines re-post needed — the agent sees the original kind:48106
|
|
// guidelines via EOSE replay when it subscribes to the ephemeral channel.
|
|
if transcription_auto_enabled {
|
|
start_auto_enabled_transcription(&state, &eph_id).await;
|
|
} else {
|
|
state.emit_huddle_state_changed();
|
|
}
|
|
|
|
Ok(result)
|
|
}
|