fix(gateway,supplier): demo sessions stick to their supplier; free pinned suppliers

The demo endpoint lost its session on nearly every second message: affinity
required clients to re-echo the assistant reply byte-identically, so clients
that strip <think> blocks or reshape whitespace (Open WebUI, litellm) missed
every turn, opened a fresh escrow session on the next free single-slot
supplier, and left the old one pinned. Open WebUI's background title/tag-gen
requests each pinned another supplier — one message could exhaust all three
kimi replicas ("no available chat supplier").

- affinity: lenient assistant-content matching (think-strip + whitespace
  collapse); user/system/tool messages and tool_calls stay byte-strict;
  strict prefixes outrank lenient ones. The mirror is never rewritten with
  client text, so receipt hashes stay valid.
- demoChat: trim trailing assistant/system messages (regenerate semantics)
  instead of 400 "the last message must have role user or tool".
- supplier: roll the transcript back when the LLM call fails so retries
  don't duplicate the delta (ChatSessionStore.truncateTranscript).
- demoChat: LRU-evict a demo session when no supplier is available (supplier
  /v1/chat/end frees the slot, sweeper Accepts the escrow) and retry the
  open with selectCandidates ignoreStatusFor to bypass the ~20s-stale
  indexer status poll. Max 2 evictions per request.
- sweeper: idle janitor closes demo sessions unused for DEMO_SESSION_IDLE_MS
  (default 3 min, below the supplier's 5 min CHAT_IDLE_TIMEOUT_MS) so
  abandoned one-off sessions release their suppliers quickly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
david
2026-07-09 02:29:10 +02:00
parent 3d2b4bb203
commit 9a4f9f0699
16 changed files with 636 additions and 43 deletions

View File

@@ -62,7 +62,7 @@ export function normalizeChatMessage(raw: unknown): ChatMessage | null {
return msg;
}
function toolCallsEqual(a?: ToolCall[], b?: ToolCall[]): boolean {
export function toolCallsEqual(a?: ToolCall[], b?: ToolCall[]): boolean {
const la = a ?? [];
const lb = b ?? [];
if (la.length !== lb.length) return false;