fix(gateway,supplier): demo sessions stick to their supplier; free pinned suppliers

The demo endpoint lost its session on nearly every second message: affinity
required clients to re-echo the assistant reply byte-identically, so clients
that strip <think> blocks or reshape whitespace (Open WebUI, litellm) missed
every turn, opened a fresh escrow session on the next free single-slot
supplier, and left the old one pinned. Open WebUI's background title/tag-gen
requests each pinned another supplier — one message could exhaust all three
kimi replicas ("no available chat supplier").

- affinity: lenient assistant-content matching (think-strip + whitespace
  collapse); user/system/tool messages and tool_calls stay byte-strict;
  strict prefixes outrank lenient ones. The mirror is never rewritten with
  client text, so receipt hashes stay valid.
- demoChat: trim trailing assistant/system messages (regenerate semantics)
  instead of 400 "the last message must have role user or tool".
- supplier: roll the transcript back when the LLM call fails so retries
  don't duplicate the delta (ChatSessionStore.truncateTranscript).
- demoChat: LRU-evict a demo session when no supplier is available (supplier
  /v1/chat/end frees the slot, sweeper Accepts the escrow) and retry the
  open with selectCandidates ignoreStatusFor to bypass the ~20s-stale
  indexer status poll. Max 2 evictions per request.
- sweeper: idle janitor closes demo sessions unused for DEMO_SESSION_IDLE_MS
  (default 3 min, below the supplier's 5 min CHAT_IDLE_TIMEOUT_MS) so
  abandoned one-off sessions release their suppliers quickly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
david
2026-07-09 02:29:10 +02:00
parent 3d2b4bb203
commit 9a4f9f0699
16 changed files with 636 additions and 43 deletions

View File

@@ -30,5 +30,6 @@ GATEWAY_DB_DIR=./data/gateway
# KEY_RATE_MAX=60 # requests per API key per window
# KEY_RATE_WINDOW_MS=60000
# SWEEPER_INTERVAL_MS=60000
# DEMO_SESSION_IDLE_MS=180000 # close demo chat sessions idle this long (frees the supplier slot)
# WALLET_HEALTH_INTERVAL_MS=600000
# SDK_REGISTRY_MAX=500 # max cached per-key Marketplace instances