Two structural fixes for demo concurrency (3 single-slot suppliers thrashed
under Open WebUI's ~3 conversations per message):
Multi-slot suppliers (MAX_CHAT_SESSIONS, default 1):
- SupplierState is now capacity-aware: a Set of active escrow refs with
free-until-full status semantics, ref-scoped release(ref) (also fixes the
latent bug where handler catch-alls released another request's slot), and
a walletMutex serializing every wallet-spending chain op — Claim, Submit,
and wallet-health consolidation queue instead of racing the single
working/collateral UTxO. /status gains active_sessions/max_sessions.
Entry-ticket settle mode (CHAT_SETTLE_MODE=ticket, default full, both sides):
- The buyer's escrow post at session open is the ONLY chain op. The supplier
verifies the Open escrow as an entry ticket but never Claims/Submits
(/v1/chat/start → {status:"ticket"}; /v1/chat/end → receipt-less
{status:"ended"}); no hard-cap timer, so sessions may outlive deliver_by.
Used tickets can't reopen sessions (records retained; 409 session_exists).
- Buyer SDK: startChat/endChat are settle-mode aware, driven by the
supplier's response — mixed-mode rollout windows settle correctly.
- Gateway: ticket closes bill cost 0; the sweeper Reclaims Open tickets
after deliver_by WITHOUT closing live sessions (the conversation and its
affinity mirror survive their reclaimed ticket), gates reclaims on
deliver_by, and keeps an in-process settled-refs memory so terminal
indexer rows aren't retried every sweep forever.
Deployment (kimi-chat trio + gateway): ticket mode, 8 slots per supplier,
idle policy gateway 10 min / supplier 15 min (fixes the inversion where the
supplier idle-closed before the gateway janitor).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The demo endpoint lost its session on nearly every second message: affinity
required clients to re-echo the assistant reply byte-identically, so clients
that strip <think> blocks or reshape whitespace (Open WebUI, litellm) missed
every turn, opened a fresh escrow session on the next free single-slot
supplier, and left the old one pinned. Open WebUI's background title/tag-gen
requests each pinned another supplier — one message could exhaust all three
kimi replicas ("no available chat supplier").
- affinity: lenient assistant-content matching (think-strip + whitespace
collapse); user/system/tool messages and tool_calls stay byte-strict;
strict prefixes outrank lenient ones. The mirror is never rewritten with
client text, so receipt hashes stay valid.
- demoChat: trim trailing assistant/system messages (regenerate semantics)
instead of 400 "the last message must have role user or tool".
- supplier: roll the transcript back when the LLM call fails so retries
don't duplicate the delta (ChatSessionStore.truncateTranscript).
- demoChat: LRU-evict a demo session when no supplier is available (supplier
/v1/chat/end frees the slot, sweeper Accepts the escrow) and retry the
open with selectCandidates ignoreStatusFor to bypass the ~20s-stale
indexer status poll. Max 2 evictions per request.
- sweeper: idle janitor closes demo sessions unused for DEMO_SESSION_IDLE_MS
(default 3 min, below the supplier's 5 min CHAT_IDLE_TIMEOUT_MS) so
abandoned one-off sessions release their suppliers quickly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MVP demo: one operator-funded API key anyone can plug into an IDE/harness,
running real escrow-per-session marketplace conversations (llm.chat.v1)
behind the standard OpenAI /chat/completions wire shape.
- shared: ChatMessage gains tool role + tool_calls; normalizeChatMessage/
chatMessagesEquivalent keep supplier transcript and gateway mirror
hash-identical for receipts (field presence is part of the identity)
- supplier: /v1/chat/message accepts {messages,tools,tool_choice} alongside
legacy {content}; callOpenAiStream forwards tools, accumulates fragmented
delta.tool_calls, allows content-less tool turns; done frame carries the
assistant message verbatim + finish_reason + usage
- gateway: demo flag on api_keys (additive migration), create-demo-key CLI,
withdraw 403 + hidden deposit address for demo keys, per-IP demo rate
limit (demo keys skip the shared per-key limit), models filtered to chat
capability
- gateway executor (demoChat.ts): transcript-prefix affinity maps stateless
calls onto open sessions, forwards only unseen deltas, transparently
reopens on supplier idle-close; never falls back to one-shot (its settle
path assumes <=1 in-flight escrow per wallet)
- mutex scope: key mutex held only for chain work (open/close); off-chain
turns serialize on per-session locks so replicas serve users concurrently
- sweeper: chat-aware reclaim guard (35min vs 11min) and abandoned-session
cleanup; terminal state confirmed via on-chain queryUtxo (indexer keeps
lineage rows at their old state forever)
- deploy: kimi-chat replicas 2+3 compose files; chat idle timeout 2.5min
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New multi-tenant gateway package: external developers point any OpenAI SDK at
the marketplace. Each API key is its own custodial buyer wallet (AES-256-GCM
sealed under GATEWAY_MASTER_KEY); every request posts an on-chain escrow from
that user's wallet.
- POST /openai/v1/chat/completions (one escrow/call, llm.text.generate.v1,
buffered + pseudo-stream), chat sessions (one escrow/session, llm.chat.v1,
real SSE), GET /openai/v1/models, /signup, /account, /account/withdraw.
- Settles by resolving the Submitted ref via the indexer + awaited Accept;
state-aware reclaim sweeper + per-wallet auto-consolidation for collateral.
- Reuses the buyer Marketplace SDK via new buyer exports ./sdk; new shared
buildWithdrawTx (custodial exit). Runs via tsx; Dockerfile + mainnet compose.
Design in docs/gateway.md. Typecheck passes for all packages; 20 vitest tests
pass. Not yet exercised against a live chain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
