MAX_CHAT_SESSIONS=1 — vuk is a single-slot ~4.5 tok/s rig, one session
owns it at a time. Idle timeout / settle mode mirror the Hetzner -chat
fleet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New manually-deployed host (62.238.38.167) with its own in-repo traefik
project (LE HTTP-01) and one text supplier proxying to the home rig at
100.82.111.46:8002. Outside CD by design; runbook in the README.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The PR #12 runbook and compose pre-flight headers claimed a wildcard
*.vector.apexfusion.org record covers new supplier hosts. It does not:
each mp-suppliers-* host is an individual DNS-only A record to
91.98.147.172, and on-box Let's Encrypt HTTP-01 issuance requires the
record to be unproxied.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eight new mainnet suppliers backed by Hetzner's OpenAI-compatible
inference API (https://inference.hetzner.com/api). Pure configuration —
the LLM_BACKEND=openai path is reused unchanged, following the
HuggingFace-router precedent:
kimi-code[-chat] Kimi-K2.7-Code (ctx 262144)
glm[-chat] GLM-5.2-NVFP4 (ctx 512000)
ds-flash-htz[-chat] DeepSeek-V4-Flash-0731 (ctx 512000)
qwen35b[-chat] Qwen/Qwen3.6-35B-A3B-FP8 (ctx 262144)
Uniform 0.2 AP3X per job, default 1 AP3X bonds, no upstream max_tokens
(advert cap = model context length), 300s SLA (1800s for chat sessions).
All four models smoke-tested for non-empty content and tools round-trip.
Runbook: docs/HETZNER_SETUP.md. Adverts, wallets, and host env files are
operator steps after merge; first `up -d` is manual (CD skips projects
with no running containers).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep SSH key material in $RUNNER_TEMP instead of ~/.ssh since a
self-hosted runner's home persists across jobs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Merges to main now deploy automatically once CI passes:
- .github/workflows/deploy-mainnet.yml: fires on successful CI runs of
main (or manually via workflow_dispatch with an optional force input),
SSHes into the mainnet host with the DEPLOY_SSH_KEY secret (host key
pinned) and runs deploy/mainnet/deploy.sh at the new commit via
git show FETCH_HEAD:, so deploy logic ships with the code it deploys.
- deploy/mainnet/deploy.sh: hard-resets /root/agents-marketplace to
origin/main, maps the old..new diff to affected compose projects
(packages/, patches/, contracts/ and root manifests rebuild all image
builds; the 17 supplier projects share one Dockerfile so builds after
the first are cache hits), builds everything up front, then does a
rolling up -d with a per-project health gate. Suppliers drain first:
/status is polled until not "working" (DRAIN_TIMEOUT_SECS, default
600s) so an in-flight job is not killed and its bond forfeited.
FORCE=1 redeploys the current sha, DRY_RUN=1 previews the plan, and
each deploy logs old -> new to /var/log/marketplace-deploy.log with a
printed rollback one-liner.
Verified: DRY_RUN=1 on the host against the pending 678ee86..6156900
delta selects indexer, buyer, gateway and all 17 suppliers, skips the
non-running wallet-monitor, and leaves image-only projects untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
runOcrOneShot posted the escrow before calling the supplier, so a busy
supplier's 409 stranded an already-paid escrow until reclaim - and the
buyer-side busy-retry multiplies this into several stranded escrows per
page. Add a live /status check per candidate, immediately before its
escrow posts, mirroring sessions.ts's startChat pre-check: "working"
skips to the next candidate without posting, anything else (free,
unreachable, malformed, slow) is treated as usable so the check is an
optimisation, never a gate. Bounded to a 2s timeout so a stalled
supplier cannot meaningfully delay the request.
Existing candidate-fallback, escrow-ref capture, usage recording, and
receipt labels are unchanged.
process.argv[1] arrives with backslashes on Windows, so the
forward-slash-only pattern never matched and `start` exited 0 without
starting the server, and without printing anything. Found while running
the gateway locally to verify this branch's OCR route end to end;
bundled here because it blocks that verification on Windows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
escrowRefStr was only assigned after submitOcr resolved, but submitOcr posts
the escrow internally and THEN calls the supplier over HTTP — a SupplierError
there (timeout, network failure, non-2xx, malformed response, receipt
mismatch) meant the escrow was already on chain while the failure path still
recorded escrow_ref: null. Given the OCR upstream's 120s timeout budget this
is likely the most common paid-but-failed shape, and the exact one the
reconciliation spine exists for.
Subscribe to the SDK's "progress" event (type "escrow_posted") around each
submitOcr attempt so the ref is captured the instant it lands, regardless of
whether submitOcr goes on to resolve or reject. Listener is scoped per
candidate and removed in a finally so it cannot accumulate.
Adds a regression test proving the failure-path usage row and the ocr_failed
structured log line both carry the real ref instead of null; verified it
fails without the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fixes 2 critical, 2 important, and 5 minor findings from review of 069edcb:
- C1: add a happy-path test that reaches postOcrEscrow.ts's escrowLockFloor
call and asserts the locked amount; verified it fails against a bare
economicTotal and passes against the real code.
- C2: gate /v1/ocr/extract (auth + rate limits) before the 16mb body parser
so an unauthenticated caller can no longer force a full buffer/JSON.parse.
- I2: hoist result/used/escrowRef in the OCR one-shot handler so the failure
path records escrow_ref/supplier/cost when an escrow was actually posted,
and emit an ocr_failed structured log line mirroring ocr_settled.
- I3: export venueAttribution and cover its sanitization directly; add a
stubbed happy-path route test asserting the first-party-brokered
disclosure label reaches the response.
- Minor: drop unused hfWeightsSource end to end, rename a misleading test,
make the OCR capability template's pinning language conditional on the
self-hosted upstream, tighten the /v1/ocr path-prefix guard, and make
preflight's required_lovelace use the same escrowLockFloor the builder
locks (exported from the shared tx barrel for gateway use).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the ocr.page.extract.chandra-ocr-2.v1 capability end to end: the
postOcrEscrow builder (locking the min-ada floor for the Submitted-state
datum, required since the min-ada fix), the gateway OCR route that selects
an OCR supplier by capability, posts a per-page escrow, calls the supplier,
verifies and settles, and returns content plus the signed receipt labelled
first-party-brokered. Supplier OCR handler with a hosted document-API
upstream and buyer-SDK OCR support included.
Squashed from the 2026-08-06 working subset; the deploy pack is
intentionally excluded (it lives in a private operator repo).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
OpenAI-compatible compute-backend preset (config only, reuses LLM_BACKEND=openai) for pointing a supplier node at the HuggingFace Inference Providers router. Adds a boot guard that rejects the OpenRouter-only reasoning param against the HF router, unit tests, and an operator runbook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The lockfile gained patchedDependencies (libsodium-wrappers-sumo) in
2aa1688, and the package manager hashes the patch file during install —
every image build since then fails with ENOENT on /repo/patches. No
image had been rebuilt since, so it surfaced only on the next deploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The escrow locks payment + both bonds and the validator requires exact
value preservation (value_equal) on Claim and Submit. Submit grows the
inline datum ~43 bytes (submitted_at None->Some, result_receipt_hash
None->Some 32 bytes). When capability_id is long enough, the
Submitted-state output's min-ada exceeds the locked total; lucid bumps
the continuing output, value_equal fails, and the validator errors with
no traces (opaque evaluateTransaction 3010/3012). Measured on mainnet at
utxoCostPerByte 4310: a 20-byte chat capability clears a 2.2M lock by
49,310, a 33-byte OCR capability is over by 11,030 - so short-capability
escrows settle and long ones never can, and because Claim's datum is
smaller it always passes, making a Submit-only failure look chain-side.
escrowLockFloor() computes min-ada for the Submitted state plus a margin
and locks max(economic total, that floor) in the post builders. Buyer-
side only, no contract or supplier changes; lucid never bumps once the
lock is sufficient, and the surplus returns through the >= terminal
distribution. postOcrEscrow's identical change ships with the OCR
feature branch. New unit test covers the floor across capability lengths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same pattern as the gateway (a54fdcb): the mp-suppliers-* DNS records are no
longer Cloudflare-proxied, so each supplier gets a websecure router with the
letsencrypt resolver and its :80 router becomes a permanent HTTPS redirect.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
planConsolidate returned already-healthy only for wallets with EXACTLY
2 UTxOs, but buildConsolidateWalletTx lands 3 (collateral + working +
the fee-reserve remainder as change). The shape consolidation produces
can never satisfy its own health check, so every wallet-health tick
re-consolidates and pays the ~0.5 AP3X min fee again. Measured in a
production deployment: a 45s ticker submitted 232 consolidations and
burned ~116 AP3X in about 2.5 idle hours; at the default 600s interval
the same loop leaks ~3 AP3X per hour whenever the wallet is not exactly
2 UTxOs.
Healthy is now structural: a pure-lovelace UTxO >= collateralLovelace
exists, at least 1 AP3X sits outside it, and fragmentation is bounded
(<= 6 UTxOs, new optional maxUtxos param). The health check also runs
BEFORE the low-balance guard, so a wallet that can already run script
transactions is never reshaped or refused. walletHealth gains two
independent circuit breakers - halt after 3 consecutive consolidations
that never reach healthy, halt past 6 consolidations per 24h - so any
future non-convergence costs at most ~1.5 AP3X instead of the whole
wallet, plus a docstring that matches reality.
Tests: the exact post-consolidation-shape regression, one-step
idempotence against the builder output model, candidate boundaries
(native assets, 1,628-lovelace-short), healthy-below-reshape-threshold,
and breaker behavior including error and reset paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
api.marketplace.vector.apexfusion.org is no longer Cloudflare-proxied, so
traefik now terminates TLS itself: a websecure router with the letsencrypt
ACME resolver, and the :80 router demoted to a permanent HTTPS redirect.
Other hosts on the box stay Cloudflare-terminated on :80.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- docs/ARCHITECTURE.md: settlement-asset framing (AP3X bonded in escrow,
settles on accepted work); remove references to external network
adapters that are not part of the project
- docs/buyer + docs/supplier READMEs: commission/bond/settle vocabulary
- Buyer web UI: "Accept & Settle", escrow-framed labels, gateway-managed
wallet wording; gateway landing title punctuation
- Delete EXECUTIVE_SUMMARY and local-llm-marketplace docs + exports:
M1-era internal design snapshots superseded by ARCHITECTURE.md and the
public docs site
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI has been red since the pnpm 10 migration. Four independent causes:
- better-sqlite3: pnpm 10 blocks dependency build scripts by default, so the
native binding was never compiled on CI. Allowlist it via
onlyBuiltDependencies in pnpm-workspace.yaml.
- libsodium-wrappers-sumo 0.7.16: the published ESM wrapper does
`import "./libsodium-sumo.mjs"` but the tarball does not ship that file
(it lives in the libsodium-sumo package). Patch the import to the bare
specifier; local runs only worked because of a hand-copied file in
node_modules.
- tx-accept/claim [live] tests: the live path checks deadlines against
Date.now(), and the fixed fixture timestamps expired in June. Freeze the
clock inside the fixture windows with vi.useFakeTimers.
- CML set_exunits panic ("RuntimeError: unreachable"): with
localUPLCEval: false the provider asks Ogmios for ex-units; the test mocks
answered `{}`, so the provider fell back to a spend redeemer at index 0
while the escrow input sorts to index 1 — CML panics on the unknown
redeemer key. Mock evaluateTransaction with the true index like real
Ogmios.
Also regenerate the advert-datum golden: the fixture builders passed plain JS
numbers to cbor-x (minimal ints, float64 for the timestamp) while the
production codec — whose output is live on mainnet — emits full-width uint64.
Builders now encode integer fields as BigInt so builders, codec, and golden
agree byte-for-byte.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The public Ogmios endpoint only serves a few concurrent ledger-state
queries and returns 500 for the rest, so the Promise.all burst over the
(now 19-wallet) list reliably failed several wallets every run. Query
sequentially instead (full sweep ~2.5s) and retry each wallet once after
2s so a lone blip no longer posts a false "query failed" alert.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
These were live on the box but never committed. Chat-session replicas of
the deepseek-flash suppliers (OpenRouter deepseek-v4-flash, llm.chat.v1,
ticket settle mode); secrets stay in the gitignored supplier/.env files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The signup page said '~5 ADA collateral' and two API error messages
told agents to 'deposit a little more ADA' - the coin is AP3X. Strings,
comments, and docs/gateway.md updated; code identifiers unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slack posts now fire only when a wallet newly drops below its threshold
or recovers above it; a wallet that stays low never re-posts. The
reminder_hours mechanism is removed entirely — per-wallet reminder
clocks drifted apart (Ogmios blips skip wallets from a run), so with
many chronically-low wallets some clock came due nearly every run,
collapsing the 6h cadence to the hourly cron interval.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two structural fixes for demo concurrency (3 single-slot suppliers thrashed
under Open WebUI's ~3 conversations per message):
Multi-slot suppliers (MAX_CHAT_SESSIONS, default 1):
- SupplierState is now capacity-aware: a Set of active escrow refs with
free-until-full status semantics, ref-scoped release(ref) (also fixes the
latent bug where handler catch-alls released another request's slot), and
a walletMutex serializing every wallet-spending chain op — Claim, Submit,
and wallet-health consolidation queue instead of racing the single
working/collateral UTxO. /status gains active_sessions/max_sessions.
Entry-ticket settle mode (CHAT_SETTLE_MODE=ticket, default full, both sides):
- The buyer's escrow post at session open is the ONLY chain op. The supplier
verifies the Open escrow as an entry ticket but never Claims/Submits
(/v1/chat/start → {status:"ticket"}; /v1/chat/end → receipt-less
{status:"ended"}); no hard-cap timer, so sessions may outlive deliver_by.
Used tickets can't reopen sessions (records retained; 409 session_exists).
- Buyer SDK: startChat/endChat are settle-mode aware, driven by the
supplier's response — mixed-mode rollout windows settle correctly.
- Gateway: ticket closes bill cost 0; the sweeper Reclaims Open tickets
after deliver_by WITHOUT closing live sessions (the conversation and its
affinity mirror survive their reclaimed ticket), gates reclaims on
deliver_by, and keeps an in-process settled-refs memory so terminal
indexer rows aren't retried every sweep forever.
Deployment (kimi-chat trio + gateway): ticket mode, 8 slots per supplier,
idle policy gateway 10 min / supplier 15 min (fixes the inversion where the
supplier idle-closed before the gateway janitor).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The demo endpoint lost its session on nearly every second message: affinity
required clients to re-echo the assistant reply byte-identically, so clients
that strip <think> blocks or reshape whitespace (Open WebUI, litellm) missed
every turn, opened a fresh escrow session on the next free single-slot
supplier, and left the old one pinned. Open WebUI's background title/tag-gen
requests each pinned another supplier — one message could exhaust all three
kimi replicas ("no available chat supplier").
- affinity: lenient assistant-content matching (think-strip + whitespace
collapse); user/system/tool messages and tool_calls stay byte-strict;
strict prefixes outrank lenient ones. The mirror is never rewritten with
client text, so receipt hashes stay valid.
- demoChat: trim trailing assistant/system messages (regenerate semantics)
instead of 400 "the last message must have role user or tool".
- supplier: roll the transcript back when the LLM call fails so retries
don't duplicate the delta (ChatSessionStore.truncateTranscript).
- demoChat: LRU-evict a demo session when no supplier is available (supplier
/v1/chat/end frees the slot, sweeper Accepts the escrow) and retry the
open with selectCandidates ignoreStatusFor to bypass the ~20s-stale
indexer status poll. Max 2 evictions per request.
- sweeper: idle janitor closes demo sessions unused for DEMO_SESSION_IDLE_MS
(default 3 min, below the supplier's 5 min CHAT_IDLE_TIMEOUT_MS) so
abandoned one-off sessions release their suppliers quickly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MVP demo: one operator-funded API key anyone can plug into an IDE/harness,
running real escrow-per-session marketplace conversations (llm.chat.v1)
behind the standard OpenAI /chat/completions wire shape.
- shared: ChatMessage gains tool role + tool_calls; normalizeChatMessage/
chatMessagesEquivalent keep supplier transcript and gateway mirror
hash-identical for receipts (field presence is part of the identity)
- supplier: /v1/chat/message accepts {messages,tools,tool_choice} alongside
legacy {content}; callOpenAiStream forwards tools, accumulates fragmented
delta.tool_calls, allows content-less tool turns; done frame carries the
assistant message verbatim + finish_reason + usage
- gateway: demo flag on api_keys (additive migration), create-demo-key CLI,
withdraw 403 + hidden deposit address for demo keys, per-IP demo rate
limit (demo keys skip the shared per-key limit), models filtered to chat
capability
- gateway executor (demoChat.ts): transcript-prefix affinity maps stateless
calls onto open sessions, forwards only unseen deltas, transparently
reopens on supplier idle-close; never falls back to one-shot (its settle
path assumes <=1 in-flight escrow per wallet)
- mutex scope: key mutex held only for chain work (open/close); off-chain
turns serialize on per-session locks so replicas serve users concurrently
- sweeper: chat-aware reclaim guard (35min vs 11min) and abandoned-session
cleanup; terminal state confirmed via on-chain queryUtxo (indexer keeps
lineage rows at their old state forever)
- deploy: kimi-chat replicas 2+3 compose files; chat idle timeout 2.5min
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The supplier sent a bare {model,messages,stream} to the OpenAI/OpenRouter
endpoint — no max_tokens and no reasoning control — so reasoning ("thinking")
models like kimi-k2.6 emitted reasoning tokens that could consume the
completion budget and finish with reason=length (truncated/empty answer =
failed job). deepseek-v4-flash has no thinking; kimi-k2.6 does by default.
Add two optional, env-gated request params (defaults preserve current behavior,
so ChatMock/DeepSeek-direct suppliers are unaffected):
- OPENAI_REASONING=off -> send reasoning:{enabled:false} (disable thinking:
faster, cheaper, straight answers; no reasoning-starved length truncation)
- OPENAI_MAX_TOKENS=N -> forward max_tokens:N. Must stay below the model
context (kimi context = 262144), so the advert's 262144 cap cannot be sent
verbatim (provider 400s when prompt+max_tokens > context).
openai.ts builds the body conditionally; config.ts parses the env; wired
through jobRunner.ts (text) and server.ts (chat-session stream).
Deployed on the 4 kimi + 3 deepseek-flash suppliers with OPENAI_REASONING=off
and OPENAI_MAX_TOKENS=131072 (env files are host-only/gitignored).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add 3 single-slot suppliers serving deepseek/deepseek-v4-flash via OpenRouter,
increasing concurrent capacity for the independent buyer machines. Same shape
as the kimi replicas: capability llm.text.generate.v1, price 200000 lovelace,
max_output_tokens=262144, max_processing_ms=3600000, OPENAI_TIMEOUT_MS=3600000,
sharing the one OpenRouter key (accepted concurrency ceiling; no 429/backoff).
- deploy/mainnet/docker-compose.supplier-deepseek-flash-{1,2,3}.yml (new)
- buyer/scripts/monitor-wallets.ts: register the 3 replica wallets
Per-instance wallets, .env files, and on-chain adverts are provisioned on the
mainnet host (gitignored); the 3 containers are live and Active on-chain.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add 3 parallel-capacity replicas of supplier-kimi (kimi-2/3/4) so the
marketplace can serve more concurrent jobs from the independent buyer
machines. Each is a single-slot worker on the same capability
(llm.text.generate.v1) and model (moonshotai/kimi-k2.6) at the same 0.2 AP3X
flat price, sharing the one OpenRouter key (accepted concurrency ceiling).
Also bring the original supplier-kimi up to the same config: raise its advert
caps (re-posted on-chain) and OPENAI_TIMEOUT_MS to match. All four kimi
suppliers now advertise max_output_tokens=262144, max_processing_ms=3600000
(OpenRouter's ceiling for kimi-k2.6), with OPENAI_TIMEOUT_MS=3600000 so the
upstream call isn't cut short before the advertised deadline.
- deploy/mainnet/docker-compose.supplier-kimi-{2,3,4}.yml (new)
- deploy/mainnet/docker-compose.supplier-kimi.yml: OPENAI_TIMEOUT_MS 300000 -> 3600000
- buyer/scripts/monitor-wallets.ts: register the 3 replica wallets
Per-instance wallets, .env files, and on-chain adverts are provisioned on the
mainnet host (gitignored); all four containers are live and Active on-chain.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an "API Keys" page to the buyer SPA (marketplace.vector.apexfusion.org)
that mints a custodial gateway API key by POSTing directly to the gateway's
public /signup, then shows the key once + deposit address + a base_url/curl
usage snippet. Page lives behind the SPA's existing operator login; per-user
registration is a later feature.
Gateway:
- GATEWAY_CORS_ORIGINS env -> corsOrigins allowlist; corsMiddleware echoes
Access-Control-* only for allowlisted Origins, answers OPTIONS preflight
204, no credentials. Empty allowlist = CORS off (default safe).
Buyer:
- GATEWAY_PUBLIC_URL env -> injected into __BUYER_BOOT__.gatewayUrl (only in
the authenticated boot block). SPA resolveGatewayUrl() falls back to
deriving "api." + location.host so the post-login-no-reload case (boot
block absent) still works.
- New ApiKeys page + /api-keys route + nav link.
Deploy (mainnet compose): gateway GATEWAY_CORS_ORIGINS and buyer
GATEWAY_PUBLIC_URL set to the marketplace/api hosts. NOTE: both images must be
REBUILT (gateway runs tsx from copied source; buyer SPA is vite-built at image
build) — an env-only recreate would silently ship the old code.
Typecheck + vite build clean; 205 tests pass. Adversarial multi-agent review
(4 dimensions, per-finding verification) surfaced only the rebuild-required
deploy note above; no code/security defects.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
buildLiveTxForEscrow called .complete() with no presetWalletInputs, so lucid was
free to spend the dedicated 5-ADA collateral UTxO as an ordinary input. On
back-to-back requests this left the wallet with no collateral candidate and the
subsequent Accept failed with collateral_required, stranding the escrow.
Reserve the smallest pure-ADA UTxO >= 5 ADA and fund PostEscrow from the rest
(falls back to default selection for single/large-UTxO cases, which keep an
ample pure-ADA change UTxO anyway). Benefits the buyer too (same shared path).
Adds reserveCollateralInputs unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
