Release was the one escrow spend path still hardwired to the synthetic
testTxBody backend, so suppliers had no way to resolve Submitted escrows
past the accept window on mainnet (the sweeper parks them 'for supplier
Release'). Adds buildLiveTxForRelease mirroring the Reclaim builder
(supplier payout, Constr4/tag 125 redeemer, Ogmios UPLC eval) and the
detectCborBackend dispatch in buildReleaseTx, reclaim.ts-style.
validFrom clamps to submitted_at + ACCEPT_WINDOW rather than tip-60s
alone: the back-pad by itself violates the validator whenever the tip is
less than 60s past the window.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tailnet account changed (caslav.nedeljkovic@ to teamhaleight@,
taild99fee.ts.net) and all device IPs were reassigned, breaking both
inference-proxy suppliers' hardcoded backend addresses.
- supplier-local: vuk llama.cpp 100.82.111.46 to 100.77.146.49.
- supplier-openclaw: the openclaw gateway is now exposed via Tailscale
Serve (David's reconfigure set gateway.tailscale.mode=serve, which
forces bind=loopback). Switch OPENAI_BASE_URL from the old raw
http endpoint to https://web-tools.taild99fee.ts.net, and pin that
MagicDNS name to web-tools' tailnet IP (100.105.36.0) via extra_hosts
(Serve's TLS cert is for the hostname, and containers don't inherit
the host tailscale MagicDNS).
- README: new account/IP table, Serve config (replaces the tailnet-bind
section), reachability check (the old ACL blocker is gone), updated
probe/runbook.
Verified: authenticated turn to the Serve endpoint from inference-proxy
returns a completion; vuk:8002 reachable from inference-proxy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second supplier on the inference-proxy box, proxying to an OpenClaw gateway
(agent 'main') on the web-tools KVM guest over the tailnet
(100.101.143.72:18789). Same vuk llama.cpp brain as supplier 'local'; the
product is the agent loop (searxng web search + web fetch + browser).
Capability llm.chat.v1, advertised model 'openclaw-web-agent', 0.2 AP3X,
ticket settle mode, MAX_CHAT_SESSIONS=1.
Uses the new stateful-upstream contract: OPENAI_SESSION_PASSTHROUGH=1 (escrow
ref as OpenAI 'user' → sticky per-escrow openclaw session, delta-only sends)
and OPENAI_MODEL_OVERRIDE=openclaw (endpoint only accepts 'openclaw').
- docker-compose.supplier-openclaw.yml: new supplier compose (traefik host
mp-suppliers-openclaw.vector.apexfusion.org).
- README: gateway config (bind=tailnet, chatCompletions on, GLOBAL tool
allow+deny lockdown), the tailnet-ACL blocker (inference-proxy can't yet
reach web-tools:18789), bring-up runbook, and the sudo-gated browser +
egress-firewall follow-ups.
- monitor-wallets.ts: register supplier-openclaw@inference-proxy.
Wallet generated + funded (200 AP3X); .env.openclaw written on the host.
Not yet advertised — go-live (post-advert) waits on the ACL + a healthy
end-to-end path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two opt-in envs preparing the openclaw supplier on inference-proxy:
OPENAI_SESSION_PASSTHROUGH=1 — chat-session turns speak the stateful-
upstream contract: the escrow ref is sent as the OpenAI `user` field
(OpenClaw keys one persistent agent session per `user`, so browser/
tool state survives across turns) and only the current turn's delta
goes upstream — the upstream session carries history, and re-sending
the full transcript would duplicate it into the upstream context every
turn (OpenClaw renders ALL request messages into the new agent turn).
The local transcript still accumulates fully for the receipt. Without
`user`, OpenClaw creates an ephemeral session per request (uuid key),
so default-off stays safe for stateless providers.
OPENAI_MODEL_OVERRIDE — fixed upstream model id sent instead of the
advert's buyer-facing model. OpenClaw's chat-completions endpoint 400s
on anything but "openclaw" / "openclaw/<agentId>", while the advert
must carry a unique fleet-routable id (openclaw-web-agent).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MAX_CHAT_SESSIONS=1 — vuk is a single-slot ~4.5 tok/s rig, one session
owns it at a time. Idle timeout / settle mode mirror the Hetzner -chat
fleet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New manually-deployed host (62.238.38.167) with its own in-repo traefik
project (LE HTTP-01) and one text supplier proxying to the home rig at
100.82.111.46:8002. Outside CD by design; runbook in the README.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The PR #12 runbook and compose pre-flight headers claimed a wildcard
*.vector.apexfusion.org record covers new supplier hosts. It does not:
each mp-suppliers-* host is an individual DNS-only A record to
91.98.147.172, and on-box Let's Encrypt HTTP-01 issuance requires the
record to be unproxied.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eight new mainnet suppliers backed by Hetzner's OpenAI-compatible
inference API (https://inference.hetzner.com/api). Pure configuration —
the LLM_BACKEND=openai path is reused unchanged, following the
HuggingFace-router precedent:
kimi-code[-chat] Kimi-K2.7-Code (ctx 262144)
glm[-chat] GLM-5.2-NVFP4 (ctx 512000)
ds-flash-htz[-chat] DeepSeek-V4-Flash-0731 (ctx 512000)
qwen35b[-chat] Qwen/Qwen3.6-35B-A3B-FP8 (ctx 262144)
Uniform 0.2 AP3X per job, default 1 AP3X bonds, no upstream max_tokens
(advert cap = model context length), 300s SLA (1800s for chat sessions).
All four models smoke-tested for non-empty content and tools round-trip.
Runbook: docs/HETZNER_SETUP.md. Adverts, wallets, and host env files are
operator steps after merge; first `up -d` is manual (CD skips projects
with no running containers).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep SSH key material in $RUNNER_TEMP instead of ~/.ssh since a
self-hosted runner's home persists across jobs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Merges to main now deploy automatically once CI passes:
- .github/workflows/deploy-mainnet.yml: fires on successful CI runs of
main (or manually via workflow_dispatch with an optional force input),
SSHes into the mainnet host with the DEPLOY_SSH_KEY secret (host key
pinned) and runs deploy/mainnet/deploy.sh at the new commit via
git show FETCH_HEAD:, so deploy logic ships with the code it deploys.
- deploy/mainnet/deploy.sh: hard-resets /root/agents-marketplace to
origin/main, maps the old..new diff to affected compose projects
(packages/, patches/, contracts/ and root manifests rebuild all image
builds; the 17 supplier projects share one Dockerfile so builds after
the first are cache hits), builds everything up front, then does a
rolling up -d with a per-project health gate. Suppliers drain first:
/status is polled until not "working" (DRAIN_TIMEOUT_SECS, default
600s) so an in-flight job is not killed and its bond forfeited.
FORCE=1 redeploys the current sha, DRY_RUN=1 previews the plan, and
each deploy logs old -> new to /var/log/marketplace-deploy.log with a
printed rollback one-liner.
Verified: DRY_RUN=1 on the host against the pending 678ee86..6156900
delta selects indexer, buyer, gateway and all 17 suppliers, skips the
non-running wallet-monitor, and leaves image-only projects untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
runOcrOneShot posted the escrow before calling the supplier, so a busy
supplier's 409 stranded an already-paid escrow until reclaim - and the
buyer-side busy-retry multiplies this into several stranded escrows per
page. Add a live /status check per candidate, immediately before its
escrow posts, mirroring sessions.ts's startChat pre-check: "working"
skips to the next candidate without posting, anything else (free,
unreachable, malformed, slow) is treated as usable so the check is an
optimisation, never a gate. Bounded to a 2s timeout so a stalled
supplier cannot meaningfully delay the request.
Existing candidate-fallback, escrow-ref capture, usage recording, and
receipt labels are unchanged.
process.argv[1] arrives with backslashes on Windows, so the
forward-slash-only pattern never matched and `start` exited 0 without
starting the server, and without printing anything. Found while running
the gateway locally to verify this branch's OCR route end to end;
bundled here because it blocks that verification on Windows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
escrowRefStr was only assigned after submitOcr resolved, but submitOcr posts
the escrow internally and THEN calls the supplier over HTTP — a SupplierError
there (timeout, network failure, non-2xx, malformed response, receipt
mismatch) meant the escrow was already on chain while the failure path still
recorded escrow_ref: null. Given the OCR upstream's 120s timeout budget this
is likely the most common paid-but-failed shape, and the exact one the
reconciliation spine exists for.
Subscribe to the SDK's "progress" event (type "escrow_posted") around each
submitOcr attempt so the ref is captured the instant it lands, regardless of
whether submitOcr goes on to resolve or reject. Listener is scoped per
candidate and removed in a finally so it cannot accumulate.
Adds a regression test proving the failure-path usage row and the ocr_failed
structured log line both carry the real ref instead of null; verified it
fails without the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fixes 2 critical, 2 important, and 5 minor findings from review of 069edcb:
- C1: add a happy-path test that reaches postOcrEscrow.ts's escrowLockFloor
call and asserts the locked amount; verified it fails against a bare
economicTotal and passes against the real code.
- C2: gate /v1/ocr/extract (auth + rate limits) before the 16mb body parser
so an unauthenticated caller can no longer force a full buffer/JSON.parse.
- I2: hoist result/used/escrowRef in the OCR one-shot handler so the failure
path records escrow_ref/supplier/cost when an escrow was actually posted,
and emit an ocr_failed structured log line mirroring ocr_settled.
- I3: export venueAttribution and cover its sanitization directly; add a
stubbed happy-path route test asserting the first-party-brokered
disclosure label reaches the response.
- Minor: drop unused hfWeightsSource end to end, rename a misleading test,
make the OCR capability template's pinning language conditional on the
self-hosted upstream, tighten the /v1/ocr path-prefix guard, and make
preflight's required_lovelace use the same escrowLockFloor the builder
locks (exported from the shared tx barrel for gateway use).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the ocr.page.extract.chandra-ocr-2.v1 capability end to end: the
postOcrEscrow builder (locking the min-ada floor for the Submitted-state
datum, required since the min-ada fix), the gateway OCR route that selects
an OCR supplier by capability, posts a per-page escrow, calls the supplier,
verifies and settles, and returns content plus the signed receipt labelled
first-party-brokered. Supplier OCR handler with a hosted document-API
upstream and buyer-SDK OCR support included.
Squashed from the 2026-08-06 working subset; the deploy pack is
intentionally excluded (it lives in a private operator repo).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
OpenAI-compatible compute-backend preset (config only, reuses LLM_BACKEND=openai) for pointing a supplier node at the HuggingFace Inference Providers router. Adds a boot guard that rejects the OpenRouter-only reasoning param against the HF router, unit tests, and an operator runbook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The lockfile gained patchedDependencies (libsodium-wrappers-sumo) in
2aa1688, and the package manager hashes the patch file during install —
every image build since then fails with ENOENT on /repo/patches. No
image had been rebuilt since, so it surfaced only on the next deploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The escrow locks payment + both bonds and the validator requires exact
value preservation (value_equal) on Claim and Submit. Submit grows the
inline datum ~43 bytes (submitted_at None->Some, result_receipt_hash
None->Some 32 bytes). When capability_id is long enough, the
Submitted-state output's min-ada exceeds the locked total; lucid bumps
the continuing output, value_equal fails, and the validator errors with
no traces (opaque evaluateTransaction 3010/3012). Measured on mainnet at
utxoCostPerByte 4310: a 20-byte chat capability clears a 2.2M lock by
49,310, a 33-byte OCR capability is over by 11,030 - so short-capability
escrows settle and long ones never can, and because Claim's datum is
smaller it always passes, making a Submit-only failure look chain-side.
escrowLockFloor() computes min-ada for the Submitted state plus a margin
and locks max(economic total, that floor) in the post builders. Buyer-
side only, no contract or supplier changes; lucid never bumps once the
lock is sufficient, and the surplus returns through the >= terminal
distribution. postOcrEscrow's identical change ships with the OCR
feature branch. New unit test covers the floor across capability lengths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same pattern as the gateway (a54fdcb): the mp-suppliers-* DNS records are no
longer Cloudflare-proxied, so each supplier gets a websecure router with the
letsencrypt resolver and its :80 router becomes a permanent HTTPS redirect.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
planConsolidate returned already-healthy only for wallets with EXACTLY
2 UTxOs, but buildConsolidateWalletTx lands 3 (collateral + working +
the fee-reserve remainder as change). The shape consolidation produces
can never satisfy its own health check, so every wallet-health tick
re-consolidates and pays the ~0.5 AP3X min fee again. Measured in a
production deployment: a 45s ticker submitted 232 consolidations and
burned ~116 AP3X in about 2.5 idle hours; at the default 600s interval
the same loop leaks ~3 AP3X per hour whenever the wallet is not exactly
2 UTxOs.
Healthy is now structural: a pure-lovelace UTxO >= collateralLovelace
exists, at least 1 AP3X sits outside it, and fragmentation is bounded
(<= 6 UTxOs, new optional maxUtxos param). The health check also runs
BEFORE the low-balance guard, so a wallet that can already run script
transactions is never reshaped or refused. walletHealth gains two
independent circuit breakers - halt after 3 consecutive consolidations
that never reach healthy, halt past 6 consolidations per 24h - so any
future non-convergence costs at most ~1.5 AP3X instead of the whole
wallet, plus a docstring that matches reality.
Tests: the exact post-consolidation-shape regression, one-step
idempotence against the builder output model, candidate boundaries
(native assets, 1,628-lovelace-short), healthy-below-reshape-threshold,
and breaker behavior including error and reset paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
api.marketplace.vector.apexfusion.org is no longer Cloudflare-proxied, so
traefik now terminates TLS itself: a websecure router with the letsencrypt
ACME resolver, and the :80 router demoted to a permanent HTTPS redirect.
Other hosts on the box stay Cloudflare-terminated on :80.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- docs/ARCHITECTURE.md: settlement-asset framing (AP3X bonded in escrow,
settles on accepted work); remove references to external network
adapters that are not part of the project
- docs/buyer + docs/supplier READMEs: commission/bond/settle vocabulary
- Buyer web UI: "Accept & Settle", escrow-framed labels, gateway-managed
wallet wording; gateway landing title punctuation
- Delete EXECUTIVE_SUMMARY and local-llm-marketplace docs + exports:
M1-era internal design snapshots superseded by ARCHITECTURE.md and the
public docs site
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI has been red since the pnpm 10 migration. Four independent causes:
- better-sqlite3: pnpm 10 blocks dependency build scripts by default, so the
native binding was never compiled on CI. Allowlist it via
onlyBuiltDependencies in pnpm-workspace.yaml.
- libsodium-wrappers-sumo 0.7.16: the published ESM wrapper does
`import "./libsodium-sumo.mjs"` but the tarball does not ship that file
(it lives in the libsodium-sumo package). Patch the import to the bare
specifier; local runs only worked because of a hand-copied file in
node_modules.
- tx-accept/claim [live] tests: the live path checks deadlines against
Date.now(), and the fixed fixture timestamps expired in June. Freeze the
clock inside the fixture windows with vi.useFakeTimers.
- CML set_exunits panic ("RuntimeError: unreachable"): with
localUPLCEval: false the provider asks Ogmios for ex-units; the test mocks
answered `{}`, so the provider fell back to a spend redeemer at index 0
while the escrow input sorts to index 1 — CML panics on the unknown
redeemer key. Mock evaluateTransaction with the true index like real
Ogmios.
Also regenerate the advert-datum golden: the fixture builders passed plain JS
numbers to cbor-x (minimal ints, float64 for the timestamp) while the
production codec — whose output is live on mainnet — emits full-width uint64.
Builders now encode integer fields as BigInt so builders, codec, and golden
agree byte-for-byte.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The public Ogmios endpoint only serves a few concurrent ledger-state
queries and returns 500 for the rest, so the Promise.all burst over the
(now 19-wallet) list reliably failed several wallets every run. Query
sequentially instead (full sweep ~2.5s) and retry each wallet once after
2s so a lone blip no longer posts a false "query failed" alert.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
These were live on the box but never committed. Chat-session replicas of
the deepseek-flash suppliers (OpenRouter deepseek-v4-flash, llm.chat.v1,
ticket settle mode); secrets stay in the gitignored supplier/.env files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The signup page said '~5 ADA collateral' and two API error messages
told agents to 'deposit a little more ADA' - the coin is AP3X. Strings,
comments, and docs/gateway.md updated; code identifiers unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slack posts now fire only when a wallet newly drops below its threshold
or recovers above it; a wallet that stays low never re-posts. The
reminder_hours mechanism is removed entirely — per-wallet reminder
clocks drifted apart (Ogmios blips skip wallets from a run), so with
many chronically-low wallets some clock came due nearly every run,
collapsing the 6h cadence to the hourly cron interval.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two structural fixes for demo concurrency (3 single-slot suppliers thrashed
under Open WebUI's ~3 conversations per message):
Multi-slot suppliers (MAX_CHAT_SESSIONS, default 1):
- SupplierState is now capacity-aware: a Set of active escrow refs with
free-until-full status semantics, ref-scoped release(ref) (also fixes the
latent bug where handler catch-alls released another request's slot), and
a walletMutex serializing every wallet-spending chain op — Claim, Submit,
and wallet-health consolidation queue instead of racing the single
working/collateral UTxO. /status gains active_sessions/max_sessions.
Entry-ticket settle mode (CHAT_SETTLE_MODE=ticket, default full, both sides):
- The buyer's escrow post at session open is the ONLY chain op. The supplier
verifies the Open escrow as an entry ticket but never Claims/Submits
(/v1/chat/start → {status:"ticket"}; /v1/chat/end → receipt-less
{status:"ended"}); no hard-cap timer, so sessions may outlive deliver_by.
Used tickets can't reopen sessions (records retained; 409 session_exists).
- Buyer SDK: startChat/endChat are settle-mode aware, driven by the
supplier's response — mixed-mode rollout windows settle correctly.
- Gateway: ticket closes bill cost 0; the sweeper Reclaims Open tickets
after deliver_by WITHOUT closing live sessions (the conversation and its
affinity mirror survive their reclaimed ticket), gates reclaims on
deliver_by, and keeps an in-process settled-refs memory so terminal
indexer rows aren't retried every sweep forever.
Deployment (kimi-chat trio + gateway): ticket mode, 8 slots per supplier,
idle policy gateway 10 min / supplier 15 min (fixes the inversion where the
supplier idle-closed before the gateway janitor).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The demo endpoint lost its session on nearly every second message: affinity
required clients to re-echo the assistant reply byte-identically, so clients
that strip <think> blocks or reshape whitespace (Open WebUI, litellm) missed
every turn, opened a fresh escrow session on the next free single-slot
supplier, and left the old one pinned. Open WebUI's background title/tag-gen
requests each pinned another supplier — one message could exhaust all three
kimi replicas ("no available chat supplier").
- affinity: lenient assistant-content matching (think-strip + whitespace
collapse); user/system/tool messages and tool_calls stay byte-strict;
strict prefixes outrank lenient ones. The mirror is never rewritten with
client text, so receipt hashes stay valid.
- demoChat: trim trailing assistant/system messages (regenerate semantics)
instead of 400 "the last message must have role user or tool".
- supplier: roll the transcript back when the LLM call fails so retries
don't duplicate the delta (ChatSessionStore.truncateTranscript).
- demoChat: LRU-evict a demo session when no supplier is available (supplier
/v1/chat/end frees the slot, sweeper Accepts the escrow) and retry the
open with selectCandidates ignoreStatusFor to bypass the ~20s-stale
indexer status poll. Max 2 evictions per request.
- sweeper: idle janitor closes demo sessions unused for DEMO_SESSION_IDLE_MS
(default 3 min, below the supplier's 5 min CHAT_IDLE_TIMEOUT_MS) so
abandoned one-off sessions release their suppliers quickly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MVP demo: one operator-funded API key anyone can plug into an IDE/harness,
running real escrow-per-session marketplace conversations (llm.chat.v1)
behind the standard OpenAI /chat/completions wire shape.
- shared: ChatMessage gains tool role + tool_calls; normalizeChatMessage/
chatMessagesEquivalent keep supplier transcript and gateway mirror
hash-identical for receipts (field presence is part of the identity)
- supplier: /v1/chat/message accepts {messages,tools,tool_choice} alongside
legacy {content}; callOpenAiStream forwards tools, accumulates fragmented
delta.tool_calls, allows content-less tool turns; done frame carries the
assistant message verbatim + finish_reason + usage
- gateway: demo flag on api_keys (additive migration), create-demo-key CLI,
withdraw 403 + hidden deposit address for demo keys, per-IP demo rate
limit (demo keys skip the shared per-key limit), models filtered to chat
capability
- gateway executor (demoChat.ts): transcript-prefix affinity maps stateless
calls onto open sessions, forwards only unseen deltas, transparently
reopens on supplier idle-close; never falls back to one-shot (its settle
path assumes <=1 in-flight escrow per wallet)
- mutex scope: key mutex held only for chain work (open/close); off-chain
turns serialize on per-session locks so replicas serve users concurrently
- sweeper: chat-aware reclaim guard (35min vs 11min) and abandoned-session
cleanup; terminal state confirmed via on-chain queryUtxo (indexer keeps
lineage rows at their old state forever)
- deploy: kimi-chat replicas 2+3 compose files; chat idle timeout 2.5min
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The supplier sent a bare {model,messages,stream} to the OpenAI/OpenRouter
endpoint — no max_tokens and no reasoning control — so reasoning ("thinking")
models like kimi-k2.6 emitted reasoning tokens that could consume the
completion budget and finish with reason=length (truncated/empty answer =
failed job). deepseek-v4-flash has no thinking; kimi-k2.6 does by default.
Add two optional, env-gated request params (defaults preserve current behavior,
so ChatMock/DeepSeek-direct suppliers are unaffected):
- OPENAI_REASONING=off -> send reasoning:{enabled:false} (disable thinking:
faster, cheaper, straight answers; no reasoning-starved length truncation)
- OPENAI_MAX_TOKENS=N -> forward max_tokens:N. Must stay below the model
context (kimi context = 262144), so the advert's 262144 cap cannot be sent
verbatim (provider 400s when prompt+max_tokens > context).
openai.ts builds the body conditionally; config.ts parses the env; wired
through jobRunner.ts (text) and server.ts (chat-session stream).
Deployed on the 4 kimi + 3 deepseek-flash suppliers with OPENAI_REASONING=off
and OPENAI_MAX_TOKENS=131072 (env files are host-only/gitignored).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
