Generic runbook for deploying a mainnet supplier on any operator-owned
server: own DNS/TLS via the in-repo Traefik compose, docker-run CLI forms
(no host Node), advert posted last as the go-live flag, verification via
the public indexer. Fleet-specific steps stay in HETZNER_SETUP.md.
Selecting a supplier used to render its form at the bottom of the
dashboard, below the card grid — chat sessions in particular ended up
off-screen. The form now renders directly under the page title, above
the grid, and clicking Use scrolls back to the top so the freshly
opened form is visible.
PDF_ENABLED=0 skips the JobStore/PdfJobDb wiring in runMain, so every
/v1/pdf-* route responds 503 (the pre-existing disabled path). Default
stays enabled; only "0"/"1" accepted, misconfig fails boot loudly.
OpenAI-compatible supplier backed by the HuggingFace Inference Providers router
(meta-llama/Llama-3.3-70B-Instruct, llm.text.generate.v1). Config only; mirrors
the glm one-off template with OPENAI_BASE_URL=https://router.huggingface.co.
- deploy/mainnet/docker-compose.supplier-hf-llama33.yml
- register the supplier wallet in monitor-wallets OPERATOR_SOURCES
The env file (supplier/.env.hf-llama33) holds the wallet key + HF token and is
never committed; wallet-monitor/wallets.json is gitignored runtime data.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The indexer never updates escrow rows on terminal spends (Accept /
Reclaim / Release consume the UTxO without producing a new escrow
output), so /account summed rows whose funds had long moved and
locked_in_escrow_lovelace over-reported for 35+ minutes after
settlement.
The account handler now verifies each active-state row's UTxO is still
unspent (chain.queryUtxo) before counting it, capped at 32 lookups per
request with the tail counted unverified. Unparseable refs and failed
lookups count as live on purpose: errors inflate the figure rather than
hide locked funds. supplier_bond stays in the sum - the buyer fronts
all three legs at PostEscrow (see postEscrow.ts totalLocked), so it is
the key's money while the escrow lives.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One chain call that never settled, run under the per-key Mutex with no
deadline, convoyed every subsequent request on that key silently while
/models and /account stayed healthy (2026-08-12 incident). The per-fetch
30s aborts in the Ogmios providers predate the freeze and did not prevent
it; the unbounded thing was the composition under the mutex.
Mutex.run() now races each fn against a deadline (default 180s): on
expiry the run rejects with MutexTimeoutError and the chain advances, so
queued work proceeds instead of freezing. A zombie that settles later can
at worst lose a UTxO-contention race with the run that replaced it -
strictly better than a permanent convoy. New work is refused with
MutexQueueFullError past a queue bound (default 64) instead of queueing
without limit. Serialization and rejection-tolerance semantics are
unchanged (covered by tests). Session locks in transcripts.ts construct
the same class and inherit both protections.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Release was the one escrow spend path still hardwired to the synthetic
testTxBody backend, so suppliers had no way to resolve Submitted escrows
past the accept window on mainnet (the sweeper parks them 'for supplier
Release'). Adds buildLiveTxForRelease mirroring the Reclaim builder
(supplier payout, Constr4/tag 125 redeemer, Ogmios UPLC eval) and the
detectCborBackend dispatch in buildReleaseTx, reclaim.ts-style.
validFrom clamps to submitted_at + ACCEPT_WINDOW rather than tip-60s
alone: the back-pad by itself violates the validator whenever the tip is
less than 60s past the window.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tailnet account changed (caslav.nedeljkovic@ to teamhaleight@,
taild99fee.ts.net) and all device IPs were reassigned, breaking both
inference-proxy suppliers' hardcoded backend addresses.
- supplier-local: vuk llama.cpp 100.82.111.46 to 100.77.146.49.
- supplier-openclaw: the openclaw gateway is now exposed via Tailscale
Serve (David's reconfigure set gateway.tailscale.mode=serve, which
forces bind=loopback). Switch OPENAI_BASE_URL from the old raw
http endpoint to https://web-tools.taild99fee.ts.net, and pin that
MagicDNS name to web-tools' tailnet IP (100.105.36.0) via extra_hosts
(Serve's TLS cert is for the hostname, and containers don't inherit
the host tailscale MagicDNS).
- README: new account/IP table, Serve config (replaces the tailnet-bind
section), reachability check (the old ACL blocker is gone), updated
probe/runbook.
Verified: authenticated turn to the Serve endpoint from inference-proxy
returns a completion; vuk:8002 reachable from inference-proxy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second supplier on the inference-proxy box, proxying to an OpenClaw gateway
(agent 'main') on the web-tools KVM guest over the tailnet
(100.101.143.72:18789). Same vuk llama.cpp brain as supplier 'local'; the
product is the agent loop (searxng web search + web fetch + browser).
Capability llm.chat.v1, advertised model 'openclaw-web-agent', 0.2 AP3X,
ticket settle mode, MAX_CHAT_SESSIONS=1.
Uses the new stateful-upstream contract: OPENAI_SESSION_PASSTHROUGH=1 (escrow
ref as OpenAI 'user' → sticky per-escrow openclaw session, delta-only sends)
and OPENAI_MODEL_OVERRIDE=openclaw (endpoint only accepts 'openclaw').
- docker-compose.supplier-openclaw.yml: new supplier compose (traefik host
mp-suppliers-openclaw.vector.apexfusion.org).
- README: gateway config (bind=tailnet, chatCompletions on, GLOBAL tool
allow+deny lockdown), the tailnet-ACL blocker (inference-proxy can't yet
reach web-tools:18789), bring-up runbook, and the sudo-gated browser +
egress-firewall follow-ups.
- monitor-wallets.ts: register supplier-openclaw@inference-proxy.
Wallet generated + funded (200 AP3X); .env.openclaw written on the host.
Not yet advertised — go-live (post-advert) waits on the ACL + a healthy
end-to-end path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two opt-in envs preparing the openclaw supplier on inference-proxy:
OPENAI_SESSION_PASSTHROUGH=1 — chat-session turns speak the stateful-
upstream contract: the escrow ref is sent as the OpenAI `user` field
(OpenClaw keys one persistent agent session per `user`, so browser/
tool state survives across turns) and only the current turn's delta
goes upstream — the upstream session carries history, and re-sending
the full transcript would duplicate it into the upstream context every
turn (OpenClaw renders ALL request messages into the new agent turn).
The local transcript still accumulates fully for the receipt. Without
`user`, OpenClaw creates an ephemeral session per request (uuid key),
so default-off stays safe for stateless providers.
OPENAI_MODEL_OVERRIDE — fixed upstream model id sent instead of the
advert's buyer-facing model. OpenClaw's chat-completions endpoint 400s
on anything but "openclaw" / "openclaw/<agentId>", while the advert
must carry a unique fleet-routable id (openclaw-web-agent).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MAX_CHAT_SESSIONS=1 — vuk is a single-slot ~4.5 tok/s rig, one session
owns it at a time. Idle timeout / settle mode mirror the Hetzner -chat
fleet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New manually-deployed host (62.238.38.167) with its own in-repo traefik
project (LE HTTP-01) and one text supplier proxying to the home rig at
100.82.111.46:8002. Outside CD by design; runbook in the README.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The PR #12 runbook and compose pre-flight headers claimed a wildcard
*.vector.apexfusion.org record covers new supplier hosts. It does not:
each mp-suppliers-* host is an individual DNS-only A record to
91.98.147.172, and on-box Let's Encrypt HTTP-01 issuance requires the
record to be unproxied.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eight new mainnet suppliers backed by Hetzner's OpenAI-compatible
inference API (https://inference.hetzner.com/api). Pure configuration —
the LLM_BACKEND=openai path is reused unchanged, following the
HuggingFace-router precedent:
kimi-code[-chat] Kimi-K2.7-Code (ctx 262144)
glm[-chat] GLM-5.2-NVFP4 (ctx 512000)
ds-flash-htz[-chat] DeepSeek-V4-Flash-0731 (ctx 512000)
qwen35b[-chat] Qwen/Qwen3.6-35B-A3B-FP8 (ctx 262144)
Uniform 0.2 AP3X per job, default 1 AP3X bonds, no upstream max_tokens
(advert cap = model context length), 300s SLA (1800s for chat sessions).
All four models smoke-tested for non-empty content and tools round-trip.
Runbook: docs/HETZNER_SETUP.md. Adverts, wallets, and host env files are
operator steps after merge; first `up -d` is manual (CD skips projects
with no running containers).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep SSH key material in $RUNNER_TEMP instead of ~/.ssh since a
self-hosted runner's home persists across jobs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Merges to main now deploy automatically once CI passes:
- .github/workflows/deploy-mainnet.yml: fires on successful CI runs of
main (or manually via workflow_dispatch with an optional force input),
SSHes into the mainnet host with the DEPLOY_SSH_KEY secret (host key
pinned) and runs deploy/mainnet/deploy.sh at the new commit via
git show FETCH_HEAD:, so deploy logic ships with the code it deploys.
- deploy/mainnet/deploy.sh: hard-resets /root/agents-marketplace to
origin/main, maps the old..new diff to affected compose projects
(packages/, patches/, contracts/ and root manifests rebuild all image
builds; the 17 supplier projects share one Dockerfile so builds after
the first are cache hits), builds everything up front, then does a
rolling up -d with a per-project health gate. Suppliers drain first:
/status is polled until not "working" (DRAIN_TIMEOUT_SECS, default
600s) so an in-flight job is not killed and its bond forfeited.
FORCE=1 redeploys the current sha, DRY_RUN=1 previews the plan, and
each deploy logs old -> new to /var/log/marketplace-deploy.log with a
printed rollback one-liner.
Verified: DRY_RUN=1 on the host against the pending 678ee86..6156900
delta selects indexer, buyer, gateway and all 17 suppliers, skips the
non-running wallet-monitor, and leaves image-only projects untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
runOcrOneShot posted the escrow before calling the supplier, so a busy
supplier's 409 stranded an already-paid escrow until reclaim - and the
buyer-side busy-retry multiplies this into several stranded escrows per
page. Add a live /status check per candidate, immediately before its
escrow posts, mirroring sessions.ts's startChat pre-check: "working"
skips to the next candidate without posting, anything else (free,
unreachable, malformed, slow) is treated as usable so the check is an
optimisation, never a gate. Bounded to a 2s timeout so a stalled
supplier cannot meaningfully delay the request.
Existing candidate-fallback, escrow-ref capture, usage recording, and
receipt labels are unchanged.
process.argv[1] arrives with backslashes on Windows, so the
forward-slash-only pattern never matched and `start` exited 0 without
starting the server, and without printing anything. Found while running
the gateway locally to verify this branch's OCR route end to end;
bundled here because it blocks that verification on Windows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
escrowRefStr was only assigned after submitOcr resolved, but submitOcr posts
the escrow internally and THEN calls the supplier over HTTP — a SupplierError
there (timeout, network failure, non-2xx, malformed response, receipt
mismatch) meant the escrow was already on chain while the failure path still
recorded escrow_ref: null. Given the OCR upstream's 120s timeout budget this
is likely the most common paid-but-failed shape, and the exact one the
reconciliation spine exists for.
Subscribe to the SDK's "progress" event (type "escrow_posted") around each
submitOcr attempt so the ref is captured the instant it lands, regardless of
whether submitOcr goes on to resolve or reject. Listener is scoped per
candidate and removed in a finally so it cannot accumulate.
Adds a regression test proving the failure-path usage row and the ocr_failed
structured log line both carry the real ref instead of null; verified it
fails without the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fixes 2 critical, 2 important, and 5 minor findings from review of 069edcb:
- C1: add a happy-path test that reaches postOcrEscrow.ts's escrowLockFloor
call and asserts the locked amount; verified it fails against a bare
economicTotal and passes against the real code.
- C2: gate /v1/ocr/extract (auth + rate limits) before the 16mb body parser
so an unauthenticated caller can no longer force a full buffer/JSON.parse.
- I2: hoist result/used/escrowRef in the OCR one-shot handler so the failure
path records escrow_ref/supplier/cost when an escrow was actually posted,
and emit an ocr_failed structured log line mirroring ocr_settled.
- I3: export venueAttribution and cover its sanitization directly; add a
stubbed happy-path route test asserting the first-party-brokered
disclosure label reaches the response.
- Minor: drop unused hfWeightsSource end to end, rename a misleading test,
make the OCR capability template's pinning language conditional on the
self-hosted upstream, tighten the /v1/ocr path-prefix guard, and make
preflight's required_lovelace use the same escrowLockFloor the builder
locks (exported from the shared tx barrel for gateway use).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the ocr.page.extract.chandra-ocr-2.v1 capability end to end: the
postOcrEscrow builder (locking the min-ada floor for the Submitted-state
datum, required since the min-ada fix), the gateway OCR route that selects
an OCR supplier by capability, posts a per-page escrow, calls the supplier,
verifies and settles, and returns content plus the signed receipt labelled
first-party-brokered. Supplier OCR handler with a hosted document-API
upstream and buyer-SDK OCR support included.
Squashed from the 2026-08-06 working subset; the deploy pack is
intentionally excluded (it lives in a private operator repo).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
OpenAI-compatible compute-backend preset (config only, reuses LLM_BACKEND=openai) for pointing a supplier node at the HuggingFace Inference Providers router. Adds a boot guard that rejects the OpenRouter-only reasoning param against the HF router, unit tests, and an operator runbook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
