OpenAI-compatible supplier backed by the HuggingFace Inference Providers router
(meta-llama/Llama-3.3-70B-Instruct, llm.text.generate.v1). Config only; mirrors
the glm one-off template with OPENAI_BASE_URL=https://router.huggingface.co.
- deploy/mainnet/docker-compose.supplier-hf-llama33.yml
- register the supplier wallet in monitor-wallets OPERATOR_SOURCES
The env file (supplier/.env.hf-llama33) holds the wallet key + HF token and is
never committed; wallet-monitor/wallets.json is gitignored runtime data.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The indexer never updates escrow rows on terminal spends (Accept /
Reclaim / Release consume the UTxO without producing a new escrow
output), so /account summed rows whose funds had long moved and
locked_in_escrow_lovelace over-reported for 35+ minutes after
settlement.
The account handler now verifies each active-state row's UTxO is still
unspent (chain.queryUtxo) before counting it, capped at 32 lookups per
request with the tail counted unverified. Unparseable refs and failed
lookups count as live on purpose: errors inflate the figure rather than
hide locked funds. supplier_bond stays in the sum - the buyer fronts
all three legs at PostEscrow (see postEscrow.ts totalLocked), so it is
the key's money while the escrow lives.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One chain call that never settled, run under the per-key Mutex with no
deadline, convoyed every subsequent request on that key silently while
/models and /account stayed healthy (2026-08-12 incident). The per-fetch
30s aborts in the Ogmios providers predate the freeze and did not prevent
it; the unbounded thing was the composition under the mutex.
Mutex.run() now races each fn against a deadline (default 180s): on
expiry the run rejects with MutexTimeoutError and the chain advances, so
queued work proceeds instead of freezing. A zombie that settles later can
at worst lose a UTxO-contention race with the run that replaced it -
strictly better than a permanent convoy. New work is refused with
MutexQueueFullError past a queue bound (default 64) instead of queueing
without limit. Serialization and rejection-tolerance semantics are
unchanged (covered by tests). Session locks in transcripts.ts construct
the same class and inherit both protections.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Release was the one escrow spend path still hardwired to the synthetic
testTxBody backend, so suppliers had no way to resolve Submitted escrows
past the accept window on mainnet (the sweeper parks them 'for supplier
Release'). Adds buildLiveTxForRelease mirroring the Reclaim builder
(supplier payout, Constr4/tag 125 redeemer, Ogmios UPLC eval) and the
detectCborBackend dispatch in buildReleaseTx, reclaim.ts-style.
validFrom clamps to submitted_at + ACCEPT_WINDOW rather than tip-60s
alone: the back-pad by itself violates the validator whenever the tip is
less than 60s past the window.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tailnet account changed (caslav.nedeljkovic@ to teamhaleight@,
taild99fee.ts.net) and all device IPs were reassigned, breaking both
inference-proxy suppliers' hardcoded backend addresses.
- supplier-local: vuk llama.cpp 100.82.111.46 to 100.77.146.49.
- supplier-openclaw: the openclaw gateway is now exposed via Tailscale
Serve (David's reconfigure set gateway.tailscale.mode=serve, which
forces bind=loopback). Switch OPENAI_BASE_URL from the old raw
http endpoint to https://web-tools.taild99fee.ts.net, and pin that
MagicDNS name to web-tools' tailnet IP (100.105.36.0) via extra_hosts
(Serve's TLS cert is for the hostname, and containers don't inherit
the host tailscale MagicDNS).
- README: new account/IP table, Serve config (replaces the tailnet-bind
section), reachability check (the old ACL blocker is gone), updated
probe/runbook.
Verified: authenticated turn to the Serve endpoint from inference-proxy
returns a completion; vuk:8002 reachable from inference-proxy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second supplier on the inference-proxy box, proxying to an OpenClaw gateway
(agent 'main') on the web-tools KVM guest over the tailnet
(100.101.143.72:18789). Same vuk llama.cpp brain as supplier 'local'; the
product is the agent loop (searxng web search + web fetch + browser).
Capability llm.chat.v1, advertised model 'openclaw-web-agent', 0.2 AP3X,
ticket settle mode, MAX_CHAT_SESSIONS=1.
Uses the new stateful-upstream contract: OPENAI_SESSION_PASSTHROUGH=1 (escrow
ref as OpenAI 'user' → sticky per-escrow openclaw session, delta-only sends)
and OPENAI_MODEL_OVERRIDE=openclaw (endpoint only accepts 'openclaw').
- docker-compose.supplier-openclaw.yml: new supplier compose (traefik host
mp-suppliers-openclaw.vector.apexfusion.org).
- README: gateway config (bind=tailnet, chatCompletions on, GLOBAL tool
allow+deny lockdown), the tailnet-ACL blocker (inference-proxy can't yet
reach web-tools:18789), bring-up runbook, and the sudo-gated browser +
egress-firewall follow-ups.
- monitor-wallets.ts: register supplier-openclaw@inference-proxy.
Wallet generated + funded (200 AP3X); .env.openclaw written on the host.
Not yet advertised — go-live (post-advert) waits on the ACL + a healthy
end-to-end path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two opt-in envs preparing the openclaw supplier on inference-proxy:
OPENAI_SESSION_PASSTHROUGH=1 — chat-session turns speak the stateful-
upstream contract: the escrow ref is sent as the OpenAI `user` field
(OpenClaw keys one persistent agent session per `user`, so browser/
tool state survives across turns) and only the current turn's delta
goes upstream — the upstream session carries history, and re-sending
the full transcript would duplicate it into the upstream context every
turn (OpenClaw renders ALL request messages into the new agent turn).
The local transcript still accumulates fully for the receipt. Without
`user`, OpenClaw creates an ephemeral session per request (uuid key),
so default-off stays safe for stateless providers.
OPENAI_MODEL_OVERRIDE — fixed upstream model id sent instead of the
advert's buyer-facing model. OpenClaw's chat-completions endpoint 400s
on anything but "openclaw" / "openclaw/<agentId>", while the advert
must carry a unique fleet-routable id (openclaw-web-agent).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MAX_CHAT_SESSIONS=1 — vuk is a single-slot ~4.5 tok/s rig, one session
owns it at a time. Idle timeout / settle mode mirror the Hetzner -chat
fleet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New manually-deployed host (62.238.38.167) with its own in-repo traefik
project (LE HTTP-01) and one text supplier proxying to the home rig at
100.82.111.46:8002. Outside CD by design; runbook in the README.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The PR #12 runbook and compose pre-flight headers claimed a wildcard
*.vector.apexfusion.org record covers new supplier hosts. It does not:
each mp-suppliers-* host is an individual DNS-only A record to
91.98.147.172, and on-box Let's Encrypt HTTP-01 issuance requires the
record to be unproxied.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eight new mainnet suppliers backed by Hetzner's OpenAI-compatible
inference API (https://inference.hetzner.com/api). Pure configuration —
the LLM_BACKEND=openai path is reused unchanged, following the
HuggingFace-router precedent:
kimi-code[-chat] Kimi-K2.7-Code (ctx 262144)
glm[-chat] GLM-5.2-NVFP4 (ctx 512000)
ds-flash-htz[-chat] DeepSeek-V4-Flash-0731 (ctx 512000)
qwen35b[-chat] Qwen/Qwen3.6-35B-A3B-FP8 (ctx 262144)
Uniform 0.2 AP3X per job, default 1 AP3X bonds, no upstream max_tokens
(advert cap = model context length), 300s SLA (1800s for chat sessions).
All four models smoke-tested for non-empty content and tools round-trip.
Runbook: docs/HETZNER_SETUP.md. Adverts, wallets, and host env files are
operator steps after merge; first `up -d` is manual (CD skips projects
with no running containers).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep SSH key material in $RUNNER_TEMP instead of ~/.ssh since a
self-hosted runner's home persists across jobs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Merges to main now deploy automatically once CI passes:
- .github/workflows/deploy-mainnet.yml: fires on successful CI runs of
main (or manually via workflow_dispatch with an optional force input),
SSHes into the mainnet host with the DEPLOY_SSH_KEY secret (host key
pinned) and runs deploy/mainnet/deploy.sh at the new commit via
git show FETCH_HEAD:, so deploy logic ships with the code it deploys.
- deploy/mainnet/deploy.sh: hard-resets /root/agents-marketplace to
origin/main, maps the old..new diff to affected compose projects
(packages/, patches/, contracts/ and root manifests rebuild all image
builds; the 17 supplier projects share one Dockerfile so builds after
the first are cache hits), builds everything up front, then does a
rolling up -d with a per-project health gate. Suppliers drain first:
/status is polled until not "working" (DRAIN_TIMEOUT_SECS, default
600s) so an in-flight job is not killed and its bond forfeited.
FORCE=1 redeploys the current sha, DRY_RUN=1 previews the plan, and
each deploy logs old -> new to /var/log/marketplace-deploy.log with a
printed rollback one-liner.
Verified: DRY_RUN=1 on the host against the pending 678ee86..6156900
delta selects indexer, buyer, gateway and all 17 suppliers, skips the
non-running wallet-monitor, and leaves image-only projects untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
runOcrOneShot posted the escrow before calling the supplier, so a busy
supplier's 409 stranded an already-paid escrow until reclaim - and the
buyer-side busy-retry multiplies this into several stranded escrows per
page. Add a live /status check per candidate, immediately before its
escrow posts, mirroring sessions.ts's startChat pre-check: "working"
skips to the next candidate without posting, anything else (free,
unreachable, malformed, slow) is treated as usable so the check is an
optimisation, never a gate. Bounded to a 2s timeout so a stalled
supplier cannot meaningfully delay the request.
Existing candidate-fallback, escrow-ref capture, usage recording, and
receipt labels are unchanged.
process.argv[1] arrives with backslashes on Windows, so the
forward-slash-only pattern never matched and `start` exited 0 without
starting the server, and without printing anything. Found while running
the gateway locally to verify this branch's OCR route end to end;
bundled here because it blocks that verification on Windows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
escrowRefStr was only assigned after submitOcr resolved, but submitOcr posts
the escrow internally and THEN calls the supplier over HTTP — a SupplierError
there (timeout, network failure, non-2xx, malformed response, receipt
mismatch) meant the escrow was already on chain while the failure path still
recorded escrow_ref: null. Given the OCR upstream's 120s timeout budget this
is likely the most common paid-but-failed shape, and the exact one the
reconciliation spine exists for.
Subscribe to the SDK's "progress" event (type "escrow_posted") around each
submitOcr attempt so the ref is captured the instant it lands, regardless of
whether submitOcr goes on to resolve or reject. Listener is scoped per
candidate and removed in a finally so it cannot accumulate.
Adds a regression test proving the failure-path usage row and the ocr_failed
structured log line both carry the real ref instead of null; verified it
fails without the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fixes 2 critical, 2 important, and 5 minor findings from review of 069edcb:
- C1: add a happy-path test that reaches postOcrEscrow.ts's escrowLockFloor
call and asserts the locked amount; verified it fails against a bare
economicTotal and passes against the real code.
- C2: gate /v1/ocr/extract (auth + rate limits) before the 16mb body parser
so an unauthenticated caller can no longer force a full buffer/JSON.parse.
- I2: hoist result/used/escrowRef in the OCR one-shot handler so the failure
path records escrow_ref/supplier/cost when an escrow was actually posted,
and emit an ocr_failed structured log line mirroring ocr_settled.
- I3: export venueAttribution and cover its sanitization directly; add a
stubbed happy-path route test asserting the first-party-brokered
disclosure label reaches the response.
- Minor: drop unused hfWeightsSource end to end, rename a misleading test,
make the OCR capability template's pinning language conditional on the
self-hosted upstream, tighten the /v1/ocr path-prefix guard, and make
preflight's required_lovelace use the same escrowLockFloor the builder
locks (exported from the shared tx barrel for gateway use).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the ocr.page.extract.chandra-ocr-2.v1 capability end to end: the
postOcrEscrow builder (locking the min-ada floor for the Submitted-state
datum, required since the min-ada fix), the gateway OCR route that selects
an OCR supplier by capability, posts a per-page escrow, calls the supplier,
verifies and settles, and returns content plus the signed receipt labelled
first-party-brokered. Supplier OCR handler with a hosted document-API
upstream and buyer-SDK OCR support included.
Squashed from the 2026-08-06 working subset; the deploy pack is
intentionally excluded (it lives in a private operator repo).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
OpenAI-compatible compute-backend preset (config only, reuses LLM_BACKEND=openai) for pointing a supplier node at the HuggingFace Inference Providers router. Adds a boot guard that rejects the OpenRouter-only reasoning param against the HF router, unit tests, and an operator runbook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The lockfile gained patchedDependencies (libsodium-wrappers-sumo) in
2aa1688, and the package manager hashes the patch file during install —
every image build since then fails with ENOENT on /repo/patches. No
image had been rebuilt since, so it surfaced only on the next deploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The escrow locks payment + both bonds and the validator requires exact
value preservation (value_equal) on Claim and Submit. Submit grows the
inline datum ~43 bytes (submitted_at None->Some, result_receipt_hash
None->Some 32 bytes). When capability_id is long enough, the
Submitted-state output's min-ada exceeds the locked total; lucid bumps
the continuing output, value_equal fails, and the validator errors with
no traces (opaque evaluateTransaction 3010/3012). Measured on mainnet at
utxoCostPerByte 4310: a 20-byte chat capability clears a 2.2M lock by
49,310, a 33-byte OCR capability is over by 11,030 - so short-capability
escrows settle and long ones never can, and because Claim's datum is
smaller it always passes, making a Submit-only failure look chain-side.
escrowLockFloor() computes min-ada for the Submitted state plus a margin
and locks max(economic total, that floor) in the post builders. Buyer-
side only, no contract or supplier changes; lucid never bumps once the
lock is sufficient, and the surplus returns through the >= terminal
distribution. postOcrEscrow's identical change ships with the OCR
feature branch. New unit test covers the floor across capability lengths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same pattern as the gateway (a54fdcb): the mp-suppliers-* DNS records are no
longer Cloudflare-proxied, so each supplier gets a websecure router with the
letsencrypt resolver and its :80 router becomes a permanent HTTPS redirect.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
planConsolidate returned already-healthy only for wallets with EXACTLY
2 UTxOs, but buildConsolidateWalletTx lands 3 (collateral + working +
the fee-reserve remainder as change). The shape consolidation produces
can never satisfy its own health check, so every wallet-health tick
re-consolidates and pays the ~0.5 AP3X min fee again. Measured in a
production deployment: a 45s ticker submitted 232 consolidations and
burned ~116 AP3X in about 2.5 idle hours; at the default 600s interval
the same loop leaks ~3 AP3X per hour whenever the wallet is not exactly
2 UTxOs.
Healthy is now structural: a pure-lovelace UTxO >= collateralLovelace
exists, at least 1 AP3X sits outside it, and fragmentation is bounded
(<= 6 UTxOs, new optional maxUtxos param). The health check also runs
BEFORE the low-balance guard, so a wallet that can already run script
transactions is never reshaped or refused. walletHealth gains two
independent circuit breakers - halt after 3 consecutive consolidations
that never reach healthy, halt past 6 consolidations per 24h - so any
future non-convergence costs at most ~1.5 AP3X instead of the whole
wallet, plus a docstring that matches reality.
Tests: the exact post-consolidation-shape regression, one-step
idempotence against the builder output model, candidate boundaries
(native assets, 1,628-lovelace-short), healthy-below-reshape-threshold,
and breaker behavior including error and reset paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
api.marketplace.vector.apexfusion.org is no longer Cloudflare-proxied, so
traefik now terminates TLS itself: a websecure router with the letsencrypt
ACME resolver, and the :80 router demoted to a permanent HTTPS redirect.
Other hosts on the box stay Cloudflare-terminated on :80.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- docs/ARCHITECTURE.md: settlement-asset framing (AP3X bonded in escrow,
settles on accepted work); remove references to external network
adapters that are not part of the project
- docs/buyer + docs/supplier READMEs: commission/bond/settle vocabulary
- Buyer web UI: "Accept & Settle", escrow-framed labels, gateway-managed
wallet wording; gateway landing title punctuation
- Delete EXECUTIVE_SUMMARY and local-llm-marketplace docs + exports:
M1-era internal design snapshots superseded by ARCHITECTURE.md and the
public docs site
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI has been red since the pnpm 10 migration. Four independent causes:
- better-sqlite3: pnpm 10 blocks dependency build scripts by default, so the
native binding was never compiled on CI. Allowlist it via
onlyBuiltDependencies in pnpm-workspace.yaml.
- libsodium-wrappers-sumo 0.7.16: the published ESM wrapper does
`import "./libsodium-sumo.mjs"` but the tarball does not ship that file
(it lives in the libsodium-sumo package). Patch the import to the bare
specifier; local runs only worked because of a hand-copied file in
node_modules.
- tx-accept/claim [live] tests: the live path checks deadlines against
Date.now(), and the fixed fixture timestamps expired in June. Freeze the
clock inside the fixture windows with vi.useFakeTimers.
- CML set_exunits panic ("RuntimeError: unreachable"): with
localUPLCEval: false the provider asks Ogmios for ex-units; the test mocks
answered `{}`, so the provider fell back to a spend redeemer at index 0
while the escrow input sorts to index 1 — CML panics on the unknown
redeemer key. Mock evaluateTransaction with the true index like real
Ogmios.
Also regenerate the advert-datum golden: the fixture builders passed plain JS
numbers to cbor-x (minimal ints, float64 for the timestamp) while the
production codec — whose output is live on mainnet — emits full-width uint64.
Builders now encode integer fields as BigInt so builders, codec, and golden
agree byte-for-byte.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The public Ogmios endpoint only serves a few concurrent ledger-state
queries and returns 500 for the rest, so the Promise.all burst over the
(now 19-wallet) list reliably failed several wallets every run. Query
sequentially instead (full sweep ~2.5s) and retry each wallet once after
2s so a lone blip no longer posts a false "query failed" alert.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
These were live on the box but never committed. Chat-session replicas of
the deepseek-flash suppliers (OpenRouter deepseek-v4-flash, llm.chat.v1,
ticket settle mode); secrets stay in the gitignored supplier/.env files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The signup page said '~5 ADA collateral' and two API error messages
told agents to 'deposit a little more ADA' - the coin is AP3X. Strings,
comments, and docs/gateway.md updated; code identifiers unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slack posts now fire only when a wallet newly drops below its threshold
or recovers above it; a wallet that stays low never re-posts. The
reminder_hours mechanism is removed entirely — per-wallet reminder
clocks drifted apart (Ogmios blips skip wallets from a run), so with
many chronically-low wallets some clock came due nearly every run,
collapsing the 6h cadence to the hourly cron interval.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
