Generic runbook for deploying a mainnet supplier on any operator-owned
server: own DNS/TLS via the in-repo Traefik compose, docker-run CLI forms
(no host Node), advert posted last as the go-live flag, verification via
the public indexer. Fleet-specific steps stay in HETZNER_SETUP.md.
Selecting a supplier used to render its form at the bottom of the
dashboard, below the card grid — chat sessions in particular ended up
off-screen. The form now renders directly under the page title, above
the grid, and clicking Use scrolls back to the top so the freshly
opened form is visible.
PDF_ENABLED=0 skips the JobStore/PdfJobDb wiring in runMain, so every
/v1/pdf-* route responds 503 (the pre-existing disabled path). Default
stays enabled; only "0"/"1" accepted, misconfig fails boot loudly.
OpenAI-compatible supplier backed by the HuggingFace Inference Providers router
(meta-llama/Llama-3.3-70B-Instruct, llm.text.generate.v1). Config only; mirrors
the glm one-off template with OPENAI_BASE_URL=https://router.huggingface.co.
- deploy/mainnet/docker-compose.supplier-hf-llama33.yml
- register the supplier wallet in monitor-wallets OPERATOR_SOURCES
The env file (supplier/.env.hf-llama33) holds the wallet key + HF token and is
never committed; wallet-monitor/wallets.json is gitignored runtime data.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The indexer never updates escrow rows on terminal spends (Accept /
Reclaim / Release consume the UTxO without producing a new escrow
output), so /account summed rows whose funds had long moved and
locked_in_escrow_lovelace over-reported for 35+ minutes after
settlement.
The account handler now verifies each active-state row's UTxO is still
unspent (chain.queryUtxo) before counting it, capped at 32 lookups per
request with the tail counted unverified. Unparseable refs and failed
lookups count as live on purpose: errors inflate the figure rather than
hide locked funds. supplier_bond stays in the sum - the buyer fronts
all three legs at PostEscrow (see postEscrow.ts totalLocked), so it is
the key's money while the escrow lives.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One chain call that never settled, run under the per-key Mutex with no
deadline, convoyed every subsequent request on that key silently while
/models and /account stayed healthy (2026-08-12 incident). The per-fetch
30s aborts in the Ogmios providers predate the freeze and did not prevent
it; the unbounded thing was the composition under the mutex.
Mutex.run() now races each fn against a deadline (default 180s): on
expiry the run rejects with MutexTimeoutError and the chain advances, so
queued work proceeds instead of freezing. A zombie that settles later can
at worst lose a UTxO-contention race with the run that replaced it -
strictly better than a permanent convoy. New work is refused with
MutexQueueFullError past a queue bound (default 64) instead of queueing
without limit. Serialization and rejection-tolerance semantics are
unchanged (covered by tests). Session locks in transcripts.ts construct
the same class and inherit both protections.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Release was the one escrow spend path still hardwired to the synthetic
testTxBody backend, so suppliers had no way to resolve Submitted escrows
past the accept window on mainnet (the sweeper parks them 'for supplier
Release'). Adds buildLiveTxForRelease mirroring the Reclaim builder
(supplier payout, Constr4/tag 125 redeemer, Ogmios UPLC eval) and the
detectCborBackend dispatch in buildReleaseTx, reclaim.ts-style.
validFrom clamps to submitted_at + ACCEPT_WINDOW rather than tip-60s
alone: the back-pad by itself violates the validator whenever the tip is
less than 60s past the window.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tailnet account changed (caslav.nedeljkovic@ to teamhaleight@,
taild99fee.ts.net) and all device IPs were reassigned, breaking both
inference-proxy suppliers' hardcoded backend addresses.
- supplier-local: vuk llama.cpp 100.82.111.46 to 100.77.146.49.
- supplier-openclaw: the openclaw gateway is now exposed via Tailscale
Serve (David's reconfigure set gateway.tailscale.mode=serve, which
forces bind=loopback). Switch OPENAI_BASE_URL from the old raw
http endpoint to https://web-tools.taild99fee.ts.net, and pin that
MagicDNS name to web-tools' tailnet IP (100.105.36.0) via extra_hosts
(Serve's TLS cert is for the hostname, and containers don't inherit
the host tailscale MagicDNS).
- README: new account/IP table, Serve config (replaces the tailnet-bind
section), reachability check (the old ACL blocker is gone), updated
probe/runbook.
Verified: authenticated turn to the Serve endpoint from inference-proxy
returns a completion; vuk:8002 reachable from inference-proxy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second supplier on the inference-proxy box, proxying to an OpenClaw gateway
(agent 'main') on the web-tools KVM guest over the tailnet
(100.101.143.72:18789). Same vuk llama.cpp brain as supplier 'local'; the
product is the agent loop (searxng web search + web fetch + browser).
Capability llm.chat.v1, advertised model 'openclaw-web-agent', 0.2 AP3X,
ticket settle mode, MAX_CHAT_SESSIONS=1.
Uses the new stateful-upstream contract: OPENAI_SESSION_PASSTHROUGH=1 (escrow
ref as OpenAI 'user' → sticky per-escrow openclaw session, delta-only sends)
and OPENAI_MODEL_OVERRIDE=openclaw (endpoint only accepts 'openclaw').
- docker-compose.supplier-openclaw.yml: new supplier compose (traefik host
mp-suppliers-openclaw.vector.apexfusion.org).
- README: gateway config (bind=tailnet, chatCompletions on, GLOBAL tool
allow+deny lockdown), the tailnet-ACL blocker (inference-proxy can't yet
reach web-tools:18789), bring-up runbook, and the sudo-gated browser +
egress-firewall follow-ups.
- monitor-wallets.ts: register supplier-openclaw@inference-proxy.
Wallet generated + funded (200 AP3X); .env.openclaw written on the host.
Not yet advertised — go-live (post-advert) waits on the ACL + a healthy
end-to-end path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two opt-in envs preparing the openclaw supplier on inference-proxy:
OPENAI_SESSION_PASSTHROUGH=1 — chat-session turns speak the stateful-
upstream contract: the escrow ref is sent as the OpenAI `user` field
(OpenClaw keys one persistent agent session per `user`, so browser/
tool state survives across turns) and only the current turn's delta
goes upstream — the upstream session carries history, and re-sending
the full transcript would duplicate it into the upstream context every
turn (OpenClaw renders ALL request messages into the new agent turn).
The local transcript still accumulates fully for the receipt. Without
`user`, OpenClaw creates an ephemeral session per request (uuid key),
so default-off stays safe for stateless providers.
OPENAI_MODEL_OVERRIDE — fixed upstream model id sent instead of the
advert's buyer-facing model. OpenClaw's chat-completions endpoint 400s
on anything but "openclaw" / "openclaw/<agentId>", while the advert
must carry a unique fleet-routable id (openclaw-web-agent).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MAX_CHAT_SESSIONS=1 — vuk is a single-slot ~4.5 tok/s rig, one session
owns it at a time. Idle timeout / settle mode mirror the Hetzner -chat
fleet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New manually-deployed host (62.238.38.167) with its own in-repo traefik
project (LE HTTP-01) and one text supplier proxying to the home rig at
100.82.111.46:8002. Outside CD by design; runbook in the README.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The PR #12 runbook and compose pre-flight headers claimed a wildcard
*.vector.apexfusion.org record covers new supplier hosts. It does not:
each mp-suppliers-* host is an individual DNS-only A record to
91.98.147.172, and on-box Let's Encrypt HTTP-01 issuance requires the
record to be unproxied.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eight new mainnet suppliers backed by Hetzner's OpenAI-compatible
inference API (https://inference.hetzner.com/api). Pure configuration —
the LLM_BACKEND=openai path is reused unchanged, following the
HuggingFace-router precedent:
kimi-code[-chat] Kimi-K2.7-Code (ctx 262144)
glm[-chat] GLM-5.2-NVFP4 (ctx 512000)
ds-flash-htz[-chat] DeepSeek-V4-Flash-0731 (ctx 512000)
qwen35b[-chat] Qwen/Qwen3.6-35B-A3B-FP8 (ctx 262144)
Uniform 0.2 AP3X per job, default 1 AP3X bonds, no upstream max_tokens
(advert cap = model context length), 300s SLA (1800s for chat sessions).
All four models smoke-tested for non-empty content and tools round-trip.
Runbook: docs/HETZNER_SETUP.md. Adverts, wallets, and host env files are
operator steps after merge; first `up -d` is manual (CD skips projects
with no running containers).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep SSH key material in $RUNNER_TEMP instead of ~/.ssh since a
self-hosted runner's home persists across jobs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
