feat(deploy): inference-proxy box — supplier 'local' backed by llama.cpp on vuk via tailscale

New manually-deployed host (62.238.38.167) with its own in-repo traefik project (LE HTTP-01) and one text supplier proxying to the home rig at 100.82.111.46:8002. Outside CD by design; runbook in the README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-12 10:28:13 +02:00
parent 44abbdffdc
commit 3ee09f1a9a
3 changed files with 247 additions and 0 deletions
--- a/deploy/inference-proxy/README.md
+++ b/deploy/inference-proxy/README.md
@@ -0,0 +1,96 @@
+# inference-proxy box — supplier `local`
+
+Compose projects for the **inference-proxy** host (`62.238.38.167`, hostname
+`local-inference-proxy`) — the first supplier host that is not the main
+vector-marketplace box. It fronts a llama.cpp rig (`llamacpp-multinode`) on
+the home machine `vuk`, reachable only over the tailscale tailnet at
+`100.82.111.46:8002`.
+
+> **Manual deploys only.** This box is NOT covered by the CD pipeline
+> (`deploy/mainnet/deploy.sh` targets the main box). Update with
+> `git pull && docker compose ... up -d --build`.
+
+## Layout
+
+| File | Project | Purpose |
+|---|---|---|
+| `docker-compose.traefik.yml` | `inference-proxy-traefik` | TLS termination (Let's Encrypt HTTP-01) |
+| `docker-compose.supplier-local.yml` | `marketplace-mainnet-supplier-local` | Supplier `local`, capability `llm.text.generate.v1` |
+
+Host env files (never committed):
+
+- `deploy/inference-proxy/.env` — `ACME_EMAIL=<operator email>` for traefik.
+- `supplier/.env.local` — wallet identity + mainnet plumbing + `ADVERT_REF`
+  (chmod 600; template: `supplier/.env.hetzner.example`, but backend is
+  `OPENAI_BASE_URL=http://100.82.111.46:8002` with no `OPENAI_API_KEY`, and
+  `OPENAI_TIMEOUT_MS=3600000`).
+
+## Bring-up order (fresh box)
+
+```bash
+# 1. docker + network (one-shot)
+curl -fsSL https://get.docker.com | sh
+docker network create traefik-net
+
+# 2. repo
+git clone https://github.com/Apex-Fusion/agents-marketplace.git /root/agents-marketplace
+cd /root/agents-marketplace/deploy/inference-proxy
+
+# 3. traefik
+echo 'ACME_EMAIL=<operator email>' > .env && chmod 600 .env
+docker compose -f docker-compose.traefik.yml up -d
+
+# 4. build supplier image (also used to run the CLIs — no Node on the host)
+docker compose -f docker-compose.supplier-local.yml build
+
+# 5. wallet: generate, then write all four values into supplier/.env.local
+docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier \
+  node_modules/.bin/tsx supplier/src/cli/gen-keypair.ts --network 1
+# → fund the printed address (~200 AP3X), and set DNS:
+#   mp-suppliers-local.vector.apexfusion.org → A 62.238.38.167 (DNS-only/unproxied)
+
+# 6. advert (after funding confirmed)
+docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier \
+  node_modules/.bin/tsx supplier/src/cli/post-advert.ts \
+    --capability-id llm.text.generate.v1 \
+    --model <exact id from http://100.82.111.46:8002/v1/models> \
+    --max-output-tokens 262144 \
+    --max-processing-ms 3600000 \
+    --price-lovelace 200000 \
+    --endpoint-url https://mp-suppliers-local.vector.apexfusion.org
+# → paste printed "<txHash>#0" into supplier/.env.local as ADVERT_REF
+
+# 7. supplier
+docker compose -f docker-compose.supplier-local.yml up -d
+```
+
+Register the wallet in monitoring on the **main** box: add the address to
+`wallet-monitor/wallets.json` and `buyer/scripts/monitor-wallets.ts`
+(`OPERATOR_SOURCES`).
+
+## Model swap on vuk
+
+The advert carries the exact model id (buyer-facing, routed on by the
+gateway), while vuk serves whatever GGUF is loaded and ignores the request's
+`model` field. After swapping the model on vuk:
+
+```bash
+cd /root/agents-marketplace/deploy/inference-proxy
+docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier \
+  node_modules/.bin/tsx supplier/src/cli/retire-advert.ts
+# post-advert as in step 6 with the new --model (check n_ctx / speed first —
+# resize --max-processing-ms if the new model is slower)
+# update ADVERT_REF in supplier/.env.local, then:
+docker compose -f docker-compose.supplier-local.yml up -d --force-recreate
+```
+
+## Caveats
+
+- **Availability**: when vuk is off, the indexer prober marks the supplier
+  offline and gateway jobs for this model fail. Accepted for a home rig.
+- **Throughput**: ~4.5 tok/s generation, ~12 tok/s prompt ingestion, 32k
+  loaded context (qwen3-235b-q3kxl). Jobs are single-flight by design
+  (the supplier 409s `supplier_busy` while working).
+- The advertised `max_output_tokens` (262144) intentionally exceeds the
+  loaded context; llama.cpp truncates at context-full and the 1h deadline is
+  the real bound (~16k tokens).
--- a/deploy/inference-proxy/docker-compose.supplier-local.yml
+++ b/deploy/inference-proxy/docker-compose.supplier-local.yml
@@ -0,0 +1,90 @@
+# deploy/inference-proxy/docker-compose.supplier-local.yml — Supplier — MAINNET
+#   (local llama.cpp on `vuk` via tailscale — one-off `llm.text.generate.v1`)
+#
+# Runs on the inference-proxy box (62.238.38.167), NOT the main marketplace
+# box. The compute backend is a llama.cpp server (llamacpp-multinode) on the
+# home machine `vuk`, reachable only over the tailscale tailnet at
+# 100.82.111.46:8002 (OpenAI-compatible /v1/chat/completions, no auth — the
+# tailnet is the trust boundary). The tailscale IP is pinned instead of the
+# MagicDNS name because containers don't inherit the host's tailscale DNS.
+#
+# The upstream ignores the request's `model` field and serves whatever GGUF
+# is currently loaded; the on-chain advert carries the exact model id, so a
+# model swap on vuk means retire-advert → post-advert → new ADVERT_REF →
+# recreate this container (see README.md).
+#
+# Pre-flight:
+#   1. DNS: mp-suppliers-local.vector.apexfusion.org → 62.238.38.167
+#      (individual DNS-only A record; letsencrypt HTTP-01 issuance needs the
+#       unproxied direct record)
+#   2. supplier/.env.local written on host (chmod 600) with at minimum:
+#        SUPPLIER_PRIV_KEY_HEX, SUPPLIER_ADDRESS, SUPPLIER_PKH,
+#        SUPPLIER_PUB_KEY_HEX, OGMIOS_URL, NETWORK_ID=1,
+#        VECTOR_ZERO_TIME_MS, ESCROW_REF_UTXO, ADVERT_REF_UTXO, ADVERT_REF,
+#        LIVE_CHAIN=1
+#      (no OPENAI_API_KEY — vuk's endpoint is unauthenticated)
+#   3. Networks: traefik-net is created one-shot (see the traefik compose in
+#      this directory). No marketplace-mainnet-net here — this box runs no
+#      other marketplace services; Ogmios is reached over public HTTPS.
+
+name: marketplace-mainnet-supplier-local
+
+services:
+  supplier:
+    build:
+      context: ../..
+      dockerfile: supplier/Dockerfile
+    container_name: marketplace-mainnet-supplier-local
+    restart: unless-stopped
+    env_file:
+      # Self-contained env for this supplier: wallet identity, ADVERT_REF,
+      # and the shared mainnet plumbing duplicated in full (NETWORK_ID,
+      # OGMIOS_URL, VECTOR_ZERO_TIME_MS, ref-script UTxOs, LIVE_CHAIN).
+      - path: ../../supplier/.env.local
+        required: false
+    environment:
+      LLM_BACKEND: openai
+      # No /v1 suffix — supplier/src/openai.ts appends /v1/chat/completions.
+      # vuk's tailscale IP; see header for why not the MagicDNS name.
+      OPENAI_BASE_URL: http://100.82.111.46:8002
+      # ~4.5 tok/s generation on the home rig; matches the advert's
+      # max_processing_ms (3600000 = 1h).
+      OPENAI_TIMEOUT_MS: "3600000"
+      # OPENAI_MAX_TOKENS intentionally unset: no max_tokens forwarded
+      # upstream; llama.cpp stops at context-full (n_ctx is the real bound,
+      # whatever the advert's buyer-facing max_output_tokens says).
+      PORT: "8080"
+      # Enable LiveOgmiosProvider so submitTx/awaitTx actually broadcast.
+      LIVE_CHAIN: "1"
+      # NETWORK_ID and OGMIOS_URL intentionally not pinned here — they MUST
+      # come from the operator-provided supplier/.env.local on a mainnet host.
+    healthcheck:
+      test: ["CMD-SHELL", "wget -qO- http://localhost:8080/healthz >/dev/null || exit 1"]
+      interval: 20s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
+    networks:
+      - traefik-net
+    labels:
+      - "traefik.enable=true"
+      - "traefik.docker.network=traefik-net"
+      - "traefik.http.routers.mp-suppliers-local-mainnet.entrypoints=web"
+      - "traefik.http.routers.mp-suppliers-local-mainnet.rule=Host(`mp-suppliers-local.vector.apexfusion.org`)"
+      - "traefik.http.routers.mp-suppliers-local-mainnet.middlewares=mp-suppliers-local-mainnet-https"
+      - "traefik.http.middlewares.mp-suppliers-local-mainnet-https.redirectscheme.scheme=https"
+      - "traefik.http.middlewares.mp-suppliers-local-mainnet-https.redirectscheme.permanent=true"
+      - "traefik.http.routers.mp-suppliers-local-mainnet-tls.entrypoints=websecure"
+      - "traefik.http.routers.mp-suppliers-local-mainnet-tls.rule=Host(`mp-suppliers-local.vector.apexfusion.org`)"
+      - "traefik.http.routers.mp-suppliers-local-mainnet-tls.tls.certresolver=letsencrypt"
+      - "traefik.http.services.mp-suppliers-local-mainnet.loadbalancer.server.port=8080"
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+
+networks:
+  traefik-net:
+    external: true
+    name: traefik-net
--- a/deploy/inference-proxy/docker-compose.traefik.yml
+++ b/deploy/inference-proxy/docker-compose.traefik.yml
@@ -0,0 +1,61 @@
+# deploy/inference-proxy/docker-compose.traefik.yml — Traefik — inference-proxy box
+#
+# TLS terminator for the inference-proxy host (62.238.38.167) — the first
+# supplier host that is NOT the main vector-marketplace box. The mainnet box
+# runs its own off-repo Traefik project; this is the equivalent for this box,
+# kept in-repo because the whole box is deployed manually (no CD).
+#
+# Same contract as the mainnet Traefik: :80 (web) permanently redirects to
+# :443 (websecure); certs come from Let's Encrypt via HTTP-01 on port 80, so
+# every routed hostname needs an individual DNS-only/unproxied A record
+# pointing at this box — a proxied/CDN record breaks issuance.
+#
+# Pre-flight:
+#   1. docker network create traefik-net        # one-shot
+#   2. Write deploy/inference-proxy/.env with:
+#        ACME_EMAIL=<operator email>            # not committed — repo is public
+#   3. docker compose --env-file .env -f docker-compose.traefik.yml up -d
+#      (run from this directory so the .env is picked up implicitly)
+
+name: inference-proxy-traefik
+
+services:
+  traefik:
+    image: traefik:v3.3
+    container_name: inference-proxy-traefik
+    restart: unless-stopped
+    command:
+      - "--providers.docker=true"
+      - "--providers.docker.exposedbydefault=false"
+      - "--providers.docker.network=traefik-net"
+      - "--entrypoints.web.address=:80"
+      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
+      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
+      - "--entrypoints.web.http.redirections.entrypoint.permanent=true"
+      - "--entrypoints.websecure.address=:443"
+      - "--certificatesresolvers.letsencrypt.acme.email=${ACME_EMAIL:?set ACME_EMAIL in deploy/inference-proxy/.env}"
+      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
+      - "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web"
+      - "--log.level=INFO"
+      - "--accesslog=true"
+    ports:
+      - "80:80"
+      - "443:443"
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - letsencrypt:/letsencrypt
+    networks:
+      - traefik-net
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+
+volumes:
+  letsencrypt:
+
+networks:
+  traefik-net:
+    external: true
+    name: traefik-net