diff --git a/deploy/inference-proxy/README.md b/deploy/inference-proxy/README.md new file mode 100644 index 0000000..39d57cc --- /dev/null +++ b/deploy/inference-proxy/README.md @@ -0,0 +1,96 @@ +# inference-proxy box — supplier `local` + +Compose projects for the **inference-proxy** host (`62.238.38.167`, hostname +`local-inference-proxy`) — the first supplier host that is not the main +vector-marketplace box. It fronts a llama.cpp rig (`llamacpp-multinode`) on +the home machine `vuk`, reachable only over the tailscale tailnet at +`100.82.111.46:8002`. + +> **Manual deploys only.** This box is NOT covered by the CD pipeline +> (`deploy/mainnet/deploy.sh` targets the main box). Update with +> `git pull && docker compose ... up -d --build`. + +## Layout + +| File | Project | Purpose | +|---|---|---| +| `docker-compose.traefik.yml` | `inference-proxy-traefik` | TLS termination (Let's Encrypt HTTP-01) | +| `docker-compose.supplier-local.yml` | `marketplace-mainnet-supplier-local` | Supplier `local`, capability `llm.text.generate.v1` | + +Host env files (never committed): + +- `deploy/inference-proxy/.env` — `ACME_EMAIL=` for traefik. +- `supplier/.env.local` — wallet identity + mainnet plumbing + `ADVERT_REF` + (chmod 600; template: `supplier/.env.hetzner.example`, but backend is + `OPENAI_BASE_URL=http://100.82.111.46:8002` with no `OPENAI_API_KEY`, and + `OPENAI_TIMEOUT_MS=3600000`). + +## Bring-up order (fresh box) + +```bash +# 1. docker + network (one-shot) +curl -fsSL https://get.docker.com | sh +docker network create traefik-net + +# 2. repo +git clone https://github.com/Apex-Fusion/agents-marketplace.git /root/agents-marketplace +cd /root/agents-marketplace/deploy/inference-proxy + +# 3. traefik +echo 'ACME_EMAIL=' > .env && chmod 600 .env +docker compose -f docker-compose.traefik.yml up -d + +# 4. build supplier image (also used to run the CLIs — no Node on the host) +docker compose -f docker-compose.supplier-local.yml build + +# 5. wallet: generate, then write all four values into supplier/.env.local +docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier \ + node_modules/.bin/tsx supplier/src/cli/gen-keypair.ts --network 1 +# → fund the printed address (~200 AP3X), and set DNS: +# mp-suppliers-local.vector.apexfusion.org → A 62.238.38.167 (DNS-only/unproxied) + +# 6. advert (after funding confirmed) +docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier \ + node_modules/.bin/tsx supplier/src/cli/post-advert.ts \ + --capability-id llm.text.generate.v1 \ + --model \ + --max-output-tokens 262144 \ + --max-processing-ms 3600000 \ + --price-lovelace 200000 \ + --endpoint-url https://mp-suppliers-local.vector.apexfusion.org +# → paste printed "#0" into supplier/.env.local as ADVERT_REF + +# 7. supplier +docker compose -f docker-compose.supplier-local.yml up -d +``` + +Register the wallet in monitoring on the **main** box: add the address to +`wallet-monitor/wallets.json` and `buyer/scripts/monitor-wallets.ts` +(`OPERATOR_SOURCES`). + +## Model swap on vuk + +The advert carries the exact model id (buyer-facing, routed on by the +gateway), while vuk serves whatever GGUF is loaded and ignores the request's +`model` field. After swapping the model on vuk: + +```bash +cd /root/agents-marketplace/deploy/inference-proxy +docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier \ + node_modules/.bin/tsx supplier/src/cli/retire-advert.ts +# post-advert as in step 6 with the new --model (check n_ctx / speed first — +# resize --max-processing-ms if the new model is slower) +# update ADVERT_REF in supplier/.env.local, then: +docker compose -f docker-compose.supplier-local.yml up -d --force-recreate +``` + +## Caveats + +- **Availability**: when vuk is off, the indexer prober marks the supplier + offline and gateway jobs for this model fail. Accepted for a home rig. +- **Throughput**: ~4.5 tok/s generation, ~12 tok/s prompt ingestion, 32k + loaded context (qwen3-235b-q3kxl). Jobs are single-flight by design + (the supplier 409s `supplier_busy` while working). +- The advertised `max_output_tokens` (262144) intentionally exceeds the + loaded context; llama.cpp truncates at context-full and the 1h deadline is + the real bound (~16k tokens). diff --git a/deploy/inference-proxy/docker-compose.supplier-local.yml b/deploy/inference-proxy/docker-compose.supplier-local.yml new file mode 100644 index 0000000..07a9e12 --- /dev/null +++ b/deploy/inference-proxy/docker-compose.supplier-local.yml @@ -0,0 +1,90 @@ +# deploy/inference-proxy/docker-compose.supplier-local.yml — Supplier — MAINNET +# (local llama.cpp on `vuk` via tailscale — one-off `llm.text.generate.v1`) +# +# Runs on the inference-proxy box (62.238.38.167), NOT the main marketplace +# box. The compute backend is a llama.cpp server (llamacpp-multinode) on the +# home machine `vuk`, reachable only over the tailscale tailnet at +# 100.82.111.46:8002 (OpenAI-compatible /v1/chat/completions, no auth — the +# tailnet is the trust boundary). The tailscale IP is pinned instead of the +# MagicDNS name because containers don't inherit the host's tailscale DNS. +# +# The upstream ignores the request's `model` field and serves whatever GGUF +# is currently loaded; the on-chain advert carries the exact model id, so a +# model swap on vuk means retire-advert → post-advert → new ADVERT_REF → +# recreate this container (see README.md). +# +# Pre-flight: +# 1. DNS: mp-suppliers-local.vector.apexfusion.org → 62.238.38.167 +# (individual DNS-only A record; letsencrypt HTTP-01 issuance needs the +# unproxied direct record) +# 2. supplier/.env.local written on host (chmod 600) with at minimum: +# SUPPLIER_PRIV_KEY_HEX, SUPPLIER_ADDRESS, SUPPLIER_PKH, +# SUPPLIER_PUB_KEY_HEX, OGMIOS_URL, NETWORK_ID=1, +# VECTOR_ZERO_TIME_MS, ESCROW_REF_UTXO, ADVERT_REF_UTXO, ADVERT_REF, +# LIVE_CHAIN=1 +# (no OPENAI_API_KEY — vuk's endpoint is unauthenticated) +# 3. Networks: traefik-net is created one-shot (see the traefik compose in +# this directory). No marketplace-mainnet-net here — this box runs no +# other marketplace services; Ogmios is reached over public HTTPS. + +name: marketplace-mainnet-supplier-local + +services: + supplier: + build: + context: ../.. + dockerfile: supplier/Dockerfile + container_name: marketplace-mainnet-supplier-local + restart: unless-stopped + env_file: + # Self-contained env for this supplier: wallet identity, ADVERT_REF, + # and the shared mainnet plumbing duplicated in full (NETWORK_ID, + # OGMIOS_URL, VECTOR_ZERO_TIME_MS, ref-script UTxOs, LIVE_CHAIN). + - path: ../../supplier/.env.local + required: false + environment: + LLM_BACKEND: openai + # No /v1 suffix — supplier/src/openai.ts appends /v1/chat/completions. + # vuk's tailscale IP; see header for why not the MagicDNS name. + OPENAI_BASE_URL: http://100.82.111.46:8002 + # ~4.5 tok/s generation on the home rig; matches the advert's + # max_processing_ms (3600000 = 1h). + OPENAI_TIMEOUT_MS: "3600000" + # OPENAI_MAX_TOKENS intentionally unset: no max_tokens forwarded + # upstream; llama.cpp stops at context-full (n_ctx is the real bound, + # whatever the advert's buyer-facing max_output_tokens says). + PORT: "8080" + # Enable LiveOgmiosProvider so submitTx/awaitTx actually broadcast. + LIVE_CHAIN: "1" + # NETWORK_ID and OGMIOS_URL intentionally not pinned here — they MUST + # come from the operator-provided supplier/.env.local on a mainnet host. + healthcheck: + test: ["CMD-SHELL", "wget -qO- http://localhost:8080/healthz >/dev/null || exit 1"] + interval: 20s + timeout: 5s + retries: 3 + start_period: 15s + networks: + - traefik-net + labels: + - "traefik.enable=true" + - "traefik.docker.network=traefik-net" + - "traefik.http.routers.mp-suppliers-local-mainnet.entrypoints=web" + - "traefik.http.routers.mp-suppliers-local-mainnet.rule=Host(`mp-suppliers-local.vector.apexfusion.org`)" + - "traefik.http.routers.mp-suppliers-local-mainnet.middlewares=mp-suppliers-local-mainnet-https" + - "traefik.http.middlewares.mp-suppliers-local-mainnet-https.redirectscheme.scheme=https" + - "traefik.http.middlewares.mp-suppliers-local-mainnet-https.redirectscheme.permanent=true" + - "traefik.http.routers.mp-suppliers-local-mainnet-tls.entrypoints=websecure" + - "traefik.http.routers.mp-suppliers-local-mainnet-tls.rule=Host(`mp-suppliers-local.vector.apexfusion.org`)" + - "traefik.http.routers.mp-suppliers-local-mainnet-tls.tls.certresolver=letsencrypt" + - "traefik.http.services.mp-suppliers-local-mainnet.loadbalancer.server.port=8080" + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3" + +networks: + traefik-net: + external: true + name: traefik-net diff --git a/deploy/inference-proxy/docker-compose.traefik.yml b/deploy/inference-proxy/docker-compose.traefik.yml new file mode 100644 index 0000000..89ce6ec --- /dev/null +++ b/deploy/inference-proxy/docker-compose.traefik.yml @@ -0,0 +1,61 @@ +# deploy/inference-proxy/docker-compose.traefik.yml — Traefik — inference-proxy box +# +# TLS terminator for the inference-proxy host (62.238.38.167) — the first +# supplier host that is NOT the main vector-marketplace box. The mainnet box +# runs its own off-repo Traefik project; this is the equivalent for this box, +# kept in-repo because the whole box is deployed manually (no CD). +# +# Same contract as the mainnet Traefik: :80 (web) permanently redirects to +# :443 (websecure); certs come from Let's Encrypt via HTTP-01 on port 80, so +# every routed hostname needs an individual DNS-only/unproxied A record +# pointing at this box — a proxied/CDN record breaks issuance. +# +# Pre-flight: +# 1. docker network create traefik-net # one-shot +# 2. Write deploy/inference-proxy/.env with: +# ACME_EMAIL= # not committed — repo is public +# 3. docker compose --env-file .env -f docker-compose.traefik.yml up -d +# (run from this directory so the .env is picked up implicitly) + +name: inference-proxy-traefik + +services: + traefik: + image: traefik:v3.3 + container_name: inference-proxy-traefik + restart: unless-stopped + command: + - "--providers.docker=true" + - "--providers.docker.exposedbydefault=false" + - "--providers.docker.network=traefik-net" + - "--entrypoints.web.address=:80" + - "--entrypoints.web.http.redirections.entrypoint.to=websecure" + - "--entrypoints.web.http.redirections.entrypoint.scheme=https" + - "--entrypoints.web.http.redirections.entrypoint.permanent=true" + - "--entrypoints.websecure.address=:443" + - "--certificatesresolvers.letsencrypt.acme.email=${ACME_EMAIL:?set ACME_EMAIL in deploy/inference-proxy/.env}" + - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json" + - "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web" + - "--log.level=INFO" + - "--accesslog=true" + ports: + - "80:80" + - "443:443" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - letsencrypt:/letsencrypt + networks: + - traefik-net + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3" + +volumes: + letsencrypt: + +networks: + traefik-net: + external: true + name: traefik-net