agents-marketplace/deploy/inference-proxy

inference-proxy box — supplier local

Compose projects for the inference-proxy host (62.238.38.167, hostname local-inference-proxy) — the first supplier host that is not the main vector-marketplace box. It fronts a llama.cpp rig (llamacpp-multinode) on the home machine vuk, reachable only over the tailscale tailnet at 100.82.111.46:8002.

Manual deploys only. This box is NOT covered by the CD pipeline (deploy/mainnet/deploy.sh targets the main box). Update with git pull && docker compose ... up -d --build.

Layout

File	Project	Purpose
docker-compose.traefik.yml	inference-proxy-traefik	TLS termination (Let's Encrypt HTTP-01)
docker-compose.supplier-local.yml	marketplace-mainnet-supplier-local	Supplier local, capability llm.text.generate.v1

Host env files (never committed):

• deploy/inference-proxy/.env — ACME_EMAIL=<operator email> for traefik.
• supplier/.env.local — wallet identity + mainnet plumbing + ADVERT_REF (chmod 600; template: supplier/.env.hetzner.example, but backend is OPENAI_BASE_URL=http://100.82.111.46:8002 with no OPENAI_API_KEY, and OPENAI_TIMEOUT_MS=3600000).

Bring-up order (fresh box)

# 1. docker + network (one-shot)
curl -fsSL https://get.docker.com | sh
docker network create traefik-net

# 2. repo
git clone https://github.com/Apex-Fusion/agents-marketplace.git /root/agents-marketplace
cd /root/agents-marketplace/deploy/inference-proxy

# 3. traefik
echo 'ACME_EMAIL=<operator email>' > .env && chmod 600 .env
docker compose -f docker-compose.traefik.yml up -d

# 4. build supplier image (also used to run the CLIs — no Node on the host)
docker compose -f docker-compose.supplier-local.yml build

# 5. wallet: generate, then write all four values into supplier/.env.local
docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier \
  node_modules/.bin/tsx supplier/src/cli/gen-keypair.ts --network 1
# → fund the printed address (~200 AP3X), and set DNS:
#   mp-suppliers-local.vector.apexfusion.org → A 62.238.38.167 (DNS-only/unproxied)

# 6. advert (after funding confirmed)
docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier \
  node_modules/.bin/tsx supplier/src/cli/post-advert.ts \
    --capability-id llm.text.generate.v1 \
    --model <exact id from http://100.82.111.46:8002/v1/models> \
    --max-output-tokens 262144 \
    --max-processing-ms 3600000 \
    --price-lovelace 200000 \
    --endpoint-url https://mp-suppliers-local.vector.apexfusion.org
# → paste printed "<txHash>#0" into supplier/.env.local as ADVERT_REF

# 7. supplier
docker compose -f docker-compose.supplier-local.yml up -d

Register the wallet in monitoring on the main box: add the address to wallet-monitor/wallets.json and buyer/scripts/monitor-wallets.ts (OPERATOR_SOURCES).

Model swap on vuk

The advert carries the exact model id (buyer-facing, routed on by the gateway), while vuk serves whatever GGUF is loaded and ignores the request's model field. After swapping the model on vuk:

cd /root/agents-marketplace/deploy/inference-proxy
docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier \
  node_modules/.bin/tsx supplier/src/cli/retire-advert.ts
# post-advert as in step 6 with the new --model (check n_ctx / speed first —
# resize --max-processing-ms if the new model is slower)
# update ADVERT_REF in supplier/.env.local, then:
docker compose -f docker-compose.supplier-local.yml up -d --force-recreate

Caveats

• Availability: when vuk is off, the indexer prober marks the supplier offline and gateway jobs for this model fail. Accepted for a home rig.
• Throughput: ~4.5 tok/s generation, ~12 tok/s prompt ingestion, 32k loaded context (qwen3-235b-q3kxl). Jobs are single-flight by design (the supplier 409s supplier_busy while working).
• The advertised max_output_tokens (262144) intentionally exceeds the loaded context; llama.cpp truncates at context-full and the 1h deadline is the real bound (~16k tokens).