3ee09f1a9a
New manually-deployed host (62.238.38.167) with its own in-repo traefik project (LE HTTP-01) and one text supplier proxying to the home rig at 100.82.111.46:8002. Outside CD by design; runbook in the README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
97 lines
4.0 KiB
Markdown
97 lines
4.0 KiB
Markdown
# inference-proxy box — supplier `local`
|
|
|
|
Compose projects for the **inference-proxy** host (`62.238.38.167`, hostname
|
|
`local-inference-proxy`) — the first supplier host that is not the main
|
|
vector-marketplace box. It fronts a llama.cpp rig (`llamacpp-multinode`) on
|
|
the home machine `vuk`, reachable only over the tailscale tailnet at
|
|
`100.82.111.46:8002`.
|
|
|
|
> **Manual deploys only.** This box is NOT covered by the CD pipeline
|
|
> (`deploy/mainnet/deploy.sh` targets the main box). Update with
|
|
> `git pull && docker compose ... up -d --build`.
|
|
|
|
## Layout
|
|
|
|
| File | Project | Purpose |
|
|
|---|---|---|
|
|
| `docker-compose.traefik.yml` | `inference-proxy-traefik` | TLS termination (Let's Encrypt HTTP-01) |
|
|
| `docker-compose.supplier-local.yml` | `marketplace-mainnet-supplier-local` | Supplier `local`, capability `llm.text.generate.v1` |
|
|
|
|
Host env files (never committed):
|
|
|
|
- `deploy/inference-proxy/.env` — `ACME_EMAIL=<operator email>` for traefik.
|
|
- `supplier/.env.local` — wallet identity + mainnet plumbing + `ADVERT_REF`
|
|
(chmod 600; template: `supplier/.env.hetzner.example`, but backend is
|
|
`OPENAI_BASE_URL=http://100.82.111.46:8002` with no `OPENAI_API_KEY`, and
|
|
`OPENAI_TIMEOUT_MS=3600000`).
|
|
|
|
## Bring-up order (fresh box)
|
|
|
|
```bash
|
|
# 1. docker + network (one-shot)
|
|
curl -fsSL https://get.docker.com | sh
|
|
docker network create traefik-net
|
|
|
|
# 2. repo
|
|
git clone https://github.com/Apex-Fusion/agents-marketplace.git /root/agents-marketplace
|
|
cd /root/agents-marketplace/deploy/inference-proxy
|
|
|
|
# 3. traefik
|
|
echo 'ACME_EMAIL=<operator email>' > .env && chmod 600 .env
|
|
docker compose -f docker-compose.traefik.yml up -d
|
|
|
|
# 4. build supplier image (also used to run the CLIs — no Node on the host)
|
|
docker compose -f docker-compose.supplier-local.yml build
|
|
|
|
# 5. wallet: generate, then write all four values into supplier/.env.local
|
|
docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier \
|
|
node_modules/.bin/tsx supplier/src/cli/gen-keypair.ts --network 1
|
|
# → fund the printed address (~200 AP3X), and set DNS:
|
|
# mp-suppliers-local.vector.apexfusion.org → A 62.238.38.167 (DNS-only/unproxied)
|
|
|
|
# 6. advert (after funding confirmed)
|
|
docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier \
|
|
node_modules/.bin/tsx supplier/src/cli/post-advert.ts \
|
|
--capability-id llm.text.generate.v1 \
|
|
--model <exact id from http://100.82.111.46:8002/v1/models> \
|
|
--max-output-tokens 262144 \
|
|
--max-processing-ms 3600000 \
|
|
--price-lovelace 200000 \
|
|
--endpoint-url https://mp-suppliers-local.vector.apexfusion.org
|
|
# → paste printed "<txHash>#0" into supplier/.env.local as ADVERT_REF
|
|
|
|
# 7. supplier
|
|
docker compose -f docker-compose.supplier-local.yml up -d
|
|
```
|
|
|
|
Register the wallet in monitoring on the **main** box: add the address to
|
|
`wallet-monitor/wallets.json` and `buyer/scripts/monitor-wallets.ts`
|
|
(`OPERATOR_SOURCES`).
|
|
|
|
## Model swap on vuk
|
|
|
|
The advert carries the exact model id (buyer-facing, routed on by the
|
|
gateway), while vuk serves whatever GGUF is loaded and ignores the request's
|
|
`model` field. After swapping the model on vuk:
|
|
|
|
```bash
|
|
cd /root/agents-marketplace/deploy/inference-proxy
|
|
docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier \
|
|
node_modules/.bin/tsx supplier/src/cli/retire-advert.ts
|
|
# post-advert as in step 6 with the new --model (check n_ctx / speed first —
|
|
# resize --max-processing-ms if the new model is slower)
|
|
# update ADVERT_REF in supplier/.env.local, then:
|
|
docker compose -f docker-compose.supplier-local.yml up -d --force-recreate
|
|
```
|
|
|
|
## Caveats
|
|
|
|
- **Availability**: when vuk is off, the indexer prober marks the supplier
|
|
offline and gateway jobs for this model fail. Accepted for a home rig.
|
|
- **Throughput**: ~4.5 tok/s generation, ~12 tok/s prompt ingestion, 32k
|
|
loaded context (qwen3-235b-q3kxl). Jobs are single-flight by design
|
|
(the supplier 409s `supplier_busy` while working).
|
|
- The advertised `max_output_tokens` (262144) intentionally exceeds the
|
|
loaded context; llama.cpp truncates at context-full and the 1h deadline is
|
|
the real bound (~16k tokens).
|