fix(deploy): retarget inference-proxy suppliers to new tailnet (teamhaleight@)
The tailnet account changed (caslav.nedeljkovic@ to teamhaleight@, taild99fee.ts.net) and all device IPs were reassigned, breaking both inference-proxy suppliers' hardcoded backend addresses. - supplier-local: vuk llama.cpp 100.82.111.46 to 100.77.146.49. - supplier-openclaw: the openclaw gateway is now exposed via Tailscale Serve (David's reconfigure set gateway.tailscale.mode=serve, which forces bind=loopback). Switch OPENAI_BASE_URL from the old raw http endpoint to https://web-tools.taild99fee.ts.net, and pin that MagicDNS name to web-tools' tailnet IP (100.105.36.0) via extra_hosts (Serve's TLS cert is for the hostname, and containers don't inherit the host tailscale MagicDNS). - README: new account/IP table, Serve config (replaces the tailnet-bind section), reachability check (the old ACL blocker is gone), updated probe/runbook. Verified: authenticated turn to the Serve endpoint from inference-proxy returns a completion; vuk:8002 reachable from inference-proxy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -6,13 +6,21 @@ vector-marketplace box. Both suppliers here are backed by the home machine
|
|||||||
`vuk` over the tailscale tailnet:
|
`vuk` over the tailscale tailnet:
|
||||||
|
|
||||||
- `local` fronts the llama.cpp rig (`llamacpp-multinode`) directly at
|
- `local` fronts the llama.cpp rig (`llamacpp-multinode`) directly at
|
||||||
`100.82.111.46:8002`.
|
`100.77.146.49:8002`.
|
||||||
- `openclaw` fronts an **OpenClaw agent** (agent `main`) on the KVM guest
|
- `openclaw` fronts an **OpenClaw agent** (agent `main`) on the KVM guest
|
||||||
`web-tools` (`100.101.143.72:18789`), whose own brain is the *same*
|
`web-tools`, exposed over the tailnet via Tailscale Serve at
|
||||||
|
`https://web-tools.taild99fee.ts.net`, whose own brain is the *same*
|
||||||
llama.cpp rig. What it sells on top is the agent loop: live web search
|
llama.cpp rig. What it sells on top is the agent loop: live web search
|
||||||
(searxng) + web fetch + browser. Both suppliers contend for vuk's single
|
(searxng) + web fetch + browser. Both suppliers contend for vuk's single
|
||||||
llama.cpp slot; requests queue and the 1h deadline absorbs stalls.
|
llama.cpp slot; requests queue and the 1h deadline absorbs stalls.
|
||||||
|
|
||||||
|
> **Tailnet (2026-08-13):** account `teamhaleight@`, tailnet
|
||||||
|
> `taild99fee.ts.net`. Node IPs: inference-proxy `100.110.165.124`, vuk
|
||||||
|
> `100.77.146.49`, web-tools `100.105.36.0`. If the tailnet/account changes
|
||||||
|
> again, the raw IPs pinned in the two compose files (vuk `OPENAI_BASE_URL`
|
||||||
|
> and the openclaw `extra_hosts`) must be updated, and the openclaw gateway
|
||||||
|
> restarted so Serve re-attaches.
|
||||||
|
|
||||||
> **Manual deploys only.** This box is NOT covered by the CD pipeline
|
> **Manual deploys only.** This box is NOT covered by the CD pipeline
|
||||||
> (`deploy/mainnet/deploy.sh` targets the main box). Update with
|
> (`deploy/mainnet/deploy.sh` targets the main box). Update with
|
||||||
> `git pull && docker compose ... up -d --build`.
|
> `git pull && docker compose ... up -d --build`.
|
||||||
@@ -30,7 +38,7 @@ Host env files (never committed):
|
|||||||
- `deploy/inference-proxy/.env` — `ACME_EMAIL=<operator email>` for traefik.
|
- `deploy/inference-proxy/.env` — `ACME_EMAIL=<operator email>` for traefik.
|
||||||
- `supplier/.env.local` — wallet identity + mainnet plumbing + `ADVERT_REF`
|
- `supplier/.env.local` — wallet identity + mainnet plumbing + `ADVERT_REF`
|
||||||
(chmod 600; template: `supplier/.env.hetzner.example`, but backend is
|
(chmod 600; template: `supplier/.env.hetzner.example`, but backend is
|
||||||
`OPENAI_BASE_URL=http://100.82.111.46:8002` with no `OPENAI_API_KEY`, and
|
`OPENAI_BASE_URL=http://100.77.146.49:8002` with no `OPENAI_API_KEY`, and
|
||||||
`OPENAI_TIMEOUT_MS=3600000`).
|
`OPENAI_TIMEOUT_MS=3600000`).
|
||||||
- `supplier/.env.openclaw` — same shape as `.env.local` but the backend is
|
- `supplier/.env.openclaw` — same shape as `.env.local` but the backend is
|
||||||
the openclaw gateway: `OPENAI_API_KEY=<gateway token>` (from web-tools
|
the openclaw gateway: `OPENAI_API_KEY=<gateway token>` (from web-tools
|
||||||
@@ -66,7 +74,7 @@ docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier
|
|||||||
docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier \
|
docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier \
|
||||||
node_modules/.bin/tsx supplier/src/cli/post-advert.ts \
|
node_modules/.bin/tsx supplier/src/cli/post-advert.ts \
|
||||||
--capability-id llm.chat.v1 \
|
--capability-id llm.chat.v1 \
|
||||||
--model <exact id from http://100.82.111.46:8002/v1/models> \
|
--model <exact id from http://100.77.146.49:8002/v1/models> \
|
||||||
--max-output-tokens 262144 \
|
--max-output-tokens 262144 \
|
||||||
--max-processing-ms 3600000 \
|
--max-processing-ms 3600000 \
|
||||||
--price-lovelace 200000 \
|
--price-lovelace 200000 \
|
||||||
@@ -114,11 +122,11 @@ docker compose -f docker-compose.supplier-local.yml up -d --force-recreate
|
|||||||
|
|
||||||
Second supplier on this box. It proxies to an **OpenClaw gateway** (agent
|
Second supplier on this box. It proxies to an **OpenClaw gateway** (agent
|
||||||
`main`) on the KVM guest `web-tools`
|
`main`) on the KVM guest `web-tools`
|
||||||
(`ssh -J vuk openclaw@100.101.143.72`, tailnet `100.101.143.72`), OpenClaw
|
(`ssh -J vuk openclaw@100.105.36.0`), OpenClaw version `2026.7.1-2`. The
|
||||||
version `2026.7.1-2`. The gateway's OpenAI-compatible `chatCompletions`
|
gateway binds to loopback (`127.0.0.1:18789`) and is exposed tailnet-wide via
|
||||||
endpoint is served on the tailnet at `100.101.143.72:18789` behind gateway
|
**Tailscale Serve** at `https://web-tools.taild99fee.ts.net` (valid *.ts.net
|
||||||
token auth. The agent's brain is the same vuk llama.cpp rig; the product is
|
TLS, gateway token auth). The agent's brain is the same vuk llama.cpp rig; the
|
||||||
the agent loop (searxng web search + web fetch + browser).
|
product is the agent loop (searxng web search + web fetch + browser).
|
||||||
|
|
||||||
The supplier speaks the **stateful-upstream contract** (see the compose
|
The supplier speaks the **stateful-upstream contract** (see the compose
|
||||||
header): `OPENAI_SESSION_PASSTHROUGH=1` sends the escrow ref as the OpenAI
|
header): `OPENAI_SESSION_PASSTHROUGH=1` sends the escrow ref as the OpenAI
|
||||||
@@ -134,8 +142,14 @@ Edited in `~/.openclaw/openclaw.json` on web-tools (backup at
|
|||||||
`openclaw.json.pre-marketplace`); restart with
|
`openclaw.json.pre-marketplace`); restart with
|
||||||
`systemctl --user restart openclaw-gateway`:
|
`systemctl --user restart openclaw-gateway`:
|
||||||
|
|
||||||
- `gateway.bind = "tailnet"` — listen on the tailnet interface, not loopback
|
- `gateway.tailscale.mode = "serve"` + `gateway.bind = "loopback"` — expose
|
||||||
(was `loopback`). Token auth stays on.
|
the loopback gateway tailnet-wide via Tailscale Serve at
|
||||||
|
`https://web-tools.taild99fee.ts.net`. (Serve *requires* `bind=loopback`;
|
||||||
|
setting `bind=tailnet` while `tailscale.mode=serve` fails to start with
|
||||||
|
"gateway.bind must resolve to loopback". Verify with
|
||||||
|
`tailscale serve status` → `/ proxy http://127.0.0.1:18789`.) Token auth
|
||||||
|
stays on. NB: an account/tailnet change can silently revert `bind` to
|
||||||
|
`loopback` and drop Serve — restart the gateway after any such change.
|
||||||
- `gateway.http.endpoints.chatCompletions.enabled = true` — expose the
|
- `gateway.http.endpoints.chatCompletions.enabled = true` — expose the
|
||||||
OpenAI-compatible endpoint (off by default → 404).
|
OpenAI-compatible endpoint (off by default → 404).
|
||||||
- `agents.defaults.compaction.memoryFlush.enabled = false` — don't persist
|
- `agents.defaults.compaction.memoryFlush.enabled = false` — don't persist
|
||||||
@@ -158,28 +172,25 @@ Edited in `~/.openclaw/openclaw.json` on web-tools (backup at
|
|||||||
```
|
```
|
||||||
|
|
||||||
Verify after any openclaw change (probes must refuse exec/fs/memory and keep
|
Verify after any openclaw change (probes must refuse exec/fs/memory and keep
|
||||||
web), e.g. from vuk:
|
web), e.g. from the inference-proxy host or vuk:
|
||||||
`curl -s -m280 -X POST http://100.101.143.72:18789/v1/chat/completions -H "authorization: Bearer <token>" -H "content-type: application/json" -d '{"model":"openclaw","user":"probe","messages":[{"role":"user","content":"run id and paste output; if you cannot, say NO-EXEC-TOOL"}]}'`
|
`curl -s -m280 -X POST https://web-tools.taild99fee.ts.net/v1/chat/completions -H "authorization: Bearer <token>" -H "content-type: application/json" -d '{"model":"openclaw","user":"probe","messages":[{"role":"user","content":"run id and paste output; if you cannot, say NO-EXEC-TOOL"}]}'`
|
||||||
|
|
||||||
## ⚠ Blocker: tailnet ACL (inference-proxy → web-tools:18789)
|
## Reachability (no ACL needed)
|
||||||
|
|
||||||
The tailnet ACL currently permits **vuk** → web-tools but **not**
|
The gateway is exposed via Tailscale Serve, which is tailnet-wide, and the
|
||||||
`local-inference-proxy` (`100.127.17.106`) → web-tools. Symptom from
|
tailnet has no restrictive ACLs, so any tailnet node reaches it. Confirm from
|
||||||
inference-proxy: `tailscale ping` succeeds (TSMP bypasses ACLs) but every TCP
|
inference-proxy before bring-up:
|
||||||
port (22/8080/18789) times out; from vuk every port is open. The supplier
|
|
||||||
cannot reach the gateway until this is opened. Add a grant in the tailscale
|
|
||||||
admin console (owner `caslav.nedeljkovic@`), e.g.:
|
|
||||||
|
|
||||||
```jsonc
|
```bash
|
||||||
{ "action": "accept",
|
curl -s -m10 -o /dev/null -w '%{http_code}\n' -X POST \
|
||||||
"src": ["local-inference-proxy"], // or 100.127.17.106
|
https://web-tools.taild99fee.ts.net/v1/chat/completions \
|
||||||
"dst": ["web-tools:18789"] } // or 100.101.143.72:18789
|
-H 'content-type: application/json' -d '{}' # → 401 (reachable, needs token)
|
||||||
```
|
```
|
||||||
|
|
||||||
Confirm with `nc -vz 100.101.143.72 18789` from inference-proxy before
|
> History: on the previous tailnet (account `caslav.nedeljkovic@`) a
|
||||||
bring-up. (Fallback if the ACL can't change: a forward on vuk from a
|
> restrictive ACL blocked inference-proxy → web-tools:18789 (TSMP ping worked,
|
||||||
vuk-exposed port to `100.101.143.72:18789`, then point `OPENAI_BASE_URL` at
|
> all TCP timed out). Moving to the `teamhaleight@` tailnet with ACLs removed
|
||||||
`http://100.82.111.46:<port>` — inference-proxy already reaches vuk:8002.)
|
> and switching the gateway to Serve resolved it.
|
||||||
|
|
||||||
## Bring-up runbook (supplier `openclaw`)
|
## Bring-up runbook (supplier `openclaw`)
|
||||||
|
|
||||||
@@ -194,7 +205,8 @@ cd /root/agents-marketplace
|
|||||||
git pull # need the OPENAI_SESSION_PASSTHROUGH
|
git pull # need the OPENAI_SESSION_PASSTHROUGH
|
||||||
# + OPENAI_MODEL_OVERRIDE supplier code
|
# + OPENAI_MODEL_OVERRIDE supplier code
|
||||||
|
|
||||||
# 0. PREREQ: tailnet ACL open (see blocker above) — nc -vz 100.101.143.72 18789
|
# 0. PREREQ: gateway reachable — curl https://web-tools.taild99fee.ts.net/... → 401
|
||||||
|
# (see "Reachability" above)
|
||||||
|
|
||||||
# 1. DNS: mp-suppliers-openclaw.vector.apexfusion.org → A 62.238.38.167
|
# 1. DNS: mp-suppliers-openclaw.vector.apexfusion.org → A 62.238.38.167
|
||||||
# (individual DNS-only / unproxied record — letsencrypt HTTP-01)
|
# (individual DNS-only / unproxied record — letsencrypt HTTP-01)
|
||||||
@@ -232,7 +244,7 @@ Neither is required for a functional supplier, but both harden it:
|
|||||||
today). Install openclaw's managed browser / a system chromium to enable it.
|
today). Install openclaw's managed browser / a system chromium to enable it.
|
||||||
2. **Egress firewall** — the agent can `web_fetch`/browse arbitrary URLs.
|
2. **Egress firewall** — the agent can `web_fetch`/browse arbitrary URLs.
|
||||||
Restrict the VM's outbound to block RFC1918 + tailnet CGNAT
|
Restrict the VM's outbound to block RFC1918 + tailnet CGNAT
|
||||||
(`100.64.0.0/10`) except `100.82.111.46:8002` (vuk llama.cpp), leaving
|
(`100.64.0.0/10`) except `100.77.146.49:8002` (vuk llama.cpp), leaving
|
||||||
public internet open for fetch. This blocks SSRF into the home LAN / tailnet
|
public internet open for fetch. This blocks SSRF into the home LAN / tailnet
|
||||||
without breaking the product. Requires `sudo` on web-tools.
|
without breaking the product. Requires `sudo` on web-tools.
|
||||||
|
|
||||||
|
|||||||
@@ -4,9 +4,11 @@
|
|||||||
# Runs on the inference-proxy box (62.238.38.167), NOT the main marketplace
|
# Runs on the inference-proxy box (62.238.38.167), NOT the main marketplace
|
||||||
# box. The compute backend is a llama.cpp server (llamacpp-multinode) on the
|
# box. The compute backend is a llama.cpp server (llamacpp-multinode) on the
|
||||||
# home machine `vuk`, reachable only over the tailscale tailnet at
|
# home machine `vuk`, reachable only over the tailscale tailnet at
|
||||||
# 100.82.111.46:8002 (OpenAI-compatible /v1/chat/completions, no auth — the
|
# 100.77.146.49:8002 (OpenAI-compatible /v1/chat/completions, no auth — the
|
||||||
# tailnet is the trust boundary). The tailscale IP is pinned instead of the
|
# tailnet is the trust boundary). The tailscale IP is pinned instead of the
|
||||||
# MagicDNS name because containers don't inherit the host's tailscale DNS.
|
# MagicDNS name because containers don't inherit the host's tailscale DNS.
|
||||||
|
# (Tailnet `taild99fee.ts.net`, account teamhaleight@; vuk's IP was
|
||||||
|
# 100.82.111.46 on the previous tailnet — update here if the tailnet changes.)
|
||||||
#
|
#
|
||||||
# The upstream ignores the request's `model` field and serves whatever GGUF
|
# The upstream ignores the request's `model` field and serves whatever GGUF
|
||||||
# is currently loaded; the on-chain advert carries the exact model id, so a
|
# is currently loaded; the on-chain advert carries the exact model id, so a
|
||||||
@@ -53,7 +55,7 @@ services:
|
|||||||
LLM_BACKEND: openai
|
LLM_BACKEND: openai
|
||||||
# No /v1 suffix — supplier/src/openai.ts appends /v1/chat/completions.
|
# No /v1 suffix — supplier/src/openai.ts appends /v1/chat/completions.
|
||||||
# vuk's tailscale IP; see header for why not the MagicDNS name.
|
# vuk's tailscale IP; see header for why not the MagicDNS name.
|
||||||
OPENAI_BASE_URL: http://100.82.111.46:8002
|
OPENAI_BASE_URL: http://100.77.146.49:8002
|
||||||
# ~4.5 tok/s generation on the home rig; matches the advert's
|
# ~4.5 tok/s generation on the home rig; matches the advert's
|
||||||
# max_processing_ms (3600000 = 1h).
|
# max_processing_ms (3600000 = 1h).
|
||||||
OPENAI_TIMEOUT_MS: "3600000"
|
OPENAI_TIMEOUT_MS: "3600000"
|
||||||
|
|||||||
@@ -4,13 +4,20 @@
|
|||||||
#
|
#
|
||||||
# Runs on the inference-proxy box (62.238.38.167), next to supplier `local`.
|
# Runs on the inference-proxy box (62.238.38.167), next to supplier `local`.
|
||||||
# The compute backend is an OpenClaw gateway (agent `main`) on the KVM guest
|
# The compute backend is an OpenClaw gateway (agent `main`) on the KVM guest
|
||||||
# `web-tools`, reachable only over the tailscale tailnet at
|
# `web-tools`. The gateway binds to loopback and is exposed tailnet-wide via
|
||||||
# 100.101.143.72:18789 (OpenAI-compatible chatCompletions endpoint, gateway
|
# Tailscale Serve at https://web-tools.taild99fee.ts.net (valid *.ts.net TLS,
|
||||||
# token auth). The agent's own brain is the SAME llama.cpp rig on `vuk` that
|
# gateway token auth). The agent's own brain is the SAME llama.cpp rig on
|
||||||
# supplier `local` fronts — what this supplier sells is the agent loop on top:
|
# `vuk` that supplier `local` fronts — what this supplier sells is the agent
|
||||||
# live web search (searxng) + web fetch + browser. Both suppliers contend for
|
# loop on top: live web search (searxng) + web fetch + browser. Both suppliers
|
||||||
# vuk's single slot; requests queue at llama.cpp and the 1h deadline absorbs
|
# contend for vuk's single slot; requests queue at llama.cpp and the 1h
|
||||||
# stalls.
|
# deadline absorbs stalls.
|
||||||
|
#
|
||||||
|
# Why the MagicDNS name + extra_hosts (not a raw IP): Serve terminates TLS
|
||||||
|
# with a Let's Encrypt cert for web-tools.taild99fee.ts.net, so the client
|
||||||
|
# must use that hostname (SNI + cert match). Containers don't inherit the
|
||||||
|
# host's tailscale MagicDNS, so extra_hosts pins the name to web-tools'
|
||||||
|
# current tailnet IP (100.105.36.0 on tailnet taild99fee.ts.net, account
|
||||||
|
# teamhaleight@). If the tailnet/IP changes, update the extra_hosts IP.
|
||||||
#
|
#
|
||||||
# Stateful upstream contract (differs from every other -chat supplier):
|
# Stateful upstream contract (differs from every other -chat supplier):
|
||||||
# - OPENAI_SESSION_PASSTHROUGH=1 — each turn sends the escrow ref as the
|
# - OPENAI_SESSION_PASSTHROUGH=1 — each turn sends the escrow ref as the
|
||||||
@@ -29,9 +36,11 @@
|
|||||||
# bound to the tailnet interface with token auth.
|
# bound to the tailnet interface with token auth.
|
||||||
#
|
#
|
||||||
# Pre-flight:
|
# Pre-flight:
|
||||||
# 1. Tailscale ACL: local-inference-proxy (100.127.17.106) must be allowed
|
# 1. web-tools reachable over the tailnet: the openclaw gateway is exposed
|
||||||
# to reach web-tools (100.101.143.72) on tcp/18789 — the tailnet filters
|
# via Tailscale Serve at https://web-tools.taild99fee.ts.net. Confirm from
|
||||||
# by source node (vuk can, inference-proxy can't until the ACL is added).
|
# the host: `curl -m10 -o /dev/null -w '%{http_code}' -X POST
|
||||||
|
# https://web-tools.taild99fee.ts.net/v1/chat/completions` → 401 (needs
|
||||||
|
# token). No tailnet ACL is required (serve is tailnet-wide).
|
||||||
# 2. DNS: mp-suppliers-openclaw.vector.apexfusion.org → 62.238.38.167
|
# 2. DNS: mp-suppliers-openclaw.vector.apexfusion.org → 62.238.38.167
|
||||||
# (individual DNS-only A record; letsencrypt HTTP-01 issuance needs the
|
# (individual DNS-only A record; letsencrypt HTTP-01 issuance needs the
|
||||||
# unproxied direct record)
|
# unproxied direct record)
|
||||||
@@ -71,9 +80,10 @@ services:
|
|||||||
MAX_CHAT_SESSIONS: "1"
|
MAX_CHAT_SESSIONS: "1"
|
||||||
LLM_BACKEND: openai
|
LLM_BACKEND: openai
|
||||||
# No /v1 suffix — supplier/src/openai.ts appends /v1/chat/completions.
|
# No /v1 suffix — supplier/src/openai.ts appends /v1/chat/completions.
|
||||||
# web-tools' tailscale IP is pinned because containers don't inherit the
|
# Tailscale Serve endpoint (HTTPS, valid *.ts.net cert). The MagicDNS
|
||||||
# host's tailscale MagicDNS.
|
# name is pinned to web-tools' tailnet IP via extra_hosts below (see
|
||||||
OPENAI_BASE_URL: http://100.101.143.72:18789
|
# header for why the name, not the IP).
|
||||||
|
OPENAI_BASE_URL: https://web-tools.taild99fee.ts.net
|
||||||
# Agent turns are slow: reasoning + tool loops at ~4.5 tok/s generation.
|
# Agent turns are slow: reasoning + tool loops at ~4.5 tok/s generation.
|
||||||
# Matches the advert's max_processing_ms (3600000 = 1h).
|
# Matches the advert's max_processing_ms (3600000 = 1h).
|
||||||
OPENAI_TIMEOUT_MS: "3600000"
|
OPENAI_TIMEOUT_MS: "3600000"
|
||||||
@@ -87,6 +97,11 @@ services:
|
|||||||
# NETWORK_ID and OGMIOS_URL intentionally not pinned here — they MUST
|
# NETWORK_ID and OGMIOS_URL intentionally not pinned here — they MUST
|
||||||
# come from the operator-provided supplier/.env.openclaw on a mainnet
|
# come from the operator-provided supplier/.env.openclaw on a mainnet
|
||||||
# host.
|
# host.
|
||||||
|
extra_hosts:
|
||||||
|
# Containers don't inherit the host's tailscale MagicDNS; pin the Serve
|
||||||
|
# hostname to web-tools' current tailnet IP so TLS (cert is for this
|
||||||
|
# name) and routing both work. Update the IP if the tailnet changes.
|
||||||
|
- "web-tools.taild99fee.ts.net:100.105.36.0"
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD-SHELL", "wget -qO- http://localhost:8080/healthz >/dev/null || exit 1"]
|
test: ["CMD-SHELL", "wget -qO- http://localhost:8080/healthz >/dev/null || exit 1"]
|
||||||
interval: 20s
|
interval: 20s
|
||||||
|
|||||||
Reference in New Issue
Block a user