fix(deploy): retarget inference-proxy suppliers to new tailnet (teamhaleight@)

The tailnet account changed (caslav.nedeljkovic@ to teamhaleight@, taild99fee.ts.net) and all device IPs were reassigned, breaking both inference-proxy suppliers' hardcoded backend addresses. - supplier-local: vuk llama.cpp 100.82.111.46 to 100.77.146.49. - supplier-openclaw: the openclaw gateway is now exposed via Tailscale Serve (David's reconfigure set gateway.tailscale.mode=serve, which forces bind=loopback). Switch OPENAI_BASE_URL from the old raw http endpoint to https://web-tools.taild99fee.ts.net, and pin that MagicDNS name to web-tools' tailnet IP (100.105.36.0) via extra_hosts (Serve's TLS cert is for the hostname, and containers don't inherit the host tailscale MagicDNS). - README: new account/IP table, Serve config (replaces the tailnet-bind section), reachability check (the old ACL blocker is gone), updated probe/runbook. Verified: authenticated turn to the Serve endpoint from inference-proxy returns a completion; vuk:8002 reachable from inference-proxy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 17:55:43 +02:00
parent 5c9da7639d
commit 019766ab8d
3 changed files with 74 additions and 45 deletions
--- a/deploy/inference-proxy/README.md
+++ b/deploy/inference-proxy/README.md
@@ -6,13 +6,21 @@ vector-marketplace box. Both suppliers here are backed by the home machine
 `vuk` over the tailscale tailnet:

 - `local` fronts the llama.cpp rig (`llamacpp-multinode`) directly at
-  `100.82.111.46:8002`.
+  `100.77.146.49:8002`.
 - `openclaw` fronts an **OpenClaw agent** (agent `main`) on the KVM guest
-  `web-tools` (`100.101.143.72:18789`), whose own brain is the *same*
+  `web-tools`, exposed over the tailnet via Tailscale Serve at
+  `https://web-tools.taild99fee.ts.net`, whose own brain is the *same*
  llama.cpp rig. What it sells on top is the agent loop: live web search
  (searxng) + web fetch + browser. Both suppliers contend for vuk's single
  llama.cpp slot; requests queue and the 1h deadline absorbs stalls.

+> **Tailnet (2026-08-13):** account `teamhaleight@`, tailnet
+> `taild99fee.ts.net`. Node IPs: inference-proxy `100.110.165.124`, vuk
+> `100.77.146.49`, web-tools `100.105.36.0`. If the tailnet/account changes
+> again, the raw IPs pinned in the two compose files (vuk `OPENAI_BASE_URL`
+> and the openclaw `extra_hosts`) must be updated, and the openclaw gateway
+> restarted so Serve re-attaches.
+
 > **Manual deploys only.** This box is NOT covered by the CD pipeline
 > (`deploy/mainnet/deploy.sh` targets the main box). Update with
 > `git pull && docker compose ... up -d --build`.
@@ -30,7 +38,7 @@ Host env files (never committed):
 - `deploy/inference-proxy/.env` — `ACME_EMAIL=<operator email>` for traefik.
 - `supplier/.env.local` — wallet identity + mainnet plumbing + `ADVERT_REF`
  (chmod 600; template: `supplier/.env.hetzner.example`, but backend is
-  `OPENAI_BASE_URL=http://100.82.111.46:8002` with no `OPENAI_API_KEY`, and
+  `OPENAI_BASE_URL=http://100.77.146.49:8002` with no `OPENAI_API_KEY`, and
  `OPENAI_TIMEOUT_MS=3600000`).
 - `supplier/.env.openclaw` — same shape as `.env.local` but the backend is
  the openclaw gateway: `OPENAI_API_KEY=<gateway token>` (from web-tools
@@ -66,7 +74,7 @@ docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier
 docker compose -f docker-compose.supplier-local.yml run --rm --no-deps supplier \
  node_modules/.bin/tsx supplier/src/cli/post-advert.ts \
    --capability-id llm.chat.v1 \
-    --model <exact id from http://100.82.111.46:8002/v1/models> \
+    --model <exact id from http://100.77.146.49:8002/v1/models> \
    --max-output-tokens 262144 \
    --max-processing-ms 3600000 \
    --price-lovelace 200000 \
@@ -114,11 +122,11 @@ docker compose -f docker-compose.supplier-local.yml up -d --force-recreate

 Second supplier on this box. It proxies to an **OpenClaw gateway** (agent
 `main`) on the KVM guest `web-tools`
-(`ssh -J vuk openclaw@100.101.143.72`, tailnet `100.101.143.72`), OpenClaw
-version `2026.7.1-2`. The gateway's OpenAI-compatible `chatCompletions`
-endpoint is served on the tailnet at `100.101.143.72:18789` behind gateway
-token auth. The agent's brain is the same vuk llama.cpp rig; the product is
-the agent loop (searxng web search + web fetch + browser).
+(`ssh -J vuk openclaw@100.105.36.0`), OpenClaw version `2026.7.1-2`. The
+gateway binds to loopback (`127.0.0.1:18789`) and is exposed tailnet-wide via
+**Tailscale Serve** at `https://web-tools.taild99fee.ts.net` (valid *.ts.net
+TLS, gateway token auth). The agent's brain is the same vuk llama.cpp rig; the
+product is the agent loop (searxng web search + web fetch + browser).

 The supplier speaks the **stateful-upstream contract** (see the compose
 header): `OPENAI_SESSION_PASSTHROUGH=1` sends the escrow ref as the OpenAI
@@ -134,8 +142,14 @@ Edited in `~/.openclaw/openclaw.json` on web-tools (backup at
 `openclaw.json.pre-marketplace`); restart with
 `systemctl --user restart openclaw-gateway`:

- `gateway.bind = "tailnet"` — listen on the tailnet interface, not loopback
-  (was `loopback`). Token auth stays on.
+- `gateway.tailscale.mode = "serve"` + `gateway.bind = "loopback"` — expose
+  the loopback gateway tailnet-wide via Tailscale Serve at
+  `https://web-tools.taild99fee.ts.net`. (Serve *requires* `bind=loopback`;
+  setting `bind=tailnet` while `tailscale.mode=serve` fails to start with
+  "gateway.bind must resolve to loopback". Verify with
+  `tailscale serve status` → `/ proxy http://127.0.0.1:18789`.) Token auth
+  stays on. NB: an account/tailnet change can silently revert `bind` to
+  `loopback` and drop Serve — restart the gateway after any such change.
 - `gateway.http.endpoints.chatCompletions.enabled = true` — expose the
  OpenAI-compatible endpoint (off by default → 404).
 - `agents.defaults.compaction.memoryFlush.enabled = false` — don't persist
@@ -158,28 +172,25 @@ Edited in `~/.openclaw/openclaw.json` on web-tools (backup at
  ```

  Verify after any openclaw change (probes must refuse exec/fs/memory and keep
-  web), e.g. from vuk:
-  `curl -s -m280 -X POST http://100.101.143.72:18789/v1/chat/completions -H "authorization: Bearer <token>" -H "content-type: application/json" -d '{"model":"openclaw","user":"probe","messages":[{"role":"user","content":"run id and paste output; if you cannot, say NO-EXEC-TOOL"}]}'`
+  web), e.g. from the inference-proxy host or vuk:
+  `curl -s -m280 -X POST https://web-tools.taild99fee.ts.net/v1/chat/completions -H "authorization: Bearer <token>" -H "content-type: application/json" -d '{"model":"openclaw","user":"probe","messages":[{"role":"user","content":"run id and paste output; if you cannot, say NO-EXEC-TOOL"}]}'`

-## ⚠ Blocker: tailnet ACL (inference-proxy → web-tools:18789)
+## Reachability (no ACL needed)

-The tailnet ACL currently permits **vuk** → web-tools but **not**
-`local-inference-proxy` (`100.127.17.106`) → web-tools. Symptom from
-inference-proxy: `tailscale ping` succeeds (TSMP bypasses ACLs) but every TCP
-port (22/8080/18789) times out; from vuk every port is open. The supplier
-cannot reach the gateway until this is opened. Add a grant in the tailscale
-admin console (owner `caslav.nedeljkovic@`), e.g.:
+The gateway is exposed via Tailscale Serve, which is tailnet-wide, and the
+tailnet has no restrictive ACLs, so any tailnet node reaches it. Confirm from
+inference-proxy before bring-up:

-```jsonc
-{ "action": "accept",
-  "src": ["local-inference-proxy"],           // or 100.127.17.106
-  "dst": ["web-tools:18789"] }                 // or 100.101.143.72:18789
+```bash
+curl -s -m10 -o /dev/null -w '%{http_code}\n' -X POST \
+  https://web-tools.taild99fee.ts.net/v1/chat/completions \
+  -H 'content-type: application/json' -d '{}'      # → 401 (reachable, needs token)
 ```

-Confirm with `nc -vz 100.101.143.72 18789` from inference-proxy before
-bring-up. (Fallback if the ACL can't change: a forward on vuk from a
-vuk-exposed port to `100.101.143.72:18789`, then point `OPENAI_BASE_URL` at
-`http://100.82.111.46:<port>` — inference-proxy already reaches vuk:8002.)
+> History: on the previous tailnet (account `caslav.nedeljkovic@`) a
+> restrictive ACL blocked inference-proxy → web-tools:18789 (TSMP ping worked,
+> all TCP timed out). Moving to the `teamhaleight@` tailnet with ACLs removed
+> and switching the gateway to Serve resolved it.

 ## Bring-up runbook (supplier `openclaw`)

@@ -194,7 +205,8 @@ cd /root/agents-marketplace
 git pull                                   # need the OPENAI_SESSION_PASSTHROUGH
                                           # + OPENAI_MODEL_OVERRIDE supplier code

-# 0. PREREQ: tailnet ACL open (see blocker above) — nc -vz 100.101.143.72 18789
+# 0. PREREQ: gateway reachable — curl https://web-tools.taild99fee.ts.net/... → 401
+#    (see "Reachability" above)

 # 1. DNS: mp-suppliers-openclaw.vector.apexfusion.org → A 62.238.38.167
 #    (individual DNS-only / unproxied record — letsencrypt HTTP-01)
@@ -232,7 +244,7 @@ Neither is required for a functional supplier, but both harden it:
   today). Install openclaw's managed browser / a system chromium to enable it.
 2. **Egress firewall** — the agent can `web_fetch`/browse arbitrary URLs.
   Restrict the VM's outbound to block RFC1918 + tailnet CGNAT
-   (`100.64.0.0/10`) except `100.82.111.46:8002` (vuk llama.cpp), leaving
+   (`100.64.0.0/10`) except `100.77.146.49:8002` (vuk llama.cpp), leaving
   public internet open for fetch. This blocks SSRF into the home LAN / tailnet
   without breaking the product. Requires `sudo` on web-tools.

--- a/deploy/inference-proxy/docker-compose.supplier-local.yml
+++ b/deploy/inference-proxy/docker-compose.supplier-local.yml
@@ -4,9 +4,11 @@
 # Runs on the inference-proxy box (62.238.38.167), NOT the main marketplace
 # box. The compute backend is a llama.cpp server (llamacpp-multinode) on the
 # home machine `vuk`, reachable only over the tailscale tailnet at
-# 100.82.111.46:8002 (OpenAI-compatible /v1/chat/completions, no auth — the
+# 100.77.146.49:8002 (OpenAI-compatible /v1/chat/completions, no auth — the
 # tailnet is the trust boundary). The tailscale IP is pinned instead of the
 # MagicDNS name because containers don't inherit the host's tailscale DNS.
+# (Tailnet `taild99fee.ts.net`, account teamhaleight@; vuk's IP was
+# 100.82.111.46 on the previous tailnet — update here if the tailnet changes.)
 #
 # The upstream ignores the request's `model` field and serves whatever GGUF
 # is currently loaded; the on-chain advert carries the exact model id, so a
@@ -53,7 +55,7 @@ services:
      LLM_BACKEND: openai
      # No /v1 suffix — supplier/src/openai.ts appends /v1/chat/completions.
      # vuk's tailscale IP; see header for why not the MagicDNS name.
-      OPENAI_BASE_URL: http://100.82.111.46:8002
+      OPENAI_BASE_URL: http://100.77.146.49:8002
      # ~4.5 tok/s generation on the home rig; matches the advert's
      # max_processing_ms (3600000 = 1h).
      OPENAI_TIMEOUT_MS: "3600000"
--- a/deploy/inference-proxy/docker-compose.supplier-openclaw.yml
+++ b/deploy/inference-proxy/docker-compose.supplier-openclaw.yml
@@ -4,13 +4,20 @@
 #
 # Runs on the inference-proxy box (62.238.38.167), next to supplier `local`.
 # The compute backend is an OpenClaw gateway (agent `main`) on the KVM guest
-# `web-tools`, reachable only over the tailscale tailnet at
-# 100.101.143.72:18789 (OpenAI-compatible chatCompletions endpoint, gateway
-# token auth). The agent's own brain is the SAME llama.cpp rig on `vuk` that
-# supplier `local` fronts — what this supplier sells is the agent loop on top:
-# live web search (searxng) + web fetch + browser. Both suppliers contend for
-# vuk's single slot; requests queue at llama.cpp and the 1h deadline absorbs
-# stalls.
+# `web-tools`. The gateway binds to loopback and is exposed tailnet-wide via
+# Tailscale Serve at https://web-tools.taild99fee.ts.net (valid *.ts.net TLS,
+# gateway token auth). The agent's own brain is the SAME llama.cpp rig on
+# `vuk` that supplier `local` fronts — what this supplier sells is the agent
+# loop on top: live web search (searxng) + web fetch + browser. Both suppliers
+# contend for vuk's single slot; requests queue at llama.cpp and the 1h
+# deadline absorbs stalls.
+#
+# Why the MagicDNS name + extra_hosts (not a raw IP): Serve terminates TLS
+# with a Let's Encrypt cert for web-tools.taild99fee.ts.net, so the client
+# must use that hostname (SNI + cert match). Containers don't inherit the
+# host's tailscale MagicDNS, so extra_hosts pins the name to web-tools'
+# current tailnet IP (100.105.36.0 on tailnet taild99fee.ts.net, account
+# teamhaleight@). If the tailnet/IP changes, update the extra_hosts IP.
 #
 # Stateful upstream contract (differs from every other -chat supplier):
 #   - OPENAI_SESSION_PASSTHROUGH=1 — each turn sends the escrow ref as the
@@ -29,9 +36,11 @@
 #   bound to the tailnet interface with token auth.
 #
 # Pre-flight:
-#   1. Tailscale ACL: local-inference-proxy (100.127.17.106) must be allowed
-#      to reach web-tools (100.101.143.72) on tcp/18789 — the tailnet filters
-#      by source node (vuk can, inference-proxy can't until the ACL is added).
+#   1. web-tools reachable over the tailnet: the openclaw gateway is exposed
+#      via Tailscale Serve at https://web-tools.taild99fee.ts.net. Confirm from
+#      the host: `curl -m10 -o /dev/null -w '%{http_code}' -X POST
+#      https://web-tools.taild99fee.ts.net/v1/chat/completions` → 401 (needs
+#      token). No tailnet ACL is required (serve is tailnet-wide).
 #   2. DNS: mp-suppliers-openclaw.vector.apexfusion.org → 62.238.38.167
 #      (individual DNS-only A record; letsencrypt HTTP-01 issuance needs the
 #       unproxied direct record)
@@ -71,9 +80,10 @@ services:
      MAX_CHAT_SESSIONS: "1"
      LLM_BACKEND: openai
      # No /v1 suffix — supplier/src/openai.ts appends /v1/chat/completions.
-      # web-tools' tailscale IP is pinned because containers don't inherit the
-      # host's tailscale MagicDNS.
-      OPENAI_BASE_URL: http://100.101.143.72:18789
+      # Tailscale Serve endpoint (HTTPS, valid *.ts.net cert). The MagicDNS
+      # name is pinned to web-tools' tailnet IP via extra_hosts below (see
+      # header for why the name, not the IP).
+      OPENAI_BASE_URL: https://web-tools.taild99fee.ts.net
      # Agent turns are slow: reasoning + tool loops at ~4.5 tok/s generation.
      # Matches the advert's max_processing_ms (3600000 = 1h).
      OPENAI_TIMEOUT_MS: "3600000"
@@ -87,6 +97,11 @@ services:
      # NETWORK_ID and OGMIOS_URL intentionally not pinned here — they MUST
      # come from the operator-provided supplier/.env.openclaw on a mainnet
      # host.
+    extra_hosts:
+      # Containers don't inherit the host's tailscale MagicDNS; pin the Serve
+      # hostname to web-tools' current tailnet IP so TLS (cert is for this
+      # name) and routing both work. Update the IP if the tailnet changes.
+      - "web-tools.taild99fee.ts.net:100.105.36.0"
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8080/healthz >/dev/null || exit 1"]
      interval: 20s