mirror of
https://github.com/block/buzz.git
synced 2026-10-27 12:40:30 +00:00
38bf642fcf
## Summary
CI now proves the full Buzz shared-compute join story end to end: a
member can discover another member's served model **through the Buzz
relay alone** and run inference over the mesh, while a non-member gets
nothing — the relay rejects its auth, and the mesh refuses to route for
it even holding a leaked endpoint address.
This is deliberately different from mesh-llm's own CI smokes (which
bootstrap two nodes with a hand-carried invite token / mdns): here the
**relay is the control plane**, exactly like the desktop app:
1. **Membership** — identities A and B are added via `buzz-admin`
(kind:13534 NIP-43 roster); C is not.
2. **Advertise** — each member publishes a client-signed kind:30003
discovery note carrying its MeshLLM owner binding and (for the serve
node) `serveTargets[].endpointAddr`, covered by an endpoint-binding
signature — the exact payload shape the desktop coordinator publishes.
3. **Trust** — the serve node derives its admission allowlist from the
relay (statuses ∩ roster) and requires the **exact expected {A, B}
owner-id set** before starting with `TrustPolicy::Allowlist`.
4. **Join** — the client verifies owner + endpoint bindings and
membership, then dials the relay-discovered endpoint (the desktop
join-watcher's `dial_endpoint_addr` step). No out-of-band token.
5. **Infer** — a chat completion against the client's local OpenAI
endpoint routes over QUIC to the serve node's model (CPU, SmolLM2-135M,
~105MB).
6. **Deny (differential)** — the stranger's NIP-42 auth must fail with
the relay's own membership rejection (`restricted: not a relay member` —
successful auth or any unrelated connect error fails the run), and
dialing the leaked endpoint must not produce a routed inference —
**while the trusted client re-proves inference immediately afterwards**,
so a dead serve node can't masquerade as an admission denial.
## What's in the PR
- `crates/buzz-relay/examples/mesh_relay_lifecycle_smoke.rs` — the
harness. One process per node (mesh-llm keeps process-global state under
`~/.mesh-llm`), orchestrator + serve/client/stranger roles,
byte-identical binding payloads to
`desktop/src-tauri/src/mesh_llm/identity.rs` (called out with
keep-in-sync comments). Child stdout is pumped through a reader thread
so every wait has a hard deadline; timed-out children are killed; exit
statuses are checked.
- `scripts/ci-mesh-lifecycle-smoke.sh` — provisions a membership-gated
relay (throwaway owner + signing identities via `buzz-admin
generate-key`), runs the harness, cleans up. Fails fast if :3000 is
already occupied (a stale open relay would mask gating).
- `scripts/start-relay-for-tests.sh` — gains opt-in NIP-43 membership
env passthrough (`BUZZ_REQUIRE_RELAY_MEMBERSHIP` + `RELAY_OWNER_PUBKEY`
+ `BUZZ_RELAY_PRIVATE_KEY`). Default behavior unchanged.
- `.github/workflows/mesh-lifecycle.yml` — separate, path-filtered,
non-required workflow (mesh paths, the harness's dependency crates,
`Cargo.lock`, dispatch), pinned to `ubuntu-24.04`. Caches the mesh
native runtime + HF model keyed on the lockfile hash, so a mesh pin bump
rolls the runtime cache. Uploads relay + harness logs on failure.
## Scope
This is an **independent protocol harness**: it speaks the same wire
protocol and payload shapes as the desktop but re-implements the
binding/verification logic (the desktop crate is outside the workspace).
Regressions inside the desktop's own discovery filtering are the desktop
unit tests' job; what this smoke proves is that the relay + mesh-llm SDK
+ admission stack support the lifecycle end to end.
## Relationship to mesh-llm's CI
Follows the shape mesh-llm's own CI proved stable (tiny CPU model, one
runner, multiple real mesh-llm processes over real QUIC — cf. their
`ci-two-node-client-serving-smoke.sh`), but swaps the token bootstrap
for the relay-driven lifecycle, which is the part only Buzz can test.
## Validation
Green on GitHub Actions (ubuntu-24.04) across three runs, including
after rebases onto the mesh v0.74 upgrade (#3467) and latest main:
```
PASS 1/6: relay-derived allowlist is exactly {A, B}
PASS 2/6: serve member ready + advertised model: jc-builds/SmolLM2-135M-Instruct-Q4_K_M-GGUF:Q4_K_M
PASS 3/6: client member discovered + joined via relay
PASS 4/6: inference routed over the mesh: "PONG"
PASS 5/6: relay rejected the stranger's NIP-42 auth (membership gate)
PASS 6/6: stranger denied (gossip visible, inference rejected: 503 all tunnels failed) while trusted inference still routes
PASS: full relay-driven mesh lifecycle verified
```
Also validated locally on macOS. `cargo fmt --all --check` and `cargo
clippy -p buzz-relay --all-targets -- -D warnings` pass.
## Notes
- The harness follows the repo's mesh `[dev-dependencies]` pin
automatically, so it doubles as a canary for future mesh upgrades (it
already caught the v0.73.1 → v0.74.0 bump during development).
- The stranger "deny" accepts either shape mesh-llm exhibits: no model
visibility at all, or gossip visibility with inference refused —
mesh-llm applies the receiving node's owner policy after the gossip
handshake, so admission gates *routing*, not gossip. The differential
trusted-inference re-check (PASS 6/6) is what makes that a real denial
rather than a dead server.
- Model-visibility windows are tunable via `MESH_CLIENT_WINDOW_SECS` /
`MESH_STRANGER_WINDOW_SECS` if shared runners prove slow — pin a longer
window in the workflow env rather than re-running the job.
---------
Signed-off-by: Michael Neale <michael.neale@gmail.com>
101 lines
4.0 KiB
TOML
101 lines
4.0 KiB
TOML
[package]
|
|
name = "buzz-relay"
|
|
default-run = "buzz-relay"
|
|
# Independent version: buzz-relay ships as a pinnable artifact
|
|
# (ghcr.io/block/buzz), released on its own cadence via `just release-relay`.
|
|
# It does NOT inherit the workspace version.
|
|
version = "0.2.0"
|
|
edition.workspace = true
|
|
rust-version.workspace = true
|
|
license.workspace = true
|
|
repository.workspace = true
|
|
description = "WebSocket relay server for the Buzz communications platform"
|
|
|
|
[[bin]]
|
|
name = "buzz-relay"
|
|
path = "src/main.rs"
|
|
|
|
[dependencies]
|
|
buzz-core = { workspace = true }
|
|
buzz-conformance = { workspace = true }
|
|
buzz-db = { workspace = true }
|
|
buzz-auth = { workspace = true }
|
|
buzz-pubsub = { workspace = true }
|
|
buzz-audit = { workspace = true }
|
|
buzz-search = { workspace = true }
|
|
buzz-relay-mesh = { workspace = true }
|
|
async-trait = "0.1"
|
|
postcard = { workspace = true }
|
|
axum = { workspace = true }
|
|
tokio = { workspace = true }
|
|
# `io` (ReaderStream) is needed to stream git subprocess stdout straight into
|
|
# the HTTP response body — see api::git::transport read-path streaming. The
|
|
# workspace default only enables rt+codec.
|
|
tokio-util = { workspace = true, features = ["io"] }
|
|
tower = { workspace = true }
|
|
tower-http = { workspace = true }
|
|
nostr = { workspace = true }
|
|
serde = { workspace = true }
|
|
serde_json = { workspace = true }
|
|
tracing = { workspace = true }
|
|
tracing-subscriber = { workspace = true }
|
|
tracing-opentelemetry = { workspace = true }
|
|
opentelemetry = { workspace = true }
|
|
opentelemetry_sdk = { workspace = true }
|
|
opentelemetry-otlp = { workspace = true }
|
|
thiserror = { workspace = true }
|
|
anyhow = { workspace = true }
|
|
uuid = { workspace = true }
|
|
chrono = { workspace = true }
|
|
dashmap = { workspace = true }
|
|
futures-util = { workspace = true }
|
|
deadpool-redis = { workspace = true }
|
|
redis = { workspace = true }
|
|
# Redis TLS (rediss://, e.g. ElastiCache) uses rustls, which needs a process
|
|
# CryptoProvider installed at startup. Both aws-lc-rs and ring are compiled in
|
|
# transitively, so rustls can't auto-select one — we install ring explicitly in
|
|
# main(). Mirrors buzz-acp's rustls setup for wss://.
|
|
rustls = { version = "0.23", default-features = false, features = ["ring", "std"] }
|
|
sqlx = { workspace = true }
|
|
reqwest = { workspace = true }
|
|
base64 = "0.22"
|
|
buzz-sdk = { workspace = true }
|
|
buzz-workflow = { workspace = true, features = ["reqwest"] }
|
|
buzz-media = { workspace = true }
|
|
s3 = { version = "0.37", package = "rust-s3", default-features = false, features = ["tokio-rustls-tls", "fail-on-err", "tags"] }
|
|
tempfile = "3"
|
|
bytes = "1"
|
|
infer = "0.19"
|
|
serde_yaml = { workspace = true }
|
|
sha2 = { workspace = true }
|
|
hmac = { workspace = true }
|
|
subtle = { workspace = true }
|
|
rand = { workspace = true }
|
|
hex = { workspace = true }
|
|
url = { workspace = true }
|
|
moka = { workspace = true }
|
|
metrics = { workspace = true }
|
|
metrics-exporter-prometheus = { workspace = true }
|
|
metrics-util = { workspace = true }
|
|
pulldown-cmark = { version = "0.13.4", default-features = false, features = ["html"] }
|
|
async-compression = { version = "0.4.42", features = ["tokio", "gzip"] }
|
|
|
|
[features]
|
|
dev = ["buzz-auth/dev"]
|
|
|
|
[dev-dependencies]
|
|
mesh-llm-sdk = { git = "https://github.com/Mesh-LLM/mesh-llm.git", tag = "v0.74.0", package = "mesh-llm-sdk", default-features = false, features = ["client", "serving"] }
|
|
mesh-llm-host-runtime = { git = "https://github.com/Mesh-LLM/mesh-llm.git", tag = "v0.74.0", package = "mesh-llm-host-runtime", default-features = false, features = ["dynamic-native-runtime"] }
|
|
# Relay-driven mesh lifecycle smoke (examples/mesh_relay_lifecycle_smoke.rs):
|
|
# the relay client for discovery notes and the exact ed25519 the mesh owner
|
|
# keys use for binding verification.
|
|
buzz-test-client = { path = "../buzz-test-client" }
|
|
ed25519-dalek = "=3.0.0-rc.0"
|
|
buzz-core = { workspace = true, features = ["test-utils"] }
|
|
buzz-auth = { workspace = true, features = ["dev"] }
|
|
reqwest = { workspace = true }
|
|
tokio-tungstenite = { workspace = true }
|
|
futures = "0.3"
|
|
flate2 = "1.1.9"
|
|
opentelemetry_sdk = { workspace = true, features = ["testing"] }
|