mirror of
https://github.com/block/buzz.git
synced 2026-10-27 12:40:30 +00:00
## Summary Adds `docs/database.md`, which covers two things: - **The target architecture.** `schema/tables/public/` is the declarative source of truth, written within the constraints that out-of-band, online schema tools can apply. Deployments at scale apply schema changes out of band; auto-migration stays supported for smaller and self-hosted deployments, and `migrations/` stays. The database has no triggers, functions, or foreign keys; that behavior lives in the owning application transaction or in durable application work. Data repair runs through the backfill framework in `docs/data-backfills.md`, not through migrations. - **Rules for schema changes made now:** 1. Add no triggers, functions, procedures, rules, or foreign keys, with no exceptions. New tenant-owned tables are fenced by community admission in the application. Startup and deletion validation already tolerate missing fences, but desired-state bootstraps still attach a fence trigger to every `community_id` table, so a new tenant-owned table is blocked until that loop stops (part of the fence-retirement work). The doc includes checklists for adding a tenant-owned table and a provenance-only table. 2. Migrations change schema only. They never mutate existing rows. Until the backfill framework ships, a repair may be a reviewed, idempotent, batched one-off script in `scripts/maintenance/` that takes the community deletion lock and touches only active communities. A repair that writes historical channel events must keep replica routing closed until its WAL has replayed. 3. Update every schema location: `migrations/`, `schema/tables/public/`, the manifest and sibling directories, the pgschema reconcile script where it applies, and both homes of the existing `push_gateway_*` tables (the target is for the gateway to own its tables alone). Never edit a merged migration. Add catalog-level parity tests for what you touch. 4. Every migration must be safe for the relay that is currently deployed, under both supported modes: auto-migration at startup and out-of-band application. Either way the old relay runs against the new schema for a while. Because auto-migration can apply expand and contract in one boot, a contract migration must fail closed when its backfill has not run, and a contract that depends on a writer release (such as dropping a trigger whose work moved into the app) must name that release as a required stop in the release notes, because no row check can prove it. The order is expand, deploy, backfill, then contract. The doc includes a table of changes that break the old relay and the safe form of each. 5. Bound every lock, build `migrations/` indexes concurrently (the declarative schema uses plain `CREATE INDEX`), and keep changes within what out-of-band apply can make online. AGENTS.md (Product Contract), ARCHITECTURE.md (buzz-db), and new path-local `migrations/AGENTS.md`, `schema/AGENTS.md` (a pointer), and `crates/buzz-push-gateway/migrations/AGENTS.md` point to the doc. The path-local files mean agents editing those directories get the rules automatically. AGENTS.md also had a stale claim that migrations are "auto-applied on relay startup". This PR corrects it: `BUZZ_AUTO_MIGRATE` defaults off in the binary and on in the Helm chart, and both modes are supported. ## What got simpler The rules were scattered across `schema/schema.sql` comments, AGENTS.md Common Gotcha #7, the backfill spec, `docs/thread-window-deployment.md`, and test names. They now live in one document, and the rest link to it. No code changes. ## Testing This PR changes documentation only. I checked links and diffs, and every claim in the doc points to code at the base commit: the layout and parity tests in `crates/buzz-db/src/runtime/migration.rs`, `EXPECTED_SCOPED_TABLES` and the serving-catalog check in `crates/buzz-db/src/store/deletion.rs`, and `BUZZ_AUTO_MIGRATE` in `crates/buzz-relay/src/main.rs`. The new `AGENTS.md` files aren't `.sql`, so neither the sqlx migrator nor the schema layout test picks them up; CI's layout, migration-count, and deletion-parity tests pass. At an earlier head, a real relay against isolated PostgreSQL 17 showed that a new unfenced `community_id` table serves normally and makes community deletion report `unknown=<table>`. Since then, the fence requirement at startup was removed on main; the doc now cites `migration_schema_fence_objects_are_optional_for_serving_and_deletion` for that. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
976 lines
58 KiB
Markdown
976 lines
58 KiB
Markdown
# Buzz Architecture
|
||
|
||
## 1. Executive Summary
|
||
|
||
Buzz is a self-hosted team communication platform built on the Nostr protocol (NIP-01 wire format), where AI agents and humans are first-class equals. Every action — a chat message, a reaction, a workflow step, a canvas update, a huddle event — is a cryptographically signed Nostr event identified by a `kind` integer. Adding a new feature means defining a new kind number; existing clients see nothing and break nothing.
|
||
|
||
The relay is the single source of truth. All reads and writes flow through it. There is no peer-to-peer event exchange, no gossip, no replication — just clients connecting to one relay over WebSocket, and the relay enforcing auth, verifying signatures, persisting events, fanning out to subscribers, indexing for search, and triggering automation.
|
||
|
||
A Buzz **community** is the tenant-visible workspace selected by the request host.
|
||
The self-hosted default remains one host, one relay process, one implicit
|
||
community. Multi-community deployments move that semantic boundary one level up:
|
||
`req.community = resolve_host(connection.host)` is established before AUTH,
|
||
EVENT, REQ, REST, media, git, search, workflow, or pub/sub handling. Unknown
|
||
hosts fail closed, and NIP-98/API-token stamps must agree with the host-derived
|
||
community rather than overriding it.
|
||
|
||
Deployment-root community management uses operator-signed NIP-98 HTTP requests.
|
||
`POST /operator/communities/delete` accepts only an exact normalized, archived
|
||
community whose asserted pubkey is its sole current owner. The caller supplies the
|
||
request UUID as the stable correlation/idempotency identity; the durable row
|
||
records operator-attested owner intent, mediating operator, and acknowledgement
|
||
version. Admission returns `202` at the `submitted` stage and performs no
|
||
inventory, approval,
|
||
quiescing, object-store access, or deletion execution synchronously. While that
|
||
non-aborted request exists, unarchive and ownership transfer conflict and owner
|
||
management lists suppress the archived row. Replaying the same UUID converges
|
||
to its current stage; a different UUID conflicts with the existing one-active-
|
||
request invariant until that request is aborted.
|
||
|
||
Owner consent on this path is asserted, not proven. The mediating operator
|
||
authenticates the owner and collects the deletion acknowledgement out of band,
|
||
upstream of the relay; the request itself carries only the operator's NIP-98
|
||
signature. The relay verifies operator authority and that the asserted pubkey
|
||
is the community's sole current owner, then records the owner pubkey, mediating
|
||
operator pubkey, and acknowledgement version as durable provenance for that
|
||
upstream ceremony. No owner-signed attestation is required or checked, and
|
||
owners have no self-service cancellation. Recovery is a privileged abort,
|
||
which stays open across the reversible `submitted`, `inventoried`, `approved`,
|
||
and `fenced` stages — releasing the request fence while leaving the community
|
||
archived — and closes from `drained` onward, when tenant-state destruction may
|
||
have begun.
|
||
|
||
Manual operator handoff converges on an admitted owner request only when
|
||
`buzz-admin deletions submit --requested-by` repeats the owner pubkey recorded
|
||
on the row. Owner provenance pins `requested_by` to `owner_pubkey`, so passing
|
||
the operator's own pubkey does not converge — it conflicts with the existing
|
||
one-active-request invariant instead.
|
||
|
||
The privileged one-shot `buzz-admin deletions drain` process gives already-
|
||
approved work priority. When none is ready, it may claim only an
|
||
operator-attested owner-origin `submitted` request under the same durable
|
||
generation lease used for execution, inventory it with lease-loss cancellation,
|
||
and atomically freeze the inventory plus a digest-bound `owner_automatic`
|
||
approval. The mediating operator remains the approval actor; the owner
|
||
acknowledgement is pre-inventory
|
||
intent, not a claim that the owner reviewed the digest. The retained lease then
|
||
enters the unchanged approved-request executor. Operator-origin requests never
|
||
auto-progress and still require explicit inventory and approval. Owner
|
||
admission still has no owner-facing cancellation or grace period; the
|
||
privileged abort described above remains the recovery path.
|
||
|
||
Ownership is mutable only while a community is active. Archiving freezes the
|
||
current owner. Normal transfer and deployment-root legacy convergence take the
|
||
same community-row lock as owner-deletion admission, then reject archived,
|
||
quiescing, deleted, or deletion-pending rotation without changing membership.
|
||
Initial owner bootstrap for a newly created community remains supported.
|
||
|
||
Buzz is a Rust monorepo, licensed Apache 2.0 under Block, Inc.
|
||
|
||
---
|
||
|
||
### System Architecture
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ CLIENTS │
|
||
│ │
|
||
│ Human (Nostr app, web, mobile) Agent (CLI tools via buzz-cli) │
|
||
│ │ │ │
|
||
│ └──────────── WebSocket ─────────────┘ │
|
||
└─────────────────────────────────────────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ buzz-relay (Axum) │
|
||
│ │
|
||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌─────────────────────┐ │
|
||
│ │ NIP-42 │ │ EVENT │ │ REQ │ │ HTTP bridge │ │
|
||
│ │ auth │ │ pipeline │ │ handler │ │ /events │ │
|
||
│ └──────────┘ └──────────┘ └──────────┘ │ /query │ │
|
||
│ │ /count │ │
|
||
│ ┌──────────────────────────────────────┐ │ /hooks/{id} │ │
|
||
│ │ SubscriptionRegistry │ │ /media/* │ │
|
||
│ │ DashMap: (channel_id, kind) → conns │ │ /git/* │ │
|
||
│ └──────────────────────────────────────┘ │ /info, NIP-05 │ │
|
||
│ └─────────────────────┘ │
|
||
└──────────┬──────────────┬──────────────────────────────────────────┘
|
||
│ │
|
||
┌─────▼──────┐ ┌────▼──────┐
|
||
│ Postgres │ │ Redis │
|
||
│ (events, │ │ (presence │
|
||
│ channels, │ │ SET EX, │
|
||
│ tokens, │ │ typing │
|
||
│ workflows, │ │ ZADD, │
|
||
│ audit) │ │ PUBLISH) │
|
||
└────────────┘ └───────────┘
|
||
|
||
Fan-out: sub_registry.fan_out() → conn_manager.send_to()
|
||
(in-process for local events; Redis round-trip for
|
||
events from other relay instances)
|
||
|
||
Redis PUBLISH occurs for channel-scoped events.
|
||
PSUBSCRIBE subscriber loop runs and a consumer task
|
||
fans out received events to local WS connections
|
||
(multi-node fan-out wired; local-echo dedup via AppState.local_event_ids).
|
||
|
||
┌──────────────┐
|
||
│ Postgres │ ← buzz-search (FTS over the search_tsv
|
||
│ (full-text │ generated column + GIN index)
|
||
│ search) │
|
||
└──────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
### Crate Dependency Hierarchy
|
||
|
||
```
|
||
buzz-core (zero I/O — types, verification, filter matching, kind registry)
|
||
│
|
||
├── buzz-db (Postgres: events, channels, tokens, workflows, audit)
|
||
├── buzz-auth (NIP-42, NIP-98, API tokens, scopes, rate limiting)
|
||
├── buzz-pubsub (Redis pub/sub, presence, typing indicators)
|
||
├── buzz-search (Postgres FTS: query, delete)
|
||
├── buzz-audit (hash-chain tamper-evident log)
|
||
└── buzz-workflow (YAML-as-code automation engine)
|
||
│
|
||
└── buzz-relay (ties everything together — the server)
|
||
|
||
buzz-acp (agent harness — bridges relay @mentions → AI agents via ACP/JSON-RPC)
|
||
buzz-sdk (typed Nostr event builders — used by buzz-acp and buzz-cli)
|
||
buzz-media (Blossom/S3 media storage)
|
||
buzz-cli (agent-first CLI)
|
||
buzz-admin (operator CLI: relay membership + key generation)
|
||
buzz-test-client (integration test harness + manual CLI)
|
||
```
|
||
|
||
**Key architectural principle:** The relay is the single source of truth. `buzz-relay` orchestrates all subsystems by calling them directly — it imports `buzz-db`, `buzz-auth`, `buzz-pubsub`, `buzz-search`, `buzz-audit`, and `buzz-workflow`. However, those subsystems are isolated from each other: `buzz-workflow` never calls `buzz-pubsub`, `buzz-search` never calls `buzz-db`, etc. The one sanctioned exception is `buzz-audit`, which calls `buzz_db::deletion::guard_community_write` to admit each audit append into its community before taking the chain lock. Cross-subsystem coordination happens only through the relay. In multi-community mode, the relay also owns propagation of `TenantContext`; service crates should receive community-scoped inputs rather than independently deriving tenancy from client-controlled event tags.
|
||
|
||
---
|
||
|
||
## 2. The Protocol
|
||
|
||
Buzz uses Nostr NIP-01 on the wire. Every action is a JSON event with six fields:
|
||
|
||
```json
|
||
{
|
||
"id": "<sha256 of canonical serialization>",
|
||
"pubkey": "<secp256k1 public key, hex>",
|
||
"kind": <unsigned integer>,
|
||
"tags": [["e", "<event-id>"], ["p", "<pubkey>"], ...],
|
||
"content": "<JSON payload or plain text>",
|
||
"sig": "<Schnorr signature over id>"
|
||
}
|
||
```
|
||
|
||
The `kind` integer is the only dispatch switch. The relay routes, stores, and fans out events based on kind. Clients filter subscriptions by kind. New feature = new kind number = zero breaking changes to existing clients.
|
||
|
||
### Kind Ranges
|
||
|
||
| Range | Meaning |
|
||
|-------|---------|
|
||
| 0–9999 | Standard Nostr kinds (NIP-01 through NIP-XX) |
|
||
| 10000–19999 | Replaceable events (NIP-16) |
|
||
| 20000–29999 | Ephemeral events — not stored, not audited |
|
||
| 30000–39999 | Parameterized replaceable events |
|
||
| 40000–49999 | Buzz custom kinds |
|
||
|
||
### Buzz Custom Kinds (selected)
|
||
|
||
| Kind | Name | Description |
|
||
|------|------|-------------|
|
||
| 7 | KIND_REACTION | Emoji reaction (standard NIP-25) |
|
||
| 9 | KIND_STREAM_MESSAGE | Chat message in a Stream channel (NIP-29 group chat) |
|
||
| 40002 | KIND_STREAM_MESSAGE_V2 | Stream message v2 format |
|
||
| 40003 | KIND_STREAM_MESSAGE_EDIT | Edit of a stream message |
|
||
| 43001 | KIND_JOB_REQUEST | Agent job request |
|
||
| 45001 | KIND_FORUM_POST | Forum thread root |
|
||
| 45003 | KIND_FORUM_COMMENT | Forum thread reply |
|
||
| 46001–46012 | KIND_WORKFLOW_* | Workflow execution events |
|
||
| 20001 | KIND_PRESENCE_UPDATE | Ephemeral presence heartbeat |
|
||
| 20003 | KIND_AGENT_PRESENCE | Ephemeral presence heartbeat an owner sends for its agents |
|
||
|
||
`buzz-core` defines each event kind as a `pub const u32` and exports the full registry as `ALL_KINDS: &[u32]` (127 kinds at the time of writing); `crates/buzz-core/src/kind.rs` is the source of truth for the current list. Kinds are `u32` (NIP-01 specifies unsigned integer; `u32` covers the full range). Buzz uses both standard Nostr kinds (e.g., kind 7 for reactions) and custom ranges (40000+).
|
||
|
||
Note: `KIND_AUTH` (22242) is `pub const KIND_AUTH: u32` in `buzz-core/src/kind.rs` and imported by `buzz-relay/src/handlers/event.rs`. `KIND_CANVAS` (40100) is likewise `pub const KIND_CANVAS: u32` in `buzz-core/src/kind.rs`.
|
||
|
||
### Wire Protocol (NIP-01 messages)
|
||
|
||
| Direction | Message | Purpose |
|
||
|-----------|---------|---------|
|
||
| Client → Relay | `["EVENT", <event>]` | Submit a signed event |
|
||
| Client → Relay | `["REQ", <sub_id>, <filter>, ...]` | Subscribe to events |
|
||
| Client → Relay | `["CLOSE", <sub_id>]` | Cancel a subscription |
|
||
| Client → Relay | `["AUTH", <event>]` | Authenticate (NIP-42) |
|
||
| Relay → Client | `["EVENT", <sub_id>, <event>]` | Deliver a matching event |
|
||
| Relay → Client | `["EOSE", <sub_id>]` | End of stored events |
|
||
| Relay → Client | `["OK", <event_id>, true/false, ""]` | Event acceptance result |
|
||
| Relay → Client | `["CLOSED", <sub_id>, "reason"]` | Subscription closed |
|
||
| Relay → Client | `["NOTICE", "message"]` | Informational message |
|
||
| Relay → Client | `["AUTH", <challenge>]` | Authentication challenge |
|
||
|
||
Max frame size: 65,536 bytes. Max subscriptions per connection: 1024. Max historical results per filter: 500.
|
||
|
||
---
|
||
|
||
## 3. Connection Lifecycle
|
||
|
||
Every WebSocket connection follows this exact sequence:
|
||
|
||
### Step 0: Community Binding
|
||
|
||
The server resolves `TenantContext` from the request host before any handler can
|
||
observe tenant data. The URL/domain is authoritative for the community, matching
|
||
today's "the relay URL is the workspace" behavior. In single-community mode the
|
||
configured host maps to the default community. In multi-community mode, an
|
||
unknown or unmapped host rejects generically and never falls through to a default
|
||
tenant. Client-supplied `#h` tags are still channel identifiers; they must resolve
|
||
to a channel inside the host-derived community.
|
||
|
||
### Step 1: Semaphore Acquire
|
||
|
||
`state.conn_semaphore.try_acquire_owned()` — if the relay is at connection capacity, the connection is rejected immediately before any data is read. The permit is held for the entire connection lifetime and dropped on cleanup.
|
||
|
||
### Step 2: NIP-42 Challenge
|
||
|
||
The relay immediately sends `["AUTH", "<challenge>"]`. The challenge is a random string. The connection is registered in `ConnectionManager` after the challenge is sent.
|
||
|
||
### Step 3: Authentication
|
||
|
||
The client must respond with `["AUTH", <signed-event>]` before submitting events or subscriptions. Authentication paths:
|
||
|
||
| Path | Mechanism | Use Case |
|
||
|------|-----------|---------|
|
||
| NIP-42 | Signed challenge, pubkey verified | WebSocket connections |
|
||
| NIP-98 HTTP Auth | Schnorr-signed `kind:27235` event on HTTP bridge endpoints | HTTP clients |
|
||
|
||
On success, `ConnectionState.auth_state` transitions from `Pending` → `Authenticated(AuthContext)`. On failure → `Failed`. Unauthenticated EVENT/REQ messages are rejected with `["CLOSED", ...]` or `["OK", ..., false, "auth-required: ..."]`.
|
||
|
||
### Step 4: Active Loops
|
||
|
||
Three concurrent tasks run for the lifetime of the connection:
|
||
|
||
- **recv_loop** (inline): reads frames, parses `ClientMessage`, dispatches to handlers
|
||
- **send_loop** (spawned): drains the mpsc channel, writes frames to the WebSocket
|
||
- **heartbeat_loop** (spawned): sends WebSocket ping every 30 seconds; 3 missed pongs → disconnect
|
||
|
||
A `CancellationToken` coordinates shutdown across all three loops.
|
||
|
||
Slow clients: `ConnectionState::send()` uses `try_send` — if the send buffer is full, a grace counter increments. After `SLOW_CLIENT_GRACE_LIMIT` (3) consecutive full-buffer events, the connection is cancelled. A successful send resets the counter.
|
||
|
||
### Step 5: Cleanup
|
||
|
||
On disconnect (any cause):
|
||
1. `cancel.cancel()` — signals all loops
|
||
2. Await send_loop and heartbeat_loop tasks
|
||
3. `sub_registry.remove_connection(conn_id)` — removes all subscriptions from the DashMap indexes
|
||
4. `conn_manager.deregister(conn_id)` — removes from the send-channel map
|
||
5. `drop(permit)` — releases the connection semaphore slot
|
||
|
||
---
|
||
|
||
## 4. Event Pipeline
|
||
|
||
When the relay receives `["EVENT", <event>]`, the handler in `handlers/event.rs` runs this pipeline in order:
|
||
|
||
```
|
||
1. AUTH CHECK — AuthState::Authenticated? MessagesWrite scope?
|
||
2. PUBKEY MATCH — event.pubkey == auth_context.pubkey?
|
||
3. KIND_AUTH REJECT — kind == 22242 (AUTH events never stored)
|
||
4. EPHEMERAL ROUTE — kind 20000–29999 → ephemeral sub-pipeline (see below)
|
||
5. VERIFY — spawn_blocking(verify_event) — Schnorr sig + ID hash
|
||
6. MEMBERSHIP — channel_id in event tags? → check_channel_membership
|
||
7. DB INSERT — db.insert_event (idempotent; search_tsv generated synchronously)
|
||
8. REDIS PUBLISH — pubsub.publish_event (if channel-scoped)
|
||
9. FAN-OUT — sub_registry.fan_out → conn_manager.send_to
|
||
10. AUDIT LOG — audit.log (spawned async, non-blocking)
|
||
11. WORKFLOW TRIGGER — wf.on_event (spawned async, excludes kinds 46001–46012)
|
||
```
|
||
|
||
Steps 10–11 are fire-and-forget: audit and workflow triggers are spawned as independent async tasks. A failure in either does not fail the event submission. Search has no asynchronous indexing step: Postgres maintains the generated `events.search_tsv` column as part of step 7, and `buzz-search` only queries it. The client receives `["OK", <id>, true, ""]` at the end of the pipeline, not immediately after DB insert.
|
||
|
||
Step 9 (fan-out) explicitly **excludes** global subscriptions (no `channel_id` constraint) from channel-scoped events — global subscriptions do NOT receive events from private channels, regardless of filter match. This is a deliberate security boundary: only subscriptions scoped to an accessible `channel_id` receive those events.
|
||
|
||
Workflow loop prevention: workflow execution kinds (46001–46012), relay-signed messages with `buzz:workflow` tag, and `KIND_GIFT_WRAP` are excluded from triggering workflows. All other stored events (including kind 9 stream messages) trigger workflow evaluation.
|
||
|
||
### Ephemeral Sub-Pipeline (kinds 20000–29999)
|
||
|
||
Ephemeral events bypass DB storage, audit, and search. Two sub-paths:
|
||
|
||
**Presence events (kind 20001):**
|
||
```
|
||
1. VERIFY — spawn_blocking(verify_event)
|
||
2. REDIS PRESENCE — set_presence() or clear_presence() based on content
|
||
3. LOCAL FAN-OUT — sub_registry.fan_out → conn_manager.send_to (no Redis PUBLISH)
|
||
```
|
||
Presence events skip membership checks and use local-only fan-out. Multi-node presence fan-out would require Redis pub/sub (documented as future work).
|
||
|
||
**Agent presence (kind 20003):** an owner's one heartbeat for all its agents, so an app that runs agents without a socket per agent can still show them online. Content is the status (`online`, `away` or `offline`) and each `p` tag names an agent, at most 256. WebSocket only. The relay keeps the agents the signer owns (`users.agent_owner_pubkey`, through the owner cache), skips the rest, and sets or clears their presence exactly as their own kind 20001 would, so the presence snapshot reports them unchanged. It is not fanned out, since live presence is matched by author; viewers see it on their next snapshot read.
|
||
|
||
**Other ephemeral events (e.g., typing indicators):**
|
||
```
|
||
1. VERIFY — spawn_blocking(verify_event)
|
||
2. MEMBERSHIP — check_channel_membership (if channel-scoped)
|
||
3. MARK LOCAL — state.mark_local_event (dedup before Redis round-trip)
|
||
4. REDIS PUBLISH — pubsub.publish_event (no DB write)
|
||
5. LOCAL FAN-OUT — sub_registry.fan_out → conn_manager.send_to
|
||
```
|
||
|
||
Typing indicators (kind 20002) are also accepted through HTTP `POST /events`, for clients that sign events but hold no WebSocket, such as app-hosted agents. The HTTP route requires an `h` tag, checks bans/timeouts and the community write fence, then runs steps 1–5 above.
|
||
|
||
Ephemeral events are never stored in Postgres and never appear in REQ historical queries.
|
||
|
||
### Handler Semaphore
|
||
|
||
Beyond the per-connection semaphore, a `handler_semaphore` (capacity 1024) limits concurrent EVENT and REQ processing across all connections. CLOSE is not rate-limited.
|
||
|
||
---
|
||
|
||
## 5. Subscription System
|
||
|
||
### SubscriptionRegistry
|
||
|
||
The subscription registry is a DashMap-backed structure in `subscription.rs`:
|
||
|
||
```rust
|
||
pub struct SubscriptionRegistry {
|
||
subs: DashMap<ConnId, HashMap<SubId, SubEntry>>,
|
||
channel_kind_index: DashMap<IndexKey, Vec<(ConnId, SubId)>>,
|
||
channel_wildcard_index: DashMap<Uuid, Vec<(ConnId, SubId)>>,
|
||
}
|
||
|
||
pub struct IndexKey {
|
||
pub channel_id: Uuid,
|
||
pub kind: Kind,
|
||
}
|
||
```
|
||
|
||
### Three-Tier Fan-Out
|
||
|
||
When an event arrives, `fan_out` consults three indexes in order:
|
||
|
||
| Tier | Index | Key | Use Case |
|
||
|------|-------|-----|---------|
|
||
| 1 | `channel_kind_index` | `(channel_id, kind)` | Subs with explicit channel + kind filter — O(1) lookup |
|
||
| 2 | `channel_wildcard_index` | `channel_id` | Subs with channel but no `kinds` constraint |
|
||
| 3 | `subs` (linear scan) | — | Global subs (no channel_id) — fallback scan |
|
||
|
||
Global subs (tier 3) are checked for non-channel-scoped events only. Channel-scoped events are delivered exclusively to subscriptions that carry a matching `channel_id` — global subscriptions are explicitly excluded from channel fan-out as a security boundary.
|
||
|
||
### NIP-01 Edge Cases
|
||
|
||
- `kinds: []` (explicit empty array) means "match nothing" — NOT a wildcard. Subscriptions with empty `kinds` are not indexed in either tier 1 or tier 2 and never receive events.
|
||
- `kinds` absent (no field) means "match all kinds" — indexed in tier 2 (channel wildcard) or tier 3 (global).
|
||
|
||
### REQ Handler Access Control
|
||
|
||
The REQ handler checks channel access **before** registering the subscription:
|
||
|
||
```
|
||
1. Parse filters, extract channel_id
|
||
2. Load accessible_channel_ids for this connection's pubkey
|
||
3. If channel_id not in accessible_channels → send CLOSED "restricted: not a channel member"
|
||
4. Only then: sub_registry.register(conn_id, sub_id, filters, channel_id)
|
||
```
|
||
|
||
This prevents a race where a non-member receives live fan-out events from a private channel between registration and the access check.
|
||
|
||
### Historical Query (EOSE)
|
||
|
||
After registering, the REQ handler queries Postgres for stored events matching the filters (up to 500 per filter, hard cap). These are sent as `["EVENT", sub_id, event]` frames before `["EOSE", sub_id]`. New events arriving after EOSE are delivered via the fan-out path.
|
||
|
||
**Client consumption invariant.** A client rebuilding channel state must
|
||
open its live subscription before (or overlapping) the finite history
|
||
REQ — a gap between the last backfill page and live delivery silently
|
||
drops events and rebuilds stale state (PR #3995). When the relay sends a
|
||
terminal CLOSED, the subscription is removed server-side; any client-side
|
||
ownership tied to it (chunk/scope fences) must be released in the same
|
||
step, or live delivery stops permanently while the client believes it is
|
||
subscribed (PR #6996).
|
||
|
||
---
|
||
|
||
## 6. Crate Reference
|
||
|
||
### buzz-core — Shared Types and Verification
|
||
|
||
**Zero I/O.** The foundation every other crate builds on. Explicitly prohibits tokio, sqlx, redis, and axum in its `Cargo.toml`.
|
||
|
||
**Key types:**
|
||
|
||
```rust
|
||
pub struct StoredEvent {
|
||
pub event: nostr::Event,
|
||
pub received_at: DateTime<Utc>,
|
||
pub channel_id: Option<Uuid>,
|
||
verified: bool, // private — use is_verified()
|
||
}
|
||
|
||
pub const ALL_KINDS: &[u32] // 80 entries (KIND_AUTH excluded — never stored)
|
||
```
|
||
|
||
**Key functions:**
|
||
|
||
| Function | Purpose |
|
||
|----------|---------|
|
||
| `filters_match(filters, event)` | OR across filters, AND within each filter. Includes NIP-01 prefix matching on event IDs. |
|
||
| `verify_event(event)` | Schnorr signature + SHA-256 ID check. CPU-bound — callers use `spawn_blocking`. |
|
||
| `is_not_global_unicast(ip)` | SSRF protection: enumerated-deny policy — blocks a specific set of non-public address classes and accepts everything else (including addresses not covered by an explicit deny rule, e.g. `fe00::1`). Blocked IPv4 classes: loopback, private (RFC 1918), link-local, CGNAT (RFC 6598), benchmarking (RFC 2544), IETF Protocol Assignments (192.0.0.0/24, exceptions: 192.0.0.9 PCP anycast, 192.0.0.10 TURN anycast), documentation (RFC 5737: 192.0.2/24, 198.51.100/24, 203.0.113/24), deprecated 6to4 relay anycast (192.88.99.0/24, RFC 7526), multicast (RFC 5771, 224/4), reserved/class-E (240/4). Blocked IPv6 classes: loopback, unspecified, ULA (fc00::/7), link-local (fe80::/10), deprecated site-local (fec0::/10, RFC 3879), multicast (ff00::/8), IETF Protocol Assignments envelope (2001::/23, global exceptions: 2001:1::1–::3 anycast, 2001:3::/32 AMT, 2001:4:112::/48 AS112-v6, 2001:20::/28 ORCHIDv2, 2001:30::/28 DETs), documentation (2001:db8::/32, 3fff::/20), 6to4 (2002::/16), Discard-Only (100::/64), Dummy prefix (100:0:0:1::/64), SRv6 SIDs (5f00::/16), NAT64 local-use (64:ff9b:1::/48). IPv4 embedded in mapped, compatible, NAT64 well-known (64:ff9b::/96), and SIIT IPv4-translated (::ffff:0:0:0/96) forms checked recursively. Compat alias: `is_private_ip`. |
|
||
|
||
**Does NOT:** store events, make network calls, spawn tasks, or depend on any async runtime.
|
||
|
||
---
|
||
|
||
### buzz-auth — Authentication and Authorization
|
||
|
||
Handles authentication paths, scope enforcement, and token operations.
|
||
|
||
**Auth paths:**
|
||
|
||
| Path | Entry Point | Notes |
|
||
|------|-------------|-------|
|
||
| NIP-42 | `verify_auth_event()` | Schnorr-signed challenge/response; grants `Scope::all_known()` (all 14 scopes) |
|
||
| NIP-98 HTTP Auth | `validate_nip98_auth()` | HTTP bridge endpoints; Schnorr-signed `kind:27235` event |
|
||
|
||
**Key types:**
|
||
|
||
```rust
|
||
pub struct AuthContext { pub pubkey: PublicKey, pub scopes: Vec<Scope>, pub auth_method: AuthMethod }
|
||
pub enum AuthMethod { Nip42, Nip98 }
|
||
pub enum Scope { MessagesRead, MessagesWrite, ChannelsRead, ChannelsWrite,
|
||
AdminChannels, UsersRead, UsersWrite, AdminUsers,
|
||
JobsRead, JobsWrite, SubscriptionsRead, SubscriptionsWrite,
|
||
FilesRead, FilesWrite, Unknown(String) }
|
||
pub trait ChannelAccessChecker: Send + Sync { ... }
|
||
pub trait RateLimiter: Send + Sync { ... }
|
||
```
|
||
|
||
**Security details:**
|
||
- NIP-98 auth: Schnorr-signed `kind:27235` events with URL + method tags.
|
||
- NIP-42 timestamp tolerance: ±60 seconds.
|
||
- Dev-only key derivation: `SHA-256("buzz-test-key:{username}")` — gated behind `#[cfg(any(test, feature = "dev"))]`. The `dev` feature must not be enabled in production relay deployments.
|
||
|
||
**Does NOT:** implement `RateLimiter` beyond a test stub (`AlwaysAllowRateLimiter`, gated behind `#[cfg(any(test, feature = "test-utils"))]`). No Redis-backed rate limiter exists anywhere in the codebase — rate limiting is not currently enforced. `RateLimitConfig` defines 4 tiers (human, agent-standard, agent-elevated, agent-platform) as a design target.
|
||
|
||
---
|
||
|
||
### buzz-db — Postgres Event Store
|
||
|
||
All database access. Uses `sqlx::query()` (runtime, not compile-time macros) — no `.sqlx/` offline cache required.
|
||
|
||
**Key operations:**
|
||
|
||
| Module | Responsibility |
|
||
|--------|---------------|
|
||
| `event.rs` | `insert_event` (ON CONFLICT DO NOTHING), `query_events` (QueryBuilder), `get_event_by_id` |
|
||
| `channel.rs` | Channel CRUD, membership management, role enforcement (transactional) |
|
||
| `feed.rs` | `query_mentions` (INNER JOIN event_mentions), `query_needs_action`, `query_activity` |
|
||
| `workflow.rs` | Full workflow/run/approval CRUD; SHA-256 hashed approval tokens |
|
||
| `partition.rs` | Monthly range partitioning for `events` and `delivery_log` tables: read-only catalog audit plus advisory-locked, time-bounded maintenance that creates uncovered months and advances an empty catch-all (`partition/maintenance.rs`) |
|
||
| `dm.rs` | DM channel management |
|
||
| `reaction.rs` | Reaction storage and retrieval |
|
||
| `thread.rs` | Thread/reply tracking |
|
||
| `user.rs` | User profile storage |
|
||
| `error.rs` | Database error types |
|
||
|
||
**Channel types:** `Stream`, `Forum`, `Dm`, `Workflow`, `System`
|
||
**Member roles:** `Owner`, `Admin`, `Member`, `Guest`, `Bot`
|
||
**Workflow statuses:** `Active`, `Disabled`, `Archived`
|
||
**Run statuses:** `Pending`, `Running`, `WaitingApproval`, `Completed`, `Failed`, `Cancelled`
|
||
|
||
**Key behaviors:**
|
||
- `ON CONFLICT DO NOTHING` for event dedup — returns `(StoredEvent, was_inserted: bool)`.
|
||
- Rejects `KIND_AUTH` (22242) and ephemeral (20000–29999) with distinct error variants.
|
||
- Transactional role enforcement in `add_member`/`remove_member`/`create_channel` — TOCTOU-safe.
|
||
- Soft-delete for channel members: `remove_member` sets `removed_at`; re-adding reverses it.
|
||
- Feed hard cap: `FEED_MAX_LIMIT = 100` rows regardless of caller-requested limit.
|
||
- `query_mentions` uses `INNER JOIN event_mentions` — normalized table with composite index on `(pubkey_hex, created_at)`.
|
||
- Approval tokens: `create_approval` receives the raw token and hashes it internally with SHA-256.
|
||
- DDL injection protection in partition manager: allowlist of table names + strict suffix/date validators.
|
||
|
||
### Supported Writer Contract (transition state)
|
||
|
||
The repository is moving toward a **supported-writer contract** where relay-owned
|
||
write APIs provide the only supported mutation path for tenant data. The current
|
||
shape is an explicit transition, not a flag day:
|
||
|
||
- **Trajectory:** converge from trigger/function/FK-enforced fencing toward
|
||
application-owned transaction protocols, while keeping existing DB backstops
|
||
in place until coverage and fleet gates are proven.
|
||
- **Community fence (transition foundation):** runtime admission APIs now offer
|
||
one shared community deletion lock per transaction, while deletion lifecycle
|
||
transitions take the matching exclusive lock. A supported serving transaction
|
||
never spans communities. Serving leases use the same application admission.
|
||
Startup and deletion catalog validation tolerate the universal fence objects
|
||
being absent; inventories still record live fences for older executors.
|
||
- **Replica floor (transition foundation):** runtime now provides a shared
|
||
replica-floor lock helper for compliant channel-event writers, and the writer
|
||
probe handshake takes the exclusive counterpart before sampling `S`, scanning
|
||
activity, and committing the heartbeat token last. `probe_once` remains the
|
||
sole token/fence-wall publication path; there is no separately persisted active
|
||
floor cutoff. Commit-time trigger+GUC enforcement remains authoritative for all
|
||
paths.
|
||
- **Dual enforcement (current):** application-owned lock+precheck paths run in
|
||
front of the existing trigger/function enforcement; commit-time trigger checks
|
||
remain authoritative during this phase. Removing those backstops is a separate
|
||
gated migration, after every pod tolerates their absence and no rollback below
|
||
that version is expected.
|
||
- **Role separation:** relay/runtime code owns admission and lock protocols;
|
||
operator maintenance/backfill workflows must either use those protocols or run
|
||
under explicit reviewed procedures that keep routing fences closed.
|
||
- **Unsupported shape:** direct owner SQL that bypasses supported transaction and
|
||
lock protocols is not a supported write path.
|
||
- **Fleet gates and reconciliation:** startup/fence probes verify guard catalog +
|
||
behavior, pgschema bootstraps run reconciliation, and lock/pool metrics remain
|
||
the operational evidence path for rollout safety.
|
||
|
||
The database end state (a declarative schema source, applied out of band at
|
||
scale; no triggers, functions, or foreign keys; backfills outside migrations)
|
||
and the expand → deploy → backfill → contract rules every schema change follows
|
||
are in [docs/database.md](docs/database.md).
|
||
|
||
**Does NOT:** cache queries, implement connection pooling logic (delegated to sqlx), or make network calls outside Postgres.
|
||
|
||
---
|
||
|
||
### buzz-pubsub — Redis Pub/Sub, Presence, Typing
|
||
|
||
Manages Redis pub/sub fan-out, presence tracking, and typing indicators. In multi-community mode all tenant-visible keys are prefixed or otherwise partitioned by community (`buzz:{community}:...`) so channel fan-out, presence, typing, and cache invalidation cannot cross hosts.
|
||
|
||
**Architecture:**
|
||
|
||
```
|
||
Publisher → pool connection → PUBLISH buzz:channel:{uuid}
|
||
Subscriber → dedicated PubSub → PSUBSCRIBE buzz:channel:*
|
||
→ broadcast::channel(4096)
|
||
```
|
||
|
||
The subscriber uses a **dedicated** `redis::aio::PubSub` connection — not from the pool. This is intentional: pool connections cannot hold `PSUBSCRIBE` state.
|
||
|
||
**Current state:** The subscriber loop is spawned in `buzz-relay/src/main.rs` and populates the broadcast channel. A consumer task subscribes via `pubsub.subscribe_local()`, calls `sub_registry.fan_out()` on each received event, and delivers matches to local WebSocket connections via `conn_manager.send_to()`. Multi-node fan-out is now wired end-to-end. Local-echo deduplication is implemented via `AppState.local_event_ids` — events published by the local relay instance are tracked and skipped when received via the Redis round-trip.
|
||
|
||
**Reconnection:** exponential backoff 1s → 30s (`backoff_secs * 2`). Backoff resets to 1s only after a clean stream end, not on each reconnect attempt.
|
||
|
||
**Presence:** `SET buzz:presence:{pubkey_hex} {status} EX 180` — 180-second TTL (3× the 60-second heartbeat interval). Single missed heartbeat does not cause presence flap.
|
||
|
||
**Typing indicators:**
|
||
```
|
||
ZADD buzz:typing:{channel_id} {now_unix} {pubkey_hex}
|
||
ZREMRANGEBYSCORE buzz:typing:{channel_id} -inf {now - 5.0}
|
||
EXPIRE buzz:typing:{channel_id} 60
|
||
```
|
||
5-second activity window. 60-second key TTL prevents orphaned empty sets.
|
||
|
||
**Does NOT:** implement the rate limiter. Does NOT store events. `PubSubManager` is not `Clone` — callers use `Arc<PubSubManager>`.
|
||
|
||
---
|
||
|
||
### buzz-search — Postgres FTS Integration
|
||
|
||
Full-text search via Postgres FTS. Events are searchable through the
|
||
`events.search_tsv` generated `tsvector` column (populated on insert, indexed
|
||
by a GIN index) — there is no separate search service or out-of-band indexer.
|
||
Privacy-sensitive kinds are excluded at the storage level (the `search_tsv`
|
||
`CASE WHEN kind IN (...)` yields `NULL`, which never matches `@@`). In
|
||
multi-community mode every query filter includes `community_id`, so the shared
|
||
`events` table is infrastructure, not a cross-community result space; the relay
|
||
re-authorizes every candidate hit before returning it.
|
||
|
||
**Key behaviors:**
|
||
- `SearchService::new(pool)` wraps a `PgPool`; `search(&SearchQuery)` runs a
|
||
parameterized FTS query against the `events.search_tsv` GIN index and returns
|
||
`SearchResult` (candidate `SearchHit`s).
|
||
- `ChannelScope` makes the channel constraint explicit (`Any` /
|
||
`ChannelLessOnly` / `Channels` / `ChannelsOrChannelLess`), closing the
|
||
ambiguity the old `Option<Vec<Uuid>> + bool` matrix could not express.
|
||
- Every query carries `community_id`; the FTS predicate is BitmapAnd-ed with
|
||
the community-leading btree filters so a query never crosses tenants.
|
||
- Permission filtering is **caller's responsibility** — `buzz-search` returns
|
||
candidate hits; the relay re-authorizes each one (channel membership, `#p`,
|
||
owner gates) before delivering it.
|
||
|
||
**Does NOT:** enforce channel membership or access control. Does NOT write
|
||
events (indexing is the `search_tsv` generated column on the `events` insert).
|
||
|
||
---
|
||
|
||
### buzz-audit — Hash-Chain Audit Log
|
||
|
||
Tamper-evident append-only log with SHA-256 hash chaining.
|
||
|
||
**Hash chain:** each entry stores `prev_hash` (hash of the previous entry). In multi-community mode audit heads/chains are per-community; operator metrics may aggregate, but tenant-readable audit verification walks one community chain. `verify_chain()` walks entries and recomputes hashes to detect tampering. Genesis entry uses `GENESIS_HASH` (64 zeros).
|
||
|
||
**Hash covers:** seq (big-endian bytes), timestamp (RFC3339), event_id, event_kind (big-endian), actor_pubkey, action string, channel_id (16 bytes or 16 zero bytes if None), canonical metadata JSON (BTreeMap for deterministic key ordering), prev_hash.
|
||
|
||
**Single-writer guarantee:** each append runs in one transaction that first passes community write admission (the shared community deletion lock plus the active/undeleted check, via `buzz-db`'s `DeletionStore::guard_transaction`), then takes a per-community `pg_advisory_xact_lock`. Both locks release on commit or rollback, including a dropped future.
|
||
|
||
**10 audit actions:** `EventCreated`, `EventDeleted`, `ChannelCreated`, `ChannelUpdated`, `ChannelDeleted`, `MemberAdded`, `MemberRemoved`, `AuthSuccess`, `AuthFailure`, `RateLimitExceeded`.
|
||
|
||
**Does NOT:** log `KIND_AUTH` (22242) events — returns `AuditError::AuthEventForbidden` immediately. Does NOT log ephemeral events (they never reach the audit pipeline).
|
||
|
||
---
|
||
|
||
### buzz-workflow — YAML-as-Code Automation Engine
|
||
|
||
Parses, validates, and executes channel-scoped workflow definitions. In multi-community mode workflow definitions, runs, approvals, webhook routes, and schedules inherit the host-derived community and evaluate triggers only against events in that community.
|
||
|
||
**Workflow definition structure:**
|
||
```yaml
|
||
name: "Incident Triage"
|
||
trigger:
|
||
on: message_posted
|
||
filter: "str_contains(trigger_text, 'P1')"
|
||
steps:
|
||
- id: notify
|
||
action: send_message
|
||
text: "P1 incident detected: {{trigger.text}}"
|
||
- id: page
|
||
if: "str_contains(trigger_text, 'production')"
|
||
action: request_approval
|
||
from: "{{trigger.author}}"
|
||
message: "Page on-call?"
|
||
```
|
||
|
||
Note: Both `TriggerDef` and `ActionDef` use serde internally-tagged enums. Triggers use `on:` as the tag field; actions use `action:` as the tag field. Fields are flattened into the parent struct, not nested.
|
||
|
||
**4 trigger types:** `message_posted`, `reaction_added`, `schedule`, `webhook`
|
||
|
||
**7 action types:**
|
||
|
||
| Action | Description |
|
||
|--------|-------------|
|
||
| `send_message` | Post to the workflow's channel (or override channel) |
|
||
| `send_dm` | Direct message to a user (pubkey hex or `{{trigger.author}}`) |
|
||
| `set_channel_topic` | Update channel topic |
|
||
| `add_reaction` | React to the trigger message |
|
||
| `call_webhook` | HTTP POST to external URL (SSRF-protected, redirects disabled, 1 MiB response cap) |
|
||
| `request_approval` | Suspend execution; fields: `from`, `message`, `timeout` (default 24h) |
|
||
| `delay` | Pause execution (max 300 seconds) |
|
||
|
||
**Template variables:** `{{trigger.text}}`, `{{trigger.author}}`, `{{steps.ID.output.FIELD}}`. Single-pass resolution (not recursive). Unknown variables left as literal text.
|
||
|
||
**Condition evaluation:** `evalexpr` with `HashMapContext`. Dot notation converted to underscores (`trigger.text` → `trigger_text`). Custom functions registered: `str_contains`, `str_starts_with`, `str_ends_with`, `str_len`. 100ms timeout prevents adversarial expressions from blocking.
|
||
|
||
**Concurrency:** `Arc<Semaphore>` with 100 permits. `try_acquire()` — returns `CapacityExceeded` immediately rather than queuing.
|
||
|
||
**Approval gates:** `request_approval` action returns `StepResult::Suspended` with a generated UUID token, but the engine does not yet persist the token or resume execution — runs that hit an approval gate are marked as failed (🚧 WF-08). `execute_from_step()` exists for future resumption support.
|
||
|
||
**Cron scheduler:** loop ticks every 60 seconds, evaluates cron expressions with window-based matching, and creates workflow runs for matched triggers. Fully implemented.
|
||
|
||
**Deletion and recreation:** An authorized kind:5 `a`-tag deletion commits its
|
||
public event, executable workflow removal, and visible kind:30620 removal in one
|
||
transaction. Rejected deletions are not accepted history; identical concurrent
|
||
requests have one dispatch/audit owner. Deletions older than a live definition
|
||
leave that newer version intact. Deletion is not a permanent coordinate ban:
|
||
clients may publish a distinct signed definition afterward, including a backdated
|
||
version, to recreate the workflow and enable triggers again. This intentionally
|
||
retains arrival-order recreation without a deletion watermark.
|
||
|
||
**Does NOT:** recursively resolve templates (single-pass only). Does NOT queue workflow runs when at capacity — returns `CapacityExceeded` immediately.
|
||
|
||
---
|
||
|
||
### Huddle Audio — WebSocket Opus Relay
|
||
|
||
Real-time voice lives inside `buzz-relay` (`src/audio/`), not a separate crate. A WebSocket endpoint (`wss://.../huddle/{channel_id}/audio`) authenticates each participant with a NIP-42 challenge, checks channel membership, admits them to an in-memory room, and forwards opaque Opus frames between peers. No external SFU.
|
||
|
||
**Frame protocol (v2):** 8-byte big-endian header (sequence `u16`, 48 kHz timestamp `u32`, level dBov `i8`, flags `u8`) followed by an opaque Opus payload. Invalid `level_dbov` values are clamped rather than dropped — losing a metric beats losing audio.
|
||
|
||
**Room state:** an admission guard synchronizes joins against the room's ended flag; soft cap 25 peers (hard cap 255 via `u8` peer index). Per-peer audio uses a bounded channel (drop-on-full); the control channel is separate and never drops join/leave.
|
||
|
||
**Lifecycle events:** the relay emits Nostr events for participant joined / left and huddle ended; the desktop client emits huddle started and guidelines. When the last peer leaves, the room ends and the channel archives atomically.
|
||
|
||
**Not yet built:** recording and per-track publishing (the corresponding kinds are reserved, no producer exists).
|
||
|
||
---
|
||
|
||
### buzz-relay — The Server
|
||
|
||
Axum WebSocket server. Ties all other crates together. The only crate that imports and orchestrates all subsystems.
|
||
|
||
**`AppState`** (Arc-wrapped, shared across all connections — key fields shown, not exhaustive):
|
||
|
||
```rust
|
||
pub struct AppState {
|
||
pub db: Db,
|
||
pub audit: Arc<AuditService>,
|
||
pub pubsub: Arc<PubSubManager>,
|
||
pub auth: Arc<AuthService>,
|
||
pub search: Arc<SearchService>,
|
||
pub sub_registry: Arc<SubscriptionRegistry>,
|
||
pub conn_manager: Arc<ConnectionManager>,
|
||
pub workflow_engine: Arc<WorkflowEngine>,
|
||
pub conn_semaphore: Arc<Semaphore>, // connection limit
|
||
pub handler_semaphore: Arc<Semaphore>, // 1024 concurrent handlers
|
||
pub relay_keypair: nostr::Keys, // relay identity
|
||
pub local_event_ids: moka::sync::Cache, // local-echo dedup
|
||
// + config, redis_pool, membership_cache, media_storage, shutdown state
|
||
}
|
||
```
|
||
|
||
Postgres pools use the closed physical role vocabulary `writer`, `reader`,
|
||
`audit`, and `search`. The periodic sampler retains cheap SQLx pool handles and
|
||
exports `buzz_db_pool_connections{pool_role,state}` for the bounded states
|
||
`idle` and `active`, `buzz_db_pool_max_connections{pool_role}` for capacity,
|
||
and `buzz_db_pool_configured{pool_role}`. All four roles are always present (16
|
||
raw gauge series total); absent optional pools report zero. Current pool size is
|
||
exactly the sum of its idle and active connection gauges. The legacy
|
||
`buzz_db_pool_*` writer gauges and `buzz_db_read_pool_*` reader gauges remain
|
||
for dashboard compatibility. Pool sizing and aggregate deployment connection
|
||
budgets remain configuration/deployment concerns rather than a relay pool
|
||
manager.
|
||
|
||
**`ConnectionState`** (per-connection):
|
||
|
||
```rust
|
||
pub struct ConnectionState {
|
||
pub auth_state: RwLock<AuthState>,
|
||
pub subscriptions: Mutex<HashMap<String, Vec<Filter>>>,
|
||
// + send_tx, cancel token
|
||
}
|
||
pub enum AuthState { Pending { challenge: String }, Authenticated(AuthContext), Failed }
|
||
```
|
||
|
||
**HTTP endpoints:**
|
||
|
||
| Method | Path | Handler |
|
||
|--------|------|---------|
|
||
| GET | `/` | WebSocket upgrade or NIP-11 relay info |
|
||
| GET | `/info` | NIP-11 relay info |
|
||
| GET | `/.well-known/nostr.json` | NIP-05 identity |
|
||
| GET | `/health` | Health check |
|
||
| GET | `/_liveness` | Liveness probe |
|
||
| GET | `/_readiness` | Readiness probe — local process lifecycle only |
|
||
| POST | `/events` | Submit a signed Nostr event over HTTP (same ingest path as WebSocket `EVENT`) |
|
||
| POST | `/query` | Query Nostr events over HTTP with NIP-01 filters |
|
||
| POST | `/count` | Count Nostr events over HTTP with NIP-45 filters |
|
||
| POST | `/hooks/{id}` | Workflow webhook trigger (secret-authenticated) |
|
||
| PUT | `/media/upload` | Upload media blob (Blossom, 50 MB limit) |
|
||
| GET/HEAD | `/media/{sha256_ext}` | Retrieve/probe media blob |
|
||
| GET | `/git/{owner}/{repo}/info/refs` | Git smart HTTP advertisement |
|
||
| POST | `/git/{owner}/{repo}/git-upload-pack` | Git smart HTTP fetch |
|
||
| POST | `/git/{owner}/{repo}/git-receive-pack` | Git smart HTTP push |
|
||
| POST | `/internal/git/policy` | Internal git hook policy check |
|
||
|
||
**Constants:**
|
||
|
||
| Constant | Value | Purpose |
|
||
|----------|-------|---------|
|
||
| `MAX_FRAME_BYTES` | 65,536 | Max WebSocket frame size |
|
||
| `MAX_SUBSCRIPTIONS` | 1024 | Per-connection subscription limit |
|
||
| `MAX_HISTORICAL_LIMIT` | 500 | Per-filter historical query cap |
|
||
| `handler_semaphore` capacity | 1024 | Concurrent EVENT/REQ handlers |
|
||
|
||
**Does NOT:** implement business logic — delegates to the appropriate crate for every operation.
|
||
|
||
---
|
||
|
||
### buzz-acp — Agent Communication Protocol Harness
|
||
|
||
Standalone binary that bridges Buzz relay events to AI agents via the [Agent Communication Protocol](https://agentclientprotocol.com/) (ACP).
|
||
|
||
**Architecture:**
|
||
|
||
```
|
||
Buzz Relay ──WS──→ buzz-acp ──stdio (ACP/JSON-RPC)──→ Agent (goose/codex/claude)
|
||
```
|
||
|
||
`buzz-acp` spawns AI agent subprocesses (1–32, default 1), connects to the relay via WebSocket with NIP-42 auth, discovers channels via REST API, and queues `@mention` events per channel. At most one prompt is in-flight per channel. Queued events are batched into a single prompt sent via `session/prompt` over ACP.
|
||
|
||
**Key modules:**
|
||
|
||
| Module | LOC | Responsibility |
|
||
|--------|-----|---------------|
|
||
| `relay.rs` | 3,143 | WebSocket + REST relay connection, NIP-42 auth |
|
||
| `queue.rs` | 2,565 | Per-channel event queue, batching, dedup |
|
||
| `main.rs` | 2,457 | Event loop, pool orchestration, heartbeat |
|
||
| `pool.rs` | 2,253 | N-agent pool, claim/return lifecycle |
|
||
| `config.rs` | 1,903 | CLI/env/TOML configuration |
|
||
| `acp.rs` | 1,785 | ACP client, stdio JSON-RPC, timeouts |
|
||
| `filter.rs` | 814 | Subscription rules, evalexpr filtering |
|
||
|
||
**Key behaviors:**
|
||
- Pool of 1–32 agent subprocesses with claim/return lifecycle.
|
||
- Per-channel queuing: at most one prompt in-flight per channel; subsequent @mentions queue until the agent responds.
|
||
- Crash recovery: agent subprocess crashes are detected and the agent is respawned.
|
||
- Depends on `buzz-core` (kind constants) and `buzz-sdk` (relay/REST utilities).
|
||
|
||
**Does NOT:** persist state.
|
||
|
||
---
|
||
|
||
### buzz-admin — Operator CLI
|
||
|
||
Subcommands:
|
||
|
||
| Subcommand | Purpose |
|
||
|------------|---------|
|
||
| `add-member` | Add a pubkey to the relay membership list (`--pubkey`, `--role`); accepts npub or hex; publishes kind:13534 roster |
|
||
| `remove-member` | Remove a pubkey from the relay membership list (`--pubkey`, optional `--role` guard); publishes kind:13534 roster |
|
||
| `list-members` | List all relay members |
|
||
| `generate-key` | Generate a new Nostr keypair (for bootstrapping) |
|
||
| `deletions` | Submit, inspect, approve, abort, unblock, run, or drain durable whole-community deletion requests |
|
||
| `storage-snapshot` | Run one isolated S3 accounting scan and publish its complete Postgres snapshot |
|
||
| `reconcile-channels` | Emit kind:39000/39002 discovery events for channels missing them (idempotent) |
|
||
|
||
The `buzz-admin` binary is shipped in the relay Docker image (`/usr/local/bin/buzz-admin`) and is the recommended way to manage relay membership in production. Use `./run.sh add-member`, `./run.sh remove-member`, and `./run.sh list-members` in Docker Compose deployments.
|
||
|
||
Kubernetes deployments may schedule the typed one-shot
|
||
`buzz-admin deletions drain` command directly. The pod owns its bounded
|
||
Postgres/Redis clients and S3 client; it does not call relay HTTP. Durable
|
||
requests, leases, retry timing, and checkpoints in Postgres are the handoff and
|
||
execution authority, so Kubernetes uses `Forbid` concurrency and zero Job
|
||
retries rather than introducing a second retry system.
|
||
The same drain first claims runnable approved work and, only when none exists,
|
||
may prepare one owner-origin submission. Inventory, automatic approval, and
|
||
execution share one generation lease and the existing retry/block/checkpoint
|
||
records; there is no preparation worker, command, queue, or retry authority.
|
||
|
||
---
|
||
|
||
### buzz-test-client — Integration Test Harness
|
||
|
||
**`BuzzTestClient`** wraps a WebSocket connection with a `VecDeque<RelayMessage>` buffer for message interleaving. Methods: `connect`, `connect_unauthenticated`, `authenticate`, `send_event`, `send_text_message`, `subscribe`, `close_subscription`, `recv_event`, `collect_until_eose`, `disconnect`.
|
||
|
||
**Test coverage:**
|
||
|
||
| File | Tests | Scope |
|
||
|------|-------|-------|
|
||
| `tests/e2e_relay.rs` | 27 | WebSocket protocol (auth, subscriptions, filters, limits, NIP-11) |
|
||
| `tests/e2e_media.rs` | 7 | Media upload/download (Blossom) |
|
||
| `tests/e2e_media_extended.rs` | 18 | Extended media scenarios |
|
||
| `tests/e2e_nostr_interop.rs` | 15 | Nostr interoperability: NIP-50 search, NIP-10 threads, NIP-17 gift wraps, DM discovery |
|
||
|
||
All e2e tests are `#[ignore]` — require a running relay. Total: **134 e2e tests**.
|
||
|
||
`src/main.rs` is a manual testing CLI (`buzz-test-cli`) with `--send`, `--subscribe`, `--channel`, `--url`, `--kind` flags.
|
||
|
||
Defines `parse_relay_message`, `OkResponse`, `RelayMessage` directly in `src/lib.rs`.
|
||
|
||
---
|
||
|
||
## 7. Security Model
|
||
|
||
Every security-sensitive operation uses an explicit, verified pattern. No implicit trust.
|
||
|
||
### Authentication
|
||
|
||
| Concern | Mechanism |
|
||
|---------|-----------|
|
||
| NIP-42 timestamp | ±60 second tolerance — prevents replay attacks |
|
||
| AUTH events | Never stored in Postgres, never logged in audit chain |
|
||
| NIP-98 HTTP Auth | Schnorr-signed `kind:27235` events — URL and method verification |
|
||
|
||
### Input Validation
|
||
|
||
| Concern | Mechanism |
|
||
|---------|-----------|
|
||
| Schnorr signatures | `verify_event()` in `buzz-core` — every event verified before storage |
|
||
| Event ID | SHA-256 of canonical serialization verified independently of signature |
|
||
| Frame size | `MAX_FRAME_BYTES = 65,536` — oversized frames rejected, connection closed |
|
||
| Search event IDs | 64-char hex validation before URL construction — prevents path injection |
|
||
| Workflow step IDs | Alphanumeric + underscore only — prevents evalexpr variable injection |
|
||
| Partition names | Allowlist of table names + strict suffix/date validators — prevents DDL injection |
|
||
|
||
### SSRF Protection
|
||
|
||
`is_not_global_unicast(ip)` (compat alias `is_private_ip`) in `buzz-core` is an enumerated-deny policy: it blocks a specific set of non-public address classes and accepts everything else, including addresses not covered by an explicit deny rule (e.g. `fe00::1`). Blocked IPv4 classes: loopback (127.0.0.0/8), private RFC 1918 (10/8, 172.16/12, 192.168/16), link-local (169.254/16), unspecified (0/8), broadcast, CGNAT/RFC 6598 (100.64/10), benchmarking/RFC 2544 (198.18/15), IETF Protocol Assignments (192.0.0.0/24, globally reachable exceptions: 192.0.0.9 PCP anycast RFC 7723 and 192.0.0.10 TURN anycast RFC 8155), documentation/RFC 5737 (192.0.2/24, 198.51.100/24, 203.0.113/24), deprecated 6to4 relay anycast (192.88.99.0/24, RFC 7526, global=None/blank → conservative deny), multicast/RFC 5771 (224/4), and reserved class-E (240/4). Blocked IPv6 classes: loopback (::1), unspecified (::), ULA (fc00::/7), link-local (fe80::/10), deprecated site-local (fec0::/10, RFC 3879), multicast (ff00::/8), IETF Protocol Assignments envelope (2001::/23, global exceptions: 2001:1::1–::3 PCP/TURN/DNS-SD anycast, 2001:3::/32 AMT RFC 7450, 2001:4:112::/48 AS112-v6 RFC 7535, 2001:20::/28 ORCHIDv2 RFC 7343, 2001:30::/28 DETs RFC 9374), documentation (2001:db8::/32 RFC 3849, 3fff::/20 RFC 9637), 6to4 (2002::/16, RFC 3056), Discard-Only (100::/64, RFC 6666), Dummy IPv6 Prefix (100:0:0:1::/64, RFC 9780), SRv6 SIDs (5f00::/16, RFC 9252), and NAT64 local-use (64:ff9b:1::/48, RFC 8215). IPv4 embedded in IPv4-mapped, IPv4-compatible, and NAT64 well-known (64:ff9b::/96, RFC 6052) forms is checked recursively against the IPv4 table; SIIT IPv4-translated (::ffff:0:0:0/96) follows the same path.
|
||
|
||
Applied in: `buzz-auth` (JWKS boundary), `buzz-workflow` (CallWebhook action),
|
||
desktop `link_preview` (SSRF check).
|
||
|
||
JWKS uses `public_https` by default and applies this public-address check to
|
||
every resolved answer before pinning the first permitted address. Explicit
|
||
`mesh` mode accepts HTTP only for a DNS hostname (not an IP literal) without
|
||
userinfo, query, fragment, or backslash. It checks every DNS answer against the
|
||
positive address allow-list in the
|
||
[NIP-FI JWKS snapshot contract](docs/nips/NIP-FI.md#jwks-snapshot), rejects
|
||
empty or mixed answers, and pins the first checked address. Mesh selection is
|
||
an operator assertion, not a transport property the Rust relay verifies.
|
||
Operators must enforce mutual TLS peer authentication and integrity
|
||
protection, authorize the JWKS endpoint, and restrict egress. Unauthenticated
|
||
HTTP permits key substitution and issuer impersonation.
|
||
|
||
### Audit Integrity
|
||
|
||
- Hash chain: each entry's SHA-256 covers all fields including `prev_hash` — tampering any entry breaks all subsequent hashes
|
||
- Canonical JSON: `BTreeMap` for deterministic key ordering — hash is reproducible
|
||
- Single-writer lock: `pg_advisory_lock` — prevents concurrent writes from breaking the chain
|
||
- Panic-safe: `catch_unwind` ensures lock release even on panic
|
||
|
||
### Access Control
|
||
|
||
- Channel membership is the only gate — enforced by the relay at every operation
|
||
- REQ handler checks access before subscription registration — no race window for private channel leaks
|
||
- TOCTOU-safe membership operations: all check-then-modify sequences run inside Postgres transactions
|
||
- Approval tokens: UUID (CSPRNG), stored as SHA-256 hash, single-use enforced with `AND status = 'pending'` in UPDATE
|
||
|
||
### Webhook Security
|
||
|
||
- Workflow webhooks: constant-time XOR comparison of stored UUID secret (not HMAC — compares the secret directly, not a body MAC)
|
||
- Outbound webhooks (CallWebhook): SSRF protection + redirects disabled + 1 MiB response cap
|
||
|
||
---
|
||
|
||
## 8. Infrastructure
|
||
|
||
Docker Compose provides the full local development stack. All services include health checks and resource limits.
|
||
|
||
### Services
|
||
|
||
| Service | Image | Port | Purpose |
|
||
|---------|-------|------|---------|
|
||
| Postgres | `postgres:17-alpine` | 5432 | Primary event store — events, channels, tokens, workflows, audit; full-text search (`search_tsv` GIN) |
|
||
| Redis | `redis:7-alpine` | 6379 | Pub/sub fan-out, presence (SET EX), typing (sorted sets) |
|
||
| Adminer | `adminer` | 8082 | DB web UI (dev only) |
|
||
| Silo (MinIO fork) | `pgsty/silo` | 9000 (API), 9001 (console) | S3-compatible object storage (media) |
|
||
| Prometheus | `prom/prometheus` | 9090 | Metrics collection |
|
||
|
||
### Postgres Schema (key tables)
|
||
|
||
| Table | Purpose |
|
||
|-------|---------|
|
||
| `events` | All stored Nostr events; monthly range-partitioned by `PARTITION BY RANGE` on `created_at`; multi-community mode keys every tenant-visible event by `community_id` |
|
||
| `channels` | Channel records (type, visibility, canvas, topic); `community_id` is immutable after creation in multi-community mode |
|
||
| `channel_members` | Membership with roles; soft-delete via `removed_at` |
|
||
| `workflows` | Workflow definitions (YAML stored as canonical JSON); scoped by community in multi-community mode |
|
||
| `workflow_runs` | Execution records with trigger context and trace |
|
||
| `workflow_approvals` | Approval gates (token stored as SHA-256 hash) |
|
||
| `audit_log` | Hash-chain audit entries; per-community chain/head in multi-community mode |
|
||
| `delivery_log` | Delivery tracking (partitioned; Rust module pending) |
|
||
|
||
### Redis Key Patterns
|
||
|
||
| Pattern | Type | TTL | Purpose |
|
||
|---------|------|-----|---------|
|
||
| `buzz:channel:{uuid}` | Pub/Sub channel | — | Event fan-out (single-community form; shared multi-community Redis must use `buzz:{community}:channel:{uuid}` or equivalent) |
|
||
| `buzz:presence:{pubkey_hex}` | String | 180s | Online/away status (single-community form; shared multi-community Redis must scope by community) |
|
||
| `buzz:typing:{channel_uuid}` | Sorted Set | 60s | Active typers (5s window; shared multi-community Redis must scope by community) |
|
||
|
||
### Full-Text Search (Postgres FTS)
|
||
|
||
Search runs over the `events.search_tsv` generated `tsvector` column on the
|
||
`events` table (no separate collection or service). The column is populated on
|
||
insert — `to_tsvector('simple', content)` — and excludes privacy-sensitive
|
||
kinds via `CASE WHEN kind IN (1059, 30300, 30622) THEN NULL`, so those rows are
|
||
storage-level unsearchable (a `NULL` tsvector never matches `@@`). A GIN index
|
||
(`idx_events_search_tsv`) backs the `@@` probe; in multi-community mode the
|
||
community-leading btree filters BitmapAnd with the GIN probe so every query is
|
||
fenced to its `community_id`.
|
||
|
||
---
|
||
|
||
## 9. Known Limitations
|
||
|
||
These are verified gaps in the current implementation — not design aspirations.
|
||
|
||
| # | Limitation | Detail |
|
||
|---|-----------|--------|
|
||
| 1 | **No sqlx offline query cache** | Uses `sqlx::query()` (runtime) not `sqlx::query!()` (compile-time). No `.sqlx/` directory. Queries are not validated at compile time. |
|
||
| 2 | **No rate limiting implementation** | `RateLimiter` trait exists in `buzz-auth`. Only implementation is `AlwaysAllowRateLimiter` (test stub, gated behind `#[cfg(any(test, feature = "test-utils"))]`). `RateLimitConfig` defines 4 tiers (human, agent-standard, agent-elevated, agent-platform) but none are enforced. |
|
||
| 3 | **No dedicated typing REST endpoint** | Typing indicators (kind 20002) are delivered via both local fan-out and Redis pub/sub (cross-node). There is no REST endpoint to query current typers — `/api/presence` returns online/away status only, not typing state. |
|
||
| 4 | **Huddle recording/tracks not built** | Voice, room lifecycle, and join/leave/end events are wired (see Huddle Audio above). Recording and per-track publishing have reserved kinds but no producer yet. |
|
||
| 5 | **Approval gates not wired end-to-end** | The executor returns `StepResult::Suspended` and the relay has grant/deny API endpoints with DB CRUD, but the engine intercepts before creating `WaitingApproval` rows — runs that hit an approval gate are marked as Failed (🚧 WF-08). |
|
||
| 6 | **Workflow actions partially stubbed** | The `send_dm` and `set_channel_topic` workflow actions are in the schema but return `NotImplemented` — a run that reaches one fails at execution (🚧 WF-07). |
|