beb76406c1
## Summary
This PR makes relay readiness failures diagnosable without weakening the
existing fail-closed readiness contract. It distinguishes Postgres pool
acquisition from query execution, Redis pool acquisition,
deletion-catalog validation, and the overall two-second deadline;
exports a bounded Prometheus contract for rollout dashboards; and fixes
the concurrency, shutdown, and listener-boundary semantics needed for
those signals to be trustworthy.
## Why
The previous `/_readiness` implementation exposed only an aggregate
ready/not-ready result. During a rollout, operators could not tell
whether a pod was blocked on:
- acquiring a Postgres connection;
- executing the Postgres readiness query;
- acquiring a Redis connection;
- validating the deletion catalog; or
- the shared readiness deadline.
Adding metrics to the existing handler also exposed three correctness
hazards that this PR resolves:
1. the same handler is mounted on both the public application listener
and the private Kubernetes health listener, so public requests could
otherwise distort rollout telemetry;
2. concurrent probes can finish out of order, allowing an older result
to overwrite newer current-state gauges; and
3. a probe started before SIGTERM can finish afterward, incorrectly
return `200 ready`, and resurrect ready gauges while the process is
draining.
## Behavior
### Readiness evaluation
- Postgres, Redis, and deletion-catalog checks still run under one
shared two-second deadline.
- Postgres distinguishes pool acquisition timeout/error from query
timeout/error.
- Redis distinguishes pool acquisition timeout/error. This does not
claim a Redis command round trip.
- The deletion catalog distinguishes operation timeout/error.
- Multiple failures are reported as `multiple_dependencies_failed`;
exhaustion of the shared deadline is reported as `overall_timeout` when
no more specific completed outcome wins.
- Readiness remains fail-closed: every dependency must succeed for `200
{"status":"ready"}`.
### Ordered publication and shutdown
`ReadinessCoordinator` is process-owned and uses one mutex as the
linearization point for probe generations, current-state publication,
and terminal shutdown.
- Every completed dependency attempt may contribute its truthful counter
and duration observation.
- Only the newest admissible probe generation may publish current-state
gauges.
- An older, slower probe cannot overwrite a newer probe's gauges.
- `begin_shutdown()` and probe commit serialize through the same
coordinator.
- Once shutdown commits, an in-flight probe cannot return ready or
publish ready/current dependency gauges, even if its dependency work
later succeeds.
Shutdown without dependency evaluation records only:
- `buzz_readiness_checks_total{reason="shutting_down"}`; and
- `buzz_readiness_state{check="overall"} = 0`.
It does **not** fabricate dependency failures, dependency state changes,
or zero-duration latency samples. If shutdown wins after an in-flight
evaluation actually ran, those completed dependency attempts may remain
as attempt telemetry, but they cannot overwrite shutdown-dominant
current state.
### Listener and response contract
- The private health listener's `/_readiness` route is the sole
authority for rollout readiness telemetry.
- The public application listener retains `/_readiness` for
compatibility, evaluates the same dependencies, and preserves the
existing response shape, but it emits no `buzz_readiness_*` metrics.
- Ready responses remain `200 {"status":"ready"}`.
- Shutdown responses remain `503 {"status":"shutting_down"}`.
- Failed private-health responses include the bounded `reason` plus
`postgres`, `redis`, and `deletion_catalog` booleans.
- Failed public compatibility responses retain the dependency booleans
but omit the new detailed reason.
- No header, query parameter, path value, or other request-controlled
value becomes a metric label.
## Prometheus contract
The final schema is intentionally capped at **99 raw Prometheus series
per pod**.
| Metric | Type | Labels | Raw series/pod |
|---|---|---|---:|
| `buzz_readiness_checks_total` | counter | `reason` | 12 |
| `buzz_readiness_dependency_checks_total` | counter | `dependency`,
typed `outcome` | 11 |
| `buzz_readiness_check_duration_seconds` | histogram | `check` | 72 |
| `buzz_readiness_state` | gauge | `check` | 4 |
| **Total** | | | **99** |
### Closed label sets
`reason`:
```text
ready
shutting_down
postgres_pool_timeout
postgres_pool_error
postgres_query_timeout
postgres_query_error
redis_pool_timeout
redis_pool_error
deletion_catalog_timeout
deletion_catalog_error
overall_timeout
multiple_dependencies_failed
```
Valid `dependency` / `outcome` pairs are enforced by typed enums:
- `postgres`: `success | pool_timeout | pool_error | operation_timeout |
operation_error`
- `redis`: `success | pool_timeout | pool_error`
- `deletion_catalog`: `success | operation_timeout | operation_error`
`check` is `overall | postgres | redis | deletion_catalog`.
The readiness histogram has 15 configured buckets concentrated around
the two-second deadline:
```text
0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5,
0.75, 1.0, 1.25, 1.5, 1.75, 2.0, 2.5, +Inf
```
The contract deliberately removes the redundant overall `result` label
and the histogram `outcome` label. Pod, ReplicaSet, version, rollout,
raw error, SQL, URL, tenant, user, community, pubkey, and
request-controlled values are prohibited application labels;
infrastructure enrichment can supply deployment identity outside the
application metric.
## Postgres production seam and CI
The real `Db::readiness_check` path now supports deterministic
production-seam testing while preserving the same acquisition/query
implementation used by the relay. The isolated PostgreSQL lane
automatically discovers and executes three ignored integration tests
covering:
- a held sole connection causing pool timeout, followed by recovery
after release;
- closed-pool acquisition error;
- acquisition success followed by query timeout;
- classified query error;
- cancellation while waiting for a connection;
- cancellation during an in-flight query; and
- eventual pool recovery with waiter/in-flight state balanced.
This prevents the central SQLx/Postgres behavior from being merely
compiled but never executed in CI.
## Tests and verification
- `cargo fmt --all -- --check`
- `./scripts/test-postgres-test-discovery.sh`
- complete `cargo test -p buzz-db`
- focused `cargo test -p buzz-relay readiness`
- real-router `cargo test -p buzz-relay real_health_route_`
- controlled PostgreSQL execution of all three ignored readiness tests
- production health router -> real `GET /_readiness` -> Prometheus
render assertions
- public-router requests proving zero rollout telemetry
- deterministic out-of-order A/B probe tests
- SIGTERM-during-probe tests proving shutdown dominance
- exported metric name/type/exact-label/bucket assertions
- exported raw-series allowlist and 99-series ceiling assertion
The route-to-scrape regression test uses the production Prometheus
builder and verifies the `2`, `2.5`, and `+Inf` readiness buckets. It
fails if the health recording call, health route, public/health
boundary, generation fence, shutdown fence, or bucket override is
removed.
## Risk assessment
**Medium.** This changes the live readiness publication path and adds
synchronization around probe commit/shutdown. The risk is bounded by:
- preserving the existing dependencies and shared two-second deadline;
- keeping readiness fail-closed;
- using a short, process-local mutex only at begin/commit linearization
points, not across dependency awaits;
- retaining the public compatibility endpoint while isolating its
telemetry;
- enforcing typed, low-cardinality labels and an exported series
ceiling; and
- covering the production router, Prometheus exposition, real PostgreSQL
seam, concurrency ordering, and shutdown races.
## Operational notes
- Dashboards should treat `buzz_readiness_state` as the latest sampled
current state, not as an event stream.
- `shutting_down` should be excluded from dependency-failure alerts
because no dependency failure is implied.
- Do not use `default_zero()` for missing current-state data;
missing/stale is unknown, not healthy.
- Use histogram buckets, heatmaps, max, or average until the Datadog
distribution metadata confirms percentiles are enabled; do not title a
widget p95 before that live readback.
## Non-goals
This PR does not add the broader process-startup lifecycle,
worker/listener supervision, shutdown coordinator, WebSocket/huddle
handoff, client recovery telemetry, dashboard mutations, Datadog
configuration changes, or deployment changes. Those remain separate
follow-up work.
## References
- [Staging dev relay image
runbook](https://github.com/block/buzz/blob/main/docs/staging-dev-relay-images.md)
- Readiness telemetry contract: `deploy/charts/buzz/README.md`
---------
Signed-off-by: Ravneet Arora <rarora@squareup.com>
Buzz 🐝
A workspace where humans and agents build together, on a relay you own.
Vision · Sovereign · Forge · Agents · Architecture · Releasing · Apache 2.0
People and agents building together in the same room.
What is this, really?
Buzz is a self-hostable workspace where humans and AI agents share the same rooms.
A Buzz community is the workspace a user reaches by URL. In the single-relay setup that ships today, the relay URL selects exactly one community. A hosted operator can serve many communities behind many domains or subdomains, but the client-facing rule stays the same: the URL is authoritative for the workspace, and all tenant-observable state under that URL is community-local.
It's a Nostr relay: every message, reaction, workflow step, review approval, and git event is a signed event in one log. Same shape, same identity model, same audit trail, whether the author is a person or a process.
In practice it feels like a team workspace. Under the hood it's an event log with taste and a suspicious number of Rust crates.
Yes, it's another AI-adjacent developer tool. We're sorry. The difference is what agents can actually do once they're inside: open repos, send patches, review code, run workflows, edit canvases, orchestrate other agents, drop into voice huddles, create channels, and pull in whoever needs to see it. The same affordances as a human teammate, the same audit trail, a different keypair.
Stuff you do in Buzz
- Ask the project a question and get an answer with receipts. Agents search six months of history and post the threads, not vibes.
- Let an agent triage a bug without giving it the keys to the kingdom. Agents have their own keys, their own channel memberships, and their own audit trail. Scoped by identity, not by permission flags — the same way you'd scope a teammate.
- Turn a feature branch into a room where patches, CI, review, and the merge decision live together — so the channel becomes the record of why the code exists.
- Search the conversation, the patch, the workflow run, and the approval in one place — because they're all the same kind of event.
- Let an agent run the workspace, not just talk in it. Channels, canvases, workflows, huddles — agents have the same surface area as humans, with their own keys and their own audit trail.
A look inside
 Agents are members, not bots. Add an agent to a channel the same way you add a person. |
 Spin up a room in seconds. Name it, describe it, make it private. |
 Media you can talk about. Leave comments pinned to specific frames. |
|
Why Buzz is better
One community. One identity model. One event log. Humans, agents, workflows, and repos all speak the same protocol, sign with the same kind of key, and end up in the same search index. In the default self-hosted deployment, one relay hosts one community; in a hosted multi-tenant deployment, each community keeps that same semantic boundary even when the backend shares Postgres, Redis, and object storage.
The bet is that one community can do what teams currently fake with chat, forges, bots, CI dashboards, release tools, search indexes, and a pile of glue code. Not all at once, not magically, but with one substrate instead of seven tabs pretending they know about each other.
Agents are part of the room, not haunted cron jobs.
Three little stories
Incident memory. It's 2am. You type "have we seen this error before?" An agent watching the channel pulls six months of history, posts the threads, the root causes, the fixes, and offers to page whoever shipped the last one. The whole exchange — question, answer, evidence — stays in the channel.
Branch as room. You open a feature branch. A channel appears. Patches land as NIP-34 events, CI posts results, an agent runs a first-pass review, teammates react to the parts they care about, and the merge decision lands in the same room as the evidence.
A release that writes itself. A workflow fires on a tag. An agent reads the merged PRs from the project channels, drafts the release notes, posts them for human review, gets a 👍 reaction, and ships. Every step signed. Every step searchable.
Works today · Being wired up · Strong opinions, pending code
| ✅ Works today | 🚧 Being wired up | 💭 Strong opinions, pending code |
|---|---|---|
| Relay, channels, threads, DMs, canvases, media, search, audit log | Mobile clients (iOS + Android, Flutter) | Web-of-trust reputation across relays |
| Desktop app (Tauri + React) | Workflow approval gates (infra exists, glue still drying) | Push notifications |
buzz-cli (agent-first, JSON in / JSON out) + ACP harness (Goose, Codex, Claude Code) |
Huddle lifecycle events | Culture features |
| YAML workflows: message / reaction / schedule / webhook triggers | ||
| Git events (NIP-34: patches, repo announcements, status) | ||
| Git hosting backend |
Please do not plan your compliance program around the 💭 column yet. The VISION docs are the long version of what we think this becomes.
Getting started
New to Buzz? Pick the path that matches you.
I just want to try the app
Grab a packaged build from the latest release:
| Platform | File |
|---|---|
| macOS (Apple Silicon) | Buzz_<version>_aarch64.dmg |
| macOS (Intel) | Buzz_<version>_x64.dmg |
| Linux (x86_64) | Buzz_<version>_amd64.AppImage or Buzz_<version>_amd64.deb |
| Windows (x64) | Buzz_<version>_x64-setup_alpha-unsigned.exe |
On a Mac, check the Apple menu > About This Mac: "Chip: Apple …" means Apple Silicon; "Processor: Intel …" means Intel.
The Windows build is not code-signed, so SmartScreen may show "Windows protected your PC" on first launch. If available, click More info, then Run anyway.
By default the app connects to ws://localhost:3000. To point it at a relay you're running or one someone shared with you, set BUZZ_RELAY_URL before launching, or switch the relay from inside the app. If you don't have a relay yet, follow Build & run from source below to stand one up locally.
I want my own hosted relay
To run a relay for your team without managing servers, you can deploy one to Railway in a click:
See here for details.
I work at Block
Don't build from source, and don't use the OSS release — use the internal build. It comes pre-wired to the Block relay and agent provider, so it works out of the box with nothing to configure.
Download the latest build from squareup/buzz-releases releases and install it.
I want to build & run from source
See Quick start below — this is the developer / self-host path.
Quick start
You'll need Docker and Hermit (or Rust 1.88+, Node 24+, pnpm 10+, just).
Once:
git clone https://github.com/block/buzz.git && cd buzz
. ./bin/activate-hermit # pinned toolchain (tools auto-download on first use)
just setup && just build
just setup runs just bootstrap automatically — it copies .env.example to .env if needed, downloads all required tools via Hermit, and starts Docker services + migrations.
Every day:
. ./bin/activate-hermit
just dev # starts the relay + desktop app together
Relay on ws://localhost:3000. Desktop app pops up. You're in.
For a split-terminal workflow (relay logs separate from Vite output), use just relay in one terminal and just desktop-dev in another.
Want a single-node / VPS relay instead of the local-dev stack? Use the production Compose bundle in deploy/compose/ (docker compose + Postgres, Redis, MinIO, optional Caddy/TLS). The root docker-compose.yml is for day-to-day development only.
For agents, set BUZZ_PRIVATE_KEY and use buzz-cli — JSON in, JSON out, designed for LLM tool calls.
Windows prerequisites
The agent shell tool runs commands under bash. On macOS and Linux that's already there; on Windows you need to bring it.
Install Git for Windows — it ships Git Bash, which is what buzz resolves at runtime. Once it's installed, everything works the same as on other platforms.
If you'd rather point buzz at a different bash-compatible shell, set BUZZ_SHELL to its path (e.g. BUZZ_SHELL=C:\path\to\bash.exe). The agent's tool description updates automatically to reflect whichever shell is active.
Architecture
┌─────────────────────────────────────────────────────────────────────────┐
│ Clients │
│ Human client AI agent CLI / scripts │
│ (Buzz desktop) (Goose, Codex, ...) (buzz-cli, agents) │
│ │ ┌──────────────┐ │ │
│ │ │ buzz-acp │ │ │
│ │ │ (ACP ↔ MCP) │ │ │
│ │ └──────┬───────┘ │ │
│ │ │ │ │
└───────┼──────────────────────┼───────────────────────┼──────────────────┘
│ WebSocket │ WS + REST │ WS + REST
▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ buzz-relay │
│ NIP-01 · NIP-42 auth · channel/DM/media/workflow/git REST · audit log │
└───┬──────────────────────────┬──────────────────────────┬───────────────┘
│ │ │
┌──▼───────────┐ ┌──────▼──────┐ ┌───────▼─────┐
│ Postgres │ │ Redis │ │ S3/MinIO │
│ (events + │ │ (pub/sub) │ │ (Blossom) │
│ FTS search) │ └─────────────┘ └─────────────┘
└──────────────┘
A Rust workspace of focused crates. Single source of truth: the relay. See ARCHITECTURE.md for the full breakdown.
Crate map
Core protocol — buzz-core (zero-I/O types, NIP-01 filters, Schnorr verify) · buzz-relay (Axum WS + REST)
Services — buzz-db (Postgres) · buzz-auth (NIP-42/98 Schnorr auth, rate limiting) · buzz-pubsub (Redis, presence, typing) · buzz-search (Postgres FTS) · buzz-audit (hash-chain log). Multi-community mode scopes tenant-observable rows, cache keys, search documents, workflow state, media metadata, git repo pointers, and audit chains by the host-derived community; shared infrastructure is an implementation detail, not a user-visible global workspace.
Agent surface — buzz-cli (agent-first CLI, JSON in / JSON out) · buzz-acp (ACP harness for Goose/Codex/Claude Code) · buzz-agent (ACP agent — see VISION_AGENT.md) · buzz-dev-mcp (shell + file-edit tools) · buzz-workflow (YAML automation) · buzz-persona (agent persona packs)
Git & pairing — git-sign-nostr / git-credential-nostr (nostr-signed git) · buzz-pair-relay / buzz-pairing-cli (relay pairing)
Shared — buzz-sdk (typed event builders) · buzz-media (Blossom/S3)
Tooling — buzz-admin (admin CLI) · buzz-test-client (E2E)
Going further
- VISION.md · VISION_SOVEREIGN.md · VISION_PROJECTS.md · VISION_AGENT.md — the four vision docs
- ARCHITECTURE.md — system design, kind ranges, subsystem boundaries
- TESTING.md — multi-agent E2E test suite
- CONTRIBUTING.md · CODE_OF_CONDUCT.md · SECURITY.md · GOVERNANCE.md
Configuration (env vars, defaults work for local dev)
All defaults work out of the box. Override via .env. Full reference in .env.example.
Common dev commands
just setup # Docker, migrations, desktop deps
just relay # Run the relay
just dev # Run the desktop app
just build # Build the Rust workspace
just check # fmt + clippy + desktop check
just test-unit # Unit tests (no infra required)
just test # Full suite (starts services if needed)
just ci # Everything CI runs
just reset # ⚠️ Wipe data + recreate
What it is not
- Not blockchain. Signed events are useful without making everyone buy a commemorative coin.
- Not an AI replacement plan. Buzz works best when humans stay in the loop and agents stay in the room.
- Not finished. We will tell you what works and what doesn't.
What it is: one relay where humans, agents, workflows, git events, and project memory cooperate — the beginning of a workspace that can grow past the tabs it replaces.
Buzz 🐝
Apache 2.0 · Built by Block, Inc.