beb76406c1
## Summary
This PR makes relay readiness failures diagnosable without weakening the
existing fail-closed readiness contract. It distinguishes Postgres pool
acquisition from query execution, Redis pool acquisition,
deletion-catalog validation, and the overall two-second deadline;
exports a bounded Prometheus contract for rollout dashboards; and fixes
the concurrency, shutdown, and listener-boundary semantics needed for
those signals to be trustworthy.
## Why
The previous `/_readiness` implementation exposed only an aggregate
ready/not-ready result. During a rollout, operators could not tell
whether a pod was blocked on:
- acquiring a Postgres connection;
- executing the Postgres readiness query;
- acquiring a Redis connection;
- validating the deletion catalog; or
- the shared readiness deadline.
Adding metrics to the existing handler also exposed three correctness
hazards that this PR resolves:
1. the same handler is mounted on both the public application listener
and the private Kubernetes health listener, so public requests could
otherwise distort rollout telemetry;
2. concurrent probes can finish out of order, allowing an older result
to overwrite newer current-state gauges; and
3. a probe started before SIGTERM can finish afterward, incorrectly
return `200 ready`, and resurrect ready gauges while the process is
draining.
## Behavior
### Readiness evaluation
- Postgres, Redis, and deletion-catalog checks still run under one
shared two-second deadline.
- Postgres distinguishes pool acquisition timeout/error from query
timeout/error.
- Redis distinguishes pool acquisition timeout/error. This does not
claim a Redis command round trip.
- The deletion catalog distinguishes operation timeout/error.
- Multiple failures are reported as `multiple_dependencies_failed`;
exhaustion of the shared deadline is reported as `overall_timeout` when
no more specific completed outcome wins.
- Readiness remains fail-closed: every dependency must succeed for `200
{"status":"ready"}`.
### Ordered publication and shutdown
`ReadinessCoordinator` is process-owned and uses one mutex as the
linearization point for probe generations, current-state publication,
and terminal shutdown.
- Every completed dependency attempt may contribute its truthful counter
and duration observation.
- Only the newest admissible probe generation may publish current-state
gauges.
- An older, slower probe cannot overwrite a newer probe's gauges.
- `begin_shutdown()` and probe commit serialize through the same
coordinator.
- Once shutdown commits, an in-flight probe cannot return ready or
publish ready/current dependency gauges, even if its dependency work
later succeeds.
Shutdown without dependency evaluation records only:
- `buzz_readiness_checks_total{reason="shutting_down"}`; and
- `buzz_readiness_state{check="overall"} = 0`.
It does **not** fabricate dependency failures, dependency state changes,
or zero-duration latency samples. If shutdown wins after an in-flight
evaluation actually ran, those completed dependency attempts may remain
as attempt telemetry, but they cannot overwrite shutdown-dominant
current state.
### Listener and response contract
- The private health listener's `/_readiness` route is the sole
authority for rollout readiness telemetry.
- The public application listener retains `/_readiness` for
compatibility, evaluates the same dependencies, and preserves the
existing response shape, but it emits no `buzz_readiness_*` metrics.
- Ready responses remain `200 {"status":"ready"}`.
- Shutdown responses remain `503 {"status":"shutting_down"}`.
- Failed private-health responses include the bounded `reason` plus
`postgres`, `redis`, and `deletion_catalog` booleans.
- Failed public compatibility responses retain the dependency booleans
but omit the new detailed reason.
- No header, query parameter, path value, or other request-controlled
value becomes a metric label.
## Prometheus contract
The final schema is intentionally capped at **99 raw Prometheus series
per pod**.
| Metric | Type | Labels | Raw series/pod |
|---|---|---|---:|
| `buzz_readiness_checks_total` | counter | `reason` | 12 |
| `buzz_readiness_dependency_checks_total` | counter | `dependency`,
typed `outcome` | 11 |
| `buzz_readiness_check_duration_seconds` | histogram | `check` | 72 |
| `buzz_readiness_state` | gauge | `check` | 4 |
| **Total** | | | **99** |
### Closed label sets
`reason`:
```text
ready
shutting_down
postgres_pool_timeout
postgres_pool_error
postgres_query_timeout
postgres_query_error
redis_pool_timeout
redis_pool_error
deletion_catalog_timeout
deletion_catalog_error
overall_timeout
multiple_dependencies_failed
```
Valid `dependency` / `outcome` pairs are enforced by typed enums:
- `postgres`: `success | pool_timeout | pool_error | operation_timeout |
operation_error`
- `redis`: `success | pool_timeout | pool_error`
- `deletion_catalog`: `success | operation_timeout | operation_error`
`check` is `overall | postgres | redis | deletion_catalog`.
The readiness histogram has 15 configured buckets concentrated around
the two-second deadline:
```text
0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5,
0.75, 1.0, 1.25, 1.5, 1.75, 2.0, 2.5, +Inf
```
The contract deliberately removes the redundant overall `result` label
and the histogram `outcome` label. Pod, ReplicaSet, version, rollout,
raw error, SQL, URL, tenant, user, community, pubkey, and
request-controlled values are prohibited application labels;
infrastructure enrichment can supply deployment identity outside the
application metric.
## Postgres production seam and CI
The real `Db::readiness_check` path now supports deterministic
production-seam testing while preserving the same acquisition/query
implementation used by the relay. The isolated PostgreSQL lane
automatically discovers and executes three ignored integration tests
covering:
- a held sole connection causing pool timeout, followed by recovery
after release;
- closed-pool acquisition error;
- acquisition success followed by query timeout;
- classified query error;
- cancellation while waiting for a connection;
- cancellation during an in-flight query; and
- eventual pool recovery with waiter/in-flight state balanced.
This prevents the central SQLx/Postgres behavior from being merely
compiled but never executed in CI.
## Tests and verification
- `cargo fmt --all -- --check`
- `./scripts/test-postgres-test-discovery.sh`
- complete `cargo test -p buzz-db`
- focused `cargo test -p buzz-relay readiness`
- real-router `cargo test -p buzz-relay real_health_route_`
- controlled PostgreSQL execution of all three ignored readiness tests
- production health router -> real `GET /_readiness` -> Prometheus
render assertions
- public-router requests proving zero rollout telemetry
- deterministic out-of-order A/B probe tests
- SIGTERM-during-probe tests proving shutdown dominance
- exported metric name/type/exact-label/bucket assertions
- exported raw-series allowlist and 99-series ceiling assertion
The route-to-scrape regression test uses the production Prometheus
builder and verifies the `2`, `2.5`, and `+Inf` readiness buckets. It
fails if the health recording call, health route, public/health
boundary, generation fence, shutdown fence, or bucket override is
removed.
## Risk assessment
**Medium.** This changes the live readiness publication path and adds
synchronization around probe commit/shutdown. The risk is bounded by:
- preserving the existing dependencies and shared two-second deadline;
- keeping readiness fail-closed;
- using a short, process-local mutex only at begin/commit linearization
points, not across dependency awaits;
- retaining the public compatibility endpoint while isolating its
telemetry;
- enforcing typed, low-cardinality labels and an exported series
ceiling; and
- covering the production router, Prometheus exposition, real PostgreSQL
seam, concurrency ordering, and shutdown races.
## Operational notes
- Dashboards should treat `buzz_readiness_state` as the latest sampled
current state, not as an event stream.
- `shutting_down` should be excluded from dependency-failure alerts
because no dependency failure is implied.
- Do not use `default_zero()` for missing current-state data;
missing/stale is unknown, not healthy.
- Use histogram buckets, heatmaps, max, or average until the Datadog
distribution metadata confirms percentiles are enabled; do not title a
widget p95 before that live readback.
## Non-goals
This PR does not add the broader process-startup lifecycle,
worker/listener supervision, shutdown coordinator, WebSocket/huddle
handoff, client recovery telemetry, dashboard mutations, Datadog
configuration changes, or deployment changes. Those remain separate
follow-up work.
## References
- [Staging dev relay image
runbook](https://github.com/block/buzz/blob/main/docs/staging-dev-relay-images.md)
- Readiness telemetry contract: `deploy/charts/buzz/README.md`
---------
Signed-off-by: Ravneet Arora <rarora@squareup.com>
Buzz Helm Chart
Buzz is a Nostr-based messaging platform for human–agent collaboration: a single relay binary serving WebSocket + REST + web UI, backed by PostgreSQL, Redis, and S3-compatible object storage.
This chart has two operating profiles selected by values:
| Profile | When | What you get |
|---|---|---|
| Production (default) | Self-hosted multi-tenant, regulated, or GitOps-managed | External managed Postgres/Redis/S3, secrets.existingSecret:, no chart-side autogen, HA-capable (replicaCount ≥ 2) |
| Quickstart (eval) | Eval, single-node, one-off demo | In-cluster Postgres + Redis + MinIO subcharts/Deployments, chart auto-generates relay + service secrets, single replica |
Quickstart (eval only)
helm install buzz oci://ghcr.io/block/buzz/charts/buzz --version 0.1.8 \
--create-namespace --namespace buzz \
--set quickstart=true \
--set postgresql.enabled=true \
--set redis.enabled=true \
--set minio.enabled=true \
--set relayUrl=wss://buzz.example.com \
--set ownerPubkey=<64-char-hex-pubkey>
This brings up everything in-cluster — Postgres, Redis, and MinIO (with
its bucket created by a post-install Job) — and composes the relay's
BUZZ_S3_ENDPOINT plus autogenerated credentials automatically. No external services required. The quickstart=true flag is an
intent marker surfaced in NOTES.txt; the bundled services are opted in via the
four *.enabled flags above (see ci/quickstart-values.yaml for the exact set
CI installs). Eval-only: every bundled service is a single replica with no HA.
For immutable delivery, pin the OCI digest instead of a tag. image.digest
overrides image.tag when both are present:
image:
repository: ghcr.io/block/buzz
digest: sha256:<64-lowercase-hex-characters>
Production (GitOps)
The chart is designed for ArgoCD and Flux. Both render charts with helm template, in which mode Helm's lookup function returns empty — any chart-side randAlphaNum call would regenerate secrets on every sync. The chart-managed Secret path is only safe for helm install / helm upgrade.
Production deploys MUST use secrets.existingSecret:. The Secret is consumed for any keys present and ignored for keys missing — extras are harmless.
To enable relay-proxied KLIPY search, add BUZZ_KLIPY_API_KEY to that Secret.
The key stays in the relay pod; clients discover the public buzz-gif
extension and gif descriptor in NIP-11, then receive KLIPY-hosted media URLs.
See docs/gif-search.md for the protocol and
security boundaries.
See:
examples/argocd-app.yaml— ArgoCD Applicationexamples/flux-helmrelease.yaml— Flux HelmRelease v2examples/secret-sample.yaml— Secret schema
Required inputs
| Key | What | When required |
|---|---|---|
relayUrl |
Public wss:// URL clients connect to |
Always |
ownerPubkey |
64-char lowercase hex Nostr pubkey of the relay operator | When relay.requireRelayMembership=true (default) |
secrets.existingSecret |
Name of pre-created Secret | Production / GitOps |
externalPostgresql.url / externalRedis.url / s3.endpoint |
External service URLs | Production — when the matching bundled service is disabled (the default) |
The chart fails at helm install / helm template time with a clear message if any of these are missing or malformed (see templates/_validate.tpl).
S3 URL addressing
Buzz uses one URL style for both media and Git/CAS object-store requests:
s3.addressingStyle |
Request shape | Use for |
|---|---|---|
path (default) |
https://endpoint/bucket/key |
Bundled MinIO and endpoints whose DNS does not resolve bucket subdomains |
virtual |
https://bucket.endpoint/key |
AWS-style providers and new Railway Storage Buckets |
The chart always renders s3.addressingStyle as
BUZZ_S3_ADDRESSING_STYLE and s3.region as BUZZ_S3_REGION. The region
defaults to us-east-1, keeping bundled MinIO and the in-pod
buzz-admin deletions workflow operable without an ambient AWS_REGION.
Production providers must set their credential region explicitly when it
differs. Existing releases that previously omitted s3.region will begin
rendering BUZZ_S3_REGION=us-east-1 after upgrade, even if an image or
relay.extraEnv entry supplied AWS_REGION; set s3.region to the provider's
actual credential region before upgrading. Only path and virtual addressing
styles are accepted; invalid
values fail chart rendering and relay startup. The bundled MinIO quickstart
deliberately keeps path because its Service DNS resolves one endpoint
hostname, not arbitrary <bucket>.<service> names.
For a Railway Storage Bucket, map its variables to chart values in the service or generated Helm configuration:
s3:
endpoint: "${{Object Storage.ENDPOINT}}"
bucket: "${{Object Storage.BUCKET}}"
region: "${{Object Storage.REGION}}"
addressingStyle: virtual
Store BUZZ_S3_ACCESS_KEY=${{Object Storage.ACCESS_KEY_ID}} and
BUZZ_S3_SECRET_KEY=${{Object Storage.SECRET_ACCESS_KEY}} in the Secret named by
secrets.existingSecret. Railway's Credentials tab is authoritative for older
buckets, which may still require path. The setting changes request routing and
SigV4 signing, so do not put the bucket into s3.endpoint; pass Railway's base
ENDPOINT and BUCKET separately.
Object storage is contacted during relay startup only when
BUZZ_GIT_CONFORMANCE_PROBE is enabled (the relay default). A probe failure is
startup-fatal, so Kubernetes readiness never opens. If an operator explicitly
disables that probe through relay.extraEnv, /_readiness does not test object
storage; configuration is still parsed strictly, but reachability and addressing
errors surface on the first storage operation.
Readiness telemetry contract
Only requests served by the private health listener (BUZZ_HEALTH_PORT) emit
rollout readiness telemetry. The compatibility /_readiness route on the public
app listener returns health but does not change these metrics.
| Metric | Type | Labels |
|---|---|---|
buzz_readiness_checks_total |
counter | reason from the closed readiness-reason set |
buzz_readiness_dependency_checks_total |
counter | dependency, typed bounded outcome |
buzz_readiness_check_duration_seconds |
histogram | check only |
buzz_readiness_state |
gauge | check only; latest publishable generation |
The schema has a ceiling of 99 raw Prometheus series per pod: 12 overall
reasons, 11 valid dependency/outcome pairs, 72 histogram series, and 4 gauges.
Do not add pod, ReplicaSet, version, rollout, error text, SQL, URL, tenant,
user, community, pubkey, header, query, or other request-controlled labels.
Shutdown without dependency evaluation increments only
buzz_readiness_checks_total{reason="shutting_down"} and sets the overall
state to zero; it does not fabricate dependency failures or latency samples.
Relay Pod extensions
The chart exposes narrow extension points for init containers, volumes, relay
volume mounts, and image command/argument overrides. extraManifests creates
independent Kubernetes resources but cannot modify the chart-managed relay
Deployment. These extension values insert fields into that Deployment, avoiding
duplication of its environment, probes, security context, secrets, and
chart-owned volumes.
For example, an init container can copy a wrapper binary into a shared volume and make that wrapper the relay entrypoint:
extraInitContainers:
- name: install-wrapper
image: example.com/wrapper-init:v1
args: [/opt/wrapper/wrapper]
securityContext:
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
resources:
requests:
cpu: 10m
memory: 16Mi
volumeMounts:
- name: wrapper
mountPath: /opt/wrapper
extraVolumes:
- name: wrapper
emptyDir: {}
relay:
command: [/opt/wrapper/wrapper]
args: [/usr/local/bin/buzz-relay]
extraVolumeMounts:
- name: wrapper
mountPath: /opt/wrapper
These values are raw Kubernetes fragments rendered with toYaml, not tpl.
The chart does not validate cross-field relationships: extension names must not
collide with chart-owned containers or volumes, mounts must reference existing
volumes, and each init container must define an appropriate security context
and resources. Empty relay.command and relay.args arrays preserve the image
defaults; non-empty values override its entrypoint and arguments respectively.
Device pairing relay
The chart can run Buzz's stateless pairing WebSocket relay as an independent Deployment and Service using the same image as the main relay:
pairingRelay:
enabled: true
url: wss://pairing.example.com
pairingRelay.url is advertised in the main relay's NIP-11 document so Buzz
clients connect directly to the dedicated endpoint. The chart does not create
an Ingress or HTTPRoute for the pairing Service; route the public hostname to
<release>-buzz-pairing:5000 with your platform's ingress configuration.
HA (production)
replicaCount > 1 hard-requires Redis:
- Redis (
redis.enabled=true,externalRedis.url, orREDIS_URLinexistingSecret) — forbuzz-pubsubfan-out
It does not require ReadWriteMany git storage. Git ref/object state is object-store-backed (each request hydrates an ephemeral repo from S3-compatible storage; writer serialization is the object-store pointer CAS — see docs/git-on-object-storage.md), and repo-name uniqueness lives in Postgres. Each replica can use its own ReadWriteOnce volume; no shared filesystem is needed.
The chart template-fails if the Redis invariant is broken at replicaCount > 1. No silent degradation.
Relay autoscaling
The optional HPA scales the relay on the larger recommendation from CPU or average active WebSockets per pod:
autoscaling:
enabled: true
minReplicas: 5
maxReplicas: 15
targetCPUUtilizationPercentage: 65
websocketMetricEnabled: true
websocketMetricName: buzz_ws_connections_active
targetWebsocketConnections: 5000
CPU scaling requires Kubernetes Metrics Server. Set websocketMetricEnabled: false
for a CPU-only HPA. WebSocket scaling additionally requires a custom-metrics
adapter (for example Prometheus Adapter) configured to expose the relay's
buzz_ws_connections_active gauge as a pod metric with the name in
websocketMetricName. The chart creates the HPA but deliberately does not
install or configure a cluster-wide metrics adapter. Scale-down is gradual by
default so long-lived WebSocket connections have time to drain.
Upgrades
Schema migrations are embedded in the relay binary via sqlx::migrate! and run at startup, gated by BUZZ_AUTO_MIGRATE (default true). Multiple replicas race-safely behind a Postgres advisory lock. helm upgrade is the entire upgrade procedure.
Migration 0032 is a hard compatibility boundary for relay versions that publish repaired channel rosters. The relay verifies the roster-fence trigger catalog and behavior before opening listeners and refuses to start if 0032 is missing or inert. Apply migrations before rolling the relay; for large installations, prefer a controlled buzz-admin migrate job with PostgreSQL lock monitoring before the code rollout.
If you prefer decoupling migrations from serving, set migrate.autoMigrate=false. In that mode the chart does not run migrations for you — you own running buzz-admin migrate (separate Pod / one-shot Job) against the database before every helm install / helm upgrade. Readiness probes only verify DB connectivity, not schema freshness, so a pod will appear healthy against an unmigrated schema and fail under load. A pre-upgrade Helm Job for this is on the chart roadmap; the values knob migrate.preUpgradeJob.enabled is reserved.
Backups
Save these. Losing any of them is data loss. See NOTES.txt printed by helm install for the live list:
BUZZ_RELAY_PRIVATE_KEY— relay identity. Rotating it = new identity (federation peers will not recognize the relay).- PostgreSQL database — the canonical event store.
- S3 bucket — media blobs (chart default bucket:
buzz-media). - Git PVC — repo on-disk state served by the relay's git endpoint.
- Owner private key — held by the operator, not by this chart. Restore by re-installing with the same
ownerPubkey.
Honest limitations (v1)
- Bundled MinIO is eval-only. The quickstart profile runs an in-cluster
MinIO (single replica, no HA,
lookup-autogenerated credentials) so the relay starts with zero external object storage. Production leavesminio.enabledoff and pointss3.endpoint(orBUZZ_S3_*inexistingSecret) at managed S3-compatible storage. The bundled Deployment is not GitOps-safe and is not intended for production traffic. - Minimal-mode is not yet supported. The relay's
BUZZ_PUBSUB=local/ filesystem media paths are upstream work in progress — even quickstart currently stands up real Redis and S3 rather than the relay's single-node fallbacks. (Full-text search already runs in Postgres, so no separate search service is provisioned.) - Cosign signing of the published chart is a follow-up (the relay image is
attested via
actions/attest-build-provenance; the chart is not yet). The chart itself is published to GHCR — see Releasing.
Releasing
The chart is published to GHCR as an OCI artifact at
oci://ghcr.io/block/buzz/charts/buzz by the helm chart workflow
(.github/workflows/helm-chart.yml), versioned independently of the desktop app
and the relay image via its own chart-v* tags. Every PR/main push still
lints, unit-tests, and render-checks the chart; only a chart-v* tag publishes,
so an in-progress main can never overwrite a released version.
To cut a release, push a chart-release/<version> branch whose <version>
matches Chart.yaml's version; merging it auto-tags chart-v<version> and
dispatches the publish job (same lane machinery as the desktop and relay
releases — see .github/workflows/auto-tag-on-release-pr-merge.yml). The publish
job fails loudly if the tag version and Chart.yaml version disagree.
Development
# Render every fixture
for f in ci/*-values.yaml tests/fixtures/*-values.yaml; do
helm template buzz . -f "$f" >/dev/null && echo "ok: $f"
done
# Unit tests
helm plugin install https://github.com/helm-unittest/helm-unittest
helm unittest .
# Lint
helm dependency build .
ct lint --config ../../../ct.yaml --charts .