Files
buzz/deploy/charts/buzz
ravarora2 91ab9d31b8 Add operation-aware database pool acquisition metrics (#7195)
## Why

Buzz already reports coarse writer/reader database checkout waits, but
those
signals cannot explain which startup or serving operation is blocked by
pool
pressure. That makes rollout diagnosis and postmortems ambiguous: a
readiness
probe, NIP-42 authentication, authorization check, reconnect history
repair,
event write, and background maintenance can all wait on the same pool
while
appearing identical.

This PR implements Package 2A of the pod-handoff plan: operation-aware
pool
borrow causality. It is observability-only; it does not change
configured pool
sizes, SQL semantics, transaction ordering, or timeout policy. Physical
DNS/TCP/TLS/Postgres authentication and session initialization remain
the
separate Package 2B boundary.

## Metric contract

The final contract separates three questions:

| Question | Metric |
|---|---|
| How long did checkout wait? |
`buzz_db_pool_acquire_duration_seconds{pool_role,operation}` |
| How did the attempt end? |
`buzz_db_pool_acquire_attempts_total{pool_role,operation,outcome}` |
| Who is waiting now for a tracked operation? |
`buzz_db_pool_waiters{pool_role,operation}` |

Outcomes are `success`, `timeout`, `error`, and `cancelled`. Operations
are
`bootstrap`, `readiness`, `tenant_resolution`, `authentication`,
`authorization`, `subscription_history`, `event_write`, and
`maintenance`.

Only these eleven pool/operation pairs are constructible:

```text
writer/bootstrap                 reader/bootstrap
writer/readiness
writer/tenant_resolution
writer/authentication
writer/authorization             reader/authorization
writer/subscription_history      reader/subscription_history
writer/event_write
writer/maintenance
```

The duration histogram intentionally does not carry `outcome`. Result
data
remains available on the terminal counter for historical counts and
rates,
without multiplying the expensive histogram family. Nine finite buckets
plus
`+Inf`, sum, and count produce 12 duration series per valid pair.
Together with
four outcome counters and one waiter gauge, the new-family ceiling is
exactly
187 raw Prometheus series per pod, asserted from the production
exporter.

The existing coarse acquisition families remain temporarily for
dashboard
compatibility while the new series are validated in staging.

The operation-specific waiter family covers the explicitly routed,
deployment-critical operations above; it is not a census of every
possible
SQLx checkout in the process. The dashboard pairs it with SQLx pool
active/idle/max gauges for whole-pool capacity context, and treats a
missing
operation series as unknown rather than healthy zero.

## What changed

### Cancellation-safe acquisition ownership

- Add writer- and reader-specific typed operation APIs so invalid label
pairs
  cannot be constructed and store modules cannot emit reader labels.
- Own every polled acquisition with one RAII terminal guard.
- Record exactly one duration and terminal outcome for success, timeout,
error,
  or cancellation.
- Emit nothing for a future that is created but never polled.
- Balance the operation-specific waiter count exactly once on every
terminal or
  dropped future.
- Periodically refresh every expected waiter pair, including healthy
zero, so
missing telemetry is not presented as zero. Reader pairs are emitted
only
when a distinct read pool is configured; a writer-only pod cannot
fabricate
  healthy reader-zero state.

### Production attribution

Route the deployment-critical acquisition paths through caller-owned
semantic
entry points, including:

- writer and reader bootstrap;
- the real post-#7149 readiness acquisition and deletion-catalog
validation;
- tenant resolution and community lifecycle checks;
- NIP-42 allowlist authentication;
- membership, moderation, invite, operator, Git, agent-owner, and policy
  authorization;
- operator community create/list/archive/unarchive, reverse host/channel
tenant
  resolution, and REQ row-community conformance lookups;
- writer/reader subscription history, feed, thread, and routed fallback
paths;
- primary and command event writes, replaceable events, mention
indexing,
  reaction/channel/member/archive side effects, and thread metadata;
- push matching, usage rollups/leadership, replica-fence startup and
recurring
probes, periodic reconciliation, channel/deletion reapers, partitions,
and
  other bounded maintenance/bootstrap paths.

Shared helpers now accept caller-owned intent or expose named semantic
variants
instead of assigning one misleading operation to every caller. No known
P0 path
uses `other`.

### Readiness and size-one-pool correctness

- Rebase on the post-#7149 readiness implementation and instrument the
actual
`Db::readiness_check` acquisition rather than the superseded ping-only
seam.
- Acquire once for deletion-catalog validation and run its queries on
that
  connection, preserving the shared readiness deadline.
- Scope the channel-roster catalog checkout before the behavior probe so
a
  size-one writer pool cannot self-deadlock during startup verification.

### Exporter, documentation, and CI

- Register metric HELP/type/unit metadata through the production
Prometheus
  builder.
- Configure dedicated checkout buckets at 1ms, 5ms, 10ms, 25ms, 50ms,
150ms,
  500ms, 1s, and 3s.
- Add a production scrape-contract test for exact names, labels,
buckets,
  valid pairs, sensitive-label exclusion, and the 187-series ceiling.
- Add source mutation guards for the P0 semantic entry points and
raw-checkout
  bypasses.
- Add an exact backend-integration CI selector for the production
attribution,
  cancellation, readiness, and size-one-pool PostgreSQL tests.
- Document the frozen label vocabulary, valid combinations, semantics,
and
  cardinality budget in the Helm chart README.

## Dashboard intent

The new Stage 2 row in **Buzz Startup & Rollout Safety** is
deployment-first:

- baseline-versus-candidate attempts, failure rates, cancellation rates,
and
  maximum wait by operation;
- outcome counts and percentages over time by SHA/ReplicaSet;
- acquisition wait heatmap, average, and maximum through the rollout;
- historical waiter pressure beside writer active/idle/max context;
- per-pod postmortem drilldown, including terminated pods;
- a smaller current-waiter table with explicit stale/missing semantics.

Percentile widgets remain disabled until Datadog metadata confirms
percentile
support for the new distribution. Current gauges use no fill,
interpolation, or
`default_zero`; missing means unknown.

## Risk assessment

Moderate. The patch touches many database acquisition call sites, but
preserves
the selected physical pool and executes the same SQL on the acquired
connection. The main risks are incorrect semantic attribution,
cancellation
double-counting, and a helper accidentally acquiring twice. Typed APIs,
production-method PostgreSQL tests, source guards, the raw scrape
contract, and
the size-one-pool regression cover those risks.

No tenant, community, user, pubkey, event, channel, SQL, URL, pod,
version,
ReplicaSet, or request-controlled value is emitted as an application
metric
label. Deployment identity is supplied by infrastructure enrichment.

## Verification

- `cargo fmt --all -- --check` — passed.
- `cargo clippy -p buzz-db -p buzz-relay --all-targets --all-features --
-D warnings`
  — passed.
- `cargo test -p buzz-db` — 122 passed, 0 failed, 263 ignored;
  source-contract integration test: 3 passed, 0 failed.
- Focused relay compatibility, metric-contract, and readiness tests —
passed.
- `scripts/test-postgres-test-discovery.sh` — passed.
- Full `buzz-relay` package run from the identical tree reached 1,015
passes;
the six media-test failures all stopped in their shared local PostgreSQL
setup with `Sqlx(PoolTimedOut)` because Docker/PostgreSQL was
unavailable.
The same six failed in isolation, while every changed exact test passed.
- Exact implementation head:
  `f92910b353086e9edf85918ca5f72190edbbe22f`.
- Exact multi-architecture staging image:
  `dev-sha-f92910b353086e9edf85918ca5f72190edbbe22f-run-33607968668-1`

(`sha256:161712c8ed2e265a15df9b63e02248d5973481f875ff129d7d2ae78a09d487a2`).
- Focused staging GitOps PR:

<https://github.com/squareup/builderbot-platform-core-infrastructure/pull/299>
— merged after renderer, inventory, infrastructure test, Kargo, Semgrep,
and
Intersect gates passed; the source/generated-artifact diff was exactly
two
  image lines.
- Exact GitHub head reports 47 terminal checks: 35 successful and 12
  intentionally skipped. PostgreSQL, unit, lint, security, both server
cross-compiles, backend integration, relay E2E, desktop, mobile, image,
  Helm, Semgrep, zizmor, and DCO gates are green.
- Datadog readback identifies two exact-image pods,
  `buzz-d79c8d8f7-ckv2l` and `buzz-d79c8d8f7-qzqdp`, in ReplicaSet
  `buzz-d79c8d8f7`; both report the full source SHA above.
- Both pods report all eleven allowed pool/operation waiter pairs at
current
zero, with no invalid pair. The acceptance window observed nonzero
success
  receipts for readiness, tenant resolution, authorization, subscription
history, event write, and maintenance, and no timeout, error, or
cancelled
outcome. Maximum observed wait was about 101 ms for maintenance and 50
ms
  for reader subscription history.

The main **Buzz Startup & Rollout Safety** dashboard now has a live
Stage 2
database row with eight widgets and nineteen fully scoped queries. Final
readback preserved all seven top-level groups, found zero under-scoped
Row 6
queries, and confirmed the tracked-operation waiter boundary in the
panel
descriptions.

Generated with Codex.

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>
2026-09-02 17:04:34 -04:00
..

Buzz Helm Chart

Buzz is a Nostr-based messaging platform for human–agent collaboration: a single relay binary serving WebSocket + REST + web UI, backed by PostgreSQL, Redis, and S3-compatible object storage.

This chart has two operating profiles selected by values:

Profile When What you get
Production (default) Self-hosted multi-tenant, regulated, or GitOps-managed External managed Postgres/Redis/S3, secrets.existingSecret:, no chart-side autogen, HA-capable (replicaCount ≥ 2)
Quickstart (eval) Eval, single-node, one-off demo In-cluster Postgres + Redis + MinIO subcharts/Deployments, chart auto-generates relay + service secrets, single replica

Quickstart (eval only)

helm install buzz oci://ghcr.io/block/buzz/charts/buzz --version 0.1.8 \
  --create-namespace --namespace buzz \
  --set quickstart=true \
  --set postgresql.enabled=true \
  --set redis.enabled=true \
  --set minio.enabled=true \
  --set relayUrl=wss://buzz.example.com \
  --set ownerPubkey=<64-char-hex-pubkey>

This brings up everything in-cluster — Postgres, Redis, and MinIO (with its bucket created by a post-install Job) — and composes the relay's BUZZ_S3_ENDPOINT plus autogenerated credentials automatically. No external services required. The quickstart=true flag is an intent marker surfaced in NOTES.txt; the bundled services are opted in via the four *.enabled flags above (see ci/quickstart-values.yaml for the exact set CI installs). Eval-only: every bundled service is a single replica with no HA.

For immutable delivery, pin the OCI digest instead of a tag. image.digest overrides image.tag when both are present:

image:
  repository: ghcr.io/block/buzz
  digest: sha256:<64-lowercase-hex-characters>

Production (GitOps)

The chart is designed for ArgoCD and Flux. Both render charts with helm template, in which mode Helm's lookup function returns empty — any chart-side randAlphaNum call would regenerate secrets on every sync. The chart-managed Secret path is only safe for helm install / helm upgrade.

Production deploys MUST use secrets.existingSecret:. The Secret is consumed for any keys present and ignored for keys missing — extras are harmless.

To enable relay-proxied KLIPY search, add BUZZ_KLIPY_API_KEY to that Secret. The key stays in the relay pod; clients discover the public buzz-gif extension and gif descriptor in NIP-11, then receive KLIPY-hosted media URLs. See docs/gif-search.md for the protocol and security boundaries.

See:

Required inputs

Key What When required
relayUrl Public wss:// URL clients connect to Always
ownerPubkey 64-char lowercase hex Nostr pubkey of the relay operator When relay.requireRelayMembership=true (default)
secrets.existingSecret Name of pre-created Secret Production / GitOps
externalPostgresql.url / externalRedis.url / s3.endpoint External service URLs Production — when the matching bundled service is disabled (the default)

The chart fails at helm install / helm template time with a clear message if any of these are missing or malformed (see templates/_validate.tpl).

S3 URL addressing

Buzz uses one URL style for both media and Git/CAS object-store requests:

s3.addressingStyle Request shape Use for
path (default) https://endpoint/bucket/key Bundled MinIO and endpoints whose DNS does not resolve bucket subdomains
virtual https://bucket.endpoint/key AWS-style providers and new Railway Storage Buckets

The chart always renders s3.addressingStyle as BUZZ_S3_ADDRESSING_STYLE and s3.region as BUZZ_S3_REGION. The region defaults to us-east-1, keeping bundled MinIO and the in-pod buzz-admin deletions workflow operable without an ambient AWS_REGION. Production providers must set their credential region explicitly when it differs. Existing releases that previously omitted s3.region will begin rendering BUZZ_S3_REGION=us-east-1 after upgrade, even if an image or relay.extraEnv entry supplied AWS_REGION; set s3.region to the provider's actual credential region before upgrading. Only path and virtual addressing styles are accepted; invalid values fail chart rendering and relay startup. The bundled MinIO quickstart deliberately keeps path because its Service DNS resolves one endpoint hostname, not arbitrary <bucket>.<service> names.

For a Railway Storage Bucket, map its variables to chart values in the service or generated Helm configuration:

s3:
  endpoint: "${{Object Storage.ENDPOINT}}"
  bucket: "${{Object Storage.BUCKET}}"
  region: "${{Object Storage.REGION}}"
  addressingStyle: virtual

Store BUZZ_S3_ACCESS_KEY=${{Object Storage.ACCESS_KEY_ID}} and BUZZ_S3_SECRET_KEY=${{Object Storage.SECRET_ACCESS_KEY}} in the Secret named by secrets.existingSecret. Railway's Credentials tab is authoritative for older buckets, which may still require path. The setting changes request routing and SigV4 signing, so do not put the bucket into s3.endpoint; pass Railway's base ENDPOINT and BUCKET separately.

Object storage is contacted during relay startup only when BUZZ_GIT_CONFORMANCE_PROBE is enabled (the relay default). A probe failure is startup-fatal, so Kubernetes readiness never opens. If an operator explicitly disables that probe through relay.extraEnv, /_readiness does not test object storage; configuration is still parsed strictly, but reachability and addressing errors surface on the first storage operation.

Readiness telemetry contract

Only requests served by the private health listener (BUZZ_HEALTH_PORT) emit rollout readiness telemetry. The compatibility /_readiness route on the public app listener returns health but does not change these metrics.

Metric Type Labels
buzz_readiness_checks_total counter reason from the closed readiness-reason set
buzz_readiness_dependency_checks_total counter dependency, typed bounded outcome
buzz_readiness_check_duration_seconds histogram check only
buzz_readiness_state gauge check only; latest publishable generation

The schema has a ceiling of 99 raw Prometheus series per pod: 12 overall reasons, 11 valid dependency/outcome pairs, 72 histogram series, and 4 gauges. Do not add pod, ReplicaSet, version, rollout, error text, SQL, URL, tenant, user, community, pubkey, header, query, or other request-controlled labels. Shutdown without dependency evaluation increments only buzz_readiness_checks_total{reason="shutting_down"} and sets the overall state to zero; it does not fabricate dependency failures or latency samples.

Operation-aware database pool acquisition contract

The operation-aware families separate three questions: who is waiting now, how completed/abandoned attempts ended, and how long checkout waits took. Outcome remains on the terminal counter for historical deployment comparison; it is intentionally absent from the expensive duration histogram.

These families cover the explicitly routed deployment-critical operations listed below; they are not a count of every SQLx checkout in Buzz. In particular, a zero operation waiter does not prove that the shared SQLx pool has no uninstrumented waiter. Interpret it beside the pool active, idle, and maximum gauges when diagnosing total capacity pressure.

Metric Type Labels
buzz_db_pool_acquire_duration_seconds histogram pool_role, operation
buzz_db_pool_acquire_attempts_total counter pool_role, operation, outcome
buzz_db_pool_waiters gauge pool_role, operation; tracked operations only, periodically refreshed including zero

Outcomes are success, timeout, error, and cancelled. Operations are bootstrap, readiness, tenant_resolution, authentication, authorization, subscription_history, event_write, and maintenance. Only the following eleven pairs are valid:

writer/bootstrap                 reader/bootstrap
writer/readiness
writer/tenant_resolution
writer/authentication
writer/authorization             reader/authorization
writer/subscription_history      reader/subscription_history
writer/event_write
writer/maintenance

Nine finite checkout buckets plus +Inf, sum, and count yield 12 histogram series per valid pair. The new contract therefore has a hard ceiling of 187 raw Prometheus series per pod: 11 × (12 + 4 + 1). The two legacy acquisition families remain temporarily for dashboard compatibility and are not part of that new-family budget. No other operation or request-controlled/sensitive label is valid.

Relay Pod extensions

The chart exposes narrow extension points for init containers, volumes, relay volume mounts, and image command/argument overrides. extraManifests creates independent Kubernetes resources but cannot modify the chart-managed relay Deployment. These extension values insert fields into that Deployment, avoiding duplication of its environment, probes, security context, secrets, and chart-owned volumes.

For example, an init container can copy a wrapper binary into a shared volume and make that wrapper the relay entrypoint:

extraInitContainers:
  - name: install-wrapper
    image: example.com/wrapper-init:v1
    args: [/opt/wrapper/wrapper]
    securityContext:
      runAsNonRoot: true
      runAsUser: 65532
      runAsGroup: 65532
      allowPrivilegeEscalation: false
      capabilities:
        drop: [ALL]
    resources:
      requests:
        cpu: 10m
        memory: 16Mi
    volumeMounts:
      - name: wrapper
        mountPath: /opt/wrapper

extraVolumes:
  - name: wrapper
    emptyDir: {}

relay:
  command: [/opt/wrapper/wrapper]
  args: [/usr/local/bin/buzz-relay]
  extraVolumeMounts:
    - name: wrapper
      mountPath: /opt/wrapper

These values are raw Kubernetes fragments rendered with toYaml, not tpl. The chart does not validate cross-field relationships: extension names must not collide with chart-owned containers or volumes, mounts must reference existing volumes, and each init container must define an appropriate security context and resources. Empty relay.command and relay.args arrays preserve the image defaults; non-empty values override its entrypoint and arguments respectively.

Device pairing relay

The chart can run Buzz's stateless pairing WebSocket relay as an independent Deployment and Service using the same image as the main relay:

pairingRelay:
  enabled: true
  url: wss://pairing.example.com

pairingRelay.url is advertised in the main relay's NIP-11 document so Buzz clients connect directly to the dedicated endpoint. The chart does not create an Ingress or HTTPRoute for the pairing Service; route the public hostname to <release>-buzz-pairing:5000 with your platform's ingress configuration.

HA (production)

replicaCount > 1 hard-requires Redis:

  • Redis (redis.enabled=true, externalRedis.url, or REDIS_URL in existingSecret) — for buzz-pubsub fan-out

It does not require ReadWriteMany git storage. Git ref/object state is object-store-backed (each request hydrates an ephemeral repo from S3-compatible storage; writer serialization is the object-store pointer CAS — see docs/git-on-object-storage.md), and repo-name uniqueness lives in Postgres. Each replica can use its own ReadWriteOnce volume; no shared filesystem is needed.

The chart template-fails if the Redis invariant is broken at replicaCount > 1. No silent degradation.

Relay autoscaling

The optional HPA scales the relay on the larger recommendation from CPU or average active WebSockets per pod:

autoscaling:
  enabled: true
  minReplicas: 5
  maxReplicas: 15
  targetCPUUtilizationPercentage: 65
  websocketMetricEnabled: true
  websocketMetricName: buzz_ws_connections_active
  targetWebsocketConnections: 5000

CPU scaling requires Kubernetes Metrics Server. Set websocketMetricEnabled: false for a CPU-only HPA. WebSocket scaling additionally requires a custom-metrics adapter (for example Prometheus Adapter) configured to expose the relay's buzz_ws_connections_active gauge as a pod metric with the name in websocketMetricName. The chart creates the HPA but deliberately does not install or configure a cluster-wide metrics adapter. Scale-down is gradual by default so long-lived WebSocket connections have time to drain.

Upgrades

Schema migrations are embedded in the relay binary via sqlx::migrate! and run at startup, gated by BUZZ_AUTO_MIGRATE (default true). Multiple replicas race-safely behind a Postgres advisory lock. helm upgrade is the entire upgrade procedure.

Migration 0032 is a hard compatibility boundary for relay versions that publish repaired channel rosters. The relay verifies the roster-fence trigger catalog and behavior before opening listeners and refuses to start if 0032 is missing or inert. Apply migrations before rolling the relay; for large installations, prefer a controlled buzz-admin migrate job with PostgreSQL lock monitoring before the code rollout.

If you prefer decoupling migrations from serving, set migrate.autoMigrate=false. In that mode the chart does not run migrations for you — you own running buzz-admin migrate (separate Pod / one-shot Job) against the database before every helm install / helm upgrade. Readiness probes only verify DB connectivity, not schema freshness, so a pod will appear healthy against an unmigrated schema and fail under load. A pre-upgrade Helm Job for this is on the chart roadmap; the values knob migrate.preUpgradeJob.enabled is reserved.

Backups

Save these. Losing any of them is data loss. See NOTES.txt printed by helm install for the live list:

  1. BUZZ_RELAY_PRIVATE_KEY — relay identity. Rotating it = new identity (federation peers will not recognize the relay).
  2. PostgreSQL database — the canonical event store.
  3. S3 bucket — media blobs (chart default bucket: buzz-media).
  4. Git PVC — repo on-disk state served by the relay's git endpoint.
  5. Owner private key — held by the operator, not by this chart. Restore by re-installing with the same ownerPubkey.

Honest limitations (v1)

  • Bundled MinIO is eval-only. The quickstart profile runs an in-cluster MinIO (single replica, no HA, lookup-autogenerated credentials) so the relay starts with zero external object storage. Production leaves minio.enabled off and points s3.endpoint (or BUZZ_S3_* in existingSecret) at managed S3-compatible storage. The bundled Deployment is not GitOps-safe and is not intended for production traffic.
  • Minimal-mode is not yet supported. The relay's BUZZ_PUBSUB=local / filesystem media paths are upstream work in progress — even quickstart currently stands up real Redis and S3 rather than the relay's single-node fallbacks. (Full-text search already runs in Postgres, so no separate search service is provisioned.)
  • Cosign signing of the published chart is a follow-up (the relay image is attested via actions/attest-build-provenance; the chart is not yet). The chart itself is published to GHCR — see Releasing.

Releasing

The chart is published to GHCR as an OCI artifact at oci://ghcr.io/block/buzz/charts/buzz by the helm chart workflow (.github/workflows/helm-chart.yml), versioned independently of the desktop app and the relay image via its own chart-v* tags. Every PR/main push still lints, unit-tests, and render-checks the chart; only a chart-v* tag publishes, so an in-progress main can never overwrite a released version.

To cut a release, push a chart-release/<version> branch whose <version> matches Chart.yaml's version; merging it auto-tags chart-v<version> and dispatches the publish job (same lane machinery as the desktop and relay releases — see .github/workflows/auto-tag-on-release-pr-merge.yml). The publish job fails loudly if the tag version and Chart.yaml version disagree.

Development

# Render every fixture
for f in ci/*-values.yaml tests/fixtures/*-values.yaml; do
  helm template buzz . -f "$f" >/dev/null && echo "ok: $f"
done

# Unit tests
helm plugin install https://github.com/helm-unittest/helm-unittest
helm unittest .

# Lint
helm dependency build .
ct lint --config ../../../ct.yaml --charts .