Files
Luke Tornquist 0e36dbc823 fix(db): keep the reader pool warm and drop its pre-acquire ping (#8495)
## Summary

The reader pool's 150ms `acquire_timeout` covers the whole acquire: the
pre-acquire idle-connection ping **and** any new-connection dial (TCP +
TLS + SCRAM). When readers slow down for a moment, the timeout:

- abandons in-flight dials mid-handshake (Postgres logs `could not
accept SSL connection: EOF detected`),
- closes healthy idle connections whose ping overran instead of
returning them to the pool, and
- falls every miss back to the writer, which then exhausts its own pool.

Across a fleet of relay pods this shows up as a burst of abandoned
handshakes on the readers that starts on nearly every pod at once,
followed seconds later by writer pool-acquire timeouts. The database
itself reports no statement, lock, or connection-limit errors before the
burst: the client is tearing down its own reader connections.

This PR removes that amplifier. It does not change the request budget,
writer fallback, or the writer pool.

### Change (`crates/buzz-db/src/runtime/mod.rs`)

- **Warm floor:** reader `min_connections` is now `max / 5`, at least 1,
derived from the existing reader pool size (no new config knob). sqlx
re-dials the floor in background maintenance, outside the request
budget, from its reaper and whenever a connection is closed instead of
released. The floor is per relay process, so standing reader connections
are the floor times the number of relay processes; size the reader's
`max_connections` for it.
- **Floor fills at boot:** with `max_lifetime`/`idle_timeout` set, a
lazy sqlx 0.9 pool does not dial `min_connections` at construction. The
existing boot ping now closes its probe connection instead of returning
it, which triggers sqlx's floor maintenance. Probe acquires that overrun
150ms retry with bounded backoff (250ms, 500ms, 1s, 2s). The doc comment
that said sqlx dials the minimum eagerly was wrong for this config and
is corrected.
- **`test_before_acquire(false)` on the reader:** the routed read's
`BEGIN … READ ONLY` already proves the connection and fails closed to
the writer (`reader_validation_error`). This removes one round trip per
checkout and the warm-connection churn.
- **Bounded fence proof:** with no pre-acquire ping, the proof (Aurora
capability probe, `BEGIN`, heartbeat observation) is the first round
trip on a checkout. It now runs under `READER_PROOF_TIMEOUT` (1s), so a
reader that stops answering without closing its sockets (partition, host
loss, failover) fails closed to the writer as the new route reason
`reader_proof_timeout` instead of hanging the request until TCP gives
up. 1s is well above a normal round trip, so a brief reader stall waits
on the reader rather than flooding the writer.
- **Bounded release:** every connection returned to the reader pool
first passes a release check bounded at 5s (`Db::read_pool_options`).
sqlx's own release ping has no timeout, so without this a socket whose
reader went silent would hold its slot until the kernel gives up; with
every slot held that way, the pool cannot dial the reader that replaced
it, and all routed reads fall back to the writer. A check that times out
or errors closes the socket and frees the slot, and floor maintenance
dials a replacement. A connection still inside a transaction is closed
instead of returned: a checkout dropped while its `BEGIN` was in flight
(a timed-out proof, a cancelled caller) has no `ROLLBACK` queued,
because sqlx counts the transaction only once `BEGIN` is acknowledged.
Closing it ends the session, and the server rolls it back. An expired
proof is therefore simply dropped. Maintenance checkouts (fleet usage
collection) also keep their close-on-drop: when one is abandoned at its
deadline, its socket is closed under sqlx's 5s close bound rather than
going through the release path.
- **Pinned isolation default:** reader sessions set
`default_transaction_isolation=read committed` in the startup packet,
which takes precedence over database and role defaults. The release
check depends on this, because it detects an open routed-read
transaction by its `REPEATABLE READ` isolation.
- **Metric:** each close is counted in
`buzz_db_reader_release_closed_total{reason}` (`timeout`, `error`,
`open_transaction`). Operator notes for the floor,
`reader_proof_timeout`, and this counter are in
`deploy/charts/buzz/README.md`.

### What got simpler

- The reader checkout path does one less round trip and no longer closes
healthy connections on a slow ping.
- Boot warm-up reuses sqlx's own `min_connections` maintenance rather
than re-implementing it.
- One release rule now covers every reader checkout, so the
open-transaction leak on cancelled reads no longer needs special
handling per call site.
- No new config knob.
- The reader test proxy now lives once in `test_support` instead of
inside `store/usage.rs`.
- Tests build their reader pools from the production options rather than
hand-copied ones.

### Tradeoffs

- The reader is now dialed at boot (previously "no reader dial at
boot"). If the reader is down at boot, this only produces log noise, and
reads still fail closed to the writer.
- The release check is one trivial statement (`SELECT
current_setting(...)`) per released reader connection. It runs in sqlx's
release task, after the response, so request latency is unchanged.
- The check detects an open transaction by comparing the current
isolation level with the session default. That is only sound because the
reader session default is pinned to `READ COMMITTED` and routed reads
begin `REPEATABLE READ`.
- **Residual window:** after the release check passes, sqlx sends its
own release ping (one more round trip), and that ping has no timeout. A
serving-read connection whose endpoint goes silent between those two
round trips still holds its slot until TCP gives up. Only releases
already mid-ping at that instant are exposed: later releases fail the
bounded check first and are closed. sqlx behaves the same way for every
pool on `main`. Maintenance checkouts avoid it through close-on-drop.
Bounding it for serving reads would mean replacing sqlx's release path
or putting a pooler in front of the reader, so it is left as a
follow-up.
- Boot fill is best effort. Each boot-probe retry still uses the 150ms
serving acquire budget, so if cold handshakes consistently take longer
than that, all five attempts fail and the floor is not filled at boot.
It fills the first time any reader connection is closed, which the
reaper and the release check both do, and until then reads dial as
before.
- This also closes pre-existing debt: a caller that dropped a routed
read mid-`BEGIN` (client disconnect) could leave a connection `idle in
transaction`. The release check covers that path too.

### Related issue

N/A (production incident follow-up; none found).

### Testing

- Postgres tests (`runtime/tests/reader_pool_postgres_tests.rs`):
- the boot ping leaves the reader pool at exactly its floor, all idle
(base leaves one connection);
- a warm reader whose heartbeat read blocks (exclusive lock) falls back
to the writer as `reader_proof_timeout` within one proof budget;
- a warm reader connection terminated server-side falls back as
`reader_validation_error` with no client error;
- a proof that times out mid-`BEGIN` on a paused reader (a test proxy
that holds bytes, like `docker pause`) falls back as
`reader_proof_timeout`. The reader database default is set to
`REPEATABLE READ` for this test. Once the reader resumes, it leaves no
`idle in transaction` reader backend, and the close is counted as
`open_transaction`. The test fails if the release check is made blind to
open transactions, or if the isolation pin is removed.
- Failover: every established reader session is stalled while new
sessions get through. Each stalled warm connection costs one
`reader_proof_timeout`, then routed reads reach the reader again through
new connections within 8s, and the closes are counted as `timeout`. The
test fails with the release bound removed.
- `store/usage.rs`: a fleet collection abandoned at its deadline on a
paused reader frees its slot within 3s while the reader is still paused,
which is faster than the 5s release check. The test fails with
maintenance close-on-drop removed.
- Unit tests: the min-connections mapping, the reader pool options, and
that construction still dials nothing.
- Blox, at `f3e3bf158`: `just fmt-check` and `just clippy` are clean,
and `buzz-db` unit tests are 191/191.
- Blox, at `01745b21e`: fmt is clean, the targeted reader tests passed 3
of 3 runs, and the release-bound and transaction-blind mutations both
turn the targeted tests red.
- Blox, at `2f8e482fb` (before the rebase onto `main`): fmt and clippy
are clean, the targeted reader tests passed 3 of 3 runs, `buzz-db` unit
tests pass, and each of four mutations turns its targeted test red:
release bound, transaction-blind check, isolation pin, and maintenance
close-on-drop.
- Blox, at the rebased head `e80bc2bed`: the same four mutations each
turn their targeted test red, and fmt, clippy, the targeted reader tests
(3 of 3 runs) and `buzz-db` unit tests are clean.
- CI Postgres lane at `e80bc2bed`: 1130/1130 passed, including every new
reader-pool test.
- Live check at `e80bc2bed`: a relay with a two-slot reader pool behind
a proxy, which was then frozen for 18s. All 246 history reads succeeded,
and the slowest took 1.0s. During the freeze, 2 proof timeouts and 2
release-timeout closes were counted. Reads returned to the reader within
36ms of the resume, and no reader backend was left `idle in
transaction`.
- Rebase note: `main` added a reader `after_connect` hook that tightens
`statement_timeout`. It now lives in `Db::read_pool_options`, so the
tests' reader pools get it too.

---------

Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 14:04:39 -04:00
..

Buzz Helm Chart

Buzz is a Nostr-based messaging platform for human–agent collaboration: a single relay binary serving WebSocket + REST + web UI, backed by PostgreSQL, Redis, and S3-compatible object storage.

This chart has two operating profiles selected by values:

Profile When What you get
Production (default) Self-hosted multi-tenant, regulated, or GitOps-managed External managed Postgres/Redis/S3, secrets.existingSecret:, no chart-side autogen, HA-capable (replicaCount ≥ 2)
Quickstart (eval) Eval, single-node, one-off demo In-cluster Postgres + Redis + MinIO subcharts/Deployments, chart auto-generates relay + service secrets, single replica

Quickstart (eval only)

helm install buzz oci://ghcr.io/block/buzz/charts/buzz --version 0.1.10 \
  --create-namespace --namespace buzz \
  --set quickstart=true \
  --set postgresql.enabled=true \
  --set redis.enabled=true \
  --set minio.enabled=true \
  --set relayUrl=wss://buzz.example.com \
  --set ownerPubkey=<64-char-hex-pubkey>

This brings up everything in-cluster — Postgres, Redis, and MinIO (with its bucket created by a post-install Job) — and composes the relay's BUZZ_S3_ENDPOINT plus autogenerated credentials automatically. No external services required. The quickstart=true flag is an intent marker surfaced in NOTES.txt; the bundled services are opted in via the four *.enabled flags above (see ci/quickstart-values.yaml for the exact set CI installs). Eval-only: every bundled service is a single replica with no HA.

For immutable delivery, pin the OCI digest instead of a tag. image.digest overrides image.tag when both are present:

image:
  repository: ghcr.io/block/buzz
  digest: sha256:<64-lowercase-hex-characters>

Bundled object storage uses Silo, a maintained MinIO fork, with matching pgsty/silo and pgsty/mc release images pinned by multi-platform digest. Both images allow anonymous pulls and support Linux amd64 and arm64; the upstream Quay MinIO images no longer allow anonymous pulls. The existing minio names and MINIO_* settings remain compatible. Keep the Helm and Compose pins aligned when upgrading, and run Buzz's object-store conformance probe against the new release before deploying it.

For an existing MinIO data volume, follow Silo's migration guide and take a backup before upgrading. Replacing the container image upgrades the storage server as well as changing its registry.

Production (GitOps)

The chart is designed for ArgoCD and Flux. Both render charts with helm template, in which mode Helm's lookup function returns empty — any chart-side randAlphaNum call would regenerate secrets on every sync. The chart-managed Secret path is only safe for helm install / helm upgrade.

Production deploys MUST use secrets.existingSecret:. The Secret is consumed for any keys present and ignored for keys missing — extras are harmless.

To enable relay-proxied KLIPY search, add BUZZ_KLIPY_API_KEY to that Secret. The key stays in the relay pod; clients discover the public buzz-gif extension and gif descriptor in NIP-11, then receive KLIPY-hosted media URLs. See docs/gif-search.md for the protocol and security boundaries.

See:

Required inputs

Key What When required
relayUrl Public wss:// URL clients connect to Always
ownerPubkey 64-char lowercase hex Nostr pubkey of the relay operator When relay.requireRelayMembership=true (default)
secrets.existingSecret Name of pre-created Secret Production / GitOps
externalPostgresql.url / externalRedis.url / s3.endpoint External service URLs Production — when the matching bundled service is disabled (the default)

The chart fails at helm install / helm template time with a clear message if any of these are missing or malformed (see templates/_validate.tpl).

S3 URL addressing

Buzz uses one URL style for both media and Git/CAS object-store requests:

s3.addressingStyle Request shape Use for
path (default) https://endpoint/bucket/key Bundled MinIO and endpoints whose DNS does not resolve bucket subdomains
virtual https://bucket.endpoint/key AWS-style providers and new Railway Storage Buckets

The chart always renders s3.addressingStyle as BUZZ_S3_ADDRESSING_STYLE and s3.region as BUZZ_S3_REGION. The region defaults to us-east-1, keeping bundled MinIO and the in-pod buzz-admin deletions workflow operable without an ambient AWS_REGION. Production providers must set their credential region explicitly when it differs. Existing releases that previously omitted s3.region will begin rendering BUZZ_S3_REGION=us-east-1 after upgrade, even if an image or relay.extraEnv entry supplied AWS_REGION; set s3.region to the provider's actual credential region before upgrading. Only path and virtual addressing styles are accepted; invalid values fail chart rendering and relay startup. The bundled MinIO quickstart deliberately keeps path because its Service DNS resolves one endpoint hostname, not arbitrary <bucket>.<service> names.

For a Railway Storage Bucket, map its variables to chart values in the service or generated Helm configuration:

s3:
  endpoint: "${{Object Storage.ENDPOINT}}"
  bucket: "${{Object Storage.BUCKET}}"
  region: "${{Object Storage.REGION}}"
  addressingStyle: virtual

Store BUZZ_S3_ACCESS_KEY=${{Object Storage.ACCESS_KEY_ID}} and BUZZ_S3_SECRET_KEY=${{Object Storage.SECRET_ACCESS_KEY}} in the Secret named by secrets.existingSecret. Railway's Credentials tab is authoritative for older buckets, which may still require path. The setting changes request routing and SigV4 signing, so do not put the bucket into s3.endpoint; pass Railway's base ENDPOINT and BUCKET separately.

Object storage is contacted during relay startup only when BUZZ_GIT_CONFORMANCE_PROBE is enabled (the relay default). A probe failure is startup-fatal, so the process exits and Kubernetes readiness never opens. If an operator explicitly disables that probe through relay.extraEnv, configuration is still parsed strictly, but reachability and addressing errors surface on the first storage operation. /_readiness tests no external dependency in either case — see the readiness contract below.

Community deletion operator job

operatorJobs.deletionDrain is a disabled-by-default, typed CronJob for /usr/local/bin/buzz-admin deletions drain. It runs inside the relay image with bounded Job lifetime/history, concurrencyPolicy: Forbid, backoffLimit: 0, and no relay HTTP call. Postgres deletion requests, leases, retries, and checkpoints remain the execution authority.

The pod receives only DATABASE_URL, REDIS_URL, and required S3 configuration/credential variables. It does not receive the relay private key, git-hook secret, relay URL, service links, or a generic environment registry. The chart disables the ordinary Kubernetes API service-account token mount; platform workload-identity admission may still inject its own projected token and provider environment variables. Schedule, deadline, history, termination grace, resources, service account, pod labels, and pod annotations are independently configurable under operatorJobs.deletionDrain.

See docs/operator-community-deletion.md for enablement, permissions, the staffed first-run procedure, failure recovery, and the current explicit approval/alerting boundaries.

Early-startup telemetry contract

buzz_process_lifecycle JSON records are the authoritative history for the fixed phases crypto_init, tracing_init, config_load, key_load, and metrics_bind, plus the aggregate process_telemetry result. They use bounded status/reason values and never contain raw configuration, keys, URLs, or errors. These phases intentionally do not emit metrics. Most run before the Prometheus exporter exists, and one uniform log-only contract preserves every phase's real event time and failure without assigning an eventual scrape time to earlier work.

Startup work after metrics_bind (database connect through public listener bind) runs with a live recorder, so each step logs Startup phase started and Startup phase finished (phase, elapsed_ms, and status = succeeded/degraded/failed, the same values lifecycle records use) and sets two gauges labelled phase:

  • buzz_startup_phase_current is 1 while the step runs and 0 once it ends. A pod that has not opened its health port shows the step it is stuck on.
  • buzz_startup_phase_seconds is the finished step's duration.

degraded means the step hit a non-fatal error and startup continued — for example, some large channel rosters failed reconciliation or a NIP-FI issuer did not warm. failed means startup aborted inside the step.

These are gauges, not counters, because each value is written once per boot and a monotonic-counter scrape drops a series' first sample. Each is written once, so it ages out after the gauge idle timeout: the idle_timeout_secs field of the Prometheus metrics exporter started log (2700 s with default intervals). A step that runs longer than that timeout loses its _current series, so a pod stuck that long shows no current step. The phase values are the closed StartupStep vocabulary in crates/buzz-relay/src/startup_steps.rs; they never reuse an early lifecycle phase name.

Readiness contract

/_readiness reports local process lifecycle only. It performs no Postgres, Redis, or deletion-catalog I/O: shutting_down returns 503, and any other state returns 200. Shared dependencies are shared by every replica, so gating the probe on them removed the whole deployment from the load balancer at once and left a reconnect burst with nowhere to land. There is no separate "starting" state — the health listener does not bind until the database, migrations, Redis, and pub/sub are up, so a process that can answer has booted. With BUZZ_NIP_FI_COMMUNITY_SOURCE=database and a mode other than off, the first NIP-FI binding snapshot also loads before any listener binds (startup phase nip_fi_snapshot_load); a failed load exits the process, so a pod never reports ready without one. Later refresh failures do not affect readiness — watch buzz_nip_fi_binding_snapshot_loaded_timestamp_seconds instead. The source's env vars (BUZZ_NIP_FI_COMMUNITY_SOURCE, BUZZ_NIP_FI_SNAPSHOT_MAX_ROWS, BUZZ_NIP_FI_SNAPSHOT_REFRESH_TIMEOUT_MS, BUZZ_NIP_FI_RESOLVER_NEGATIVE_CACHE_MAX, BUZZ_NIP_FI_RESOLVER_MAX_INFLIGHT) and alerts are in docs/operator-nip-fi-bindings.md.

Shared-dependency health moved to /_status on the same private health listener, under a dependencies object carrying the postgres, redis, deletion_catalog, and aggregate reason fields the readiness body used to return. Do not wire /_status to a Kubernetes probe.

/_status is a cached read. It performs no Postgres, Redis, or deletion-catalog I/O of its own. One background loop per pod evaluates the three dependencies every 30 seconds, awaiting each evaluation before taking the next tick, so a pod never has more than one evaluation in flight no matter how often — or how rarely — the endpoint is read. The loop publishes the dependency metrics below and caches the report /_status serves. Its first cycle runs at startup, and each evaluation is bounded by a two-second budget.

Every dependencies object therefore states how old its report is:

Field Meaning
sample: "not_yet_sampled" the first cycle has not completed; no postgres/redis/deletion_catalog/reason fields are present, because there is no observation to report
sample: "fresh" the report is at most two cadences (60s) old
sample: "stale" the report outlived two cadences, so the sampler missed at least one cycle — read the verdict as history, not as current state
sample_age_seconds age of the report at request time (absent when not_yet_sampled)
sample_interval_seconds the sampling cadence, 30

Polling /_status more often than the cadence returns the same cached report; it does not make the data fresher and adds no dependency load.

Readiness telemetry contract

Only requests served by the private health listener (BUZZ_HEALTH_PORT) emit rollout telemetry. The compatibility /_readiness route on the public app listener returns the same lifecycle answer but does not change these metrics.

Metric Type Labels Source
buzz_readiness_checks_total counter reason ∈ {ready, shutting_down} /_readiness
buzz_readiness_state gauge check="overall"; latest private probe observation, 1 ready or 0 shutting down /_readiness
buzz_readiness_dependency_checks_total counter dependency, typed bounded outcome dependency sampler
buzz_readiness_check_duration_seconds histogram check only dependency sampler
buzz_readiness_dependency_sample_completed_timestamp_seconds gauge none; Unix time the cached report completed completion publisher

The three dependency families keep their buzz_readiness_* names for dashboard continuity, but nothing about them is request-driven any more: the 30-second sampler publishes them whether or not anyone reads /_status, so a quiet endpoint no longer produces a flat dashboard during the outage it exists to explain.

buzz_readiness_dependency_sample_completed_timestamp_seconds carries when the cached report completed, in Unix seconds. The relay sampler is the only owner allowed to advance that epoch, and it writes it immediately after the cache is replaced. A separate bounded publisher re-emits the stored epoch often enough to survive local gauge idle-timeout; republishing never advances the timestamp. If a republish races a newer completion, one scrape can briefly see the older epoch, but the republish path verifies after writing and repairs to the newer stored epoch before that republish call returns. So the value stands still when sampling stops, and the time elapsed since it was written is whatever the reader computes at read time.

Following the buzz_storage_sweep_age_seconds convention, the series is not emitted until the first sample completes: its absence means "not yet sampled", not "fresh".

That is the whole server-side contract. Freshness alerting is built from this gauge in the monitoring provider, and the monitor query, thresholds, and per-pod tag grouping belong with the deployment's monitor configuration rather than in this chart — they depend on the provider's query grammar and on the tags its agent attaches, neither of which this repo owns.

Two properties of the neighboring families are worth knowing when building that alerting. buzz_readiness_dependency_checks_total and buzz_readiness_check_duration_seconds are cumulative, so a sampler that stops leaves their last values exported and scraped indefinitely: those series stay present and flat rather than disappearing. And per-report freshness for a human reading a single pod is already on the sample, sample_age_seconds, and sample_interval_seconds fields of /_status above.

The schema has a ceiling of 87 raw Prometheus series per pod: 2 probe reasons, 11 valid dependency/outcome pairs, 72 histogram series, and 2 gauges (overall lifecycle + completion timestamp). Do not add pod, ReplicaSet, version, rollout, error text, SQL, URL, tenant, user, community, pubkey, header, query, or other request-controlled labels. A readiness probe records no dependency attempt or latency sample at all. The readiness gauge is not a monotonic lifecycle mirror: shutdown changes the authoritative lifecycle flag, and the next private readiness probe observes and publishes that state.

Community admission telemetry

Metric Type Labels
buzz_community_admission_checks_total counter outcome ∈ {active, inactive, check_error}

Counts the durable community-active check run before a socket is admitted. Admission is fail-closed: only active serves. inactive (a confirmed archival answer) and check_error (the lookup itself failed, so the tenant lifecycle is unknown) both refuse the socket before any AUTH or REQ frame is read; the client sees an ordinary dial failure and retries. The two outcomes stay distinct so a rise in check_error reads as database pressure rather than archival. outcome is the only dimension — community and error text are request-controlled and must never become labels.

Operation-aware database pool acquisition contract

The operation-aware families separate four questions: when an operation asked for a connection, who is waiting now, how completed/abandoned attempts ended, and how long checkout waits took. Outcome remains on the terminal counter for historical deployment comparison; it is intentionally absent from the expensive duration histogram.

These families cover the explicitly routed deployment-critical operations listed below; they are not a count of every SQLx checkout in Buzz. In particular, a zero operation waiter does not prove that the shared SQLx pool has no uninstrumented waiter. Interpret it beside the pool active, idle, and maximum gauges when diagnosing total capacity pressure.

Metric Type Labels
buzz_db_pool_acquire_started_total counter pool_role, operation
buzz_db_pool_acquire_duration_seconds histogram pool_role, operation
buzz_db_pool_acquire_attempts_total counter pool_role, operation, outcome
buzz_db_pool_waiters gauge pool_role, operation; tracked operations only, periodically refreshed including zero

Outcomes are success, timeout, error, and cancelled. Operations are bootstrap, readiness, tenant_resolution, authentication, authorization, subscription_history, event_write, and maintenance. Only the following twelve pairs are valid:

writer/bootstrap                 reader/bootstrap
writer/readiness
writer/tenant_resolution
writer/authentication
writer/authorization             reader/authorization
writer/subscription_history      reader/subscription_history
writer/event_write
writer/maintenance               reader/maintenance

Nine finite checkout buckets plus +Inf, sum, and count yield 12 histogram series per valid pair. The new contract therefore has a hard ceiling of 216 raw Prometheus series per pod: 12 × (1 + 12 + 4 + 1). The two legacy acquisition families remain temporarily for dashboard compatibility and are not part of that new-family budget. No other operation or request-controlled/sensitive label is valid.

Writer connection setup contract

Writer connection setup is separate from checkout. At boot, SQLx constructs the writer pool and creates its minimum physical connections. Later it may create more when the pool grows or replaces a broken or expired connection. Every connected writer session must install the created-at floor, install the session timeouts, verify READ COMMITTED isolation, and reach ready before SQLx can give it to a caller.

Metric Type Labels
buzz_db_connection_step_started_total counter pool_role, step
buzz_db_connection_step_duration_seconds histogram pool_role, step
buzz_db_connection_step_attempts_total counter pool_role, step, outcome

The fixed steps are writer_pool, physical_connect, created_at_floor, session_timeouts, isolation, and ready. Measurable phases emit start, duration, and terminal evidence. physical_connect and ready are success milestones: SQLx 0.9 exposes after_connect only after DNS, network, TLS, and authentication finish, so Buzz does not invent separate timings for those internal phases. Raw connect failures before after_connect are classified on the aggregate writer_pool phase during initial construction. Outcomes are succeeded, failed, timed_out, and cancelled where valid.

For a measurable phase on one pod, subtract all terminal outcomes from its start counter to derive the number of attempts currently in progress. The session phases run in order, so a later phase start also proves the earlier phases succeeded for that attempt.

Four start counters, four 13-series histograms, and fifteen terminal counters create a hard ceiling of 71 raw Prometheus series per pod. Connection ordinals, database URLs, hosts, usernames, SQL, and raw errors are forbidden as metric labels.

When audit logging is enabled, the buzz_db_connection_* metric totals combine the main writer pool and the separate audit writer pool.

Physical database pool utilization and configuration contract

Every physical Postgres pool the relay owns reports current utilization, capacity, and configuration under role-labelled families. The role vocabulary is closed — writer, reader, audit, search — and all four roles are always present, so a series never appears or disappears when an optional pool is enabled or disabled. An unconfigured optional pool reports buzz_db_pool_configured 0 with zero utilization and capacity rather than going missing.

Metric Type Labels
buzz_db_pool_connections gauge pool_role, state
buzz_db_pool_max_connections gauge pool_role
buzz_db_pool_configured gauge pool_role; 1 configured, 0 not configured

States are idle and active; their sum is the pool's current size. buzz_db_pool_max_connections reports capacity separately, so summing the connections family counts each current connection exactly once. The three families are fixed at 16 raw Prometheus series per pod (4 × 2 current utilization plus 4 maximum capacity plus 4 configured). The audit and search roles are statistics-only: pool ownership, capacities, timeouts, and query routing are unchanged, and these roles emit no acquisition telemetry.

The pre-existing unlabelled buzz_db_pool_* (writer) and buzz_db_read_pool_* (reader) gauges are still exported unchanged for dashboard compatibility. The reader family remains absent when no read replica is configured; use buzz_db_pool_configured{pool_role="reader"} for the explicit signal.

Pool sizing and the aggregate connection budget across all four roles remain deployment configuration concerns. The relay does not enforce a combined connection ceiling.

The reader pool keeps a warm floor of one fifth of its maximum (at least one) established connections per relay pod, so standing reader connections are that floor times the pod count; size the replica's max_connections for it. Each routed read proves the replica is caught up before serving from it. A proof that does not answer within 1s falls back to the writer, counted as buzz_db_route_decision{decision="writer",reason="reader_proof_timeout"}. A sustained rise in that series means the replica is reachable but not answering, for example during a failover. Every connection returned to the reader pool passes a release check first. A connection is closed instead of returned, and the pool dials a replacement, when the check does not answer within 5s (reason="timeout"), fails (error), or finds a transaction still open (open_transaction). Each close increments buzz_db_reader_release_closed_total{reason}, a counter with those three series. A rise in timeout means reader sockets stopped answering.

Relay Pod extensions

The chart exposes narrow extension points for init containers, volumes, relay volume mounts, and image command/argument overrides. extraManifests creates independent Kubernetes resources but cannot modify the chart-managed relay Deployment. These extension values insert fields into that Deployment, avoiding duplication of its environment, probes, security context, secrets, and chart-owned volumes.

For example, an init container can copy a wrapper binary into a shared volume and make that wrapper the relay entrypoint:

extraInitContainers:
  - name: install-wrapper
    image: example.com/wrapper-init:v1
    args: [/opt/wrapper/wrapper]
    securityContext:
      runAsNonRoot: true
      runAsUser: 65532
      runAsGroup: 65532
      allowPrivilegeEscalation: false
      capabilities:
        drop: [ALL]
    resources:
      requests:
        cpu: 10m
        memory: 16Mi
    volumeMounts:
      - name: wrapper
        mountPath: /opt/wrapper

extraVolumes:
  - name: wrapper
    emptyDir: {}

relay:
  command: [/opt/wrapper/wrapper]
  args: [/usr/local/bin/buzz-relay]
  extraVolumeMounts:
    - name: wrapper
      mountPath: /opt/wrapper

These values are raw Kubernetes fragments rendered with toYaml, not tpl. The chart does not validate cross-field relationships: extension names must not collide with chart-owned containers or volumes, mounts must reference existing volumes, and each init container must define an appropriate security context and resources. Empty relay.command and relay.args arrays preserve the image defaults; non-empty values override its entrypoint and arguments respectively.

Storage accounting worker

storageAccounting.enabled=true adds a CronJob that runs buzz-admin storage-snapshot against the relay's S3 bucket and writes a durable snapshot row, keeping the expensive object walk off the relay Pods.

Snapshot identity contract

Every snapshot row records the code that produced it in code_sha, taken from the BUZZ_STORAGE_SNAPSHOT_CODE_SHA environment variable the chart derives from the deployed image identity (image.digest when set, otherwise image.tag, otherwise Chart.AppVersion). The value is never hashed or sanitized.

Datadog version label

Every Pod the chart renders — relay, pairing relay, storage accounting, and the deletion drain operator job — carries a chart-owned tags.datadoghq.com/version label derived from the deployed image: image.tag when set, otherwise image.digest, otherwise Chart.AppVersion. The tag is preferred because it is the readable build name (sha-<sha7>, or main-<UTC commit time>-<sha7> for Kargo-promoted main builds) dashboards use; promotion tooling writes it together with the digest that actually pins the image.

relay:
  podLabels:
    tags.datadoghq.com/env: production
    tags.datadoghq.com/service: buzz
    # tags.datadoghq.com/version: NOT set here — the chart owns it.

Precedence is explicit: a tags.datadoghq.com/version supplied under any workload's podLabels is ignored. Every other label — including tags.datadoghq.com/env and tags.datadoghq.com/service, which describe the deployment rather than the image — passes through unchanged. Remove any wrapper-maintained version pins when upgrading; a pin equal to image.tag renders identically, so the upgrade does not roll Pods for that label.

A Kubernetes label value is capped at 63 bytes, must begin and end with an alphanumeric, and may otherwise contain only [-._a-zA-Z0-9] — a far narrower grammar than an OCI tag. image.tag is deliberately left unconstrained, so the label is derived through a total mapping rather than a check that could reject a tag the chart used to accept:

Revision Label
sha256:<64 hex> digest the hex without sha256:, first 63 characters
already a valid label value, lowercase, no __, and not exactly 63 lowercase hex characters emitted byte for byte (e.g. 1.2.3-rc.4, sha-1a2b3c4)
anything else the first 63 hex characters of the revision's SHA-256

The passthrough row requires Datadog's normal form because Datadog lowercases tag values and collapses __: passing ReleaseA and releasea through unchanged would report one version for two revisions. Every emitted label is therefore already normalized, so distinct labels stay distinct in Datadog.

When both image.tag and image.digest are set, the label names the tag — even if the tag and digest disagree. The digest still pins what actually runs (repository@digest), so keep the two in step when promoting.

Exactly 63 lowercase hex characters is a reserved shape — it is what the first and third rows emit, so the passthrough row must not be able to emit it too. A tag of that shape is hashed rather than preserved. Shorter hex tags and ordinary 40-character git SHAs are unaffected.

Arbitrary OCI revisions outnumber 63-byte label values, so no mapping onto this grammar can be injective and the chart does not claim one. What it does claim is collision resistance: every case retains 252 bits of SHA-256, the same margin the digest case has always relied on, and the reserved shape keeps that margin across cases rather than only within one. Two earlier iterations fell short of this — one kept a readable prefix with only 40 bits of hash (a colliding tag pair was brute-forced in about a second), and one let a passthrough tag reproduce a hashed label with no hash work at all, by copying a rendered label into image.tag.

The image reference in the Pod spec and BUZZ_STORAGE_SNAPSHOT_CODE_SHA always keep the exact revision; the label is the joinable telemetry key. See docs/deployment-identity.md.

Device pairing relay

The chart can run Buzz's stateless pairing WebSocket relay as an independent Deployment and Service using the same image as the main relay:

pairingRelay:
  enabled: true
  url: wss://pairing.example.com

pairingRelay.url is advertised in the main relay's NIP-11 document so Buzz clients connect directly to the dedicated endpoint. The chart does not create an Ingress or HTTPRoute for the pairing Service; route the public hostname to <release>-buzz-pairing:5000 with your platform's ingress configuration.

HA (production)

replicaCount > 1 hard-requires Redis:

  • Redis (redis.enabled=true, externalRedis.url, or REDIS_URL in existingSecret) — for buzz-pubsub fan-out

It does not require ReadWriteMany git storage. Git ref/object state is object-store-backed (each request hydrates an ephemeral repo from S3-compatible storage; writer serialization is the object-store pointer CAS — see docs/git-on-object-storage.md), and repo-name uniqueness lives in Postgres. Each replica can use its own ReadWriteOnce volume; no shared filesystem is needed.

The chart template-fails if the Redis invariant is broken at replicaCount > 1. No silent degradation.

Relay autoscaling

The optional HPA scales the relay on the larger recommendation from CPU or average active WebSockets per pod:

autoscaling:
  enabled: true
  minReplicas: 5
  maxReplicas: 15
  targetCPUUtilizationPercentage: 65
  websocketMetricEnabled: true
  websocketMetricName: buzz_ws_connections_active
  targetWebsocketConnections: 5000

CPU scaling requires Kubernetes Metrics Server. Set websocketMetricEnabled: false for a CPU-only HPA. WebSocket scaling additionally requires a custom-metrics adapter (for example Prometheus Adapter) configured to expose the relay's buzz_ws_connections_active gauge as a pod metric with the name in websocketMetricName. The chart creates the HPA but deliberately does not install or configure a cluster-wide metrics adapter. Scale-down is gradual by default so long-lived WebSocket connections have time to drain.

Upgrades

Schema migrations are embedded in the relay binary via sqlx::migrate! and run at startup, gated by BUZZ_AUTO_MIGRATE (default true). Multiple replicas race-safely behind a Postgres advisory lock. helm upgrade is the entire upgrade procedure.

Migration 0032 is a hard compatibility boundary for relay versions that publish repaired channel rosters. The relay verifies the roster-fence trigger catalog and behavior before opening listeners and refuses to start if 0032 is missing or inert. Apply migrations before rolling the relay; for large installations, prefer a controlled buzz-admin migrate job with PostgreSQL lock monitoring before the code rollout.

If you prefer decoupling migrations from serving, set migrate.autoMigrate=false. In that mode the chart does not run migrations for you — you own running buzz-admin migrate (separate Pod / one-shot Job) against the database before every helm install / helm upgrade. Readiness probes only verify DB connectivity, not schema freshness, so a pod will appear healthy against an unmigrated schema and fail under load. A pre-upgrade Helm Job for this is on the chart roadmap; the values knob migrate.preUpgradeJob.enabled is reserved.

Backups

Save these. Losing any of them is data loss. See NOTES.txt printed by helm install for the live list:

  1. BUZZ_RELAY_PRIVATE_KEY — relay identity. Rotating it = new identity (federation peers will not recognize the relay).
  2. PostgreSQL database — the canonical event store.
  3. S3 bucket — media blobs (chart default bucket: buzz-media).
  4. Git PVC — repo on-disk state served by the relay's git endpoint.
  5. Owner private key — held by the operator, not by this chart. Restore by re-installing with the same ownerPubkey.

Honest limitations (v1)

  • Bundled MinIO is eval-only. The quickstart profile runs an in-cluster MinIO (single replica, no HA, lookup-autogenerated credentials) so the relay starts with zero external object storage. Production leaves minio.enabled off and points s3.endpoint (or BUZZ_S3_* in existingSecret) at managed S3-compatible storage. The bundled Deployment is not GitOps-safe and is not intended for production traffic.
  • Minimal-mode is not yet supported. The relay's BUZZ_PUBSUB=local / filesystem media paths are upstream work in progress — even quickstart currently stands up real Redis and S3 rather than the relay's single-node fallbacks. (Full-text search already runs in Postgres, so no separate search service is provisioned.)
  • Cosign signing of the published chart is a follow-up (the relay image is attested via actions/attest-build-provenance; the chart is not yet). The chart itself is published to GHCR — see Releasing.

Releasing

The chart is published to GHCR as an OCI artifact at oci://ghcr.io/block/buzz/charts/buzz by the helm chart workflow (.github/workflows/helm-chart.yml), versioned independently of the desktop app and the relay image via its own chart-v* tags. Every PR/main push still lints, unit-tests, and render-checks the chart; only a chart-v* tag publishes, so an in-progress main can never overwrite a released version.

To cut a release, push a chart-release/<version> branch whose <version> matches Chart.yaml's version; merging it auto-tags chart-v<version> and dispatches the publish job (same lane machinery as the desktop and relay releases — see .github/workflows/auto-tag-on-release-pr-merge.yml). The publish job fails loudly if the tag version and Chart.yaml version disagree.

Development

# Render every fixture
for f in ci/*-values.yaml tests/fixtures/*-values.yaml; do
  helm template buzz . -f "$f" >/dev/null && echo "ok: $f"
done

# Unit tests
helm plugin install https://github.com/helm-unittest/helm-unittest
helm unittest .

# Lint
helm dependency build .
ct lint --config ../../../ct.yaml --charts .