mirror of
https://github.com/block/buzz.git
synced 2026-10-27 20:41:57 +00:00
73a2a05c973bb97593f2d456c23a2ccdfe1a7bec
62 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tom Brow
|
73a2a05c97 |
feat(push): support platform-managed gateway runtime pods (#8113)
Platform-owned service meshes need runtime pod identity and a replacement network policy without patching rendered gateway manifests. Add runtime-only pod labels, service account and shutdown-grace settings, plus an explicit named replacement-policy mode. Migration isolation remains independent; the default render is unchanged. Chart 0.3.4 publication and live deployment verification follow in the separate release and deployment work. ### Validation Added tests cover runtime-only settings, reserved selector labels, minimum shutdown grace, replacement-policy acknowledgement and conflicting settings. The complete chart render suite and release-contract tests pass. Default rendering matches the current main chart byte-for-byte. Repository-wide `just ci` reached unit tests but failed in existing buzz-acp tests outside this chart change. Live mesh behavior has not been exercised. --------- Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
Tom Brow
|
c9143ed8f7 |
chore(push): release gateway chart 0.3.3 (#8107)
Prepares publication of the Buzz push gateway chart 0.3.3, including the optional migration Pod annotations already merged in #8105. This PR adds release notes only; chart templates, values, and schema are unchanged from the reviewed implementation. Merging the `push-chart-release/0.3.3` branch triggers the normal release workflow: create `push-chart-v0.3.3`, validate the chart, then publish `oci://ghcr.io/block/buzz/charts/buzz-push-gateway:0.3.3`. Keep this PR in draft until publication is explicitly authorized. It does not deploy a gateway. ### Validation Focused migration annotation renders, release contract checks, and local chart packaging pass. The merged implementation passed chart CI and Jude's automated review in #8105. The earlier repository-wide local `just ci` run failed in unchanged ACP tests; it was not rerun for release notes. The registry manifest digest can only be verified after publication. --------- Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
Tom Brow
|
fd885b5530 |
feat(push): support migration pod annotations (#8105)
Allow operators to configure migration Pod annotations independently of runtime Pod annotations. This lets clusters with automatic sidecar injection opt the migration Job out without changing the gateway Deployment or Helm hook metadata. The new string-map setting defaults to empty. Chart version 0.3.3 is prepared here; publication requires a separate approved chart release PR. ### Validation The full chart render suite and focused migration annotation tests pass. Tests verify unchanged default resources and hooks, independent runtime annotations, and rejection of non-string annotation values. Independent agent review found no blockers. Repository-wide `just ci` exited 1 in unchanged `buzz-acp` tests: idle/stdout keepalive timing and steering assertions (one failure in the first group, four in the second). Subsequent recipes were not reached. No rerun or unrelated code fix was attempted. Human testing remains pending. Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
Luke Tornquist
|
6918dd3a83 |
Schedule the deletion drain safely (#7827)
## Why The deletion engine already exposes a one-shot `buzz-admin deletions drain` command, but operators have no safe way to schedule it. A generic workload wrapper could expose arbitrary commands or inherit relay credentials, which would add a second control plane and widen the blast radius. ## What This PR adds an opt-in Kubernetes CronJob for the typed deletion drain command. It is stacked on #7818, which adds owner-request admission. ## How The chart keeps the job disabled by default. The CronJob uses `Forbid`, zero Kubernetes retries, bounded runtime and history, and only the database, Redis, and object-store settings required by the executor. PostgreSQL leases, checkpoints, and retry timing remain the sole durable retry authority. The closed `_operator-jobs.tpl` helper centralizes safe CronJob mechanics without creating an arbitrary command or environment registry. This leaves the touched system simpler than a general job framework would. ## Risk The deployment blast radius is low because the job is disabled by default. When enabled, it can run destructive deletion work that the database has already approved, so the chart limits concurrency, retries, credentials, and execution time. ## Testing Rendered every `deploy/charts/buzz/ci/*.yaml` fixture with `helm template`; all passed at `e68cb753bb6673aa1baa60046969bdb2d4738b62`. Ran the full independent Desktop, Tauri, and web lanes; all passed at `e68cb753bb6673aa1baa60046969bdb2d4738b62`. Flutter package resolution made no progress within three minutes, so the mobile lane is not claimed green. ## Bigger picture A later stack layer must inventory and approve admitted owner requests. Until then, this scheduler cannot make a new owner request destructive. Generated with Codex --------- Signed-off-by: tornquist <tornquist@squareup.com> Signed-off-by: Codex <noreply@openai.com> Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> |
||
|
Luke Tornquist
|
b37e477211 |
Make relay readiness process-local (#7341)
## Summary Make Kubernetes readiness depend only on the relay process lifecycle, so a shared Postgres or Redis slowdown cannot withdraw every pod at once. Move dependency checks to one fixed-cadence loop per pod. The loop permits only one evaluation at a time, emits metrics after each sample, and caches a timestamped report. `/_status` reads that cache without calling Postgres, Redis, or the deletion catalog. It reports `not_yet_sampled`, `fresh`, or `stale`. Each completed sample also writes a Unix timestamp gauge, so Datadog can compute sample age at query time. Fail closed when the community lifecycle lookup errors. An inconclusive tenant-state lookup no longer admits a socket to AUTH or REQ handling. ### Related issue None found. This addresses the elevated relay HTTP 500 and reconnect incident investigated on 2026-09-03. ### Tradeoff The readiness gauge now reports the latest private-probe observation, not the shutdown transition itself. After shutdown starts, a scrape can still see `1` until the next readiness probe refreshes the gauge to `0`. Kubernetes routing is unaffected because it uses the readiness response from the authoritative process state. A terminal `0` was never guaranteed: even a transition-owned gauge could update in process and then exit before Prometheus scraped it. If no scrape occurs before exit, both designs miss the terminal sample. The new design adds at most the interval until the next readiness probe when a scrape occurs during shutdown. Dependency status is now sampled every 30 seconds rather than on demand. A slow evaluation delays the next sample instead of stacking more work. The status payload exposes the cached report age. The completion timestamp stays unchanged when sampling stops, so `time() - timestamp` crosses the stale threshold without a server aging loop. Outcome counters and duration histograms remain cumulative and cannot provide no-data detection. ### Testing Gimli exercised exact SHA `edff1ec4a5362e132e367bf7449d2b0a68f1e9a9` on an isolated Blox stack. Repeated `/_status` bursts caused no dependency calls, metrics advanced without status traffic, stale status remained visible, and lifecycle lookup failures admitted no AUTH or REQ handling. Follow-up verification at `fdbb394ed598c94cde85ed01cb11e9de0ee8f00d` confirmed sampler-owned outcome and duration emission without status traffic and bound the real startup seam. The final timestamp-gauge change is covered by targeted red/green tests and the same boot regression at `c0f36e22b527c2fd6a63a548d036d006ffa17171`. ### Complexity Readiness remains one process-local sample per probe. Dependency I/O now has one owner and one execution path; request volume cannot increase dependency-check load. One completion timestamp replaces the prior server-side age update path; Datadog computes age without another loop. Community admission also returns to one fail-closed rule for inactive and unknown tenant state. Generated with [Claude Code](https://claude.ai/code) --------- Signed-off-by: tornquist <tornquist@squareup.com> Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Amp <amp@ampcode.com> |
||
|
Tom Brow
|
781d39510c |
feat(push): support configurable HTTP(S) delivery URLs (#7877)
The relay can use a configurable HTTP(S) gateway delivery URL and signs the same URL it sends. The gateway verifies NIP-98 against the received request URL, including authority, port, path and query. App Attest, scoped grants, replay checks and delivery quotas are unchanged. Forwarding proxies must preserve Host and supply the original request scheme through X-Forwarded-Proto. The NIP relay-delivery wording now defines the configured HTTP(S) URL as the signed URL and labels the public URL as an example. <!-- buzz-review-completed --> --------- Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
Tyler
|
797012ff01 |
fix(ci): consume the published MinIO image (#7870)
Switch every CI job that starts MinIO to the public `ghcr.io/block/buzz-minio:latest` image published by #7869. Integration jobs pull the image; image builds stay in the separate, path-filtered publisher workflow. - Apply the CI Compose override to both desktop integration shards, backend integration, relay E2E, and mesh lifecycle, including script-mediated Compose calls. Both the server and bucket initializer use the Buzz image. - Override both MinIO images in Helm's quickstart CI values, covering the server, bucket initializer, and relay's wait-for-bucket init container in the gated kind installation. - Select integration and mesh checks when either Compose file changes, with regression coverage using the real pinned `dorny/paths-filter` action. ### Publication verified #7869 is merged, and [the first main-branch publication](https://github.com/block/buzz/actions/runs/36021389137/job/107706874954) succeeded. An anonymous pull using an empty Docker configuration returned the publisher's digest: `sha256:b8470bbeafbf57b20c86cf63804682b714bdcfdbb517f3770247e321e623f48f`. This PR is rebased onto main and ready for consumer integration checks. ### Validation - Ran the repository's MinIO smoke script against the actual published AMD64 image: healthy startup, bucket initialization, object upload/read/delete, and rejection of anonymous reads all passed. Used the rendered CI Compose services with isolated local ports and resource names. - Verified `mc` runs and writes its configuration as Helm's non-root UID/GID 65532 with dropped capabilities and no privilege escalation. - Helm 3.16.4 lint and quickstart rendering passed; all three rendered MinIO container references use the Buzz image. - All 14 CI selection cases and the required-context isolation check passed. Mesh workflow passes actionlint; shared workflows pass with shellcheck disabled for pre-existing findings. - Fresh repository-wide review traced workflow and script consumers and found no remaining CI image omissions. Full application integration checks and a real kind installation have not been run locally. --------- Signed-off-by: Tyler Longwell <tlongwell@squareup.com> |
||
|
ravarora2
|
4142d2b84e |
Use worker snapshots for relay storage metrics (#7845)
## Summary Make the relay read completed storage-accounting snapshots from PostgreSQL by default. An environment with no snapshot emits no storage metrics. When a worker publishes its first result, the relay picks it up on the next successful leader usage tick, without a restart or configuration change. This changes how storage usage is measured and reported. **S3 remains the object store. PostgreSQL stores the completed counts, byte totals, community breakdown, and calculation metadata—not the objects themselves.** ### Why move relay storage accounting entirely to database reads? Today the relay supports two sources: its own S3 scan (`inline`, the default) and the worker's saved snapshot (`external`). That requires operators to deploy a worker and then coordinate a separate relay-mode change in each environment. The worker already owns the S3 listing and calculation. It runs in a separate process with its own object cap, memory limit, and deadline. Keeping a second scan inside the serving relay retains the resource pressure and failure modes that this isolation is meant to remove. PostgreSQL provides a durable handoff. The worker replaces one completed snapshot atomically after a successful calculation. The relay reads that row and exposes its values through the existing metrics endpoint. A relay restart or leadership change can load the last completed result without scanning the bucket again. There is deliberately no S3 fallback when a snapshot is absent or a read fails. Such a fallback would put the expensive scan back in the relay precisely when a worker is absent or unhealthy. It would also leave two calculation paths with different resource limits and freshness behavior. ### What happens if no worker exists? **A worker that has never published is a normal, successful inactive state.** The snapshot table must exist through the normal database schema, but it may contain zero rows. The reader returns success, emits no storage totals or failed-load gauge, and checks again on the next usage tick. It does not exit the relay, start a worker, or list S3. The relay checks for a completed database row; it does not inspect Kubernetes Jobs or CronJobs. Worker presence and snapshot presence are therefore different: | Situation | Storage-reader behavior | |---|---| | No worker and no snapshot row | Return success, emit no storage metrics, and query again next tick. | | Worker starts later and publishes its first row | Read and emit it on the next successful leader usage tick; no restart or mode change. | | Worker is removed or fails, but its last valid row remains | Continue reporting the last completed totals with their increasing age. | | Query succeeds but a previously present row has been removed | Clear the cache and stop refreshing storage series; existing exporter series expire through the idle timeout. | | Database read fails, times out, or returns an invalid snapshot | Log a warning, set `buzz_storage_snapshot_load_ok=0`, retain any last-good cached totals, and retry next tick. | | Snapshot table is missing | Treat this as a schema error, not normal worker absence; follow the failed-read behavior above. | | `BUZZ_STORAGE_METRICS=off` | Skip the snapshot query and all storage metric emission. | No row means “not measured,” not zero bytes. A valid completed snapshot of an empty bucket can report zero. If a read fails before any good snapshot has been cached, only failed-load health is emitted. These reads happen in the existing background metrics task, not the relay startup path. A storage-read failure does not request process exit or change readiness directly. A wider database outage can still affect the relay's existing database-dependent operations and readiness. ### Read, parse, and emit path ```text Worker: S3 listing -> BucketSnapshot -> JSON -> PostgreSQL Relay: PostgreSQL -> JSON -> BucketSnapshot -> existing metric gauges Collector: relay /metrics endpoint -> Datadog ``` 1. `main.rs::run_storage_sweep_tick` calls the reader from the existing leader-only usage task. The default interval remains 300 seconds. 2. `storage_sweep.rs::run_storage_metrics_tick` coordinates the read, cache update, failure health, and metric emission. 3. `refresh_persisted_snapshot` calls the existing `Db::load_storage_accounting_snapshot()` method. A five-second timeout covers pool acquisition and the query. It uses the relay's existing pool. 4. The existing query in `buzz-db/src/store/storage_accounting.rs` reads `snapshot`, `completed_at`, `duration_ms`, `max_objects`, and `code_sha` from the singleton row. 5. `serde_json::from_value(stored.snapshot)` decodes the JSON into the existing `buzz_media::BucketSnapshot` type. Rust infers that type from `cache_persisted_snapshot`'s argument. The generated `Deserialize` implementation handles all fields, including the UUID-keyed `per_community` map. 6. `emit_cached_storage_metrics` sets the existing physical/logical totals and community byte/object gauges. The existing Prometheus exporter exposes them for Datadog collection; this code does not send snapshot JSON to Datadog. The worker's `BucketSnapshot`/`CommunityStorage` structures, JSON field names, SQL publication method, and database schema are unchanged. The old external-mode JSON conversion moves out of `main.rs` into the reader helper. No new format or hand-written field parser is needed. Duration and object-cap metadata are validated before replacing the cache. The five-second relay read timeout is separate from the worker's 30-second startup acquisition budget introduced in #7770. This PR does not change the worker's pool or startup logic. ### Configuration and code changes | Configuration | Before | After | |---|---|---| | Unset, `inline`, or `on` | Relay-local S3 scan | Read completed database snapshots. `inline` logs one migration warning. | | `external` or `snapshot` | Read completed database snapshots | Continue reading snapshots. | | `off` | Disabled | Disabled. | | Unknown or empty value | Disabled with a configuration error | Same. | Existing `inline` settings intentionally become reader aliases. Merely changing the default would leave older deployments on their explicit `inline` setting and preserve the need for coordinated configuration PRs. Remove relay-local scan scheduling, in-flight scan state, scan configuration, and obsolete scan-attempt tests and health metrics. Preserve the physical/logical total and per-community metric names, leader-only emission, community scope filtering, and cleanup of old community labels. The shared relay-state field remains in place; its comment now describes a cached worker result. The Helm chart defaults to `relayMode: external` while leaving `storageAccounting.enabled: false`. Its rendering test verifies that the reader is configured even when no worker is created. The reader regression tests cover missing rows, later publication, updates, invalid data, timeout, removal, reactivation, and the explicit off switch. `docs/storage-accounting.md` describes this operating model. ### Freshness and rollout impact `buzz_storage_snapshot_age_seconds` uses the worker's original completion timestamp. Re-reading an old row never makes it fresh. `buzz_storage_snapshot_load_ok=1` means the row was read and decoded; it does not prove that the worker's latest attempt succeeded. Monitor snapshot age against the worker's schedule and allowed runtime, and monitor Job failures separately. The old relay scan-attempt health gauges are retired because the relay no longer runs those scans. Deploy this relay version through the usual release process. Once it reaches an environment, workers can be enabled independently. Already-released charts that still set `inline` work with the new binary; future chart releases default explicitly to `external`. An intentional `off` override still requires an operator to enable the reader. **Environments with no completed snapshot lose relay-calculated storage totals after this upgrade.** That is the intended tradeoff for removing scans from the serving process. Environments with a prior snapshot continue showing its last completed totals and age. This PR does not install workers in additional environments or deploy the new relay image. ### Related issue Related to #4601. Follows the worker startup fix in #7770. The separate fleet-usage collector work in #7176 overlaps this code and may need reconciliation when it merges. ### Testing Built and ran the relay locally against isolated PostgreSQL and Redis with the legacy `inline` setting, five-second usage ticks, and a 15-second metric idle timeout. The same process was exercised through these steps: 1. Start with an empty snapshot table: relay becomes ready and exposes no storage series. 2. Publish a snapshot containing 123 bytes: the community gauge becomes 123. 3. Replace it with 456 bytes: the gauge becomes 456. 4. Remove the row: storage series expire without fabricating a zero-byte measurement. 5. Publish 789 bytes: metrics reactivate without restarting the relay; readiness remains healthy. The local S3 endpoint recorded zero requests. The unrelated Git conformance startup probe was disabled for this isolated reader check; the test covers storage accounting, not every other use of S3 in the relay. No new relay image was deployed to staging or production during this test. The broad local integration run also hit an existing `p0_pool_acquisitions_use_typed_operation_pairs_without_other` failure in `buzz-db/tests/observability_source.rs`. The same source-only test fails on unmodified `main` at `0ef7a2222`; this PR changes neither the test nor its failing source. The full local `just ci` run stopped at mobile native-asset setup: the `objective_c` build hook received no SDK path from `xcrun`. Mobile tests did not run. Generated with Codex Signed-off-by: Ravneet Arora <rarora@squareup.com> |
||
|
Tom Brow
|
0ef7a2222e |
fix(mobile-infra): render push grant lifetimes as decimal in chart 0.3.2 (#7820)
A valid 30-day push gateway grant lifetime could render in scientific notation (e.g. `2.592e+06`) in the chart, which the gateway binary does not accept. Always ormat the integer as decimal text, preserving the strict gateway parser. ### Testing Added regression coverage that feeds actual Helm-rendered environment values into the production config parser. Covers defaults, values files, command-line overrides, exact lifetime boundaries, and invalid input rejection. The default case failed before the fix and passes afterward. Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
Tom Brow
|
4e65148e76 |
release: push gateway chart 0.3.1 (#7749)
Publishes chart 0.3.1 for `buzz-push-gateway`, containing #7717. Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
Tom Brow
|
fe808eae0e |
fix(push): label plaintext push gateway service as HTTP (#7717)
The gateway serves plaintext HTTP on its public container port, but the chart currently advertises that Service port as `https`. Label it `http` so protocol-aware proxies discover the application's actual protocol. Numeric port 8080, target port `public`, and the application version remain unchanged. Consumers that refer to the old Service port name must update that reference. Prepare chart version 0.3.1, which is absent from the current release tags and OCI registry. This ordinary fix branch does not trigger the chart release lane. Publishing and downstream dependency updates remain separate steps. This corrects protocol metadata; it is not evidence of a TLS-origination failure. Istio controls upstream TLS through DestinationRules and auto-mTLS independently of this port label. Validation: added rendered-Service and container-port assertions across default, production, and Gateway API fixtures. The assertion fails with the old label and passes with the fix. The full chart render/lint suite and release-contract checks pass. Repository-wide `just ci` is still running locally. --------- Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
ravarora2
|
4c2086c084 |
feat(db): expose connection setup metrics (#7286)
## Why
Current pool metrics show checkout outcomes and pool state after the
fact. They do not show when a checkout began or which writer-connection
setup step failed. During startup and pool growth, operators need to
distinguish pool saturation from a slow or unsafe connection setup.
## What
- Count every instrumented database checkout when it starts, with a
fixed operation label.
- Measure writer-pool creation and every physical writer connection
across `physical_connect`, `created_at_floor`, `session_timeouts`,
`isolation`, and `ready`.
- Publish fixed-cardinality start, terminal-outcome, duration, and
waiter metrics without database URLs, SQL, raw errors, connection
ordinals, or per-connection lifecycle logs.
- Document how to identify the active setup bottleneck during a rollout
or incident.
## How
The existing typed checkout wrapper records starts and current waiters.
The huddle-history path uses that wrapper instead of a raw pool
checkout.
The production SQLx `after_connect` hook records start, terminal
outcome, and duration for each sequential safety step. A drop guard
records cancellation once if setup exits before a terminal result. Fixed
enums keep every label bounded.
For a phase, `started_total - sum(attempts_total)` is the number of
in-progress attempts on that pod. Because setup is sequential, a later
phase starting also proves the earlier phases succeeded.
The database path emits metrics only. This PR intentionally does not add
per-connection lifecycle receipts or direct `stderr` writes.
## Risk
Medium. This changes the production writer-pool `after_connect` hook and
adds one counter update to instrumented checkout paths. It does not
change the database safety statements or their failure behavior. The
telemetry has fixed labels and no per-connection identifiers.
## Testing
At exact head `ffb5fcb114a9c986ffd0cb2cfc9d413a32d5b1aa`, a release
relay was started against isolated local PostgreSQL, Redis, and MinIO
services. The main health endpoint returned `ok`, readiness returned
`{"status":"ready"}`, the connection-step metrics were exported, and no
database lifecycle receipts were emitted. A release CLI then created a
channel, sent a message, and read the same event back successfully.
The earlier staging deployment used pre-rebase head
`f58e9480a4f068db0c591f604fd6800fdd4bfc45`. The deployed
multi-architecture image came from [GitHub Actions run
33780888255](https://github.com/block/buzz/actions/runs/33780888255),
manifest
`sha256:b9351fa644e08376cbe1999f9bee311d33d1799a68eadef7929c4f862a832fec`.
The [staging
deployment](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/317)
brought both pods in ReplicaSet `buzz-6c8758bd7d` to Ready with zero
restarts, and the connection metrics produced data in the [rollout
dashboard](https://app.datadoghq.com/dashboard/tm7-qxr-wt2/buzz-startup--rollout-safety).
That staging image predates the cleanup that removed per-connection
lifecycle logs; the metric schema and database safety statements are
unchanged.
## Verification
- `cargo test -p buzz-db -- --test-threads=1`: 128 tests passed across
the package and integration target; 255 opt-in tests remained ignored.
- Four opt-in production-path PostgreSQL regressions passed: initial
minimum connections, post-startup pool growth, isolation failure, and
session-timeout setup failure.
- All 35 relay media tests passed against the isolated PostgreSQL
database.
- Rust formatting, workspace clippy, desktop/Tauri clippy, web checks,
mobile analysis, security checks, and file-size policy checks passed.
- The repository-wide unit stage also exposed three unrelated existing
`buzz-acp` failures: one timing-sensitive keepalive test and two
environment-default tests.
## Bigger picture
This is the database-metrics part of the startup and rollout
observability work. The early-startup lifecycle foundation merged in
#7258, so the rebase removed that duplicate commit from this PR.
Originating discussion:
buzz://message?channel=6ac85131-70cd-4bda-a031-38d34114934e&id=fa2bed181c092697210a60bb6eedc55a665d5c1c6cabc1413a04686647011f71
Generated with Codex
---------
Signed-off-by: Ravneet Arora <rarora@squareup.com>
|
||
|
ravarora2
|
6b1e45d2f6 |
Isolate S3 storage metrics from the relay (#7543)
## Why **This PR moves the S3 storage scan out of the serving relay and into a separate batch job.** The job saves a complete result in PostgreSQL. The relay reads that result, exports the existing storage totals, and reports the snapshot's freshness and database-read health. ```text Today: relay -> scan S3 -> calculate totals -> Datadog metrics After rollout: CronJob -> scan S3 -> PostgreSQL snapshot -> relay -> Datadog metrics ``` The code in this PR is the new pipeline. It adds the worker, database snapshot, relay read mode, and Helm CronJob template. It does **not** enable the pipeline in production. The chart keeps the CronJob disabled, and the relay keeps its current inline mode by default. [squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244) will set the production limits, enable the job, and switch the relay to the saved snapshot after this PR merges. The relay currently lists the media bucket once an hour and calculates storage totals inside the serving process. The production bucket now exceeds the relay's one million object cap, so each sweep fails and the last good per-community values remain stale. The Datadog dashboard can no longer show current storage use by community. Raising that cap inside the serving relay would move a larger, memory-heavy scan into the process that owns live WebSocket connections. The scan retains state for unique blob hashes and community bindings, so its memory use grows with the bucket. A large scan could raise relay memory use, trigger restarts, and drop user connections. The goal is to restore current metrics without putting a production-scale bucket scan in the relay's failure domain. No media objects move or change. Only the process that calculates the totals and the handoff to the relay change. ## What - Add `buzz-admin storage-snapshot`, a run-once worker that scans S3 and saves one complete accounting snapshot in PostgreSQL. - Add `external` relay mode to read the saved snapshot. Keep the current `inline` mode as the default, and add `off` as a fail-closed option. - Add an opt-in Helm CronJob for the worker. The 10 million object cap exists only on this Job. - Keep the current fleet and per-community gauges, and add snapshot age, generation duration, configured cap, cap utilization, and database-load health gauges. - Add an additive migration for the latest complete snapshot and its run metadata. ## Overall plan 1. Merge this PR to add the worker, database handoff, relay modes, and disabled-by-default chart support. This step does not change production behavior. 2. Publish the relay image and Helm chart `0.1.9` from `main`. 3. Update [squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244) to pin those final artifacts and rerender its manifests. 4. Use the infrastructure PR to deploy the CronJob with a 10 million object cap, a one-hour deadline, and a 20 GiB memory limit. 5. Start one manual Job during a staffed window and verify its snapshot, duration, memory use, relay stability, and Datadog output. 6. Keep the daily schedule and `external` relay mode only after that gate passes, then add alerts for stale snapshots, snapshot-load failure, failed jobs, cap pressure, duration, and worker memory. The code and infrastructure remain separate so this PR can ship the mechanism without silently enabling a production-scale scan. The infrastructure PR owns the production settings and rollout gate. ## How ### Complete snapshot handoff The worker takes a deployment-wide PostgreSQL advisory lock before it lists S3. This prevents scheduled and manual workers from overlapping, even if Kubernetes starts both. The final database replacement runs through that same lock-owning session. If the session dies during the scan, publication fails instead of letting the stale worker overwrite its successor. A worker that no longer owns the lock cannot publish through another pooled connection. The worker reads S3 in pages of 1,000 objects and folds each page into physical totals, logical per-community usage, and anomaly counts. It stops before folding a page that would exceed `--max-objects` and emits structured progress every 100,000 objects. The worker replaces the singleton database row only after the full scan succeeds. A listing error, malformed page, cap breach, serialization error, lost lock session, or database error cannot publish a partial snapshot. The last completed snapshot stays available. Each stored result includes its completion time, duration, cap, and image or source revision. PostgreSQL is the handoff between the batch worker and the relay. The worker writes one complete deployment-wide snapshot, and the relay reads that snapshot on its normal leader metrics tick. No new network service or worker endpoint is required. ### Relay behavior `BUZZ_STORAGE_METRICS=inline` keeps the existing in-process sweep and its one million object default. `external` loads the newest completed worker snapshot on each leader metrics tick. `off` emits no storage-family metrics. An unknown value fails closed to `off`. External mode republishes the existing fleet, per-community, and anomaly metric names, so those current queries remain valid. Snapshot handoff health uses a separate family: `buzz_storage_snapshot_age_seconds`, `buzz_storage_snapshot_duration_seconds`, `buzz_storage_snapshot_max_objects`, `buzz_storage_snapshot_cap_utilization`, and `buzz_storage_snapshot_load_ok`. The `buzz_storage_sweep_*` gauges remain inline-only because they describe a scan attempted by the relay process. External mode does not synthesize `sweep_ok=1` from the last successful database row: an old row proves only that an earlier worker completed. Kubernetes CronJob/Job status is authoritative for the latest worker result, including non-zero exit, deadline expiry, and OOM termination. The relay keeps the last complete snapshot available when a worker run fails. Snapshot age shows that no replacement arrived on schedule, while `buzz_storage_snapshot_load_ok` distinguishes staleness from a missing row, database read failure, or invalid payload. If a community disappears, changes host, or leaves the configured emission scope, the relay zeroes the old labeled series instead of leaving stale values behind. Per-community bytes are logical referenced media usage. If two communities reference the same blob, each community receives the blob's bytes. Fleet physical totals still count each S3 object once. This PR preserves those existing metric semantics. ### Deployment boundary The CronJob is disabled by default. When enabled, it uses `concurrencyPolicy: Forbid`, no retry, a configurable active deadline, bounded history, explicit CPU and memory resources, a read-only root filesystem, and no Istio sidecar by default. The worker receives only `DATABASE_URL`, S3 settings, optional S3 credential references, and the code revision. It does not receive the relay private key, Git hook HMAC secret, Redis URL, or the full relay secret as environment variables. Operators may reuse the Buzz service account or select a dedicated one. ### Non-goals - This PR does not enable the worker or `external` mode in production. - This PR does not change media uploads, object ownership, retention, or deletion. - This PR does not change the names or meaning of the existing fleet, per-community, or anomaly storage gauges. - This PR does not add a lease service or allow multiple workers to publish concurrently. ## Risk The application and chart defaults do not enable the new worker or raise the serving relay's cap. The database change is an additive deployment-global singleton table. The relay's default remains `inline`, so merging this PR alone does not change the active metrics path. The main rollout risk is worker memory use because the fold retains state for distinct blob hashes and community bindings. The first production-sized run must confirm peak memory and duration before the schedule is trusted. The 10 million cap and one-hour deadline bound the first deployment, but they do not replace measurement. External mode also needs stale-snapshot, snapshot-load, and job-failure monitors because the relay intentionally keeps publishing the last complete result after a failed job or read error. This behavior avoids gaps and partial data, while snapshot age and load health tell operators when the result or PostgreSQL handoff needs attention. Job status remains necessary for immediate failures and termination reasons that a killed process cannot record itself. ## Testing No production S3 scan was run from this PR. Automated regressions verify that external mode keeps stale last-good totals visible without emitting inline attempt-health gauges, reports database-load failure while retaining the cache, and leaves inline telemetry unchanged. Mutation checks also prove those tests fail if external mode is routed through inline emission or a failed load is reported as healthy. After deployment, create one manual Job from the CronJob during a staffed window. Require a completed Job, exactly one new snapshot, fewer objects than the configured cap, acceptable duration and peak memory, no relay restart regression, restored Datadog points, and no OpenMetrics sample truncation. ## Next steps 1. Complete review and merge this PR. 2. Publish chart `0.1.9` and the final main-branch relay image. 3. Replace the provisional pins in [squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244), rerender, and require clean CI. 4. Run the staffed manual Job gate and record object count, peak memory, duration, snapshot revision, and relay stability. 5. Confirm the relay reads the new snapshot in `external` mode and the existing `block.buzz_relay.*` storage series update again. 6. Enable the daily 03:17 UTC schedule and add dashboard panels and monitors for snapshot age, snapshot-load health, Job failures, cap pressure, duration, and worker memory. If production must roll back to an older relay image, set `BUZZ_STORAGE_METRICS=off` first. Older images do not understand `external` and would resume the failing inline scan. ## Bigger picture This is the application half of the storage-metrics repair. The paired infrastructure PR owns the production resource limits, schedule, artifact pins, secret boundary, and monitored rollout. Keeping those changes separate lets this PR remain safe by default while still providing a complete path to current per-community S3 totals. Related issue: N/A. No matching issue or pull request exists in `block/buzz`. Generated with Codex --------- Signed-off-by: Ravneet Arora <rarora@squareup.com> Co-authored-by: Codex <noreply@openai.com> |
||
|
Tom Brow
|
6dfd145cde |
release(mobile-infra): buzz-push-gateway 0.3.0 (#7685)
Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
Luke Tornquist
|
213092074c |
Instrument database pool roles (#7356)
## What
Name the relay's four Postgres pool roles—writer, reader, audit, and
search—and export one fixed-cardinality utilization metric contract for
all of them. Existing writer and reader gauges remain unchanged.
## Why
Audit and search already use separate pools, but they are missing from
pool pressure telemetry. Operators cannot see all process-level
connection demand before setting a deployment-wide connection budget.
## How
A closed `DbPoolRole` vocabulary now labels
`buzz_db_pool_connections{pool_role,state}` and
`buzz_db_pool_configured{pool_role}`. The relay retains cheap pool
clones for statistics only; ownership, capacities, timeouts, query
routing, and failure behavior do not change. The docs also replace the
stale async search-index description with the generated-column behavior
used today.
This simplifies four ad hoc observability states into one bounded model
without adding a pool manager or changing service boundaries.
## Risk
Low to moderate. This changes relay metrics and pool construction
plumbing, but not SQL execution or routing. Legacy metric names remain
available for current dashboards.
## Testing
- Push gate at `3f5468a0463bdfc5879bcdd3878fa06b42ca84c2`: all 14 Rust
test lanes passed; desktop Tauri checks passed.
- Blox: `cargo test --workspace --all-targets` with repository-declared
Postgres services.
- Blox: `cargo clippy --workspace --all-targets -- -D warnings`.
- Blox: `cargo fmt --all -- --check`.
## Bigger picture
Aggregate deployment budget enforcement remains deferred. This PR
exposes the per-role facts that a later deployment-policy change can
consume.
Generated with Claude Code
---------
Signed-off-by: tornquist <tornquist@squareup.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Tom Brow
|
76f7e5ef0c |
feat(push): make gateway origin configurable (#7261)
Make the push gateway origin configurable across the relay, gateway, mobile client, and deployment chart. This is a fresh deployment. The gateway has only run on personal development infrastructure, and push has no existing enabled users. Legacy migration, configured-to-unconfigured mobile transitions, and rolling upgrades from pre-launch gateway binaries are not supported. Populated pre-launch databases are rejected with an informative error before migrations run; their data is untouched. Mobile builds may omit the gateway URL, leaving push unavailable. Once push is enabled, subsequent builds must retain that configuration. ### Validation - All applicable CI checks pass at `5f546cfaf403f0386f146d213adfb4392f4ec29c`, including Rust, mobile/Swift, desktop/integration, security, image builds, and Helm checks. An unrelated agent cancellation test passed on retry. - PostgreSQL CI passed 384 tests, including the production-entry-point regression for refusing populated pre-launch databases before migrations, fresh initialization, and subsequent initialized deployments. - Focused local validation passed all 10 gateway PostgreSQL tests under the CI discovery filter and all 3 unconfigured-mobile tests. Repository-wide PostgreSQL discovery validation and the complete pre-push hooks passed. - Physical-device APNs/App Attest behavior and deployment to a live cluster were not verified in this work. --------- Signed-off-by: Tom Brow <tomb@block.xyz> Co-authored-by: Codex <noreply@openai.com> |
||
|
Salman Mohammed
|
f3940ff21e |
fix(ci): pull pinned MinIO images from Quay (#7599)
## Summary CI is failing in `main` before tests start because Docker Hub denies pulls for `minio/minio` and `minio/mc`. Retrying the failed jobs produced the same errors. Use MinIO’s Quay images in the development, harness, deployment Compose, and Helm configurations. Pin each image to its multi-platform digest and retain the release tags already used by deployment templates. Update the architecture reference to match. ### Related issue No duplicate issue or PR found. Addresses the image-pull failures in [main CI attempt 2](https://github.com/block/buzz/actions/runs/34645306855/attempts/2). ### Testing - Pulled both pinned images for Linux ARM64 and AMD64. - Started an isolated Compose stack derived from the updated MinIO service definitions, once per architecture (AMD64 under local emulation). - Verified the existing health check passed and the bucket initializer exited with code 0. - Uploaded an object with `mc cp`, read it with `mc cat`, compared its content, and deleted it with `mc rm` on both architectures. - Confirmed the initialized bucket was private and removed the temporary stack and volume. Generated with Codex Signed-off-by: Salman Mohammed <smohammed@squareup.com> |
||
|
ravarora2
|
88687876f7 |
feat(relay): add early startup lifecycle logs (#7258)
## Why Early relay failures can currently appear as a container restart without a trustworthy in-process account of whether crypto, structured logging, configuration, relay identity, or the metrics listener failed. Most of those steps happen before the Prometheus exporter exists, so their chronology belongs in logs rather than metrics. Implements the logs-only early-startup slice of #7238. Post-bind Prometheus exporter supervision is tracked separately in #7284. ## What changed - create a process lifecycle recorder before the Tokio runtime and emit a fixed, versioned JSON schema directly to stderr; - record started and exactly one terminal event for `crypto_init`, `tracing_init`, `config_load`, `key_load`, `metrics_bind`, and the aggregate `process_telemetry` phase; - keep every status and reason bounded and suppress raw errors that could contain credentials, keys, URLs, or other secrets; - return typed metrics-install errors so `metrics_bind` can be classified without logging raw values, while preserving the existing public `metrics::install` API; - document the logs-only evidence contract and add real child-process regressions for success and failure paths. This PR adds **no startup metric families** and no dashboard contract. Existing application metrics remain unchanged. ## Verification Exact head: `8faf7526822a119efa035e58b2b3c59aa67fc81d` - `cargo fmt --all -- --check` - `cargo clippy -p buzz-relay --all-targets -- -D warnings` - relay binary target: 13 passed, 1 PostgreSQL-only test ignored - real relay child-process lifecycle target: 9 passed - full relay package library target: 1,023 passed, 89 ignored; the same six media tests failed at `crates/buzz-relay/src/api/media.rs:1145` with `Sqlx(PoolTimedOut)` because local PostgreSQL is unavailable - three independent exact-head reviews found no correctness, security, compatibility, lifecycle-accounting, logs-only-scope, or test-adequacy finding All exact-head GitHub CI gates are green, including lint, unit tests, PostgreSQL, relay/backend/desktop integration, both Linux server cross-compiles, Windows/macOS builds, and security checks. ## Staging verification - exact multi-architecture image: `dev-sha-8faf7526822a119efa035e58b2b3c59aa67fc81d-run-33708188952-1` - immutable manifest: `sha256:26cad28266a6bb0b0e7081eb6091d374e5489f8bb78c475a4a65737dee86cc67` - image workflow: https://github.com/block/buzz/actions/runs/33708188952 - focused staging deployment: https://github.com/squareup/builderbot-platform-core-infrastructure/pull/314 - replacement ReplicaSet `buzz-d68764bc7` has two Ready pods with zero restarts - Datadog received one complete, contiguous sequence 1-12 from each pod; both end with `process_telemetry/terminal/succeeded` at 3 ms - queries scoped to the replacement ReplicaSet return no data for the removed `buzz_startup_phase_terminal` or `buzz_startup_phase_duration_seconds` families The experimental Row 7 was removed from the Buzz Startup & Rollout Safety dashboard. This logs-only PR deliberately adds no replacement dashboard row. --- **Update Sep 3, 12:26 ET:** Clarified the review boundary: this PR does not close the broader #7238. Later exporter-task termination is pre-existing runtime behavior and is now explicitly tracked in #7284; no production code or staged image changed in this update. Generated with Codex Signed-off-by: Ravneet Arora <rarora@squareup.com> |
||
|
ravarora2
|
91ab9d31b8 |
Add operation-aware database pool acquisition metrics (#7195)
## Why
Buzz already reports coarse writer/reader database checkout waits, but
those
signals cannot explain which startup or serving operation is blocked by
pool
pressure. That makes rollout diagnosis and postmortems ambiguous: a
readiness
probe, NIP-42 authentication, authorization check, reconnect history
repair,
event write, and background maintenance can all wait on the same pool
while
appearing identical.
This PR implements Package 2A of the pod-handoff plan: operation-aware
pool
borrow causality. It is observability-only; it does not change
configured pool
sizes, SQL semantics, transaction ordering, or timeout policy. Physical
DNS/TCP/TLS/Postgres authentication and session initialization remain
the
separate Package 2B boundary.
## Metric contract
The final contract separates three questions:
| Question | Metric |
|---|---|
| How long did checkout wait? |
`buzz_db_pool_acquire_duration_seconds{pool_role,operation}` |
| How did the attempt end? |
`buzz_db_pool_acquire_attempts_total{pool_role,operation,outcome}` |
| Who is waiting now for a tracked operation? |
`buzz_db_pool_waiters{pool_role,operation}` |
Outcomes are `success`, `timeout`, `error`, and `cancelled`. Operations
are
`bootstrap`, `readiness`, `tenant_resolution`, `authentication`,
`authorization`, `subscription_history`, `event_write`, and
`maintenance`.
Only these eleven pool/operation pairs are constructible:
```text
writer/bootstrap reader/bootstrap
writer/readiness
writer/tenant_resolution
writer/authentication
writer/authorization reader/authorization
writer/subscription_history reader/subscription_history
writer/event_write
writer/maintenance
```
The duration histogram intentionally does not carry `outcome`. Result
data
remains available on the terminal counter for historical counts and
rates,
without multiplying the expensive histogram family. Nine finite buckets
plus
`+Inf`, sum, and count produce 12 duration series per valid pair.
Together with
four outcome counters and one waiter gauge, the new-family ceiling is
exactly
187 raw Prometheus series per pod, asserted from the production
exporter.
The existing coarse acquisition families remain temporarily for
dashboard
compatibility while the new series are validated in staging.
The operation-specific waiter family covers the explicitly routed,
deployment-critical operations above; it is not a census of every
possible
SQLx checkout in the process. The dashboard pairs it with SQLx pool
active/idle/max gauges for whole-pool capacity context, and treats a
missing
operation series as unknown rather than healthy zero.
## What changed
### Cancellation-safe acquisition ownership
- Add writer- and reader-specific typed operation APIs so invalid label
pairs
cannot be constructed and store modules cannot emit reader labels.
- Own every polled acquisition with one RAII terminal guard.
- Record exactly one duration and terminal outcome for success, timeout,
error,
or cancellation.
- Emit nothing for a future that is created but never polled.
- Balance the operation-specific waiter count exactly once on every
terminal or
dropped future.
- Periodically refresh every expected waiter pair, including healthy
zero, so
missing telemetry is not presented as zero. Reader pairs are emitted
only
when a distinct read pool is configured; a writer-only pod cannot
fabricate
healthy reader-zero state.
### Production attribution
Route the deployment-critical acquisition paths through caller-owned
semantic
entry points, including:
- writer and reader bootstrap;
- the real post-#7149 readiness acquisition and deletion-catalog
validation;
- tenant resolution and community lifecycle checks;
- NIP-42 allowlist authentication;
- membership, moderation, invite, operator, Git, agent-owner, and policy
authorization;
- operator community create/list/archive/unarchive, reverse host/channel
tenant
resolution, and REQ row-community conformance lookups;
- writer/reader subscription history, feed, thread, and routed fallback
paths;
- primary and command event writes, replaceable events, mention
indexing,
reaction/channel/member/archive side effects, and thread metadata;
- push matching, usage rollups/leadership, replica-fence startup and
recurring
probes, periodic reconciliation, channel/deletion reapers, partitions,
and
other bounded maintenance/bootstrap paths.
Shared helpers now accept caller-owned intent or expose named semantic
variants
instead of assigning one misleading operation to every caller. No known
P0 path
uses `other`.
### Readiness and size-one-pool correctness
- Rebase on the post-#7149 readiness implementation and instrument the
actual
`Db::readiness_check` acquisition rather than the superseded ping-only
seam.
- Acquire once for deletion-catalog validation and run its queries on
that
connection, preserving the shared readiness deadline.
- Scope the channel-roster catalog checkout before the behavior probe so
a
size-one writer pool cannot self-deadlock during startup verification.
### Exporter, documentation, and CI
- Register metric HELP/type/unit metadata through the production
Prometheus
builder.
- Configure dedicated checkout buckets at 1ms, 5ms, 10ms, 25ms, 50ms,
150ms,
500ms, 1s, and 3s.
- Add a production scrape-contract test for exact names, labels,
buckets,
valid pairs, sensitive-label exclusion, and the 187-series ceiling.
- Add source mutation guards for the P0 semantic entry points and
raw-checkout
bypasses.
- Add an exact backend-integration CI selector for the production
attribution,
cancellation, readiness, and size-one-pool PostgreSQL tests.
- Document the frozen label vocabulary, valid combinations, semantics,
and
cardinality budget in the Helm chart README.
## Dashboard intent
The new Stage 2 row in **Buzz Startup & Rollout Safety** is
deployment-first:
- baseline-versus-candidate attempts, failure rates, cancellation rates,
and
maximum wait by operation;
- outcome counts and percentages over time by SHA/ReplicaSet;
- acquisition wait heatmap, average, and maximum through the rollout;
- historical waiter pressure beside writer active/idle/max context;
- per-pod postmortem drilldown, including terminated pods;
- a smaller current-waiter table with explicit stale/missing semantics.
Percentile widgets remain disabled until Datadog metadata confirms
percentile
support for the new distribution. Current gauges use no fill,
interpolation, or
`default_zero`; missing means unknown.
## Risk assessment
Moderate. The patch touches many database acquisition call sites, but
preserves
the selected physical pool and executes the same SQL on the acquired
connection. The main risks are incorrect semantic attribution,
cancellation
double-counting, and a helper accidentally acquiring twice. Typed APIs,
production-method PostgreSQL tests, source guards, the raw scrape
contract, and
the size-one-pool regression cover those risks.
No tenant, community, user, pubkey, event, channel, SQL, URL, pod,
version,
ReplicaSet, or request-controlled value is emitted as an application
metric
label. Deployment identity is supplied by infrastructure enrichment.
## Verification
- `cargo fmt --all -- --check` — passed.
- `cargo clippy -p buzz-db -p buzz-relay --all-targets --all-features --
-D warnings`
— passed.
- `cargo test -p buzz-db` — 122 passed, 0 failed, 263 ignored;
source-contract integration test: 3 passed, 0 failed.
- Focused relay compatibility, metric-contract, and readiness tests —
passed.
- `scripts/test-postgres-test-discovery.sh` — passed.
- Full `buzz-relay` package run from the identical tree reached 1,015
passes;
the six media-test failures all stopped in their shared local PostgreSQL
setup with `Sqlx(PoolTimedOut)` because Docker/PostgreSQL was
unavailable.
The same six failed in isolation, while every changed exact test passed.
- Exact implementation head:
`f92910b353086e9edf85918ca5f72190edbbe22f`.
- Exact multi-architecture staging image:
`dev-sha-f92910b353086e9edf85918ca5f72190edbbe22f-run-33607968668-1`
(`sha256:161712c8ed2e265a15df9b63e02248d5973481f875ff129d7d2ae78a09d487a2`).
- Focused staging GitOps PR:
<https://github.com/squareup/builderbot-platform-core-infrastructure/pull/299>
— merged after renderer, inventory, infrastructure test, Kargo, Semgrep,
and
Intersect gates passed; the source/generated-artifact diff was exactly
two
image lines.
- Exact GitHub head reports 47 terminal checks: 35 successful and 12
intentionally skipped. PostgreSQL, unit, lint, security, both server
cross-compiles, backend integration, relay E2E, desktop, mobile, image,
Helm, Semgrep, zizmor, and DCO gates are green.
- Datadog readback identifies two exact-image pods,
`buzz-d79c8d8f7-ckv2l` and `buzz-d79c8d8f7-qzqdp`, in ReplicaSet
`buzz-d79c8d8f7`; both report the full source SHA above.
- Both pods report all eleven allowed pool/operation waiter pairs at
current
zero, with no invalid pair. The acceptance window observed nonzero
success
receipts for readiness, tenant resolution, authorization, subscription
history, event write, and maintenance, and no timeout, error, or
cancelled
outcome. Maximum observed wait was about 101 ms for maintenance and 50
ms
for reader subscription history.
The main **Buzz Startup & Rollout Safety** dashboard now has a live
Stage 2
database row with eight widgets and nineteen fully scoped queries. Final
readback preserved all seven top-level groups, found zero under-scoped
Row 6
queries, and confirmed the tracked-operation waiter boundary in the
panel
descriptions.
Generated with Codex.
---------
Signed-off-by: Ravneet Arora <rarora@squareup.com>
|
||
|
ravarora2
|
beb76406c1 |
feat(relay): add detailed readiness metrics (#7149)
## Summary
This PR makes relay readiness failures diagnosable without weakening the
existing fail-closed readiness contract. It distinguishes Postgres pool
acquisition from query execution, Redis pool acquisition,
deletion-catalog validation, and the overall two-second deadline;
exports a bounded Prometheus contract for rollout dashboards; and fixes
the concurrency, shutdown, and listener-boundary semantics needed for
those signals to be trustworthy.
## Why
The previous `/_readiness` implementation exposed only an aggregate
ready/not-ready result. During a rollout, operators could not tell
whether a pod was blocked on:
- acquiring a Postgres connection;
- executing the Postgres readiness query;
- acquiring a Redis connection;
- validating the deletion catalog; or
- the shared readiness deadline.
Adding metrics to the existing handler also exposed three correctness
hazards that this PR resolves:
1. the same handler is mounted on both the public application listener
and the private Kubernetes health listener, so public requests could
otherwise distort rollout telemetry;
2. concurrent probes can finish out of order, allowing an older result
to overwrite newer current-state gauges; and
3. a probe started before SIGTERM can finish afterward, incorrectly
return `200 ready`, and resurrect ready gauges while the process is
draining.
## Behavior
### Readiness evaluation
- Postgres, Redis, and deletion-catalog checks still run under one
shared two-second deadline.
- Postgres distinguishes pool acquisition timeout/error from query
timeout/error.
- Redis distinguishes pool acquisition timeout/error. This does not
claim a Redis command round trip.
- The deletion catalog distinguishes operation timeout/error.
- Multiple failures are reported as `multiple_dependencies_failed`;
exhaustion of the shared deadline is reported as `overall_timeout` when
no more specific completed outcome wins.
- Readiness remains fail-closed: every dependency must succeed for `200
{"status":"ready"}`.
### Ordered publication and shutdown
`ReadinessCoordinator` is process-owned and uses one mutex as the
linearization point for probe generations, current-state publication,
and terminal shutdown.
- Every completed dependency attempt may contribute its truthful counter
and duration observation.
- Only the newest admissible probe generation may publish current-state
gauges.
- An older, slower probe cannot overwrite a newer probe's gauges.
- `begin_shutdown()` and probe commit serialize through the same
coordinator.
- Once shutdown commits, an in-flight probe cannot return ready or
publish ready/current dependency gauges, even if its dependency work
later succeeds.
Shutdown without dependency evaluation records only:
- `buzz_readiness_checks_total{reason="shutting_down"}`; and
- `buzz_readiness_state{check="overall"} = 0`.
It does **not** fabricate dependency failures, dependency state changes,
or zero-duration latency samples. If shutdown wins after an in-flight
evaluation actually ran, those completed dependency attempts may remain
as attempt telemetry, but they cannot overwrite shutdown-dominant
current state.
### Listener and response contract
- The private health listener's `/_readiness` route is the sole
authority for rollout readiness telemetry.
- The public application listener retains `/_readiness` for
compatibility, evaluates the same dependencies, and preserves the
existing response shape, but it emits no `buzz_readiness_*` metrics.
- Ready responses remain `200 {"status":"ready"}`.
- Shutdown responses remain `503 {"status":"shutting_down"}`.
- Failed private-health responses include the bounded `reason` plus
`postgres`, `redis`, and `deletion_catalog` booleans.
- Failed public compatibility responses retain the dependency booleans
but omit the new detailed reason.
- No header, query parameter, path value, or other request-controlled
value becomes a metric label.
## Prometheus contract
The final schema is intentionally capped at **99 raw Prometheus series
per pod**.
| Metric | Type | Labels | Raw series/pod |
|---|---|---|---:|
| `buzz_readiness_checks_total` | counter | `reason` | 12 |
| `buzz_readiness_dependency_checks_total` | counter | `dependency`,
typed `outcome` | 11 |
| `buzz_readiness_check_duration_seconds` | histogram | `check` | 72 |
| `buzz_readiness_state` | gauge | `check` | 4 |
| **Total** | | | **99** |
### Closed label sets
`reason`:
```text
ready
shutting_down
postgres_pool_timeout
postgres_pool_error
postgres_query_timeout
postgres_query_error
redis_pool_timeout
redis_pool_error
deletion_catalog_timeout
deletion_catalog_error
overall_timeout
multiple_dependencies_failed
```
Valid `dependency` / `outcome` pairs are enforced by typed enums:
- `postgres`: `success | pool_timeout | pool_error | operation_timeout |
operation_error`
- `redis`: `success | pool_timeout | pool_error`
- `deletion_catalog`: `success | operation_timeout | operation_error`
`check` is `overall | postgres | redis | deletion_catalog`.
The readiness histogram has 15 configured buckets concentrated around
the two-second deadline:
```text
0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5,
0.75, 1.0, 1.25, 1.5, 1.75, 2.0, 2.5, +Inf
```
The contract deliberately removes the redundant overall `result` label
and the histogram `outcome` label. Pod, ReplicaSet, version, rollout,
raw error, SQL, URL, tenant, user, community, pubkey, and
request-controlled values are prohibited application labels;
infrastructure enrichment can supply deployment identity outside the
application metric.
## Postgres production seam and CI
The real `Db::readiness_check` path now supports deterministic
production-seam testing while preserving the same acquisition/query
implementation used by the relay. The isolated PostgreSQL lane
automatically discovers and executes three ignored integration tests
covering:
- a held sole connection causing pool timeout, followed by recovery
after release;
- closed-pool acquisition error;
- acquisition success followed by query timeout;
- classified query error;
- cancellation while waiting for a connection;
- cancellation during an in-flight query; and
- eventual pool recovery with waiter/in-flight state balanced.
This prevents the central SQLx/Postgres behavior from being merely
compiled but never executed in CI.
## Tests and verification
- `cargo fmt --all -- --check`
- `./scripts/test-postgres-test-discovery.sh`
- complete `cargo test -p buzz-db`
- focused `cargo test -p buzz-relay readiness`
- real-router `cargo test -p buzz-relay real_health_route_`
- controlled PostgreSQL execution of all three ignored readiness tests
- production health router -> real `GET /_readiness` -> Prometheus
render assertions
- public-router requests proving zero rollout telemetry
- deterministic out-of-order A/B probe tests
- SIGTERM-during-probe tests proving shutdown dominance
- exported metric name/type/exact-label/bucket assertions
- exported raw-series allowlist and 99-series ceiling assertion
The route-to-scrape regression test uses the production Prometheus
builder and verifies the `2`, `2.5`, and `+Inf` readiness buckets. It
fails if the health recording call, health route, public/health
boundary, generation fence, shutdown fence, or bucket override is
removed.
## Risk assessment
**Medium.** This changes the live readiness publication path and adds
synchronization around probe commit/shutdown. The risk is bounded by:
- preserving the existing dependencies and shared two-second deadline;
- keeping readiness fail-closed;
- using a short, process-local mutex only at begin/commit linearization
points, not across dependency awaits;
- retaining the public compatibility endpoint while isolating its
telemetry;
- enforcing typed, low-cardinality labels and an exported series
ceiling; and
- covering the production router, Prometheus exposition, real PostgreSQL
seam, concurrency ordering, and shutdown races.
## Operational notes
- Dashboards should treat `buzz_readiness_state` as the latest sampled
current state, not as an event stream.
- `shutting_down` should be excluded from dependency-failure alerts
because no dependency failure is implied.
- Do not use `default_zero()` for missing current-state data;
missing/stale is unknown, not healthy.
- Use histogram buckets, heatmaps, max, or average until the Datadog
distribution metadata confirms percentiles are enabled; do not title a
widget p95 before that live readback.
## Non-goals
This PR does not add the broader process-startup lifecycle,
worker/listener supervision, shutdown coordinator, WebSocket/huddle
handoff, client recovery telemetry, dashboard mutations, Datadog
configuration changes, or deployment changes. Those remain separate
follow-up work.
## References
- [Staging dev relay image
runbook](https://github.com/block/buzz/blob/main/docs/staging-dev-relay-images.md)
- Readiness telemetry contract: `deploy/charts/buzz/README.md`
---------
Signed-off-by: Ravneet Arora <rarora@squareup.com>
|
||
|
Tom Brow
|
42b42447b0 |
feat(mobile): prepare buzz-push-gateway for deployment (#7158)
A few things to prepare for deploying the Buzz push gateway as implemented in #6269: - Cut `buzz-push-gateway` chart `0.2.0` from the current config (certificate auth, dogfood client) - Datadog-compatible metrics - Align the desired schema, metrics docs, and production ingress guidance with the current dogfood deployment contract ## Links - [BUZZ-17: Publish the MVP-compatible gateway chart](https://linear.app/squareup/issue/BUZZ-17/publish-the-mvp-compatible-gateway-chart) - [BUZZ-26: Add Datadog-compatible push-gateway metrics and correct source drift](https://linear.app/squareup/issue/BUZZ-26/add-datadog-compatible-push-gateway-metrics-and-correct-source-drift) --------- Signed-off-by: Tom Brow <tomb@block.xyz> Co-authored-by: Codex <noreply@openai.com> |
||
|
Tom Brow
|
c432a111ca |
feat(mobile): push notifications MVP (#6269)
This PR implements MVP, iOS-only,
[NIP-PL](
|
||
|
Will Pfleger
|
86b9142a09 |
refactor(relay): NIP-98 admin auth with Operator/Moderator roles and NIP-11 discovery (#3777)
Adds authenticated, role-based moderation to the relay admin API. On
`main` the admin API is read-only and gated only by `Host`/`Origin`
matching; this branch adds NIP-98 authentication, a two-tier
Operator/Moderator principal model, mutation and staffing endpoints, and
NIP-11 auto-discovery so clients never type the admin URL by hand.
## Authentication (`BUZZ_ADMIN_AUTH`)
`BUZZ_ADMIN_AUTH` accepts `nip98` or `disabled`. Leaving it unset
defaults to `nip98` (fail-secure). Configuration fails closed: any other
value aborts startup, while a lingering `BUZZ_ADMIN_TOKEN` is ignored
with a startup warning — token (bearer) authentication is not supported.
`Host`/`Origin` matching is retained in every mode as defense-in-depth.
- **`nip98`** (default) — per-request signed NIP-98 (kind 27235) events,
resolved to an Operator or Moderator principal with per-person
attribution and individual revocability. Read-write per resolved
principal.
- **`disabled`** — no credential; relies entirely on network-layer
controls (reverse proxy, VPN, firewall) and logs a `WARN` on every boot.
Always read-only: `authorize()` resolves no principal, so mutation and
staffing routes always `403`.
## Roles
Buzz has two independent authority axes after this change.
**Relay-level** roles (new here) are deployment-global: they act across
every community on the relay, through the admin API. **Community-level**
roles (pre-existing, unchanged by this PR) are tenant-scoped: they act
inside one community, through signed Nostr moderation commands.
### Relay level (new)
| Role | Description |
|---|---|
| **Operator** | Full control of the deployment's moderation surface:
read all reports, feedback, and attachments across every community;
resolve reports with enforcement (`delete`/`kick`/`ban`/`timeout`) or
decisions (`dismiss`/`escalate`); reopen and cancel; update feedback
status; and manage the Operator/Moderator roster via the staffing
endpoints. |
| **Moderator** | Day-to-day triage: everything an Operator can do
except staffing — cannot view or change the roster. |
How a pubkey acquires a relay role (resolution order; config always
outranks DB):
1. Listed in `RELAY_OPERATOR_PUBKEYS` → **Operator** (source `config`)
2. Equals `RELAY_OWNER_PUBKEY` while `RELAY_OPERATOR_PUBKEYS` is empty →
**Operator** (source `owner_fallback`, a break-glass grant for
self-hosters that deactivates once any operator is configured)
3. Row in the `relay_operators` table → **Operator** or **Moderator**
(source `db`, managed via the staffing endpoints)
4. No match → `403`
### Community level (pre-existing, unchanged)
| Role | Description |
|---|---|
| **Owner** (community) | Full authority within their community: every
moderation action (delete, kick, ban/unban, timeout/untimeout, resolve
reports, view queue) plus member, role, and invite management. No guard
rails. |
| **Admin** (community) | Same community-wide moderation capabilities as
owner, except an admin cannot ban or time out the owner or a fellow
admin — only the owner may action an admin. Manages members and invites;
only the owner grants the admin role. |
| **Member** (community) | Standard participant; no moderation
capability. |
| **Owner / Admin** (channel) | Channel-local authority only: delete
messages and kick users within their own channel. |
| **Member / Guest / Bot** (channel) | No moderation authority. |
There is no community-level Moderator tier in v1; relay-level Moderator
is the only role by that name.
## Escalation scoping
The operator report queue is an escalation backstop, not the community's
day-to-day triage surface (per `VISION_MODERATION`, the severe class is
the platform's to review rather than the community's). Two rules enforce
that:
- **Escalated-by-default listing.** `GET /reports` with no `status`
parameter returns only `escalated` reports. An explicit
`status=<open|resolved|dismissed|escalated>` filter is always honored as
given, and full visibility across every status stays available for
platform-safety and legal review via `scope=all` (which lists reports
regardless of status). `scope` accepts only `all` and is ignored when an
explicit `status` is present.
- **Auto-escalated `illegal` reports.** Member reports whose category is
`illegal` are ingested with `status=escalated` rather than `open`, so
the severe class reaches the operator backstop without waiting for a
community admin to forward it. Every other category still lands `open`.
Auto-escalation only sets the queue status — it records no moderator
decision and stamps no resolver, so an auto-escalated report is
indistinguishable downstream from an admin-escalated one: the reopen
route returns it to `open` on the same terms, keyed only on status,
never on how the report became escalated.
## Principal resolution and NIP-98 admission
`resolve_admin_principal()` returns `AdminPrincipal { pubkey, role,
source }` per the resolution order above; `None` never falls through as
a role. Admission is ordered so the replay guard is a privilege, not a
public surface: signature/URL/method/payload-hash verification first,
roster check second, and only then is the deployment-scoped replay id
atomically consumed — a validly-signing but unrostered key never
allocates a replay slot. Redis failure fails closed.
## Report resolution, recovery, and enforcement provenance
`POST /reports/{id}/resolve` is a crash-safe enforcement state machine:
decision-only outcomes (`dismiss`/`escalate`) are a single
CAS-plus-audit transaction; enforcement
(`delete`/`kick`/`ban`/`timeout`) claims the report
(`open`→`processing`), runs the durable mutation, then finalizes — a
re-drive resumes at the step marker and converges to exactly-one
enforcement, fenced by a lease and an outbox claim token.
Person-directed enforcement on an `event`-kind report derives its target
from the stored event's author (server-owned truth, never the reporter's
`p` tag) via a single `derive_enforcement_target` shared by the HTTP
driver and the recovery worker. If the reported event was purged before
its author could be read, person-directed actions are rejected pre-claim
and the report stays `open`; `delete` needs only the event id and is
exempt.
`GET /reports/{id}` and the resolve response carry an `activeAction`
field surfacing the enforcement that actually executed — a report
dismissed after a reopen still reports the ban that ran. `POST
/reports/{id}/reopen` returns a terminal report to `open` (idempotent on
`requestId`). `POST /reports/{id}/cancel` is the sole recovery path for
a pre-mutation `failed` action, attributed via
`relay_admin_actions.cancelled_by`.
## Feedback
`GET /feedback` and `/feedback/{id}` survive a tenant purge: provenance
columns are severed to `NULL` rather than cascade-deleted, and the
attachment path fails closed to `404` on a severed row. `PATCH
/feedback/{id}` updates lifecycle `status`
(`new`/`reviewed`/`archived`).
## Staffing and probe
`GET/PUT/DELETE /operators/{pubkey}` are Operator-only; mutating a
config-backed pubkey returns `409 Conflict`. `GET /operators` returns
the union of config and DB principals with per-entry `source`. `GET
/probe` reports auth mode, role, source, `canAct`, and `canStaff` for
the desktop console.
## NIP-11 auto-discovery
The NIP-11 relay-information document gains an optional `admin_api`
field carrying the canonical admin origin (`scheme://host[:port]`, no
path), present iff `BUZZ_ADMIN_HOST` is set and omitted otherwise. The
scheme follows the same loopback rule as NIP-98 `u`-tag verification via
a shared `scheme_for_host` helper, so the advertised origin and the
origin the relay verifies against can never diverge.
## Operator API origin decoupling
`RELAY_OPERATOR_API_ORIGIN` is no longer required at boot when
`RELAY_OPERATOR_PUBKEYS` is set — it is used only by the
community-provisioning endpoints, which fail closed at request time
(with a boot-time `WARN`) until it is set. The admin console needs no
origin.
## Admin-web adaptation
The standalone `admin-web` dashboard signs each request as a NIP-98
event via a NIP-07 browser extension, discovers the auth mode with a
single unauthenticated probe (`200` → `disabled`, anything else →
`nip98`, fail-secure), and carries no token entry surface. Playwright
coverage exercises the NIP-98 and CSP paths.
## Security hardening
Three findings from security review are folded in:
- **Append-only roster audit.** `PUT`/`DELETE /operators/{pubkey}`
mutate the deployment-wide root of trust, but the upsert overwrites
`role`/`added_by` in place and the delete removes the only row — so a
grant→revoke sequence left no trace of who was ever granted or by whom.
Each mutation now writes an `relay_operator_audit` row (actor, target,
`grant`/`revoke`, pre-image `prev_role`, `new_role`, timestamp) inside
the same transaction as the mutation. A per-target transaction-scoped
advisory lock serializes concurrent mutations of the same pubkey before
the pre-image read, so the recorded `prev_role` is always the true
predecessor even under a concurrent-grant race. Chronology is keyed on a
`BIGINT GENERATED ALWAYS AS IDENTITY` `seq` column, not the wall clock:
the serializing lock guarantees insertion order and `seq` captures it,
so ordered reads (`ORDER BY seq`) follow the true privilege chain even
across a backward NTP step that a `clock_timestamp()` ordering would
invert. `created_at` (`clock_timestamp()`) is retained as informational
occurrence time only. Append-only by construction — no `UPDATE`/`DELETE`
path and no API surface. A no-op delete writes nothing.
- **`expirationSecs` overflow.** The timeout path built `Utc::now() +
Duration::seconds(secs as i64)` from an attacker-controlled `u64`:
`i64::MAX` panicked the handler, and a wrapped-negative magnitude minted
a *past* expiry that still passed validation. `compute_timeout_until`
now rejects zero, rejects magnitudes above a documented
`MAX_TIMEOUT_SECS` (365 days), and uses checked
`try_seconds`/`checked_add_signed` so no input can panic or produce a
past expiry — over-cap, zero, `i64::MAX`, and wrapping-negative inputs
all return a clean `4xx`.
- **Uppercase-hex config-backed bypass.** Config pubkeys are lowercased
at parse, but the `409` immutability check raw-string-compared the path
param while `decode_hex_pubkey` accepted uppercase — so `PUT
/operators/{UPPERCASE}` skipped the guard and wrote a shadow row for the
same 32 bytes. The validated param is now canonicalized (lowercased)
before the `409` check, DB write, `DELETE`, and response body.
## Migrations
- `0035_relay_operators.sql` — `relay_operators` roster table
(deployment-global), `actor_authority` on `moderation_actions`,
`processing` status plus `active_action_id` on `moderation_reports`,
`status` on `product_feedback`.
- `0036_relay_admin_actions.sql` — enforcement-action table with a
`request_id` idempotency key, a `step_marker` for crash recovery, and a
`cancelled_by` attribution column.
- `0037_relay_admin_action_lease.sql` — lease fencing for the action
worker.
- `0038_relay_admin_outbox_claim_token.sql` — fenced claim token on the
outbox worker.
- `0039_relay_operator_audit.sql` — append-only `relay_operator_audit`
trail for roster mutations (see Security hardening).
`docs/admin/README.md` documents the full principal model, NIP-98 event
requirements, capabilities by role, the startup error matrix, and the
discovery field.
## Production blast radius
A relay without `BUZZ_ADMIN_HOST` is completely unaffected — the admin
surface stays disabled and `BUZZ_ADMIN_AUTH` is ignored; a lingering
`BUZZ_ADMIN_TOKEN` logs a startup warning and must be removed. Where
`BUZZ_ADMIN_HOST` **is** set, unset `BUZZ_ADMIN_AUTH` defaults to
`nip98` (per-person signed auth); `BUZZ_ADMIN_AUTH=disabled` reproduces
`main`'s prior `Host`/`Origin`-only gating but is read-only (mutation
routes `403`). The five migrations add tables and columns without
touching existing data.
---
Related: [block/buzz#4768](https://github.com/block/buzz/pull/4768)
(desktop admin console consuming the `admin_api` field),
[squareup/bb-public#339](https://github.com/squareup/bb-public/pull/339)
(Phase 4 rollout config)
---------
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
|
||
|
klopez4212
|
8471049c43 |
feat(desktop): add KLIPY GIF search to composers (#5554)
## Summary - Supersedes #1913 with a KLIPY-hosted URL implementation. - Adds KLIPY GIF search and trending results to desktop message and forum composers. - Keeps selected GIFs hosted by KLIPY; Buzz stores only the external URL and media metadata (no imeta tag, since relays only accept hash-backed local `/media/` entries). - Aligns the Emoji/GIF picker with Buzz's standard segmented control, theme surfaces, and motion behavior. ## Relay-to-provider boundary - The relay proxies KLIPY search/share so the `BUZZ_KLIPY_API_KEY` never reaches the desktop; the key stays server-side behind a redacted `Debug` impl. - The dedicated GIF `reqwest` client sets `redirect::Policy::none()`. Because the API key rides in the request path, following a provider `3xx` could replay a key-bearing URL to an attacker-chosen host (an SSRF/key-disclosure primitive). With redirects disabled, a `3xx` returns as a non-success status that the handlers map to a generic `502`; the `Location` target is never read or forwarded. - Admission reuses the established NIP-98, membership, replay, and per-pubkey rate-limit gates, with an upstream response-size cap and allowlisting so KLIPY error bodies never cross the relay boundary. ## Accessibility - Under `prefers-reduced-motion: reduce`, the picker grid renders a static provider poster (a normalized `jpg` asset) instead of the animated preview, or a named static placeholder when no poster is available. It reacts to preference changes while mounted. `no-preference` keeps the animated preview. - Selected GIFs carry their title through `ImetaMedia`'s `displayLabel`, so the composer thumbnail, preview dialog, editor, lightbox, and remove control all derive one non-empty accessible name instead of an empty `Attachment ` label. Ordinary hashed uploads keep their existing hash-derived names. --------- Signed-off-by: kenny lopez <klopez4212@gmail.com> Signed-off-by: Kenny Lopez <klopez4212@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Signed-off-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz> Co-authored-by: Princess Donut <b238ea756dee4d98afa5883fc7f1de61eeabe65bf700e3a5a5a80db5e42e2c2b@buzz.block.builderlab.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz> |
||
|
Luke Tornquist
|
f249710331 |
Qualify canonical relay images for staged delivery (#6781)
## Why Phase 1 of progressive delivery needs an explicit eligibility boundary before deployment automation can safely consume relay artifacts. Today the canonical Docker workflow can publish independently of the same-SHA CI result, the relay does not expose its source/build identity, and the Helm chart only renders mutable tags. The existing `ghcr.io/block/buzz-staging-dev` workflow in #6709 remains a manual pre-merge preview lane. It is deliberately not a canonical promotion input. ## What changed - Compile the full source SHA and GitHub Actions build identity into canonical relay binaries and expose it at `/_status`. - Gate creation of tagged multi-architecture manifests on the latest exact-SHA `CI` push run succeeding on `main` or `release`. - Preserve the signed SLSA provenance and add a signed, release-only deployment-eligibility predicate with source, build run, qualifying CI run, and exact compatible Buzz chart version. - Keep failed or pending builds untagged and without an eligibility predicate; preview-package artifacts are excluded by package and signer contract. - Add optional `image.digest` support to Buzz chart `0.1.8`, retaining the existing tag fallback. - Document verification and the intentional boundary between image/chart compatibility and backwards-compatible database migrations. ## Risk and rollout Canonical tag publication now waits for same-SHA CI and fails closed when the newest matching run fails. Per-architecture blobs may remain untagged after a failed qualification; they are not promotion inputs. Debug manifests are also CI-gated but do not receive the release eligibility predicate. There are no migration or schema-orchestration changes. The companion BuilderBot infrastructure PR will pin a separately verified canonical main digest and expose a read-only running-deployment inspection command. Signed eligibility is available only for artifacts published after this workflow lands. ## Verification - Blox: workflow contract fixtures, including stale-green rejection and predicate structure - Blox: `actionlint` for `ci.yml` and `docker.yml` - Blox: 46 Helm unit tests, chart lint, and production/quickstart render fixtures - Blox: relay status unit test, workspace Rust formatting, and workspace all-target clippy - Live evidence: exact-SHA CI selector chose successful run `32857345153` for source `9d1e4b257657f382d3111ce748f3da8d063b7671` - Existing canonical digest provenance verified with `gh attestation verify` ## References - #6709 — developer-preview precursor; intentionally separate from canonical eligibility - `docs/deployment-identity.md` *Generated with Codex* --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> |
||
|
Brad Seiler
|
d12dea4e67 | Support community deletion in versioned media buckets (#6738) | ||
|
Wes
|
24ec6a468e |
Repair stale large channel roster snapshots (#6251)
## Summary - detect relay-authored NIP-29 kind 39002 roster snapshots truncated by the former 1,000-member query cap and repair stale large rosters during relay startup - serialize canonical roster capture and replacement with membership writes, preserving tenant, channel, signer, pubkey, and role boundaries through mixed-version deployments - install migration 0032's fail-closed roster fence on the partitioned events table and verify its catalog shape plus behavior before opening relay listeners ## Rollout Migration 0032 is a hard schema-before-code compatibility boundary. Apply migrations before rolling this relay version. Startup refuses to open listeners when the parent/partition triggers are missing, disabled, mis-shaped, or behaviorally inert. For large installations, prefer `buzz-admin migrate` and monitor lock acquisition as documented in the chart README. ## Validation Exact head: `bcbba271f54bc0046a6683007e5a2b70403a11d5` - rebased onto `569308c23c9c2bf620dd3a9a5e4baecbcfa22e16`; the nine-file feature patch is byte-identical to pre-rebase head `be8ea0084f4d4c78c7c2550baad4399e4df8ce73` - pre-push hook passed at exact head: branch-skew, file-size, full Rust unit suite, Desktop Tauri clippy, and Desktop Tauri tests - `cargo fmt --all -- --check` - `cargo test -p buzz-relay group_members_snapshot_keeps_members_past_one_thousand -- --nocapture` - focused CI-mode Playwright regression: `selected relay agents revoked after the invite prompt cause no side effects` passed at exact head - prior exact-patch validation: `large_roster_reconciliation_candidates_respect_snapshot_count_and_signer`, mixed-writer locking/rollback, migration admission, partition trigger coverage, and desired-schema parity regressions ## Review Independent DB/relay review found no blocking issues in the exact feature patch. The concurrency fence holds the established replacement and membership locks on one transaction/connection through replacement; failures roll back both soft-delete and insert. Reconciliation remains tenant/channel/signer scoped and validates exact normalized pubkey-plus-role membership. The prior red Desktop shard was unrelated to this backend-only diff: its mocked mention test exercises no relay, database, or migration path. It reproduced as a timing flake on the old head, passed on retry/base, and now passes locally after rebasing onto current main. --------- Signed-off-by: Wes <wesbillman@users.noreply.github.com> Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz> |
||
|
Kalvin C
|
8a2c9af2db |
feat(deletion): add durable whole-community deletion (#4425)
## Summary Adds a durable, operator-controlled V1 for deleting an entire Buzz community without deleting another tenant's data. The workflow is exposed through `buzz-admin deletions`: - `sweep` records independent fleet storage-taxonomy observations - `submit`, `list`, `inspect`, and `approve` manage a deletion request - `unblock` resumes a fail-closed request after an operator records remediation identity and reason - `run` and `drain` execute bounded work Requests advance through a PostgreSQL-backed state machine and stop at `retention_pending` after logical deletion has been independently verified across PostgreSQL, object storage, and Redis. This PR ships the engine and CLI, not a continuously running worker or Kubernetes packaging. For V1, a cluster/VM administrator invokes `/usr/local/bin/buzz-admin` from the existing relay image, for example with `kubectl exec` or an equivalent container/VM exec path. ## What whole-community V1 removes For the target community, V1 removes: - rows from the allowlisted community-scoped PostgreSQL catalog, including members, profiles, authored events and bodies, DMs, reactions, mentions, memberships, tokens, workflows, moderation, audit, feedback, and rate-limit state - media sidecars and upload-attribution records under `_meta/<community>/` and `_uploads/<community>/` - Git repository pointers under `repos/<community>/` - Redis keys under `buzz:<community>:*` The community row survives as a permanent tombstone, and deletion control-plane records remain as evidence of the request, approval, execution, and result. ## Safety model Deletion is not a broad `DELETE CASCADE` followed by optimistic cleanup. The destructive boundaries are durable and fail closed. ### 1. Inventory and approval - `submit` resolves the target and freezes the schema plus summary-only storage inventory. - Approval is bound to the exact request, community, and frozen inventory digest. - Unsupported manifest versions, malformed keys inside the target's owned prefixes, live scoped-table/write-fence coverage drift, frozen-inventory mismatch, and approval mismatch block execution rather than guessing. Migration and catalog revision numbers are not authorization gates; the executor validates the live safety shape instead. - Storage inventory is server-side prefix scoped to exactly: - `_meta/<community>/` - `_uploads/<community>/` - `repos/<community>/` - The deletion path never lists the whole shared bucket and has no arbitrary per-community object cap. Its listing work is proportional to the target community's bindings, not total fleet storage. - Fleet-wide taxonomy sweeps remain independent observability. They report unknown writer shapes but do not gate deletion submission, fencing, or destructive progress. Maintainers must add deletion taxonomy coverage whenever a new community-owned object-key class is introduced; writer-coverage tests bind the current media and Git writers to that contract. ### 2. Quiesce, fence, and destructive freeze - Writes continue through submission, inventory, and approval. They stop when execution moves the target into `quiescing` and then establishes the durable fence. - Already-admitted external effects finish under heartbeated serving-write leases; the exact admitted lease may renew while the community is quiescing, but new lease acquisition is rejected. The executor drains admitted leases before destructive work. - Invite minting after quiescing begins fails as typed `AccessDenied` (HTTP 503 at the relay boundary) before an invite can be persisted. - Database triggers enforce the community write fence across the complete catalog of community-scoped tables. Startup/readiness and destructive execution validate that catalog so a newly added but unfenced table cannot silently escape. - **Named isolation assumption — fresh write snapshot.** Every writer transaction that can reach a community-fenced relation must use PostgreSQL `READ COMMITTED`; each guarded write therefore observes a statement snapshot no older than acquisition of the community deletion lock. `REPEATABLE READ` and `SERIALIZABLE` can retain a pre-fence snapshot and are unsupported for writers. The writer pool refuses non-`READ COMMITTED` sessions at connection setup, and both SQL fence functions reject an explicit per-transaction isolation override with SQLSTATE `25000`. Configuration-delivered bad isolation can surface through SQLx as a pool-acquire timeout because every `after_connect` attempt is rejected; the precise `community writes require READ COMMITTED isolation` reason remains observable when the SQL guard is reached. Read-only replica transactions are outside this assumption. - Holding the shared advisory lock until the guarded write executes is a separate liveness condition: under `READ COMMITTED`, releasing it early does not permit resurrection because the trigger rechecks the fence, but it can turn a fleet sweep into a statement-wide SQLSTATE `55000` abort. - After the fence closes writers, storage is re-enumerated into chunked side-table rows. Per-prefix counts and digests bind those concrete keys to the destructive manifest. - Manifest chunk insertion, update, and deletion are protected after freeze. This closes the race where an unbound key could otherwise appear after the manifest was committed. ### 3. Checkpointed destruction - Target-owned object bindings are deleted from the frozen destructive manifest in bounded batches with durable progress. - The concrete key list lives in chunked side-table rows rather than one request-row JSON value. It supports large communities, resumable execution, and terminal cleanup. - Missing objects are accepted as idempotent crash-window outcomes; malformed ownership, changed evidence, and unexplained target-prefix drift fail closed. - PostgreSQL purging remains scoped by `community_id`, including the guarded NIP-RS hard-delete path discovered with real Desktop kind `30078` read-state data. - Redis cleanup explicitly scans and `UNLINK`s only `buzz:<community_id>:*`. Natural expiry is insufficient because some keys, including tunnel generation counters used as fencing state, are deliberately persistent. ### 4. Independent verification - PostgreSQL logical absence is checked after purge. - The three target-owned storage prefixes are freshly inventoried again and must be empty. - Redis requires two complete empty namespace scans. - Only after all three stores pass does the request advance through `logically_verified` to `retention_pending`. ## What V1 deliberately does not erase ### Shared content-addressed storage Per-community deletion removes bindings, metadata, attribution records, and Git pointers. It does **not** physically delete fleet-shared CAS bytes that another community may still reference: - media blobs and thumbnails - Git manifests, packs, and indexes (`manifests/`, `packs/`, and `idx/`) Safe reclamation requires a separate fleet-wide reachability and retention GC. Unknown keys elsewhere in the shared bucket do not block one community's deletion; malformed or unrecognized keys inside that community's three owned prefixes still fail closed. ### External retained copies The online logical-deletion proof does not erase object versions/replicas, database backups/WAL, CDN copies, provider retention copies, or observability exports. Those require their own retention and purge controls. ### Member-only erasure This PR erases a whole community. It does not implement the different operation "erase one npub while preserving the community." Removing membership or accepting NIP-09 is not member erasure. A member-only workflow would need to find and selectively remove or redact authored event content and pubkeys, profile data, DMs, reactions, mentions, memberships/roles, tokens, workflows/subscriptions, upload attribution, moderation/audit history, repository attribution, and identity embedded in tags or JSON. It would also need explicit rules for ownership transfer, surviving replies and thread metadata, audit-chain integrity, immutable Git history, and shared-CAS reachability. That requires a pubkey-level fence and selective graph rewrite; it is a separate deletion product, not a safe extension of this whole-tenant worker. ## In scope - migration `0029_community_deletion.sql`: requests, approvals, leases, manifest chunks, checkpoints, tombstones, and the universal write-fence catalog - durable executor leases, generations, heartbeats, retry/block state, and resumable stage transitions - operator-driven `sweep`, `submit`, `list`, `inspect`, `approve`, `unblock`, `run`, and `drain` commands - serving-path fences for database writes and external effects across event ingest, media, Git, workflow, push, invites, mesh/tunnel, and related paths - target-prefix-only storage inventory, summary manifests, post-fence destructive chunks, and bounded batch deletion - exact community Redis namespace purge and two-pass absence verification - cross-community isolation, crash/resume, manifest-integrity, writer-taxonomy, and schema/migration regressions - desired-state `schema/schema.sql` support without requiring a SQLx migration ledger ## Deferred / not covered - dedicated Helm/chart worker Deployment, service account, secrets, probes, resources, and network policy - autonomous `buzz-admin deletions worker` poll loop and worker-only health server - least-privilege separation among migration, relay-serving, and destructive execution roles - fleet-wide shared-CAS physical GC - backup/provider/CDN/observability retention completion - member-only erasure - provider-native conditional-delete improvements - a general force-continue escape hatch; permanent safety failures remain fail closed unless an operator remediates the cause and records an audited `unblock` The removed continuous-worker implementation remains deferred; no remote follow-up branch is claimed by this PR. ## Validation ### Current PR head and repository state Current pushed head: `359d8402ee15f049768f54156f67b953c7a7e2ed`, rebased onto `cc9a2f783375e51a6e8d1f2f9d01d5f7e22813d1` (`origin/main` at push time). The complete PR diff is now 47 files, 9,834 additions, and 517 deletions. The bespoke source-scanner stack was removed to keep this PR scoped to community deletion. Tyler/team requested the underlying fenced-write safety behavior, not `ast-grep`, `crates/buzz-db/tests/community_fenced_writes.rs`, its 27 fixtures, or the new `scripts/lints/community_*.yml` rules. Those scanner-specific files, dependencies, Hermit links, and runner wiring are absent from the current tree. The production database write fence, startup/destructive live-catalog validation, and deletion behavior remain. Source validation on this exact SHA passed: - `cargo fmt --all -- --check` - `bash -n scripts/run-tests.sh` - `cargo nextest run -p buzz-db --all-targets`: 102 passed, 173 skipped, 0 failed - `cargo nextest run -p buzz-deletion --all-targets`: 10 passed, 9 skipped, 0 failed - `cargo nextest run -p buzz-admin --all-targets`: 1 passed, 0 failed - affected-package/all-target Clippy with warnings denied - lockfile consistency - Helm 3.16.4 lint and all 44 chart unit tests - Helm region controls using that fixture: default `BUZZ_S3_REGION=us-east-1`, explicit `eu-west-2` override, and blank-region schema rejection The prior Kubernetes battery below was run against `928992237358a3294621ac0280830b77155abc04`. It remains useful evidence for the patch-equivalent production deletion implementation, but it is **not** claimed as exact-SHA evidence for current head `359d8402ee15f049768f54156f67b953c7a7e2ed`; the current cleanup removes only scanner/test/tooling infrastructure. CI restarted for the new head after the rebase and is pending. Human review remains `CHANGES_REQUESTED`. ### Prior-head live Kubernetes deletion and safety gates The full program used one immutable image, real PostgreSQL, Redis, MinIO, and a three-relay Kubernetes release: - source: `928992237358a3294621ac0280830b77155abc04` (**prior head**) - image: `buzz-e2e:sha-928992237358` - immutable image digest: `sha256:a1a204f4618ac22d9e210be5e5290645a15d79831ae30b0e44379357c8e4a895` - evidence root: `/tmp/buzz-e2e/20260807T033025Z-928992237358-full-gates/` - evidence-manifest digest: `82875c5bc9bea7370b796a7aef3457b3a1c8306c84c59e0f7388bbb5ad30e865` Passed gates at that prior head: - **Chart/operator region:** default `us-east-1`, explicit nondefault propagation, blank-region schema rejection, live in-pod environment, and an in-pod taxonomy sweep over 18 objects with zero unknown. - **Fenced writers and lifecycle:** open-write/fence ordering; 100-attempt anti-starvation; invite, push matcher, and exhausted-reaper bystander isolation; non-`READ-COMMITTED` rejection; manifest/tombstone contracts; eight-failure stage block and audited `unblock`. - **Destructive lifecycle:** submit → approve → run → `retention_pending`; PostgreSQL tombstone and Redis/S3 verification true; zero retries/errors; terminal reruns rejected with exit 5. - **Fresh 10,001-object crash boundary:** exactly two chunks (10,000 + 1). The executor deleted chunk 0 from MinIO while its PostgreSQL stamp was row-lock-blocked, was killed with `SIGKILL`, left one object and both stamps absent, then resumed the same request under generation 2 to zero objects and terminal state. - **Independent dead-owner recovery:** a dedicated executor claimed generation 1, blocked before effects, and was killed through containerd with `SIGKILL` (no TERM cleanup). The request remained owned and unreclaimable before lease expiry; a successor claimed generation 2 after 60 seconds and completed with two attempts and zero retries. - **Three-pod socket isolation:** ordinary NIP-42 and joined huddle-audio target witnesses on every replica received exact `1008 / community deleted`; healthy-tenant witnesses on those pods remained live; deleted-host reconnect returned HTTP 404. - **Health/provenance:** all replicas independently returned ready and retained the exact image digest before/after destructive runs and an audio-enabled rolling restart; PostgreSQL, Redis, and MinIO were healthy at close. Instrument corrections were retained as evidence rather than counted as product failures: a foreground PostgreSQL forward caused an initial `PoolTimedOut`; Kubernetes pod deletion exercised graceful TERM rather than dead-owner recovery; shell-background socket witnesses died with their parent; and the first image build hit the corporate TLS proxy. Detached forwarding/witnesses, containerd `SIGKILL`, and the configured internal CA/Artifactory mirror produced the discriminating runs without weakening product security. ### Prior-head cleanup For the prior-head Kubernetes run, the Helm release was removed, namespace absence was verified, run-owned Screen sessions were absent, and that source worktree remained clean. The evidence manifest was independently recomputed and every indexed artifact passed `shasum -a 256 -c`. The current `359d8402` source worktree is also clean after the scanner-only cleanup and push. --------- Signed-off-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz> Signed-off-by: npub1dccv64krpcpse5cmkzfeh998cftungyatw3djt8jwdw6g43f7fyqzzmrf7 <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz> Signed-off-by: Kalvin Chau <kalvin@block.xyz> Signed-off-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz> Signed-off-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz> Co-authored-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz> Co-authored-by: npub1dccv64krpcpse5cmkzfeh998cftungyatw3djt8jwdw6g43f7fyqzzmrf7 <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz> Co-authored-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz> |
||
|
Jordan Mecom
|
769ac70b74 |
fix(media): require authenticated reads (#4610)
This change requires a valid signed Blossom authorization request and current relay membership for every media GET and HEAD request. It removes the unauthenticated compatibility path and updates desktop reads to send the required authorization. This blocks anonymous retrieval and access after relay-membership revocation. It does not yet bind a blob to its originating channel, so someone removed from a private channel can still read a known blob while remaining a relay member. That channel-ACL follow-up remains required before closing the full finding. ## Testing - `git diff --check origin/main...codex/security-media-read-auth` - Rebased onto `origin/main` at `5c98932` - Full CI pending Originating Buzz thread: `buzz://message?channel=3928fe05-df61-4b5d-b9c7-d623b9b10ea1&id=3c6c02312f763fbe0d2bfc33a6c1a362f91d0354f3d18b039cf7a0558c1439d1` --------- Signed-off-by: Jordan Mecom <jm@squareup.com> Signed-off-by: Alex Rosenzweig <arosenzweig@squareup.com> Signed-off-by: Eli Foster <efoster@squareup.com> Co-authored-by: Eli Foster <efoster@squareup.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Brad Seiler
|
e14fff74d0 |
relay: fuzz WebSocket 1012 restart-close timing on graceful drain (BUZZ_DRAIN_JITTER_MS) (#4542)
## Problem On SIGTERM the relay sends every live WebSocket a **1012 Service Restart** close frame via `ConnectionManager::drain_all()` — all in the same instant (`main.rs` shutdown task → `state.rs::drain_all`). On a pod holding thousands of sessions, that makes every client reconnect simultaneously: the thundering-herd reconnect behind the DB pool-timeout bursts observed on each rolling deploy. Client-side jitter can't fix this — the desktop client *resets* its backoff to base on a 1012 and reconnects with only ±25% jitter (`relayClientSession.ts`), so the spread has to come from the server. ## Change Add `BUZZ_DRAIN_JITTER_MS` (default `0` = unchanged behavior). The two paths are kept **deliberately separate** so the default is byte-for-byte the previously shipped shutdown: - **Jitter off (`0`/unset, the default):** the original synchronous, all-at-once `drain_all()` runs unchanged — queue the 1012 on each connection's control channel, cancel, return. No new machinery on the default path. - **Jitter on (`> 0`):** a separate async `drain_all_jittered(jitter_ms)` spreads each connection's restart close over an independent uniform delay in **`[1, jitter_ms]`**. Each delayed close travels a dedicated `RestartClose` channel; the writer flushes the 1012 frame and **acknowledges the flush over a oneshot**, so drain waits for confirmed delivery (up to `RESTART_CLOSE_ACK_TIMEOUT` = 5s) rather than assuming it, falling back to cancellation if the channel is full/closed or the ack times out. The drain future is **owned and awaited** by the shutdown task, and the 30s hard-drain backstop is aborted only after a clean drain — so a clean roll exits `0`. The two methods can be unified and the old one dropped later once the jittered path is proven for all cases. - **`config.rs`** — `drain_jitter_ms`: non-negative parse, clamped to `MAX_DRAIN_JITTER_MS` = **20s** (leaving 10s of the 30s budget for flush). Junk fails loudly at startup; **empty/whitespace-only is treated as unset (jitter off)** so a `BUZZ_DRAIN_JITTER_MS=""` kill switch does not crashloop the relay (matches the sibling env vars in this file). - **`state.rs`** — `drain_all()` (unchanged synchronous default) + `drain_all_jittered()` (jittered + flush-ack). Both set the sticky `draining` flag before the first await. A registration that lands mid-shutdown always self-signals via the **immediate** control-frame + cancel path — jitter smears already-established sockets, not late arrivals. - **`main.rs`** — shutdown task dispatches: `drain_jitter_ms == 0` → `drain_all()`, else `drain_all_jittered(...).await`. ## Safety - **Default off is the currently-committed path.** With jitter unset/0 the shutdown runs the original synchronous `drain_all()` — no restart channel, no ack wait. Safe to deploy dark and dial up. - **Shutdown-boundary race preserved.** Sticky flag set before any await; a late registration self-signals its close with no jitter. - **Owned + backstopped.** The jittered drain future is awaited; the 30s hard-drain `process::exit(1)` remains the ceiling. `MAX_DRAIN_JITTER_MS` (20s) + `RESTART_CLOSE_ACK_TIMEOUT` (5s) = 25s, inside the 30s budget; 5s pre-sleep + 25s = 30s against `terminationGracePeriodSeconds: 60`. ## Known behavior to note (not a blocker, flagged from review) On a **successful** flush the jittered path deliberately does not cancel the connection token — teardown then depends on the client echoing our Close, or on process exit. Compliant clients echo; a silent client rides to the 30s hard exit. The default (jitter-off) path cancels deterministically as before. ## Tests - `config::tests::drain_jitter_defaults_off_and_rejects_junk` — default off, `20000`, clamp `60000`→`20000`, explicit `0`, junk `"soon"` fails, **empty `""` and whitespace-only treated as off**. - `state::tests::drain_all_is_immediate` — default path queues frame + cancels synchronously. - `state::tests::drain_all_sends_restart_close_and_cancels_every_conn`, `drain_all_full_control_buffer_still_cancels`, `register_after_drain_self_signals_restart_close_and_cancel`. - `state::tests::drain_all_jittered_defers_close_until_within_jitter_window` (paused time). - `state::tests::drain_all_jittered_waits_for_writer_acknowledgement_without_cancelling`. - `state::tests::drain_all_jittered_cancels_when_restart_channel_is_full_or_closed`. - `state::tests::drain_all_jittered_cancels_when_flush_ack_times_out` (paused time — the 5s ack-timeout fallback). Validation at `46c690940`: `cargo fmt -p buzz-relay --check`, `cargo clippy -p buzz-relay --all-targets -- -D warnings`, and the drain/config unit suite all clean. Local live SIGTERM test with a real relay process + 200 NIP-42-authenticated sockets — see the PR comment for the before/after distribution and exit codes. ## Rollout Ship with default `0`, then set `BUZZ_DRAIN_JITTER_MS` (e.g. 10000–20000) on bb-block first, watch the roll-window pool-timeout metric, then bb-public. `""` is a safe kill switch. Complements the preStop `sleep` (stops routing before close). --------- Signed-off-by: npub1srl70fhzyu3fsnahl06vw2czvqc2w3ds37hyzvjnk8ve8f03ngcqg9le2w <80ffe7a6e22722984fb7fbf4c72b026030a745b08fae413253b1d993a5f19a30@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: Brad Seiler <seiler@squareup.com> Co-authored-by: npub1srl70fhzyu3fsnahl06vw2czvqc2w3ds37hyzvjnk8ve8f03ngcqg9le2w <80ffe7a6e22722984fb7fbf4c72b026030a745b08fae413253b1d993a5f19a30@buzz.block.builderlab.xyz> Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> |
||
|
Nick Kampe
|
36cf932ff0 |
docs(chart): fix ArgoCD example for native OCI sources (full artifact repoURL + path) (#3426)
## Problem `examples/argocd-app.yaml` uses the split form: ```yaml repoURL: oci://ghcr.io/block/buzz/charts chart: buzz targetRevision: 0.1.0 ``` On ArgoCD >= 3.0 (native OCI sources), the `chart` field is **ignored** for `oci://` repoURLs, so ArgoCD tries to pull the `charts` path itself and fails with `403 … repository:block/buzz/charts:pull denied` — a misleading error that reads like an auth problem. Additionally, spec validation rejects the Application without a `path` (`spec.source.repoURL and either spec.source.path or spec.source.chart are required`), since `chart` isn't recognized for OCI. Hit both on ArgoCD 3.4.4 following the example verbatim. ## Fix Use the full chart artifact path as `repoURL`, add `path: "."`, bump the pinned example version to the latest published chart (0.1.6), and leave a comment explaining both traps: ```yaml repoURL: oci://ghcr.io/block/buzz/charts/buzz path: . targetRevision: 0.1.6 ``` Verified working in production (ArgoCD 3.4.4, anonymous GHCR pull, chart 0.1.6). Related open PRs/issues: none found. --------- Signed-off-by: Kampe <blindside328@gmail.com> Signed-off-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@buzz.block.builderlab.xyz> Co-authored-by: Kampe <blindside328@gmail.com> Co-authored-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@buzz.block.builderlab.xyz> |
||
|
Kalvin C
|
7012d86d52 |
feat: configure S3 URL addressing style (#3400)
## Summary - add one strict `BUZZ_S3_ADDRESSING_STYLE=path|virtual` setting shared by media and Git/CAS storage - preserve path-style defaults for bundled Compose/Helm MinIO while supporting Railway's virtual-hosted bucket contract - fail startup on invalid or non-Unicode values before dependency connection, and validate the Helm value with the same two choices - document operator mappings and why endpoint and bucket remain separate for routing and SigV4 signing ## Best-practice rationale AWS documents both URL forms and favors virtual-hosted addressing for S3, while compatibility endpoints such as the bundled MinIO deployment can require path style. `rust-s3` defaults to virtual/subdomain addressing and provides `with_path_style()` for the explicit compatibility case. Some providers buckets only support as virtual-hosted bucket styles. This PR therefore uses one explicit, provider-neutral switch rather than endpoint heuristics or fallback behavior, while retaining `path` as Buzz's backward-compatible default. Sources: - https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html - https://docs.rs/rust-s3/0.37.0/s3/bucket/struct.Bucket.html - https://docs.railway.com/storage-buckets#url-style - https://github.com/minio/minio/blob/master/docs/config/README.md#domain ## Validation - `cargo fmt --all` - `cargo check --workspace --all-targets` - targeted `buzz-media` and `buzz-relay` parsing/client-construction tests for defaults, strict errors, and both URL styles - Helm unittest: 45/45 passed - Compose config/render validation passed - local MinIO path-mode relay startup passed the Git A3 conformance probe and became ready - unreachable object storage failed startup and readiness never opened - push hooks completed the broader Rust and desktop suites successfully --------- Signed-off-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz> Signed-off-by: Kalvin Chau <kalvin@block.xyz> Co-authored-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz> |
||
|
Dave Grochowski
|
12d63c67be |
release(chart): publish 0.1.7 (#3393)
## Why Publish chart 0.1.7 after the feature PR merged from a fork and therefore intentionally skipped the internal-branch auto-tag job. ## What - Trigger the `chart-release/0.1.7` release lane - Update the quickstart example to reference chart 0.1.7 ## Risk Assessment Low — the chart implementation is already merged and tested; this PR creates its immutable release tag and OCI artifact. ## References - Chart implementation: https://github.com/block/buzz/pull/3322 - `helm unittest` 0.8.2: 43/43 tests passed - Local pre-push checks passed Generated with Amp Signed-off-by: David Grochowski <dgrochowski@squareup.com> Co-authored-by: Amp <amp@ampcode.com> |
||
|
Kalvin C
|
1e307e178a |
chore(compose): remove stale typesense env vars (#3332)
Search migrated to Postgres FTS (commit
|
||
|
Dave Grochowski
|
af4d861516 |
feat(chart): add relay pod extension points (#3322)
## Why Allow operators to install wrapper binaries and override the relay entrypoint without maintaining a duplicated Deployment outside the OSS chart. `extraManifests` can create independent resources but cannot extend the chart-managed relay Pod. ## What - Add opt-in init-container, volume, volume-mount, command, and args extension points - Preserve image defaults when extensions are empty and compose generic init containers with the MinIO readiness gate - Document the distinction from `extraManifests`, add schema coverage, and release chart 0.1.7 ## Risk Assessment Low — all new values are opt-in, and default rendered manifests are unchanged apart from version-derived metadata. Merge publishes a new chart version without modifying existing installations. ## References - [OpenTelemetry Collector Pod extensions](https://github.com/open-telemetry/opentelemetry-helm-charts/blob/main/charts/opentelemetry-collector/templates/_pod.tpl) alongside [extraManifests](https://github.com/open-telemetry/opentelemetry-helm-charts/blob/main/charts/opentelemetry-collector/templates/extraManifests.yaml) - [Argo CD extraObjects](https://github.com/argoproj/argo-helm/blob/main/charts/argo-cd/templates/extra-manifests.yaml) alongside component-scoped Pod extension hooks - `helm unittest` 0.8.2: 43/43 tests passed - Helm lint, schema validation, fixture renders, and chart packaging passed - Oracle review found no functional issues; its literal no-`tpl` regression test recommendation is included Generated with Amp --------- Signed-off-by: David Grochowski <dgrochowski@squareup.com> Co-authored-by: Amp <amp@ampcode.com> |
||
|
Tyler
|
9b0f744804 |
resolve findings (#3150)
Fixes all six HIGH findings from the buzz security report, one commit per finding. Independently reviewed to approval by Max at `0158ae542`, plus a deep isolated live pass (clean-room compose stack, weird ports, full product matrix) at the same head — see the buzz-security thread for evidence. `fe65c07c3` merges current `origin/main` on top (new commit, no rebase), inheriting the nostr 0.44.6 bump (#3135) and relay-admin ban gate (#3128). ## Findings and fixes | Finding | Commit | Fix | |---|---|---| | 003 — quinn-proto RUSTSEC-2026-0185 | `e5dcdec72` | Bump quinn-proto 0.11.14 → 0.11.16 (lockfile-only) | | 002/004 — linkify-it quadratic-parse DoS (GHSA-22p9-wv53-3rq4, GHSA-v245-v573-v5vm) | `923b3c20f` | pnpm override `linkify-it: ^5.0.2`; `pnpm why` confirms a single 5.0.2 copy | | 001 — media reads served unauthenticated by default | `0f277e3e2` | Helm `requireMediaGetAuth` defaults to `true` + rendered-chart test pinning the default | | 006 — removed workflow owners retain webhook-exfiltration authority | `4749bd56c` | Fail-closed per-fire authority gate (current owner/admin membership) on **all four** trigger doors (on_event, scheduler pre-claim, manual trigger, webhook — masked as generic 404), save-time gate for `call_webhook` defs, durable disable-on-removal wired to kinds 9001 + 9022 | | 005 — git Smart-HTTP reads ignore channel membership | `e648f2dba` + `0158ae542` | `authorize_git_read`: caller's **current active membership** in the repo's bound channel, checked before any hydration/subprocess on all three read doors (`info_refs` for both services + `upload_pack` POST). Uniform generic 404 denials (no membership probing), no repo-owner bypass, first-`buzz-channel`-tag binding semantics fail closed on ambiguous duplicates (mutation-verified test). Resolution follows the live kind:30617 announcement, so deleted/replaced announcements deny immediately. The committed `e2e-git-perms.sh` guest scenario previously asserted the vulnerability — now asserts denial. | ## Behavior changes to be aware of 1. **Unbound repos fail closed for git reads.** `buzz repos create` emits no `buzz-channel` tag, so CLI-created repos without a binding are unreadable via git HTTP. Correct per finding 005's fail-closed posture; a follow-up could bind CLI-created repos at creation time. 2. **006 is conservative:** a workflow disabled on owner removal does not auto-re-enable if the owner is re-added — explicit re-enable required. 3. Merge conflict resolution in `fe65c07c3`: kept main's `@radix-ui/react-dismissable-layer` 1.1.19 bump alongside the linkify-it security override (`pnpm-workspace.yaml` + lockfile). ## Verification at the merge head `fe65c07c3` (same shell) - buzz-relay `--lib`: 761 passed / 1 failed — the lone red is the known pre-existing `mesh_demo::demo_join_forwarded_arm_round_trips_echo` 504 flake, present on main - SEC-005 module incl. PG behavioral matrix: 8/8 (removed-member, never-member, owner-no-bypass, deleted-30617, malformed/ambiguous binding, owner-mismatch all denied) - buzz-workflow 153/0, buzz-db 84/0; `clippy --all-targets -D warnings` + `fmt --check` clean - Desktop JS 3637/3637, tsc clean, biome clean, file-size/px-text/pubkey-truncation gates clean - `helm lint` + `helm unittest` (40/40) on `deploy/charts/buzz` - All five pre-push hooks green (desktop-check, desktop-test, rust-tests, desktop-tauri-test, branch-skew) Prior review evidence at `0158ae542` (pre-merge): Max's independent exact-head approval + clean-room live regression pass (`WORK_LOGS/2026-07-27_SECURITY_HIGH_LIVE_TEST.md` in his workspace). Max will re-run the deep local pass at this post-merge head before merge. --------- Signed-off-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: Tyler Longwell <tlongwell@block.xyz> |
||
|
Will Pfleger
|
bd37a4d584 |
feat(media): add S3-truth per-community storage sweep (#2044)
Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@sprout-oss.stage.blox.sqprod.co> |
||
|
Tyler
|
0fb820f9bf | Revert "feat(relay): inventory unreachable Git objects" (#2275) | ||
|
thomaspblock
|
3afc9dae15 | feat(relay): inventory unreachable Git objects (#2264) | ||
|
thomaspblock
|
a4d82ec722 | perf(relay): cache Git pack hydration (#2169) | ||
|
Tyler
|
b1b4e7d55c |
release(helm): buzz chart 0.1.6 (#2109)
Signed-off-by: Tyler Longwell <tlongwell@block.xyz> Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Tyler Longwell <tlongwell@block.xyz> |
||
|
Tyler
|
29c48883d3 |
Route lag-tolerant reads to an optional Postgres read replica (#2084)
Signed-off-by: Tyler Longwell <tlongwell@block.xyz> Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Tyler Longwell <tlongwell@block.xyz> |
||
|
Tyler
|
5a656c9818 |
fix(chart): support CPU-only relay autoscaling (#2086)
Signed-off-by: npub1jh9wn95s0472h86ahapupaf7m6kx4v9sx2n0atj2hltcfer8k06s5n3pyf <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub1jh9wn95s0472h86ahapupaf7m6kx4v9sx2n0atj2hltcfer8k06s5n3pyf <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@sprout-oss.stage.blox.sqprod.co> |
||
|
Tyler
|
f584bbd971 |
feat(chart): add relay horizontal autoscaling (#2077)
Signed-off-by: npub1jh9wn95s0472h86ahapupaf7m6kx4v9sx2n0atj2hltcfer8k06s5n3pyf <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub1jh9wn95s0472h86ahapupaf7m6kx4v9sx2n0atj2hltcfer8k06s5n3pyf <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@sprout-oss.stage.blox.sqprod.co> |
||
|
Tyler
|
f308762852 |
feat(media): require auth for relay media reads (#1926)
Signed-off-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Tyler Longwell <tlongwell@block.xyz> |
||
|
Tyler
|
2318b3096c |
Route Git scratch through configured volume (#1884)
Signed-off-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> |
||
|
Tyler
|
4883ea1ad2 |
release: push gateway chart 0.1.0 (#1855)
Signed-off-by: Tyler Longwell <tlongwell@block.xyz> Co-authored-by: Tyler Longwell <tlongwell@block.xyz> |
||
|
Tyler
|
ccb021d713 |
Relay mesh: cross-pod tunnel + huddle transport (buzz-relay-mesh) (#1670)
Signed-off-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co> Signed-off-by: npub17jjz49l9jjmhhk7cac63j8yt9z555n9cw8vk7v5jz4vzw4ppld5qgj57cc <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@sprout-oss.stage.blox.sqprod.co> Signed-off-by: Tyler Longwell <tlongwell@squareup.com> Signed-off-by: tlongwell-block <109685178+tlongwell-block@users.noreply.github.com> Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Co-authored-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub1cc3ha7z055mu0rwwu7806t2wt8mj3pvu0uv5mfp2c50dahaqhczshdalg6 <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub17jjz49l9jjmhhk7cac63j8yt9z555n9cw8vk7v5jz4vzw4ppld5qgj57cc <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub1jh9wn95s0472h86ahapupaf7m6kx4v9sx2n0atj2hltcfer8k06s5n3pyf <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> |
||
|
Tyler
|
1c006822e4 |
feat(push): add public APNs gateway (#1770)
Signed-off-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> |
||
|
Tyler
|
9b47c8548f |
Add optional standalone pairing relay to Helm chart (#1799)
Signed-off-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> |