mirror of
https://github.com/block/buzz.git
synced 2026-10-31 14:31:02 +00:00
codex/plugin-runtime-support
17 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Jarrod Sibbison
|
c4c86006f2 |
feat(relay): deliver pubkey mentions to relay companions (#7793)
We are introducing a companion service to the relay that needs pubkey event mentions for pubkeys it manages. This PR introduces an outbox for pubkey mentions that will be sent to these "operator listener" services. ## Summary Deliver mentions of registered pubkeys to deployment-global operator listeners. - Configure listeners with `BUZZ_OPERATOR_LISTENERS`; listeners manage target pubkeys through NIP-98-authenticated `POST`/`DELETE /operator/listener/pubkeys` requests. - During supported event ingestion, match `p` tags to registrations and insert one outbox row per listener in the event transaction. - A background worker claims up to 10 rows, sends NIP-98-signed notifications, and retries transient failures with exponential backoff for up to 9 attempts (idle polling backs off from 250 ms to 2 s). - Outbox rows are ineligible for delivery after 15 minutes and are reaped every 5 minutes. Registered pubkeys live for 30 days; re-registering renews them, and a daily reaper removes expired registrations. - Keep persistence app-managed: no database triggers, functions, foreign keys, or separate match queue. ## Changed behavior Listener outbox rows are enqueued inside the shared event-insert and thread-metadata transactions; enqueue failure rolls back the event. `event_mentions` behavior is unchanged: ordinary event/reaction mention indexing remains post-commit best effort with warning-only failures. ## Testing - `cargo fmt --all` - `cargo test -p buzz-db --lib` - `cargo test -p buzz-relay --lib config::tests::operator_listener_routes` - `cargo check -p buzz-db -p buzz-relay` |
||
|
Kalvin C
|
5a349db91c |
feat(relay): add opt-in newest-first thread windows (#7823)
## Summary Add an opt-in, newest-first thread window to the authenticated HTTP query path. A filter with `thread_window: true` returns: - reply rows ordered by `(created_at DESC, id ASC)` with an exact composite cursor; - optional bounded reactions, edits, deletions, and delete-of-aux closure; - one relay-signed `kind:39007` bounds event bound to the normalized host, authenticated reader, channel, root, request cursor, kinds, depth, limit, and aux choice. Filters without the flag keep the existing oldest-first thread behavior. Channel windows and `kind:39006` are unchanged. No client opts into this mode in this PR. ### Why extend NIP-CW NIP-CW already defines relay-computed, cursor-paged conversation views over extended NIP-01 filters. Thread windows use the same protocol ideas: ordinary signed rows, composite keyset pagination, optional aux closure, and relay-signed bounds. Adding a thread mode keeps those related wire contracts together instead of creating a second NIP with duplicated terminology and compatibility rules. Extending a draft, optional NIP with an additive mode is normal protocol evolution. Existing fields and meanings are not reinterpreted: - `top_level: true` remains channel mode; - `kind:39006` remains channel-window bounds; - `thread_window: true` selects the new mode; - `kind:39007` has a distinct identity and root/request binding; - extension-unaware clients never send the selector; - legacy thread queries remain unchanged. NIP-CW documents only the wire contract and relay semantics. Database plans, replica internals, and rollout procedures remain implementation concerns. ### Why thread windows use independent modules The implementation deliberately keeps `thread_window.rs` separate in `buzz-core`, `buzz-db`, and `buzz-relay` rather than adding branches throughout the existing channel-window path. The modes share pagination concepts, but their invariants differ: - channel mode selects top-level rows; thread mode requires one root and depths `1..=depth_limit`; - thread mode accepts only conversation row kinds and rejects unknown/conflicting fields; - thread bounds bind the root and authenticated reader because cross-channel aux visibility can differ by reader; - thread aux includes the root and requires complete two-hop closure; - thread mode refreshes writer authorization before selection and before signing; - historical cursor pages may use a proved replica transaction, but fallback must restart both aux hops on the writer; - legacy thread traversal is oldest-first and must not inherit descending cursor semantics. Forcing these rules into channel-window code would create a mode flag across parsing, SQL predicates, bounds construction, authorization, replica fallback, and tests. That abstraction would not remove complexity. It would distribute it. Separate modules keep each contract local and leave the established channel and legacy paths untouched. ### Technical design - Strict request parsing normalizes UUIDs, event IDs, kinds, defaults, and cursors before computing the response binding. - SQL applies community, root, channel, depth, kind, deletion, and cursor predicates before `limit + 1`. - Ordered metadata scanning plus exact event-key lookup avoids the stale-statistics plan that previously exceeded the four-second SQL budget on a 10k-reply thread. - Aux work is bounded by target batches, query count, raw candidate count, response bytes, SQL timeouts, and one shared HTTP deadline. - Fresh authorization-set comparison catches both grants and revocations during closure. The relay returns retryable `503` rather than signing an incomplete page. - Replica failure permanently degrades the request to the writer and restarts closure from its original targets without resetting budgets. - Migration `0049_thread_window_index.sql` adds the mixed-direction index used by newest-first scans. It validates a prebuilt index and bounds startup lock/build time. ### Tradeoffs - The stale-statistics-safe plan is intentionally less clever. Measured warm 100k deep-cursor execution increased from roughly 0.30 ms to 4.80 ms, plus about 8 ms planning, in exchange for avoiding a reproduced multi-second timeout after rapid 10k-thread insertion. - Bounds are authoritative only for the served page. There is no cross-page snapshot guarantee, and replica insertion coverage does not imply latest edit/deletion/aux visibility. - Strict parsing rejects unsupported constraints instead of silently serving a different row set. - Required aux closure fails closed. The relay does not omit aux and sign plausible-but-incomplete bounds. - This PR ships relay capability only. Client rendering, retry UX, live-history overlap, and compatibility fallback remain separate work. ### Related issue No existing issue found. This work follows the thread-loading design and review discussion. ### Testing - `cargo test -p buzz-core`: 258 unit tests and 2 doctests passed. - `cargo check --workspace --all-targets`: passed at the rebased implementation head. - `cargo test -p buzz-db`: 131 non-PostgreSQL tests passed; 275 infrastructure tests were ignored as declared. - Focused PostgreSQL migration/runtime regressions: 3/3 passed against an isolated PostgreSQL container, including both corrected migration-ledger tests. - `cargo fmt --all --check`: passed. - Pre-rebase feature validation passed 398/398 PostgreSQL tests, signed-wire multi-tenant/auth/pagination/aux/revocation probes, clean Compose boot/reboot, and full core tests. - Independent HA validation exercised three relay pods and a physical PostgreSQL standby on the same runtime logic before the final documentation-only cleanup. - Independent final-delta review confirmed exactly two changed files, byte-identical executable migration SQL after comments/diagnostic normalization, matching prebuild/catalog/recovery instructions, and `git diff --check`; no runtime receipts were relabeled to the new SHA. - The pre-push hook ran repository checks but its Rust unit lane exited 100 without exposing an individual failed test in the captured multiplexed output. The independently run affected-package checks above passed; current-head repository CI remains required. Known unrelated baseline behavior remains: the mesh-demo echo test can return 504, and same-second archive → unarchive can deduplicate its membership notification. Neither path is changed here. --------- Signed-off-by: Kalvin Chau <kalvin@block.xyz> Signed-off-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz> Signed-off-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz> Co-authored-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz> Co-authored-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz> |
||
|
Jordan Mecom
|
7881447c19 |
fix(audit): frame hash inputs with TLV (#7492)
Hash new audit entries from a TLV buffer built with one-byte field tags, big-endian u64 byte lengths, and raw values. Absent optional fields are omitted; present empty values retain their tag and zero length. This prevents bytes from moving between fields without restricting object IDs or inferring boundaries from field contents. Store an explicit `hash_version`. Migration 0048 labels existing rows as version 1, and new writers use version 2. Historical hashes are unchanged and mixed-version chains verify. The legacy encoding retains its original ambiguity; TLV protects new entries. Apply the migration before deploying the updated writers and verifiers. Tests cover both boundary collisions, absent versus empty fields, stable digests, unknown versions, mixed-version chains, and stored-row tampering. All 16 audit and 128 database unit tests passed. The six PostgreSQL audit tests passed against both migrated and fresh schemas, and a populated-row migration check preserved the historical hash. Full `just ci` passed. Related audit fixes: #4216 and #4249; issue #4173. Signed-off-by: Jordan Mecom <jm@squareup.com> |
||
|
Will Pfleger
|
ce41be9c4b |
fix(relay): fire kick live side effects at convergence; persist target; fence re-add race with held lock (#7298)
## What
Ensures admin kick live side effects (subscription eviction + workflow
disable) fire correctly on every path — fresh HTTP, crash recovery, and
upgrade recovery from pre-migration rows — without a race window where a
concurrent legitimate re-add can be revoked.
### Finding 1 — Lock held through effects
Replace `verify_member_still_removed` (which acquired + released the
advisory lock before returning) with `membership_removal_fence`, a RAII
guard that keeps the `pg_advisory_xact_lock` alive until the caller
drops it.
Update `apply_kick_live_side_effects` to use the fence on every path
(fresh and recovery), holding it through eviction and workflow-disable.
Remove the `is_recovery` parameter entirely.
**The bug:** the old code released the lock before returning a bool; a
concurrent `add_member` could commit between the check and the effects,
silently evicting a re-added member.
**New guarantee:** any `add_member` either serializes before the fence
(fence observes `removed_at IS NULL`, skips effects) or waits until
after effects complete.
### Finding 2 — Persisted-context path scoped to kick only
Gate the persisted-context branch in `admin_action_worker.rs` on `action
== "kick"`. Non-kick actions (ban, timeout, delete) always re-derive
from the report.
**The bug:** any action with `enforcement_target_pubkey` set took the
persisted-context branch, which hard-coded `target_event_id = None`. A
stranded delete either failed pre-marker recovery ("delete requires
target_event_id") or finalized post-marker without the tombstone outbox
row.
### Finding 3 — Legacy kick row upgrade recovery (Carl's
CHANGES_REQUESTED)
Migration 0047 (renumbered from 0045 to avoid collision with two main
merges: 0045_retain_push_revocation_tombstones and
0046_storage_accounting_snapshots) adds
`enforcement_target_pubkey`/`enforcement_channel_id` without
backfilling.
**The bug:** the convergence gate read only the persisted columns and
rejected NULL/NULL rows as corrupt, causing every recovery retry to fail
forever. A cancellation cannot release a post-mutation action, so these
rows remain permanently stuck.
**Fix:** when persisted columns are absent, fall back to the function
parameters (which the recovery worker re-derives from the report row).
Error only when both sources are absent (genuinely unresolvable). This
preserves new-writer safety while allowing pre-migration rows to
converge.
An old writer in a rolling deployment can also create NULL rows after
the DDL applies; the same fallback handles these.
### Finding 4 — Test moved to PG lane
`kick_live_side_effects_clears_membership_cache_and_evicts_subscription`
was in the infra-free unit lane but called `membership_removal_fence`,
which requires a real Postgres connection. Moved to the PG fixture lane
(`#[ignore = "requires Postgres"]`) with a real member row so the fence
query sees the expected `removed_at` state.
## Tests
- **`crash_recovery_redrive_fires_kick_live_side_effects`** (existing):
re-drive via real recovery worker path, asserts all three side effects.
-
**`crash_recovery_after_readd_preserves_membership_subscriptions_and_workflows`**
(updated): uses real `add_member` (not direct SQL) — re-add completes
before fence is acquired.
- **`membership_removal_fence_blocks_concurrent_add_member`**
(existing): holds the fence explicitly, spawns `add_member`, asserts it
blocks until the fence is dropped.
- **`stranded_delete_pre_marker_recovers_via_worker`** (existing):
stranded delete (crash before mutation) converges via `recover_one` to
succeeded.
- **`legacy_kick_row_pre_marker_recovers_via_worker`** (new): old-writer
shaped row (NULL columns, no step_marker) converges through the full
state machine via recovery worker. Falsifiable: reverting the
convergence-gate fallback leaves the action stuck in enforcing.
- **`legacy_kick_row_post_marker_recovers_via_worker`** (new): same but
with step_marker set (post-marker crash). Falsifiable: same revert.
-
**`kick_live_side_effects_clears_membership_cache_and_evicts_subscription`**
(moved to PG lane): verifies cache eviction and subscription removal
against a real member row.
## CI
`Rust / Rust Lint` and `Windows Rust` failures in the prior run were
main-inherited (same lanes fail on main at the prior base). The branch
is now at main's current head.
---------
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>
Co-authored-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz>
|
||
|
ravarora2
|
4c2086c084 |
feat(db): expose connection setup metrics (#7286)
## Why
Current pool metrics show checkout outcomes and pool state after the
fact. They do not show when a checkout began or which writer-connection
setup step failed. During startup and pool growth, operators need to
distinguish pool saturation from a slow or unsafe connection setup.
## What
- Count every instrumented database checkout when it starts, with a
fixed operation label.
- Measure writer-pool creation and every physical writer connection
across `physical_connect`, `created_at_floor`, `session_timeouts`,
`isolation`, and `ready`.
- Publish fixed-cardinality start, terminal-outcome, duration, and
waiter metrics without database URLs, SQL, raw errors, connection
ordinals, or per-connection lifecycle logs.
- Document how to identify the active setup bottleneck during a rollout
or incident.
## How
The existing typed checkout wrapper records starts and current waiters.
The huddle-history path uses that wrapper instead of a raw pool
checkout.
The production SQLx `after_connect` hook records start, terminal
outcome, and duration for each sequential safety step. A drop guard
records cancellation once if setup exits before a terminal result. Fixed
enums keep every label bounded.
For a phase, `started_total - sum(attempts_total)` is the number of
in-progress attempts on that pod. Because setup is sequential, a later
phase starting also proves the earlier phases succeeded.
The database path emits metrics only. This PR intentionally does not add
per-connection lifecycle receipts or direct `stderr` writes.
## Risk
Medium. This changes the production writer-pool `after_connect` hook and
adds one counter update to instrumented checkout paths. It does not
change the database safety statements or their failure behavior. The
telemetry has fixed labels and no per-connection identifiers.
## Testing
At exact head `ffb5fcb114a9c986ffd0cb2cfc9d413a32d5b1aa`, a release
relay was started against isolated local PostgreSQL, Redis, and MinIO
services. The main health endpoint returned `ok`, readiness returned
`{"status":"ready"}`, the connection-step metrics were exported, and no
database lifecycle receipts were emitted. A release CLI then created a
channel, sent a message, and read the same event back successfully.
The earlier staging deployment used pre-rebase head
`f58e9480a4f068db0c591f604fd6800fdd4bfc45`. The deployed
multi-architecture image came from [GitHub Actions run
33780888255](https://github.com/block/buzz/actions/runs/33780888255),
manifest
`sha256:b9351fa644e08376cbe1999f9bee311d33d1799a68eadef7929c4f862a832fec`.
The [staging
deployment](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/317)
brought both pods in ReplicaSet `buzz-6c8758bd7d` to Ready with zero
restarts, and the connection metrics produced data in the [rollout
dashboard](https://app.datadoghq.com/dashboard/tm7-qxr-wt2/buzz-startup--rollout-safety).
That staging image predates the cleanup that removed per-connection
lifecycle logs; the metric schema and database safety statements are
unchanged.
## Verification
- `cargo test -p buzz-db -- --test-threads=1`: 128 tests passed across
the package and integration target; 255 opt-in tests remained ignored.
- Four opt-in production-path PostgreSQL regressions passed: initial
minimum connections, post-startup pool growth, isolation failure, and
session-timeout setup failure.
- All 35 relay media tests passed against the isolated PostgreSQL
database.
- Rust formatting, workspace clippy, desktop/Tauri clippy, web checks,
mobile analysis, security checks, and file-size policy checks passed.
- The repository-wide unit stage also exposed three unrelated existing
`buzz-acp` failures: one timing-sensitive keepalive test and two
environment-default tests.
## Bigger picture
This is the database-metrics part of the startup and rollout
observability work. The early-startup lifecycle foundation merged in
#7258, so the rebase removed that duplicate commit from this PR.
Originating discussion:
buzz://message?channel=6ac85131-70cd-4bda-a031-38d34114934e&id=fa2bed181c092697210a60bb6eedc55a665d5c1c6cabc1413a04686647011f71
Generated with Codex
---------
Signed-off-by: Ravneet Arora <rarora@squareup.com>
|
||
|
ravarora2
|
6b1e45d2f6 |
Isolate S3 storage metrics from the relay (#7543)
## Why **This PR moves the S3 storage scan out of the serving relay and into a separate batch job.** The job saves a complete result in PostgreSQL. The relay reads that result, exports the existing storage totals, and reports the snapshot's freshness and database-read health. ```text Today: relay -> scan S3 -> calculate totals -> Datadog metrics After rollout: CronJob -> scan S3 -> PostgreSQL snapshot -> relay -> Datadog metrics ``` The code in this PR is the new pipeline. It adds the worker, database snapshot, relay read mode, and Helm CronJob template. It does **not** enable the pipeline in production. The chart keeps the CronJob disabled, and the relay keeps its current inline mode by default. [squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244) will set the production limits, enable the job, and switch the relay to the saved snapshot after this PR merges. The relay currently lists the media bucket once an hour and calculates storage totals inside the serving process. The production bucket now exceeds the relay's one million object cap, so each sweep fails and the last good per-community values remain stale. The Datadog dashboard can no longer show current storage use by community. Raising that cap inside the serving relay would move a larger, memory-heavy scan into the process that owns live WebSocket connections. The scan retains state for unique blob hashes and community bindings, so its memory use grows with the bucket. A large scan could raise relay memory use, trigger restarts, and drop user connections. The goal is to restore current metrics without putting a production-scale bucket scan in the relay's failure domain. No media objects move or change. Only the process that calculates the totals and the handoff to the relay change. ## What - Add `buzz-admin storage-snapshot`, a run-once worker that scans S3 and saves one complete accounting snapshot in PostgreSQL. - Add `external` relay mode to read the saved snapshot. Keep the current `inline` mode as the default, and add `off` as a fail-closed option. - Add an opt-in Helm CronJob for the worker. The 10 million object cap exists only on this Job. - Keep the current fleet and per-community gauges, and add snapshot age, generation duration, configured cap, cap utilization, and database-load health gauges. - Add an additive migration for the latest complete snapshot and its run metadata. ## Overall plan 1. Merge this PR to add the worker, database handoff, relay modes, and disabled-by-default chart support. This step does not change production behavior. 2. Publish the relay image and Helm chart `0.1.9` from `main`. 3. Update [squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244) to pin those final artifacts and rerender its manifests. 4. Use the infrastructure PR to deploy the CronJob with a 10 million object cap, a one-hour deadline, and a 20 GiB memory limit. 5. Start one manual Job during a staffed window and verify its snapshot, duration, memory use, relay stability, and Datadog output. 6. Keep the daily schedule and `external` relay mode only after that gate passes, then add alerts for stale snapshots, snapshot-load failure, failed jobs, cap pressure, duration, and worker memory. The code and infrastructure remain separate so this PR can ship the mechanism without silently enabling a production-scale scan. The infrastructure PR owns the production settings and rollout gate. ## How ### Complete snapshot handoff The worker takes a deployment-wide PostgreSQL advisory lock before it lists S3. This prevents scheduled and manual workers from overlapping, even if Kubernetes starts both. The final database replacement runs through that same lock-owning session. If the session dies during the scan, publication fails instead of letting the stale worker overwrite its successor. A worker that no longer owns the lock cannot publish through another pooled connection. The worker reads S3 in pages of 1,000 objects and folds each page into physical totals, logical per-community usage, and anomaly counts. It stops before folding a page that would exceed `--max-objects` and emits structured progress every 100,000 objects. The worker replaces the singleton database row only after the full scan succeeds. A listing error, malformed page, cap breach, serialization error, lost lock session, or database error cannot publish a partial snapshot. The last completed snapshot stays available. Each stored result includes its completion time, duration, cap, and image or source revision. PostgreSQL is the handoff between the batch worker and the relay. The worker writes one complete deployment-wide snapshot, and the relay reads that snapshot on its normal leader metrics tick. No new network service or worker endpoint is required. ### Relay behavior `BUZZ_STORAGE_METRICS=inline` keeps the existing in-process sweep and its one million object default. `external` loads the newest completed worker snapshot on each leader metrics tick. `off` emits no storage-family metrics. An unknown value fails closed to `off`. External mode republishes the existing fleet, per-community, and anomaly metric names, so those current queries remain valid. Snapshot handoff health uses a separate family: `buzz_storage_snapshot_age_seconds`, `buzz_storage_snapshot_duration_seconds`, `buzz_storage_snapshot_max_objects`, `buzz_storage_snapshot_cap_utilization`, and `buzz_storage_snapshot_load_ok`. The `buzz_storage_sweep_*` gauges remain inline-only because they describe a scan attempted by the relay process. External mode does not synthesize `sweep_ok=1` from the last successful database row: an old row proves only that an earlier worker completed. Kubernetes CronJob/Job status is authoritative for the latest worker result, including non-zero exit, deadline expiry, and OOM termination. The relay keeps the last complete snapshot available when a worker run fails. Snapshot age shows that no replacement arrived on schedule, while `buzz_storage_snapshot_load_ok` distinguishes staleness from a missing row, database read failure, or invalid payload. If a community disappears, changes host, or leaves the configured emission scope, the relay zeroes the old labeled series instead of leaving stale values behind. Per-community bytes are logical referenced media usage. If two communities reference the same blob, each community receives the blob's bytes. Fleet physical totals still count each S3 object once. This PR preserves those existing metric semantics. ### Deployment boundary The CronJob is disabled by default. When enabled, it uses `concurrencyPolicy: Forbid`, no retry, a configurable active deadline, bounded history, explicit CPU and memory resources, a read-only root filesystem, and no Istio sidecar by default. The worker receives only `DATABASE_URL`, S3 settings, optional S3 credential references, and the code revision. It does not receive the relay private key, Git hook HMAC secret, Redis URL, or the full relay secret as environment variables. Operators may reuse the Buzz service account or select a dedicated one. ### Non-goals - This PR does not enable the worker or `external` mode in production. - This PR does not change media uploads, object ownership, retention, or deletion. - This PR does not change the names or meaning of the existing fleet, per-community, or anomaly storage gauges. - This PR does not add a lease service or allow multiple workers to publish concurrently. ## Risk The application and chart defaults do not enable the new worker or raise the serving relay's cap. The database change is an additive deployment-global singleton table. The relay's default remains `inline`, so merging this PR alone does not change the active metrics path. The main rollout risk is worker memory use because the fold retains state for distinct blob hashes and community bindings. The first production-sized run must confirm peak memory and duration before the schedule is trusted. The 10 million cap and one-hour deadline bound the first deployment, but they do not replace measurement. External mode also needs stale-snapshot, snapshot-load, and job-failure monitors because the relay intentionally keeps publishing the last complete result after a failed job or read error. This behavior avoids gaps and partial data, while snapshot age and load health tell operators when the result or PostgreSQL handoff needs attention. Job status remains necessary for immediate failures and termination reasons that a killed process cannot record itself. ## Testing No production S3 scan was run from this PR. Automated regressions verify that external mode keeps stale last-good totals visible without emitting inline attempt-health gauges, reports database-load failure while retaining the cache, and leaves inline telemetry unchanged. Mutation checks also prove those tests fail if external mode is routed through inline emission or a failed load is reported as healthy. After deployment, create one manual Job from the CronJob during a staffed window. Require a completed Job, exactly one new snapshot, fewer objects than the configured cap, acceptable duration and peak memory, no relay restart regression, restored Datadog points, and no OpenMetrics sample truncation. ## Next steps 1. Complete review and merge this PR. 2. Publish chart `0.1.9` and the final main-branch relay image. 3. Replace the provisional pins in [squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244), rerender, and require clean CI. 4. Run the staffed manual Job gate and record object count, peak memory, duration, snapshot revision, and relay stability. 5. Confirm the relay reads the new snapshot in `external` mode and the existing `block.buzz_relay.*` storage series update again. 6. Enable the daily 03:17 UTC schedule and add dashboard panels and monitors for snapshot age, snapshot-load health, Job failures, cap pressure, duration, and worker memory. If production must roll back to an older relay image, set `BUZZ_STORAGE_METRICS=off` first. Older images do not understand `external` and would resume the failing inline scan. ## Bigger picture This is the application half of the storage-metrics repair. The paired infrastructure PR owns the production resource limits, schedule, artifact pins, secret boundary, and monitored rollout. Keeping those changes separate lets this PR remain safe by default while still providing a complete path to current per-community S3 totals. Related issue: N/A. No matching issue or pull request exists in `block/buzz`. Generated with Codex --------- Signed-off-by: Ravneet Arora <rarora@squareup.com> Co-authored-by: Codex <noreply@openai.com> |
||
|
Luke Tornquist
|
213092074c |
Instrument database pool roles (#7356)
## What
Name the relay's four Postgres pool roles—writer, reader, audit, and
search—and export one fixed-cardinality utilization metric contract for
all of them. Existing writer and reader gauges remain unchanged.
## Why
Audit and search already use separate pools, but they are missing from
pool pressure telemetry. Operators cannot see all process-level
connection demand before setting a deployment-wide connection budget.
## How
A closed `DbPoolRole` vocabulary now labels
`buzz_db_pool_connections{pool_role,state}` and
`buzz_db_pool_configured{pool_role}`. The relay retains cheap pool
clones for statistics only; ownership, capacities, timeouts, query
routing, and failure behavior do not change. The docs also replace the
stale async search-index description with the generated-column behavior
used today.
This simplifies four ad hoc observability states into one bounded model
without adding a pool manager or changing service boundaries.
## Risk
Low to moderate. This changes relay metrics and pool construction
plumbing, but not SQL execution or routing. Legacy metric names remain
available for current dashboards.
## Testing
- Push gate at `3f5468a0463bdfc5879bcdd3878fa06b42ca84c2`: all 14 Rust
test lanes passed; desktop Tauri checks passed.
- Blox: `cargo test --workspace --all-targets` with repository-declared
Postgres services.
- Blox: `cargo clippy --workspace --all-targets -- -D warnings`.
- Blox: `cargo fmt --all -- --check`.
## Bigger picture
Aggregate deployment budget enforcement remains deferred. This PR
exposes the per-role facts that a later deployment-policy change can
consume.
Generated with Claude Code
---------
Signed-off-by: tornquist <tornquist@squareup.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Tom Brow
|
76f7e5ef0c |
feat(push): make gateway origin configurable (#7261)
Make the push gateway origin configurable across the relay, gateway, mobile client, and deployment chart. This is a fresh deployment. The gateway has only run on personal development infrastructure, and push has no existing enabled users. Legacy migration, configured-to-unconfigured mobile transitions, and rolling upgrades from pre-launch gateway binaries are not supported. Populated pre-launch databases are rejected with an informative error before migrations run; their data is untouched. Mobile builds may omit the gateway URL, leaving push unavailable. Once push is enabled, subsequent builds must retain that configuration. ### Validation - All applicable CI checks pass at `5f546cfaf403f0386f146d213adfb4392f4ec29c`, including Rust, mobile/Swift, desktop/integration, security, image builds, and Helm checks. An unrelated agent cancellation test passed on retry. - PostgreSQL CI passed 384 tests, including the production-entry-point regression for refusing populated pre-launch databases before migrations, fresh initialization, and subsequent initialized deployments. - Focused local validation passed all 10 gateway PostgreSQL tests under the CI discovery filter and all 3 unconfigured-mobile tests. Repository-wide PostgreSQL discovery validation and the complete pre-push hooks passed. - Physical-device APNs/App Attest behavior and deployment to a live cluster were not verified in this work. --------- Signed-off-by: Tom Brow <tomb@block.xyz> Co-authored-by: Codex <noreply@openai.com> |
||
|
ravarora2
|
91ab9d31b8 |
Add operation-aware database pool acquisition metrics (#7195)
## Why
Buzz already reports coarse writer/reader database checkout waits, but
those
signals cannot explain which startup or serving operation is blocked by
pool
pressure. That makes rollout diagnosis and postmortems ambiguous: a
readiness
probe, NIP-42 authentication, authorization check, reconnect history
repair,
event write, and background maintenance can all wait on the same pool
while
appearing identical.
This PR implements Package 2A of the pod-handoff plan: operation-aware
pool
borrow causality. It is observability-only; it does not change
configured pool
sizes, SQL semantics, transaction ordering, or timeout policy. Physical
DNS/TCP/TLS/Postgres authentication and session initialization remain
the
separate Package 2B boundary.
## Metric contract
The final contract separates three questions:
| Question | Metric |
|---|---|
| How long did checkout wait? |
`buzz_db_pool_acquire_duration_seconds{pool_role,operation}` |
| How did the attempt end? |
`buzz_db_pool_acquire_attempts_total{pool_role,operation,outcome}` |
| Who is waiting now for a tracked operation? |
`buzz_db_pool_waiters{pool_role,operation}` |
Outcomes are `success`, `timeout`, `error`, and `cancelled`. Operations
are
`bootstrap`, `readiness`, `tenant_resolution`, `authentication`,
`authorization`, `subscription_history`, `event_write`, and
`maintenance`.
Only these eleven pool/operation pairs are constructible:
```text
writer/bootstrap reader/bootstrap
writer/readiness
writer/tenant_resolution
writer/authentication
writer/authorization reader/authorization
writer/subscription_history reader/subscription_history
writer/event_write
writer/maintenance
```
The duration histogram intentionally does not carry `outcome`. Result
data
remains available on the terminal counter for historical counts and
rates,
without multiplying the expensive histogram family. Nine finite buckets
plus
`+Inf`, sum, and count produce 12 duration series per valid pair.
Together with
four outcome counters and one waiter gauge, the new-family ceiling is
exactly
187 raw Prometheus series per pod, asserted from the production
exporter.
The existing coarse acquisition families remain temporarily for
dashboard
compatibility while the new series are validated in staging.
The operation-specific waiter family covers the explicitly routed,
deployment-critical operations above; it is not a census of every
possible
SQLx checkout in the process. The dashboard pairs it with SQLx pool
active/idle/max gauges for whole-pool capacity context, and treats a
missing
operation series as unknown rather than healthy zero.
## What changed
### Cancellation-safe acquisition ownership
- Add writer- and reader-specific typed operation APIs so invalid label
pairs
cannot be constructed and store modules cannot emit reader labels.
- Own every polled acquisition with one RAII terminal guard.
- Record exactly one duration and terminal outcome for success, timeout,
error,
or cancellation.
- Emit nothing for a future that is created but never polled.
- Balance the operation-specific waiter count exactly once on every
terminal or
dropped future.
- Periodically refresh every expected waiter pair, including healthy
zero, so
missing telemetry is not presented as zero. Reader pairs are emitted
only
when a distinct read pool is configured; a writer-only pod cannot
fabricate
healthy reader-zero state.
### Production attribution
Route the deployment-critical acquisition paths through caller-owned
semantic
entry points, including:
- writer and reader bootstrap;
- the real post-#7149 readiness acquisition and deletion-catalog
validation;
- tenant resolution and community lifecycle checks;
- NIP-42 allowlist authentication;
- membership, moderation, invite, operator, Git, agent-owner, and policy
authorization;
- operator community create/list/archive/unarchive, reverse host/channel
tenant
resolution, and REQ row-community conformance lookups;
- writer/reader subscription history, feed, thread, and routed fallback
paths;
- primary and command event writes, replaceable events, mention
indexing,
reaction/channel/member/archive side effects, and thread metadata;
- push matching, usage rollups/leadership, replica-fence startup and
recurring
probes, periodic reconciliation, channel/deletion reapers, partitions,
and
other bounded maintenance/bootstrap paths.
Shared helpers now accept caller-owned intent or expose named semantic
variants
instead of assigning one misleading operation to every caller. No known
P0 path
uses `other`.
### Readiness and size-one-pool correctness
- Rebase on the post-#7149 readiness implementation and instrument the
actual
`Db::readiness_check` acquisition rather than the superseded ping-only
seam.
- Acquire once for deletion-catalog validation and run its queries on
that
connection, preserving the shared readiness deadline.
- Scope the channel-roster catalog checkout before the behavior probe so
a
size-one writer pool cannot self-deadlock during startup verification.
### Exporter, documentation, and CI
- Register metric HELP/type/unit metadata through the production
Prometheus
builder.
- Configure dedicated checkout buckets at 1ms, 5ms, 10ms, 25ms, 50ms,
150ms,
500ms, 1s, and 3s.
- Add a production scrape-contract test for exact names, labels,
buckets,
valid pairs, sensitive-label exclusion, and the 187-series ceiling.
- Add source mutation guards for the P0 semantic entry points and
raw-checkout
bypasses.
- Add an exact backend-integration CI selector for the production
attribution,
cancellation, readiness, and size-one-pool PostgreSQL tests.
- Document the frozen label vocabulary, valid combinations, semantics,
and
cardinality budget in the Helm chart README.
## Dashboard intent
The new Stage 2 row in **Buzz Startup & Rollout Safety** is
deployment-first:
- baseline-versus-candidate attempts, failure rates, cancellation rates,
and
maximum wait by operation;
- outcome counts and percentages over time by SHA/ReplicaSet;
- acquisition wait heatmap, average, and maximum through the rollout;
- historical waiter pressure beside writer active/idle/max context;
- per-pod postmortem drilldown, including terminated pods;
- a smaller current-waiter table with explicit stale/missing semantics.
Percentile widgets remain disabled until Datadog metadata confirms
percentile
support for the new distribution. Current gauges use no fill,
interpolation, or
`default_zero`; missing means unknown.
## Risk assessment
Moderate. The patch touches many database acquisition call sites, but
preserves
the selected physical pool and executes the same SQL on the acquired
connection. The main risks are incorrect semantic attribution,
cancellation
double-counting, and a helper accidentally acquiring twice. Typed APIs,
production-method PostgreSQL tests, source guards, the raw scrape
contract, and
the size-one-pool regression cover those risks.
No tenant, community, user, pubkey, event, channel, SQL, URL, pod,
version,
ReplicaSet, or request-controlled value is emitted as an application
metric
label. Deployment identity is supplied by infrastructure enrichment.
## Verification
- `cargo fmt --all -- --check` — passed.
- `cargo clippy -p buzz-db -p buzz-relay --all-targets --all-features --
-D warnings`
— passed.
- `cargo test -p buzz-db` — 122 passed, 0 failed, 263 ignored;
source-contract integration test: 3 passed, 0 failed.
- Focused relay compatibility, metric-contract, and readiness tests —
passed.
- `scripts/test-postgres-test-discovery.sh` — passed.
- Full `buzz-relay` package run from the identical tree reached 1,015
passes;
the six media-test failures all stopped in their shared local PostgreSQL
setup with `Sqlx(PoolTimedOut)` because Docker/PostgreSQL was
unavailable.
The same six failed in isolation, while every changed exact test passed.
- Exact implementation head:
`f92910b353086e9edf85918ca5f72190edbbe22f`.
- Exact multi-architecture staging image:
`dev-sha-f92910b353086e9edf85918ca5f72190edbbe22f-run-33607968668-1`
(`sha256:161712c8ed2e265a15df9b63e02248d5973481f875ff129d7d2ae78a09d487a2`).
- Focused staging GitOps PR:
<https://github.com/squareup/builderbot-platform-core-infrastructure/pull/299>
— merged after renderer, inventory, infrastructure test, Kargo, Semgrep,
and
Intersect gates passed; the source/generated-artifact diff was exactly
two
image lines.
- Exact GitHub head reports 47 terminal checks: 35 successful and 12
intentionally skipped. PostgreSQL, unit, lint, security, both server
cross-compiles, backend integration, relay E2E, desktop, mobile, image,
Helm, Semgrep, zizmor, and DCO gates are green.
- Datadog readback identifies two exact-image pods,
`buzz-d79c8d8f7-ckv2l` and `buzz-d79c8d8f7-qzqdp`, in ReplicaSet
`buzz-d79c8d8f7`; both report the full source SHA above.
- Both pods report all eleven allowed pool/operation waiter pairs at
current
zero, with no invalid pair. The acceptance window observed nonzero
success
receipts for readiness, tenant resolution, authorization, subscription
history, event write, and maintenance, and no timeout, error, or
cancelled
outcome. Maximum observed wait was about 101 ms for maintenance and 50
ms
for reader subscription history.
The main **Buzz Startup & Rollout Safety** dashboard now has a live
Stage 2
database row with eight widgets and nineteen fully scoped queries. Final
readback preserved all seven top-level groups, found zero under-scoped
Row 6
queries, and confirmed the tracked-operation waiter boundary in the
panel
descriptions.
Generated with Codex.
---------
Signed-off-by: Ravneet Arora <rarora@squareup.com>
|
||
|
Will Pfleger
|
04babf0265 |
chore(db): drop Phase-A NIP-FI relay-side authority ledger (#7221)
Removes the dead relay-side authority ledger introduced by migrations 0041 and 0042, and the dead `require_attested_key` verifier knob from `buzz-auth`. Both are unreachable by design under NIP-FI spec v2 (#7214, squash `d4420eb47`), which makes OSS Buzz stateless for identity: the relay neither stores nor verifies an authority chain. ## What changes **`migrations/0044_drop_nip_fi_ledger.sql`** Drops all fifteen NIP-FI ledger tables and their trigger functions using `CASCADE` to resolve the circular deferred FK between `identity_bindings` and `identity_lifecycle_history`. Drops proceed in FK dependency order: selectors → history/bindings → enrollment_policies/receipts → parallel drop of auth tables. Restores `community_write_fence_excluded_table` to its pre-0041 body (removes NIP-FI table names from the exclusion array). **`schema/schema.sql`** Removes the NIP-FI section (~1885 lines of tables, functions, and triggers) and updates `community_write_fence_excluded_table` to match. **`crates/buzz-db/src/runtime/migration.rs`** - Updates the `embedded_migrator_contains_consolidated_initial_schema` sanity check: count 43→44, adds 0044 assertion block (verifies `DROP TABLE` statements and absence of NIP-FI names from `schema.sql`). - Removes ~2580 lines of NIP-FI Postgres integration tests (all `#[tokio::test] #[ignore = "requires Postgres"]` from the 0041/0042 behavioral coverage). - Removes the `extract_excluded_table_array` drift check (0042 body no longer matches `schema.sql` by design). - Adds `migration_0044_drops_populated_nip_fi_ledger_cleanly`: runs migrations to 0042, seeds rows in `authorization_operation_receipts` and `authorization_invalidation_domains`, then runs to 0044 and verifies all fifteen NIP-FI tables are absent. **`crates/buzz-auth/src/nip_fi/config.rs`** Removes `require_attested_key: bool` from `IssuerPolicy` — field, constructor parameter, accessor, and its contribution to `derive_assertion_policy_id`. **`crates/buzz-auth/src/nip_fi/verifier.rs`** `parse_nostr_pubkey_claim` no longer takes a `policy` parameter. The `None` (absent claim) arm now returns `Err(VerifierError::ClaimRejected)` unconditionally instead of conditionally on `policy.require_attested_key()`. **`crates/buzz-auth/src/nip_fi/verifier/tests.rs`** - Removes `missing_nostr_pubkey_denies_under_attested_key_policy` (the sole `require_attested_key: true` call site). - Removes `false,` from all eleven `IssuerPolicy::new` call sites. - Injects `nostr_pubkey` by default in `mint_signed_by` (spec v2 requires it unconditionally). - Updates `valid_access_token_verifies` to assert `asserted_key().is_some()`. **`crates/buzz-auth/src/nip_fi/startup/tests.rs` + `jwks/tests.rs`** Removes `false,` from all `IssuerPolicy::new` call sites and adds `nostr_pubkey` to all token-minting helpers. ## Verification - Fresh-DB migration run to head: all migrations apply cleanly in sequence. - Populated-0041/0042-DB migration through 0044: seeds rows in live NIP-FI tables, verifies all fifteen are dropped without error. Closes the dead-code inventory item from the spec-v2 cleanup plan (channel `48374f48`). Follows #7214. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> |
||
|
ravarora2
|
beb76406c1 |
feat(relay): add detailed readiness metrics (#7149)
## Summary
This PR makes relay readiness failures diagnosable without weakening the
existing fail-closed readiness contract. It distinguishes Postgres pool
acquisition from query execution, Redis pool acquisition,
deletion-catalog validation, and the overall two-second deadline;
exports a bounded Prometheus contract for rollout dashboards; and fixes
the concurrency, shutdown, and listener-boundary semantics needed for
those signals to be trustworthy.
## Why
The previous `/_readiness` implementation exposed only an aggregate
ready/not-ready result. During a rollout, operators could not tell
whether a pod was blocked on:
- acquiring a Postgres connection;
- executing the Postgres readiness query;
- acquiring a Redis connection;
- validating the deletion catalog; or
- the shared readiness deadline.
Adding metrics to the existing handler also exposed three correctness
hazards that this PR resolves:
1. the same handler is mounted on both the public application listener
and the private Kubernetes health listener, so public requests could
otherwise distort rollout telemetry;
2. concurrent probes can finish out of order, allowing an older result
to overwrite newer current-state gauges; and
3. a probe started before SIGTERM can finish afterward, incorrectly
return `200 ready`, and resurrect ready gauges while the process is
draining.
## Behavior
### Readiness evaluation
- Postgres, Redis, and deletion-catalog checks still run under one
shared two-second deadline.
- Postgres distinguishes pool acquisition timeout/error from query
timeout/error.
- Redis distinguishes pool acquisition timeout/error. This does not
claim a Redis command round trip.
- The deletion catalog distinguishes operation timeout/error.
- Multiple failures are reported as `multiple_dependencies_failed`;
exhaustion of the shared deadline is reported as `overall_timeout` when
no more specific completed outcome wins.
- Readiness remains fail-closed: every dependency must succeed for `200
{"status":"ready"}`.
### Ordered publication and shutdown
`ReadinessCoordinator` is process-owned and uses one mutex as the
linearization point for probe generations, current-state publication,
and terminal shutdown.
- Every completed dependency attempt may contribute its truthful counter
and duration observation.
- Only the newest admissible probe generation may publish current-state
gauges.
- An older, slower probe cannot overwrite a newer probe's gauges.
- `begin_shutdown()` and probe commit serialize through the same
coordinator.
- Once shutdown commits, an in-flight probe cannot return ready or
publish ready/current dependency gauges, even if its dependency work
later succeeds.
Shutdown without dependency evaluation records only:
- `buzz_readiness_checks_total{reason="shutting_down"}`; and
- `buzz_readiness_state{check="overall"} = 0`.
It does **not** fabricate dependency failures, dependency state changes,
or zero-duration latency samples. If shutdown wins after an in-flight
evaluation actually ran, those completed dependency attempts may remain
as attempt telemetry, but they cannot overwrite shutdown-dominant
current state.
### Listener and response contract
- The private health listener's `/_readiness` route is the sole
authority for rollout readiness telemetry.
- The public application listener retains `/_readiness` for
compatibility, evaluates the same dependencies, and preserves the
existing response shape, but it emits no `buzz_readiness_*` metrics.
- Ready responses remain `200 {"status":"ready"}`.
- Shutdown responses remain `503 {"status":"shutting_down"}`.
- Failed private-health responses include the bounded `reason` plus
`postgres`, `redis`, and `deletion_catalog` booleans.
- Failed public compatibility responses retain the dependency booleans
but omit the new detailed reason.
- No header, query parameter, path value, or other request-controlled
value becomes a metric label.
## Prometheus contract
The final schema is intentionally capped at **99 raw Prometheus series
per pod**.
| Metric | Type | Labels | Raw series/pod |
|---|---|---|---:|
| `buzz_readiness_checks_total` | counter | `reason` | 12 |
| `buzz_readiness_dependency_checks_total` | counter | `dependency`,
typed `outcome` | 11 |
| `buzz_readiness_check_duration_seconds` | histogram | `check` | 72 |
| `buzz_readiness_state` | gauge | `check` | 4 |
| **Total** | | | **99** |
### Closed label sets
`reason`:
```text
ready
shutting_down
postgres_pool_timeout
postgres_pool_error
postgres_query_timeout
postgres_query_error
redis_pool_timeout
redis_pool_error
deletion_catalog_timeout
deletion_catalog_error
overall_timeout
multiple_dependencies_failed
```
Valid `dependency` / `outcome` pairs are enforced by typed enums:
- `postgres`: `success | pool_timeout | pool_error | operation_timeout |
operation_error`
- `redis`: `success | pool_timeout | pool_error`
- `deletion_catalog`: `success | operation_timeout | operation_error`
`check` is `overall | postgres | redis | deletion_catalog`.
The readiness histogram has 15 configured buckets concentrated around
the two-second deadline:
```text
0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5,
0.75, 1.0, 1.25, 1.5, 1.75, 2.0, 2.5, +Inf
```
The contract deliberately removes the redundant overall `result` label
and the histogram `outcome` label. Pod, ReplicaSet, version, rollout,
raw error, SQL, URL, tenant, user, community, pubkey, and
request-controlled values are prohibited application labels;
infrastructure enrichment can supply deployment identity outside the
application metric.
## Postgres production seam and CI
The real `Db::readiness_check` path now supports deterministic
production-seam testing while preserving the same acquisition/query
implementation used by the relay. The isolated PostgreSQL lane
automatically discovers and executes three ignored integration tests
covering:
- a held sole connection causing pool timeout, followed by recovery
after release;
- closed-pool acquisition error;
- acquisition success followed by query timeout;
- classified query error;
- cancellation while waiting for a connection;
- cancellation during an in-flight query; and
- eventual pool recovery with waiter/in-flight state balanced.
This prevents the central SQLx/Postgres behavior from being merely
compiled but never executed in CI.
## Tests and verification
- `cargo fmt --all -- --check`
- `./scripts/test-postgres-test-discovery.sh`
- complete `cargo test -p buzz-db`
- focused `cargo test -p buzz-relay readiness`
- real-router `cargo test -p buzz-relay real_health_route_`
- controlled PostgreSQL execution of all three ignored readiness tests
- production health router -> real `GET /_readiness` -> Prometheus
render assertions
- public-router requests proving zero rollout telemetry
- deterministic out-of-order A/B probe tests
- SIGTERM-during-probe tests proving shutdown dominance
- exported metric name/type/exact-label/bucket assertions
- exported raw-series allowlist and 99-series ceiling assertion
The route-to-scrape regression test uses the production Prometheus
builder and verifies the `2`, `2.5`, and `+Inf` readiness buckets. It
fails if the health recording call, health route, public/health
boundary, generation fence, shutdown fence, or bucket override is
removed.
## Risk assessment
**Medium.** This changes the live readiness publication path and adds
synchronization around probe commit/shutdown. The risk is bounded by:
- preserving the existing dependencies and shared two-second deadline;
- keeping readiness fail-closed;
- using a short, process-local mutex only at begin/commit linearization
points, not across dependency awaits;
- retaining the public compatibility endpoint while isolating its
telemetry;
- enforcing typed, low-cardinality labels and an exported series
ceiling; and
- covering the production router, Prometheus exposition, real PostgreSQL
seam, concurrency ordering, and shutdown races.
## Operational notes
- Dashboards should treat `buzz_readiness_state` as the latest sampled
current state, not as an event stream.
- `shutting_down` should be excluded from dependency-failure alerts
because no dependency failure is implied.
- Do not use `default_zero()` for missing current-state data;
missing/stale is unknown, not healthy.
- Use histogram buckets, heatmaps, max, or average until the Datadog
distribution metadata confirms percentiles are enabled; do not title a
widget p95 before that live readback.
## Non-goals
This PR does not add the broader process-startup lifecycle,
worker/listener supervision, shutdown coordinator, WebSocket/huddle
handoff, client recovery telemetry, dashboard mutations, Datadog
configuration changes, or deployment changes. Those remain separate
follow-up work.
## References
- [Staging dev relay image
runbook](https://github.com/block/buzz/blob/main/docs/staging-dev-relay-images.md)
- Readiness telemetry contract: `deploy/charts/buzz/README.md`
---------
Signed-off-by: Ravneet Arora <rarora@squareup.com>
|
||
|
Tom Brow
|
42b42447b0 |
feat(mobile): prepare buzz-push-gateway for deployment (#7158)
A few things to prepare for deploying the Buzz push gateway as implemented in #6269: - Cut `buzz-push-gateway` chart `0.2.0` from the current config (certificate auth, dogfood client) - Datadog-compatible metrics - Align the desired schema, metrics docs, and production ingress guidance with the current dogfood deployment contract ## Links - [BUZZ-17: Publish the MVP-compatible gateway chart](https://linear.app/squareup/issue/BUZZ-17/publish-the-mvp-compatible-gateway-chart) - [BUZZ-26: Add Datadog-compatible push-gateway metrics and correct source drift](https://linear.app/squareup/issue/BUZZ-26/add-datadog-compatible-push-gateway-metrics-and-correct-source-drift) --------- Signed-off-by: Tom Brow <tomb@block.xyz> Co-authored-by: Codex <noreply@openai.com> |
||
|
Luke Tornquist
|
bd73490418 |
ci: run PostgreSQL tests in isolated lane (#6730)
## Why PR #6660 introduced four PostgreSQL-backed persistence tests and a focused CI selector, but an exact test-name list does not automatically cover future database tests. Broad ignored-test execution also exposed shared-schema races and ambiguity between desired-state and migration-applied schema expectations. ## What - Establish a discoverable convention: PostgreSQL unit modules use `postgres_tests`, PostgreSQL integration binaries use a `postgres_` prefix, and unrelated external-infrastructure tests use an `external_infra_` prefix. - Add a dedicated nextest PostgreSQL profile and archive-backed CI job covering the relevant crates without enumerating test names. - Create a run-scoped desired-state source database and a unique PostgreSQL database per test process for parallel-safe isolation, with cleanup on success, failure, or interruption. - Route destructive migration and migration-parity tests to clean `template0` databases while desired-state tests clone the desired-state database. - Document discovery, schema modes, required role privileges, and a bounded portable local runner. - Keep the existing infrastructure-free unit-test jobs unchanged. This is intentionally limited to test and CI harness behavior. It does not move production database code, change persistence semantics, or implement issue #20's broader shared test-utility refactor. ## Risk Low production risk because all changes are confined to tests, CI configuration, documentation, and test harness scripts. Remaining operational risks: - Cleanup retries `dropdb --force` five times across roughly four seconds. Exhaustion warns with the database name but deliberately does not mask the test result; individual PostgreSQL diagnostics are suppressed. - The seven-package boundary is duplicated between the runner and CI archive and must remain synchronized if PostgreSQL tests move to a new crate. - Portable SHA fallback branches were exercised on Linux; no macOS Blox workstation was needed for this Linux CI artifact. - GitHub's unchanged generic Unit Tests job currently fails while cold-linking `buzz-voice` because `sherpa-onnx-c-api` is absent. The branch changes no voice/build/toolchain inputs; the exact parent previously passed that job, and Blox passes the same infrastructure-free suite (1,450/1,450). A prior-head one-job retry and the final-head run both reproduced the hosted-runner failure. ## Verification Author Blox workstation `2020088`: - `cargo fmt --all -- --check` - `just clippy` - `just test-unit`: 1,450 passed; PostgreSQL tests remain skipped in fast jobs - PR #6660 focused tests: 4/4 passed through the final runner - Full lane: 282/282 passed in 10.087s with a non-superuser role limited to `CREATEDB`, `CREATEROLE`, and `pg_read_all_stats` - 272 desired-state tests - 10 migration-applied tests - 6 explicitly filtered external-infrastructure tests - Cleanup fault injection: a deliberately underprivileged 282-test run produced the expected 277 passes and 5 failures, and the post-run catalog audit found zero lane databases after retry cleanup. The restored successful run also left zero lane databases. GitHub exact final head `92c231e48f299ea2af23817763f5ae80ac013d68`: - [PostgreSQL Tests](https://github.com/block/buzz/actions/runs/32789030203/job/97630827823): 282/282 passed across 10 binaries in 98.886s; 1,240 skipped, including 6 via the profile filter - The shared relay/PostgreSQL archive predecessor completed successfully - [Unit Tests](https://github.com/block/buzz/actions/runs/32789030203/job/97626930420): unrelated `sherpa-onnx-c-api` native-link failure described above - Independent exact-head review on separate Blox workstation `2022762`: no substantive findings after all initial findings were addressed. The reviewer independently verified cleanup retry behavior, exit-status preservation, shell portability, syntax, and a clean exact-head worktree. ## References - Stacked on #6660 at exact head `561de54be4d9c2b622b7c2aa5b61bc3068f47e2a` - https://github.com/TheSentinel454/buzz/issues/20 - https://github.com/TheSentinel454/buzz/issues/20#issuecomment-5398526792 Generated with Codex. ## Update — August 24, 2026 review follow-up - Centralized the repeated PostgreSQL test URL resolution in crate-local, test-only helpers without introducing the broader shared utility refactor from issue #20. - Restored descriptive hybrid/Redis test function names. Structural `external_infra_*_tests` modules now own exclusion, and the nextest filter only recognizes module path segments. - Added a three-second source guard that scans every Rust file and fails CI when an ignored PostgreSQL test would be omitted or an external-infrastructure test would be included. Fixture tests cover accepted modules/binaries and both failure modes. - Removed the two implementation-plan documents. Performance profiling on Blox workstation `2027352` (same prebuilt archive and cargo-nextest 0.9.143 for every comparison): - Current per-test database model, 8 workers: 10.15–10.40s across three runs. The 283 `createdb` calls and 284 `dropdb` calls consumed 28.85–29.46 aggregate seconds, about 39–40% of aggregate test-process duration. - Reusing one database per worker without cleanup: 6.55–6.84s, about 35% faster, but all three runs failed because global matcher-queue tests inherited a quiesced community from an earlier test; the failing test varied with scheduling. - Reusing workers with `TRUNCATE … RESTART IDENTITY CASCADE`: all 282 tests passed, but total time regressed to 10.47–10.67s. The 265 truncate calls consumed 24.34–25.45 aggregate seconds, so truncation merely replaced most clone/drop cost. - Raising concurrency to 16 workers reduced one isolated-database run to 8.79s, but 24 workers exposed a cluster-global `pg_stat_activity` race. The profile remains at the proven-safe 8 workers. Decision: retain unique per-test databases. Harness cleanup uses `dropdb --if-exists --force`; it drops the database rather than truncating tables or deleting rows. Verification at local branch head `60086c1f6ee34a193c342255a5e8e6293b60e988`: - Discovery inventory: exactly 282 intended tests; all 6 hybrid/Redis tests present and structurally excluded. - PR #6660 focused persistence tests: 4/4 passed. - Full PostgreSQL lane: 282/282 passed three times; 1,240 skipped each run; wall time 9.72–9.96s without profiling shims. - Non-ignored tests with a desired-state database: 1,234/1,234 passed; one mesh-demo timeout from the first run passed immediately in isolation. - `cargo fmt --all -- --check` and all-target/all-feature clippy for `buzz-db`, `buzz-deletion`, and `buzz-relay` passed. Final independent re-review at exact head `42097c0136ab3fa8f0efe5720ae829c04575b9b6` on separate Blox workstation `2028455`: no substantive residual findings. The reviewer independently exercised ordinary strings, zero/one/three-hash raw strings, line/block comments, and string-contained lookalikes, then reran the full 353-file scan, shell syntax, Python AST parsing, and diff check. Independent review found that the source guard's initial regular expression could miss raw-string ignore reasons and treat a commented attribute as real. Final head `42097c0136ab3fa8f0efe5720ae829c04575b9b6` parses valid ordinary/raw Rust string literals only at attributes found in comment-sanitized source; regression fixtures cover both cases. The fixture suite, Python compilation, shell syntax, diff check, and full 353-file repository scan pass on Blox. ## Update — August 25, 2026 restack PR #6660 merged, so this follow-up was rebased from its former exact parent onto current `origin/main` (`8d2d0ff5ad42733e9949442c4b6358d0ba87f9a8`). The final candidate head is `05dcc2ab28948e3ab79bb44839a69f2ba44648a2`; the PR no longer carries #6660's pre-squash history. Fresh exact-head verification on Blox workstation `2028572`: - Discovery guard passed across 353 Rust files; inventory remained exactly 282 intended PostgreSQL tests. - PR #6660 focused persistence tests: 4/4 passed. - Full PostgreSQL lane: 282/282 passed three times; 1,243 skipped each run; wall time 9.70–9.87s. - `cargo fmt --all -- --check` and all-target/all-feature clippy for `buzz-db`, `buzz-deletion`, and `buzz-relay` passed. ## Update — August 25, 2026 final hosted verification - [GitHub PostgreSQL Tests](https://github.com/block/buzz/actions/runs/32801679226/job/97667835080): 282/282 passed in 53.787s; 1,243 skipped. - The final guard scanned all 353 Rust files successfully before the hosted lane ran. - The shared relay/archive prerequisite and infrastructure-free Unit Tests job both passed at exact final head `42097c0136ab3fa8f0efe5720ae829c04575b9b6`. --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Signed-off-by: tornquist <tornquist@squareup.com> |
||
|
Will Pfleger
|
9ab163190b |
feat(db): add NIP-FI identity and final-admission schema foundation (#6994)
PR 2 of the NIP-FI plan: the schema foundation. Establishes the durable server-side identity ledger and final-admission surface that the runtime phases build on. All of Phase A's migrations live here; later phases own their own deltas. Depends on nothing — PR 1 (#6776, merged) owned zero migration files. This PR's relations are shaped to store exactly what PR 1's verifier produces: issuer-qualified identity and the four denial classes. They meet in a later PR that writes a verified assertion into these tables in one transaction. ## Two internally-ordered migrations - `0041_nip_fi_identity_foundation.sql` (migration A) — core identity + base-lifecycle relations (5 tables): issuer-qualified `(iss, sub)` bindings, lifecycle history/selectors, enrollment policies, and operation receipts. Applies cleanly to current `main`. - `0042_nip_fi_authorization_foundation.sql` (migration B) — the final-admission surface (10 tables): authorization events + capacity, admission results, replay/receipt guards, audit, invalidation domains/floors, protected-object authority, authority epochs, and restore version deltas. Applies to A's resulting state. Fifteen NIP-FI relations total, zero dangling foreign keys. Identity is issuer-qualified throughout — no single-global-issuer assumption in any relation, no `Block`-hardcoding. A single deployment may run one issuer; that is config, not schema. ## Durable, immutable ledger posture All 15 relations are append-only (immutable `no_delete`/`no_truncate` triggers) and carry `community_id` as provenance, not ownership. Both migrations widen the single SQL source of truth `community_write_fence_excluded_table` so the relations are never fence-attached, never purged on community deletion, and never counted as tenant-scoped drift by the deletion control plane's exact-set catalog check — the same posture main already applies to `product_feedback` and `rate_limit_violations`. `schema/schema.sql` keeps one consolidated definition of that function whose exclusion array byte-matches `0042`, guarded by a parity assertion so a future consolidation cannot silently drop NIP-FI relations from the ledger. This makes a tenant's identity/authorization ledger survive community deletion, per the spec's `FI-INV-02` (durable binding) and `FI-INV-03` (tombstone monotonicity) and `NIP-FI.md`'s "durable server state" ruling. `communities(id)` FK never dangles: community rows become permanent tombstones, never hard-deleted. ## Authorization shape and cardinality contracts Authenticated `OperatorDenied` events (`actor_kind` 1–3, non-null `request_fingerprint`) carry a null `semantic_fingerprint` and commit without a denial-attempt row. The denial-attempt cardinality and shape guards are scoped to unresolved pre-auth kind-9 events (`actor_kind = 4`). Applied and no-op lifecycle receipts (`outcome_code IN (1, 3)`) require exactly one mapped success-transition event; denied lifecycle receipts (`outcome_code = 2`) require zero events from the complete core lifecycle success-transition class (kinds 1, 2, 3, 6: enrolled, revoked, rotated, retired) — any such event paired with a denied receipt would record a transition that never occurred. ## Mined vs. new Re-cut from Franco's #1476 (`0029`/`0030`) and Cea's #4772 committer schema, re-cut along FK topology and renumbered above the live `main` tip. The buzz-auth core of #1476 is Cea-authored; `Co-authored-by` reflects verified per-commit authorship of the mined schema. Zero Rust/`deletion.rs` edits — the migration-only exclusion widening keeps `EXPECTED_SCOPED_TABLES` untouched. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz> Co-authored-by: Cea Stapleton Cordasco <261786559+cea@users.noreply.github.com> Co-authored-by: Hayt <9e1c23a3fd83f61da34420e4e88ff1b16e45cafcc0cd9019eb07d4ecfa8ca9b0@buzz.block.builderlab.xyz> |
||
|
Luke Tornquist
|
3ed623bb21 |
feat(db): configurable writer session timeouts (lock, idle-txn, statement) (#6229)
## Why A wedged relay boot pod holding a relation lock can park every other writer in the fleet behind it: DB load pins at pool capacity in `Lock:relation` waits while CPU stays flat, and nothing server-side releases the lock until the holder dies. We hit exactly this in production — ~1,400 sessions queued behind one crash-looping pod's boot transaction for ~20 minutes until kubelet killed the container. ## What Applies session-level Postgres timeouts to every **writer** connection inside the existing single `after_connect` hook in `buzz-db`, all env-tunable through the same `Config::from_env → DbConfig` path as the existing pool-size knobs: | Env var | GUC | Default | Effect | |---|---|---|---| | `BUZZ_DB_LOCK_TIMEOUT_MS` | `lock_timeout` | 5000 | statements waiting on any lock fail fast instead of parking behind a wedged holder | | `BUZZ_DB_IDLE_TXN_TIMEOUT_MS` | `idle_in_transaction_session_timeout` | 60000 | reaps wedged clients idling inside an open transaction while holding locks | | `BUZZ_DB_STATEMENT_TIMEOUT_MS` | `statement_timeout` | 0 (off) | opt-in runaway-statement cap; off by default because startup migrations/backfills legitimately run long statements | `0` disables a timeout (Postgres semantics) and deliberately passes through the env parsing — unlike the pool-size knobs where `0` falls back to the default. The reader pool is untouched: replica sessions never take contended locks and already fail acquire in 150 ms. Deployers tune these via plain env vars (`.env`, or `relay.extraEnv` in the Helm chart) — no code changes needed. ## Behavior change to note With the 5 s default `lock_timeout`, a boot-time migration or backfill that waits >5 s on a lock now errors (surfacing in logs / crash-looping the pod) instead of stalling silently. That is the intended visible-failure-over-fleet-stall tradeoff; deployers with slow contended migrations can set `BUZZ_DB_LOCK_TIMEOUT_MS=0`. ## Testing - `cargo test -p buzz-db -p buzz-relay` — buzz-db green; buzz-relay has 9 failures that also fail on clean `main` in this environment (api::admin/api::media/mesh_demo — unrelated, pre-existing). - New config test covers override / `0`-passthrough / invalid-fallback for all three env vars. - Extended the existing `writer_pool_safety_hook_is_single_and_composed` source-shape test so the timeouts can't drift out of the single `after_connect` hook (SQLx replaces hooks — a second hook would silently disarm the floor guard). - `cargo fmt --check` and `cargo clippy --all-targets` clean for the touched crates. Closest existing PR/issue: none found. --- **Update Aug 28, 17:06 EDT:** Rebased onto `main` at `a3730784fc` and addressed the latest correctness review. - Ported the timeout policy onto the refactored `buzz-db::runtime` pool constructor and kept the shared env overlay for relay, admin, deletion, and audit writers. - Migration/schema-destruction connections now disable `lock_timeout` and `statement_timeout` for their intentional long wait/DDL path. This supersedes the earlier “Behavior change to note”: contended boot migrations wait for the current migration owner rather than crash-looping after five seconds. - The audit worker now preserves and retries the same entry on PostgreSQL `55P03` lock timeouts, using exponential backoff capped at one second. Other database errors retain the existing terminal error behavior, and retries emit `buzz_audit_log_lock_retries_total`. - Added CI-backed PostgreSQL regressions for writer GUC installation/migration exemption, audit-pool lock timeouts, and worker recovery. The worker regression holds the real audit advisory lock past `lock_timeout`, observes a retry, releases the lock, and proves the original entry is appended exactly once. Current verification supersedes the earlier testing notes: workspace Rust clippy passed with warnings denied; all nine infrastructure-free backend unit-test lanes passed; all three focused PostgreSQL regressions passed against PostgreSQL 17; formatting, diff checks, file-size guards, and desktop frontend checks passed. The Linux Blox workstation could not run the unrelated Tauri native lane because `glib-2.0` is absent, so that platform check is left to PR CI. --- **Update Aug 31, 11:09 EDT:** Rebased onto current `main` at `c3132c3ee9` and reran the requested audit-lock contention scenario on Blox at head `896c3fe9ed`. - `git range-diff` reports both PR commits unchanged by the rebase; the branch remains two commits and the worktree is clean. - `cargo fmt --all -- --check` and clippy with warnings denied passed for `buzz-db`, `buzz-relay`, `buzz-admin`, and `buzz-deletion`. - All three focused PostgreSQL 17 regressions passed: writer session timeout/migration exemption, audit writer timeout bounds, and audit worker recovery of the original entry exactly once. - Live protocol verification used a head-built relay and CLI, native PostgreSQL 17/Redis, `BUZZ_DB_LOCK_TIMEOUT_MS=300`, and an eight-second hold on the community audit advisory lock. The real message was accepted and persisted once while the lock was held; its audit-row count remained zero during contention while retries accumulated. After release, exactly one `event_created` audit row appeared and remained exactly one after an additional two-second duplicate check. The run recorded nine lock-timeout retries, zero audit failures, and event ID `7f8c4ffae28e78555fcf2d56396d6e6c01b3712e5411288dc79e9a54af9d9444`. Generated with Codex --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> |
||
|
Tom Brow
|
c432a111ca |
feat(mobile): push notifications MVP (#6269)
This PR implements MVP, iOS-only,
[NIP-PL](
|
||
|
Luke Tornquist
|
a3730784fc |
refactor(db): extract domain stores from database runtime (#6987)
## Summary Finish the remaining database-store extraction tracked by [TheSentinel454/buzz#2](https://github.com/TheSentinel454/buzz/issues/2) in one reviewable PR. This consolidates the previously stacked domain slices after #6782 merged. It preserves the runtime/store boundary established by #6660, #6668, #6700, and #6782 while separating database runtime infrastructure from domain-owned persistence: - `runtime/` owns pool construction and sizing, writer/reader routing, read sessions and route proofs, transaction infrastructure, observability primitives, replica fencing, health support, migrations, and cross-cutting runtime tests. - `store/` owns domain records, SQL, row parsing, locks and invariants, `Db` domain methods, focused tests, and logical-operation datastore spans. - `lib.rs` remains a 57-line compatibility facade that preserves existing crate-root paths and `Db` method signatures through re-exports. Domain coverage includes API tokens, authentication allowlists, reminders, event queries, threads, reactions, feeds, users and DMs, push, workflows/runs/approvals, relay membership and invites, product feedback, moderation/admin moderation, relay admin actions/operators, git repositories, archived identities, usage, partition maintenance, deletion, channel membership inherited from merged #6782, and the final runtime/store layout. The branch has been rebased onto current `main`. Database changes that landed there were incorporated rather than overwritten: `relay_admin_actions.rs` and `relay_operators.rs` now live under `store/`, their 27 public `Db` wrappers and existing behavior remain intact, and every wrapper has exactly one fixed-name datastore span. Concurrent changes to migration, moderation, admin moderation, and error handling are also retained. ### Exact base and head - Base: `main` at `ed11c8d8bf0a17402be5cf243724f89471530d2f` - Head: `codex/issue-2-store-extraction` at `be24430472d1a87ac5c0d6026c620cd6caea3537` ### Related issue - Structural tracker: [TheSentinel454/buzz#2](https://github.com/TheSentinel454/buzz/issues/2) - Domain trackers: [#6](https://github.com/TheSentinel454/buzz/issues/6), [#7](https://github.com/TheSentinel454/buzz/issues/7), [#12](https://github.com/TheSentinel454/buzz/issues/12), [#13](https://github.com/TheSentinel454/buzz/issues/13) - Acceptance trackers: [#17](https://github.com/TheSentinel454/buzz/issues/17), [#19](https://github.com/TheSentinel454/buzz/issues/19) This supersedes #6783, #6784, #6787, #6788, #6789, #6792, #6820, #6794, #6796, #6797, #6798, #6799, #6804, #6805, #6806, #6808, #6809, #6811, #6812, #6813, #6814, #6815, and #6890. Their discussions remain available for review history. ### #17 / #19 acceptance - Preserves the metric names, fixed labels, transaction/lock timing boundaries, and privacy/cardinality constraints introduced by #6700. - Keeps exactly one datastore span per public logical operation, including the 27 relay-admin wrappers added on `main`. - Removes `store_ownership.rs`; physical ownership and focused source guards now enforce the boundary directly. - Leaves no `impl Db`, domain SQL, focused domain test group, or datastore span in `lib.rs`. - Preserves existing public paths such as `buzz_db::channel`, `buzz_db::event`, and `buzz_db::workflow` through crate-root re-exports while keeping internal `runtime` and `store` namespaces private. ### Non-goals - No SQL, schema, locking, transaction, retry, timeout, or client-visible behavior changes. - No generic store traits, domain handles, broad `PgExecutor` migration, new store crate, raw pool accessor, or broader directory reorganization. - No tracker issues are closed by this PR. ### Risk The cumulative diff is large but structural. Risk is primarily module-path, ownership, or conflict-resolution drift. It is mitigated by preserving public re-exports, comparing the newly moved `main` implementations to their upstream source, source guards, touched-crate compilation, PostgreSQL-backed test coverage, and an independent exact-head review on a separate clean Blox workstation. ### Testing Author workstation `buzz-tornquist-pr-6987-rebase`, rebased branch ending at exact head `be24430472d1a87ac5c0d6026c620cd6caea3537`: - `cargo fmt --all --check` - `cargo clippy -p buzz-db -p buzz-relay --all-targets -- -D warnings` - `cargo test -p buzz-db --lib` — 113 passed, 240 PostgreSQL tests intentionally ignored - `cargo test -p buzz-db --test observability_source` — 2 passed - PostgreSQL-backed `buzz-db` coverage under native PostgreSQL — 235 passed in the shared serial run; the five shared-state/config-sensitive cases passed as isolated reruns against fresh schemas, including the two owner-limit tests with their fixture's `BUZZ_MAX_COMMUNITIES_PER_OWNER=3` - `cargo test -p buzz-relay --lib -- --test-threads=1` under native PostgreSQL/Redis — 991 passed; the three current-month partition-sensitive identity-archive cases passed after provisioning the August 2026 test partition; 87 infrastructure-marked tests remained ignored - Source/diff guards — relay-admin implementation bodies match current `main`; all 27 public wrapper signatures are retained; exactly one datastore span wraps each wrapper; `lib.rs` has zero `impl Db` blocks and zero datastore spans; no duplicate top-level relay-admin modules or `store_ownership.rs`; `error.rs` matches current `main` Independent clean review workstation `buzz-tornquist-pr-6987-review`, detached at exact head `be24430472d1a87ac5c0d6026c620cd6caea3537`: - `cargo fmt --all --check` - `cargo clippy -p buzz-db -p buzz-relay --all-targets -- -D warnings` - `cargo test -p buzz-db --lib` — 113 passed, 240 ignored - `cargo test -p buzz-db --test observability_source` — 2 passed - Exact-head ownership/re-export/instrumentation audit — no remaining actionable findings --------- Signed-off-by: OpenAI Codex <codex@openai.com> Signed-off-by: tornquist <tornquist@squareup.com> Co-authored-by: OpenAI Codex <codex@openai.com> |