jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-10-27 12:40:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,879 Commits 1,195 Branches 231 Tags
5dbaa4d32dccdd1ef629bfbe6d2a9e577b67634c
Commit Graph

16 Commits

Author SHA1 Message Date
Ian Oberst
a8fe2b612c feat(db): advance an empty partition catch-all under bounded locks (#7993) 
## Summary

Adds bounded, serialized advancement of an empty right-edge partition
catch-all for `events` and `delivery_log`. The repaired catalogs
currently end in an empty catch-all from 2027-01-01, and the merged
manager (#6515) only audits them. Without this change, January writes
land in the catch-all, after which only an operator can move them.

When `BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED=true`, each relay keeps six
dedicated monthlies after the current month. It does this at startup and
on every audit interval by replacing the empty catch-all with canonical
monthlies and a new canonical `<table>_p_future` after the horizon. From
today's catalogs the first run creates January–March 2027 and moves the
catch-all to April 2027.

## Safety design

- **Refuse before locking.** The following return `operator_required`
without taking any table lock:
  - a populated catch-all, or one whose occupancy is unknown;
- a catch-all with child-only triggers, which the replacements would not
inherit;
  - a misaligned bound;
  - a `DEFAULT`, anomalous or pending-detach child;
  - a name collision.
- **One winner.** A schema-scoped `pg_try_advisory_xact_lock` is taken
before any DDL. Losers report `skipped_locked` and never issue DDL. The
winner re-plans from a fresh audit before taking any table lock and
checks for collisions. A runner that lost a race therefore sees the
committed layout as a no-op, not as colliding names.
- **Writer lock order.** First `ONLY` the parent is locked `ACCESS
EXCLUSIVE` under a 2 s `lock_timeout`. Then each foreign-key counterpart
(today only `communities`) is locked in OID order `SHARE ROW EXCLUSIVE
NOWAIT`, then the catch-all `NOWAIT`.
- The counterparts are read again once the parent is locked, and that
set is the one locked. Any FK DDL on or to the parent conflicts with the
parent lock, so the set can't change after that. An FK committed while
maintenance waited is therefore locked `NOWAIT` too, so attaching never
waits on it under the parent lock. The read before the lock remains as a
cheap early refusal.
- This is the order an insert takes locks (parent, then the FK check on
`communities`), so maintenance cannot deadlock with ingest. The internal
manual runbook used to lock `communities` first; it now uses this order
too.
- `SHARE ROW EXCLUSIVE` is the strongest lock the DDL takes on
`communities`; measured on PG 16 and 17, the catch-all `DROP` takes
none, as long as the catch-all's only foreign keys are those inherited
from the parent. A catch-all with its own foreign key, or one that
another table's foreign key references, is refused: once before any
lock, and again after the catch-all lock, where the check is
authoritative. Readers and other writers' `FOR KEY SHARE` checks
proceed. A concurrent `communities` writer makes the run fail fast with
`lock_timeout`.
  - A `40P01` deadlock victim is also reported as `lock_timeout`.
- Parents that are referenced by a foreign key, or that reference an
inherited table, are refused, including when such a key commits while
maintenance waits on the parent.
- **Re-plan under lock.** The catalog is re-audited under those locks.
The transaction proceeds only if the plan is unchanged.
- **Bounded.** `statement_timeout` 5 s,
`idle_in_transaction_session_timeout` 5 s, and a 10 s client deadline
per table. A deadline-abandoned connection is closed, never returned to
the pool, and the server rolls back anything uncommitted. If the
deadline fires after `COMMIT` was sent, the change may have landed; the
next audit reports the truth. The orphaned backend keeps its locks until
its in-flight statement ends, at most `statement_timeout`.
- **Verify before commit.** The changed catalog is re-audited inside the
transaction for bounds, canonical kinds, trigger parity and catch-all
emptiness, and serving safety must not regress. It also checks that
every new child has valid, ready clones of each parent index and
validated clones of each PK/unique/FK constraint.

- **Audits tolerate a concurrent advance.** Catalog rows come from the
statement snapshot, but `pg_get_expr` renders from current catalog
state. So an audit that overlaps another relay's `DROP` of a catch-all
can see a partition that no longer exists. The read-only audit detects
this (a NULL bound, or `42P01` from the emptiness probe) and retries up
to twice with a fresh snapshot.
- **Fail fast on audit probes.** If another relay's audit is probing the
catch-all's emptiness, the catch-all `NOWAIT` lock fails and the run
reports `lock_timeout`. The parent lock is then released immediately,
and the next runner or interval completes the advance.

Uncovered-month creation (`BUZZ_PARTITION_MANAGER_CREATE_ENABLED`,
default `true`) now runs through the same locked transaction, replacing
the unlocked per-month `CREATE`. That flag remains the kill switch for
all automatic partition DDL: with it off, advancement is disabled too.
With advancement disabled (the default) the periodic loop stays
read-only and startup behaves as before, apart from the locking.

A runner that planned DDL always re-audits before returning, even when a
peer did the work, so it never reports the pre-maintenance layout. A
serialized re-plan that finds only disabled work reports
`skipped_disabled`.

## Behavior changes

- The relay audit horizon and the `buzz-admin partition-audit` default
are now 6 months (`PARTITION_MANAGER_MONTHS_AHEAD`), up from 3. Until
advancement is enabled, months already covered by the catch-all make
`degraded` true. That correctly reports a runway under six months.
- New metric `buzz_partition_maintenance_runs_total{table,outcome}` with
outcomes `noop | skipped_disabled | created | advanced | skipped_locked
| operator_required | lock_timeout | deadline | error`.
`buzz_partition_create_attempts_total` keeps its per-month `created |
skipped_covered | error` labels. `error` there still means a DDL or
collision failure; `lock_timeout` and `deadline` runs add nothing to it,
and only `buzz_partition_maintenance_runs_total` records them.
- `ensure_future_partitions` (free function and `Db` method) is removed;
`maintain_partitions` is the one entry point.
- New advisory-lock label `lock_type="partition_maintenance"`.

## Rollout

Enable `BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED` per environment in BPCI:
`bb-block-staging` first, then production. The staging canary proves a
real advance because the six-month horizon already extends past January.
Fallback: rerun the internal manual runbook, which uses the same
parent-first lock order as of 2026-10-05. It keeps `ACCESS EXCLUSIVE` on
the counterparts, because its detach/attach can drop FK triggers on
them.

## Testing

- `cargo test -p buzz-db --lib partition`: 17 unit tests. The 10 new
ones cover:
- plans: repaired layout, policy gating, no-op, refusal states including
child-only triggers, December→January;
  - the create kill switch also stopping advancement;
  - the month cap at exactly 120;
- per-month metric outcomes for transient failures versus DDL failures;
  - the mid-audit-drop retry classifier.
- `cargo test -p buzz-db --lib partition -- --ignored` (PostgreSQL 17):
48/48 on 9 consecutive runs, including 17 new tests:
  - advance to the canonical layout, with routing and idempotent rerun;
  - replacing a canonically named catch-all in one transaction;
  - populated catch-all refused in under 1 s while the parent is locked;
  - a `communities` writer making the run fail fast and roll back;
  - a `communities` reader not blocking the advance;
- an in-flight insert's FK check, and its write to `communities`, behind
queued maintenance completing without a deadlock;
- a foreign key committed while maintenance waits on the parent, with a
writer on its new counterpart, failing fast as `lock_timeout` on that
table instead of waiting under the parent lock;
- a foreign key referencing `events`, committed while maintenance waits
on the parent, being refused under the parent lock;
- a catch-all's own foreign key, or a foreign key referencing it, being
refused before the parent lock with the key intact, and a catch-all key
committed while maintenance waits on the parent being refused after the
catch-all lock;
  - a `40P01` deadlock victim being classified as `lock_timeout`;
  - parent contention timing out without blocking the other table;
  - held advisory lock skipping with no DDL;
- concurrent runners converging, with at most a transient
`lock_timeout`;
- a stale audit after a concurrent advance being a no-op, and the losing
runner returning the committed layout;
- a re-plan that finds only disabled work reporting `skipped_disabled`;
  - an audit retrying when a listed partition is dropped mid-audit;
  - a catch-all with a child-only trigger being refused;
- client deadline abandoning the attempt and closing the session (scoped
to the test's own blocked backend);
  - catch-all name collision;
  - incoming foreign key refused.
- Mutation checks:
- restoring the old counterpart-first `ACCESS EXCLUSIVE` order fails the
reader test (`lock_timeout`) and the writer test (a ~1 s deadlock wait);
- dropping `NOWAIT` from the counterpart lock, or allowing a populated
catch-all, fails the matching tests;
- removing the parent-locked counterpart re-read fails the committed-FK
tests (the parent is held 2.06 s waiting on the new counterpart), and
ignoring its refusal fails the referencing-FK test;
- locking the counterparts `SHARE ROW EXCLUSIVE` before the parent fails
the writer test;
  - removing either catch-all foreign-key check fails its test.
- Full PostgreSQL lane: 716/718. Both failures are timing-sensitive
tests outside partition code, and each passes 3/3 when run alone:
  - `hanging_redis_peer_times_out_before_the_health_listener_binds`;
-
`readiness_check_cancellation_balances_waiter_and_inflight_connection`.
- `cargo clippy -p buzz-db -p buzz-relay -p buzz-admin --all-targets -D
warnings` and `cargo fmt --check` are clean.

## Related

Refs #2396.

This PR conflicts textually with #7994 (partition audit observability
follow-ups) in `crates/buzz-db/src/store/partition.rs` and
`crates/buzz-relay/src/main.rs`. Whichever PR merges second will be
rebased onto the other.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 12:59:10 -07:00
Luke Tornquist
c6340a45a0 fix(deletion): give the scheduled drain the cold-start DB connector (#8146) 
## Summary

The scheduled `buzz-admin deletions drain` Job never gets past its first
database connection in a hosted staging deployment. Every run exits 5
after exactly 3 seconds with `pool timed out while waiting for an open
connection`.

The drain starts in a fresh pod with no mesh sidecar, and that pod's
first connection can take longer than the relay's 3-second acquisition
budget. `connect_store()` in `buzz-deletion` inherits that budget from
`DbConfig::default()`, with no retry. In the same pod, the same command
run 15 seconds later succeeds. TCP to Postgres, Redis, and S3 from a
sidecar-less pod is fine.

#7770 fixed the same failure for `buzz-admin storage-snapshot`, but only
inside `buzz-admin`. This PR moves that connector into `buzz-db` as
`Db::connect_cold_start` and uses it from both run-once workers:

- **Pool settings:** a 30s acquisition budget, and no idle spare
connections (`min_connections: 0`).
- **Retries:** startup timeouts and transient transport errors retry at
most twice, after 2s and 5s, so startup waits at most 97s.
- **No retry:** configuration, authentication, TLS, and protocol errors
fail immediately, and the error message stays free of credentials.
- **Per-worker settings:** each worker keeps its own pool size.
Storage-snapshot stays at 1 connection, and deletion stays at
`BUZZ_DB_POOL_SIZE` (default 20). Each worker also keeps its session
timeouts.
- **Startup events:** the stderr events carry a worker prefix.
Storage-snapshot still emits `storage_snapshot_db_connect_*`, and the
deletion worker now emits `deletion_db_connect_*`.

- **Error text:** the exhaustion message is now `{worker} database
startup failed after …`. For storage-snapshot that reads
`storage_snapshot database startup failed …` instead of `storage
snapshot database startup failed …`. Nothing in this repo matches on
that string.
- **Session-setup failures:** SQLx retries a failed `after_connect` hook
(writer floor guard, session timeouts, isolation assertion) until the
acquire deadline. Those failures therefore surface as `timeout`, are
retried, and fail closed after at most 97s instead of immediately.

Relay pool defaults are unchanged. The deletion CLI's operator commands
(`list`, `approve`, …) use the same connector. They normally run inside
the warm relay container, where the first attempt succeeds.

**What got simpler:** there is now one cold-start connector instead of a
`buzz-admin`-private copy that the deletion worker couldn't reach.
`storage_snapshot_startup.rs` is gone, and its unit tests moved with the
code. The two Postgres startup tests now share one harness with the
drain's tests (`postgres_storage_snapshot.rs` →
`postgres_cold_start.rs`).

### Related issue

Follow-up to #7770 (same failure, other worker) and #7827 (the drain
CronJob). No duplicate found.

### Testing

All at head `1545de7b5`:

- **GitHub CI:** green on attempt 1, including `PostgreSQL Domain /
PostgreSQL Tests`.
- **Blox:**
- fmt, `clippy --workspace --all-targets -D warnings`, and the `buzz-db`
/ `buzz-admin` / `buzz-deletion` tests pass.
- All four `postgres_cold_start` tests pass, each against a freshly
migrated database.
- Falsification: with main's `connect_store` restored, the drain retry
test fails.
- **Agent review:** two rounds. All findings are fixed or disclosed; the
error-text change is listed above.
- **Agent exercise:** real `buzz-admin deletions list` / `deletions
drain` against local Postgres 17 behind a fault proxy.
  - **5s delay:** succeeds on attempt 1 (3/3). Main fails at about 3.5s.
- **Dropped handshake, then 5s delay:** attempt 1 fails with
`UnexpectedEof`, then a 2s backoff, then attempt 2 succeeds (3/3).
- **Wrong password, missing database or malformed URL:** exit 5 after
one attempt.
- **Persistent rejection:** exit 5 after three attempts, with 2s and 5s
backoffs.
  - The deletion queue stayed empty throughout.
- **Not covered:** an approved deletion, real S3 / Pod Identity, TLS and
session-setup failures, and a live 97s stall.

---------

Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
 2026-10-08 10:30:38 -04:00
Luke Tornquist
1972b7d256 feat(admin): add community archive commands (#7107) 2026-10-07 18:31:00 -04:00
Ian Oberst
14a752fb24 fix(db): audit partition catalog before creation (#6515) 
## Why

The partition manager checks partition existence by name only and treats
Postgres 42P17 "would overlap partition" errors as success. On databases
where a catch-all partition (`events_p_future` /
`delivery_log_p_future`) covers future months, the monthly creation path
silently does nothing while real coverage erodes toward a write outage.
There is currently no visibility into partition catalog health and no
signal before writes start failing.

## What

- **Read-only catalog audit** in `buzz-db` (`partition.rs`): classifies
the live partition catalog for `events` and `delivery_log` via
`pg_inherits` and bound parsing — coverage bounds, catch-all detection,
anomalous children, and trigger parity checked recursively down to
routable leaf partitions.
- **Creation gated on the audit verdict**: the 42P17-to-success swallow
is removed; monthly CREATE runs only for months the audit proves
uncovered, a canonical-name mismatch is an error, and per-month outcome
capture replaces first-error abort. New
`BUZZ_PARTITION_MANAGER_CREATE_ENABLED` kill switch (default `true`).
- **Prometheus metrics** per table: `buzz_partition_serving_safe`,
`buzz_partition_uncovered_months` (audit window is now through +3
months), `buzz_partition_catch_all_covered_months`,
`buzz_partition_anomalous_children`,
`buzz_partition_trigger_parity_missing`/`_extra`,
`buzz_partition_audit_duration_seconds`,
`buzz_partition_audit_runs_total{outcome}`,
`buzz_partition_audit_failures_total`,
`buzz_partition_create_attempts_total`, and
`buzz_partition_audit_last_success_timestamp_seconds` for staleness
alerting.
- **Cached diagnostics**: `/_status.partition_catalog` reports partition
serving-safety, audit age, and freshness. `/_readiness` stays
process-local, consistent with #7341; missing, unsafe, or stale audits
do not remove pods from service.
- **Periodic audit task** in the relay, plus config surface for the
audit interval and the monthly-create kill switch. Catch-alls are
detected from catalog bounds rather than configured names.
- Doc correction: the mesh heartbeat comment no longer claims readiness
gating (it never was gated; behavior unchanged).

## Behavior changes & risk

- No DETACH/DROP/ATTACH and no schema migrations. The only DDL is the
pre-existing monthly CREATE path, now strictly narrower
(audit-proven-uncovered months only) and behind a kill switch.
- Partition audit health does not gate readiness. Until the first
completed audit, read-only retries back off from 5 seconds to a
60-second cap. Any completed audit, including unsafe coverage, restores
the normal interval.
- Partition-overlap errors now surface as errors instead of being
swallowed as success.
- Rollback: revert the commit, or set
`BUZZ_PARTITION_MANAGER_CREATE_ENABLED=false` to disable the creation
path while keeping the audit and metrics.

## Validation

- New Postgres-backed regression tests for catalog classification and
trigger parity (disabled child trigger, `ENABLE ALWAYS` parent, disabled
nested leaf clone in a multi-level layout, healthy-catalog zero-parity
checks). Each new regression was verified to fail against the previous
logic before the fix.
- Local affected-target validation passed against a disposable
current-schema database, with the known mesh forwarded-echo and tracing
callsite failures excluded. This is not a claim that the full suite
passes.

Follow-up work (separate PRs): bounded catch-all advance (empty-only
golden path; non-empty reattach only behind an explicit operator flag,
default off), fresh-install convergence, and proactive coverage-runway
alerting.
---

## Update — 2026-08-21

- Fixed nested-leaf trigger parity auditing so child-only row triggers
are checked on every routable descendant leaf and reported as
`{leaf}:{trigger}` under the owning root child.
- Added a Postgres regression test for a child-only trigger on a nested
leaf; mutation-reverting to immediate-child-only extra-trigger scanning
makes it fail.


Generated with Codex

Signed-off-by: coder 0 <d97ebdbb198c7237c94f84ea8bb8a73583ea067407eebd0062abbb3962527fb1@buzz.block.builderlab.xyz>
Signed-off-by: Ian Oberst <ioberst@squareup.com>
Signed-off-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: coder 0 <d97ebdbb198c7237c94f84ea8bb8a73583ea067407eebd0062abbb3962527fb1@buzz.block.builderlab.xyz>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-09-29 13:43:40 -07:00
ravarora2
a929532a04 fix(admin): allow cold storage worker DB startup (#7770) 
## Summary

The standalone S3 accounting worker can exit before its scan begins
because database startup inherits the relay’s three-second connection
budget. A cold DNS probe took about five seconds.

Give `buzz-admin storage-snapshot` its own pool: one connection, no
minimum idle connections, and a 30-second acquisition budget. Retry
startup timeouts and transport failures at most twice, after two and
five seconds. Total startup wait is bounded to 97 seconds.
Configuration, authentication, TLS, and protocol errors fail
immediately.

Log each startup attempt’s elapsed time and error category without
connection credentials. Relay defaults stay unchanged. Retries stop
before lock acquisition; the same lock-owning database session still
publishes the completed snapshot.

### Related issue

Follow-up to #7543. No duplicate startup fix found.

### Testing

Archived-command check: built a nextest archive, temporarily moved
`target/debug/buzz-admin` out of the checkout, and ran
`scripts/postgres-test-run.sh --archive-file <archive>`. Both tests
failed before the path fix and passed afterward. The tests use nextest’s
relocated executable path, with the original Cargo path retained for
local runs.

Manually restored the old startup call and ran
`scripts/postgres-test-run.sh -p buzz-admin --test
postgres_storage_snapshot -E 'test(storage_snapshot_retries_then)'`. It
failed before reaching S3; restoring the fix passed. This confirms the
test exercises the actual worker command.

Local `just ci` passed all stages before mobile tests, which stopped at
SDK discovery in the `objective_c` native hook. The default Xcode tools
report an unaccepted license. Two existing PostgreSQL tests hit
connection timeouts in the parallel run and passed individually.

After merge, build a worker image and repeat the cold-pod staging smoke
test through snapshot publication before changing relay metrics mode.

Generated with Codex

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>
 2026-09-22 13:32:20 -04:00
ravarora2
6b1e45d2f6 Isolate S3 storage metrics from the relay (#7543) 
## Why

**This PR moves the S3 storage scan out of the serving relay and into a
separate batch job.** The job saves a complete result in PostgreSQL. The
relay reads that result, exports the existing storage totals, and
reports the snapshot's freshness and database-read health.

```text
Today:         relay -> scan S3 -> calculate totals -> Datadog metrics
After rollout: CronJob -> scan S3 -> PostgreSQL snapshot -> relay -> Datadog metrics
```

The code in this PR is the new pipeline. It adds the worker, database
snapshot, relay read mode, and Helm CronJob template. It does **not**
enable the pipeline in production. The chart keeps the CronJob disabled,
and the relay keeps its current inline mode by default.
[squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244)
will set the production limits, enable the job, and switch the relay to
the saved snapshot after this PR merges.

The relay currently lists the media bucket once an hour and calculates
storage totals inside the serving process. The production bucket now
exceeds the relay's one million object cap, so each sweep fails and the
last good per-community values remain stale. The Datadog dashboard can
no longer show current storage use by community.

Raising that cap inside the serving relay would move a larger,
memory-heavy scan into the process that owns live WebSocket connections.
The scan retains state for unique blob hashes and community bindings, so
its memory use grows with the bucket. A large scan could raise relay
memory use, trigger restarts, and drop user connections.

The goal is to restore current metrics without putting a
production-scale bucket scan in the relay's failure domain. No media
objects move or change. Only the process that calculates the totals and
the handoff to the relay change.

## What

- Add `buzz-admin storage-snapshot`, a run-once worker that scans S3 and
saves one complete accounting snapshot in PostgreSQL.
- Add `external` relay mode to read the saved snapshot. Keep the current
`inline` mode as the default, and add `off` as a fail-closed option.
- Add an opt-in Helm CronJob for the worker. The 10 million object cap
exists only on this Job.
- Keep the current fleet and per-community gauges, and add snapshot age,
generation duration, configured cap, cap utilization, and database-load
health gauges.
- Add an additive migration for the latest complete snapshot and its run
metadata.

## Overall plan

1. Merge this PR to add the worker, database handoff, relay modes, and
disabled-by-default chart support. This step does not change production
behavior.
2. Publish the relay image and Helm chart `0.1.9` from `main`.
3. Update
[squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244)
to pin those final artifacts and rerender its manifests.
4. Use the infrastructure PR to deploy the CronJob with a 10 million
object cap, a one-hour deadline, and a 20 GiB memory limit.
5. Start one manual Job during a staffed window and verify its snapshot,
duration, memory use, relay stability, and Datadog output.
6. Keep the daily schedule and `external` relay mode only after that
gate passes, then add alerts for stale snapshots, snapshot-load failure,
failed jobs, cap pressure, duration, and worker memory.

The code and infrastructure remain separate so this PR can ship the
mechanism without silently enabling a production-scale scan. The
infrastructure PR owns the production settings and rollout gate.

## How

### Complete snapshot handoff

The worker takes a deployment-wide PostgreSQL advisory lock before it
lists S3. This prevents scheduled and manual workers from overlapping,
even if Kubernetes starts both.

The final database replacement runs through that same lock-owning
session. If the session dies during the scan, publication fails instead
of letting the stale worker overwrite its successor. A worker that no
longer owns the lock cannot publish through another pooled connection.

The worker reads S3 in pages of 1,000 objects and folds each page into
physical totals, logical per-community usage, and anomaly counts. It
stops before folding a page that would exceed `--max-objects` and emits
structured progress every 100,000 objects.

The worker replaces the singleton database row only after the full scan
succeeds. A listing error, malformed page, cap breach, serialization
error, lost lock session, or database error cannot publish a partial
snapshot. The last completed snapshot stays available. Each stored
result includes its completion time, duration, cap, and image or source
revision.

PostgreSQL is the handoff between the batch worker and the relay. The
worker writes one complete deployment-wide snapshot, and the relay reads
that snapshot on its normal leader metrics tick. No new network service
or worker endpoint is required.

### Relay behavior

`BUZZ_STORAGE_METRICS=inline` keeps the existing in-process sweep and
its one million object default. `external` loads the newest completed
worker snapshot on each leader metrics tick. `off` emits no
storage-family metrics. An unknown value fails closed to `off`.

External mode republishes the existing fleet, per-community, and anomaly
metric names, so those current queries remain valid. Snapshot handoff
health uses a separate family: `buzz_storage_snapshot_age_seconds`,
`buzz_storage_snapshot_duration_seconds`,
`buzz_storage_snapshot_max_objects`,
`buzz_storage_snapshot_cap_utilization`, and
`buzz_storage_snapshot_load_ok`.

The `buzz_storage_sweep_*` gauges remain inline-only because they
describe a scan attempted by the relay process. External mode does not
synthesize `sweep_ok=1` from the last successful database row: an old
row proves only that an earlier worker completed. Kubernetes CronJob/Job
status is authoritative for the latest worker result, including non-zero
exit, deadline expiry, and OOM termination.

The relay keeps the last complete snapshot available when a worker run
fails. Snapshot age shows that no replacement arrived on schedule, while
`buzz_storage_snapshot_load_ok` distinguishes staleness from a missing
row, database read failure, or invalid payload. If a community
disappears, changes host, or leaves the configured emission scope, the
relay zeroes the old labeled series instead of leaving stale values
behind.

Per-community bytes are logical referenced media usage. If two
communities reference the same blob, each community receives the blob's
bytes. Fleet physical totals still count each S3 object once. This PR
preserves those existing metric semantics.

### Deployment boundary

The CronJob is disabled by default. When enabled, it uses
`concurrencyPolicy: Forbid`, no retry, a configurable active deadline,
bounded history, explicit CPU and memory resources, a read-only root
filesystem, and no Istio sidecar by default.

The worker receives only `DATABASE_URL`, S3 settings, optional S3
credential references, and the code revision. It does not receive the
relay private key, Git hook HMAC secret, Redis URL, or the full relay
secret as environment variables. Operators may reuse the Buzz service
account or select a dedicated one.

### Non-goals

- This PR does not enable the worker or `external` mode in production.
- This PR does not change media uploads, object ownership, retention, or
deletion.
- This PR does not change the names or meaning of the existing fleet,
per-community, or anomaly storage gauges.
- This PR does not add a lease service or allow multiple workers to
publish concurrently.

## Risk

The application and chart defaults do not enable the new worker or raise
the serving relay's cap. The database change is an additive
deployment-global singleton table. The relay's default remains `inline`,
so merging this PR alone does not change the active metrics path.

The main rollout risk is worker memory use because the fold retains
state for distinct blob hashes and community bindings. The first
production-sized run must confirm peak memory and duration before the
schedule is trusted. The 10 million cap and one-hour deadline bound the
first deployment, but they do not replace measurement.

External mode also needs stale-snapshot, snapshot-load, and job-failure
monitors because the relay intentionally keeps publishing the last
complete result after a failed job or read error. This behavior avoids
gaps and partial data, while snapshot age and load health tell operators
when the result or PostgreSQL handoff needs attention. Job status
remains necessary for immediate failures and termination reasons that a
killed process cannot record itself.

## Testing

No production S3 scan was run from this PR.

Automated regressions verify that external mode keeps stale last-good
totals visible without emitting inline attempt-health gauges, reports
database-load failure while retaining the cache, and leaves inline
telemetry unchanged. Mutation checks also prove those tests fail if
external mode is routed through inline emission or a failed load is
reported as healthy.

After deployment, create one manual Job from the CronJob during a
staffed window. Require a completed Job, exactly one new snapshot, fewer
objects than the configured cap, acceptable duration and peak memory, no
relay restart regression, restored Datadog points, and no OpenMetrics
sample truncation.

## Next steps

1. Complete review and merge this PR.
2. Publish chart `0.1.9` and the final main-branch relay image.
3. Replace the provisional pins in
[squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244),
rerender, and require clean CI.
4. Run the staffed manual Job gate and record object count, peak memory,
duration, snapshot revision, and relay stability.
5. Confirm the relay reads the new snapshot in `external` mode and the
existing `block.buzz_relay.*` storage series update again.
6. Enable the daily 03:17 UTC schedule and add dashboard panels and
monitors for snapshot age, snapshot-load health, Job failures, cap
pressure, duration, and worker memory.

If production must roll back to an older relay image, set
`BUZZ_STORAGE_METRICS=off` first. Older images do not understand
`external` and would resume the failing inline scan.

## Bigger picture

This is the application half of the storage-metrics repair. The paired
infrastructure PR owns the production resource limits, schedule,
artifact pins, secret boundary, and monitored rollout. Keeping those
changes separate lets this PR remain safe by default while still
providing a complete path to current per-community S3 totals.

Related issue: N/A. No matching issue or pull request exists in
`block/buzz`.

Generated with Codex

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-17 10:34:31 -04:00
Luke Tornquist
3ed623bb21 feat(db): configurable writer session timeouts (lock, idle-txn, statement) (#6229) 
## Why

A wedged relay boot pod holding a relation lock can park every other
writer in the fleet behind it: DB load pins at pool capacity in
`Lock:relation` waits while CPU stays flat, and nothing server-side
releases the lock until the holder dies. We hit exactly this in
production — ~1,400 sessions queued behind one crash-looping pod's boot
transaction for ~20 minutes until kubelet killed the container.

## What

Applies session-level Postgres timeouts to every **writer** connection
inside the existing single `after_connect` hook in `buzz-db`, all
env-tunable through the same `Config::from_env → DbConfig` path as the
existing pool-size knobs:

| Env var | GUC | Default | Effect |
|---|---|---|---|
| `BUZZ_DB_LOCK_TIMEOUT_MS` | `lock_timeout` | 5000 | statements waiting
on any lock fail fast instead of parking behind a wedged holder |
| `BUZZ_DB_IDLE_TXN_TIMEOUT_MS` | `idle_in_transaction_session_timeout`
| 60000 | reaps wedged clients idling inside an open transaction while
holding locks |
| `BUZZ_DB_STATEMENT_TIMEOUT_MS` | `statement_timeout` | 0 (off) |
opt-in runaway-statement cap; off by default because startup
migrations/backfills legitimately run long statements |

`0` disables a timeout (Postgres semantics) and deliberately passes
through the env parsing — unlike the pool-size knobs where `0` falls
back to the default. The reader pool is untouched: replica sessions
never take contended locks and already fail acquire in 150 ms.

Deployers tune these via plain env vars (`.env`, or `relay.extraEnv` in
the Helm chart) — no code changes needed.

## Behavior change to note

With the 5 s default `lock_timeout`, a boot-time migration or backfill
that waits >5 s on a lock now errors (surfacing in logs / crash-looping
the pod) instead of stalling silently. That is the intended
visible-failure-over-fleet-stall tradeoff; deployers with slow contended
migrations can set `BUZZ_DB_LOCK_TIMEOUT_MS=0`.

## Testing

- `cargo test -p buzz-db -p buzz-relay` — buzz-db green; buzz-relay has
9 failures that also fail on clean `main` in this environment
(api::admin/api::media/mesh_demo — unrelated, pre-existing).
- New config test covers override / `0`-passthrough / invalid-fallback
for all three env vars.
- Extended the existing `writer_pool_safety_hook_is_single_and_composed`
source-shape test so the timeouts can't drift out of the single
`after_connect` hook (SQLx replaces hooks — a second hook would silently
disarm the floor guard).
- `cargo fmt --check` and `cargo clippy --all-targets` clean for the
touched crates.

Closest existing PR/issue: none found.

---
**Update Aug 28, 17:06 EDT:** Rebased onto `main` at `a3730784fc` and
addressed the latest correctness review.

- Ported the timeout policy onto the refactored `buzz-db::runtime` pool
constructor and kept the shared env overlay for relay, admin, deletion,
and audit writers.
- Migration/schema-destruction connections now disable `lock_timeout`
and `statement_timeout` for their intentional long wait/DDL path. This
supersedes the earlier “Behavior change to note”: contended boot
migrations wait for the current migration owner rather than
crash-looping after five seconds.
- The audit worker now preserves and retries the same entry on
PostgreSQL `55P03` lock timeouts, using exponential backoff capped at
one second. Other database errors retain the existing terminal error
behavior, and retries emit `buzz_audit_log_lock_retries_total`.
- Added CI-backed PostgreSQL regressions for writer GUC
installation/migration exemption, audit-pool lock timeouts, and worker
recovery. The worker regression holds the real audit advisory lock past
`lock_timeout`, observes a retry, releases the lock, and proves the
original entry is appended exactly once.

Current verification supersedes the earlier testing notes: workspace
Rust clippy passed with warnings denied; all nine infrastructure-free
backend unit-test lanes passed; all three focused PostgreSQL regressions
passed against PostgreSQL 17; formatting, diff checks, file-size guards,
and desktop frontend checks passed. The Linux Blox workstation could not
run the unrelated Tauri native lane because `glib-2.0` is absent, so
that platform check is left to PR CI.


---
**Update Aug 31, 11:09 EDT:** Rebased onto current `main` at
`c3132c3ee9` and reran the requested audit-lock contention scenario on
Blox at head `896c3fe9ed`.

- `git range-diff` reports both PR commits unchanged by the rebase; the
branch remains two commits and the worktree is clean.
- `cargo fmt --all -- --check` and clippy with warnings denied passed
for `buzz-db`, `buzz-relay`, `buzz-admin`, and `buzz-deletion`.
- All three focused PostgreSQL 17 regressions passed: writer session
timeout/migration exemption, audit writer timeout bounds, and audit
worker recovery of the original entry exactly once.
- Live protocol verification used a head-built relay and CLI, native
PostgreSQL 17/Redis, `BUZZ_DB_LOCK_TIMEOUT_MS=300`, and an eight-second
hold on the community audit advisory lock. The real message was accepted
and persisted once while the lock was held; its audit-row count remained
zero during contention while retries accumulated. After release, exactly
one `event_created` audit row appeared and remained exactly one after an
additional two-second duplicate check. The run recorded nine
lock-timeout retries, zero audit failures, and event ID
`7f8c4ffae28e78555fcf2d56396d6e6c01b3712e5411288dc79e9a54af9d9444`.

Generated with Codex

---------

Signed-off-by: Luke Tornquist <tornquist@squareup.com>
 2026-08-31 11:59:13 -04:00
Wes
e0940927ff fix(channels): return complete member rosters (#5765) 
## Summary

- return complete channel rosters instead of truncating at 1,000 members
- chunk `event_mentions` inserts inside one transaction so large kind
`39002` snapshots remain discoverable by every `p` tag
- add a targeted `buzz-admin reconcile-channels --channel <uuid>`
force-republish path for stale discovery snapshots
- cover a 1,501-member roster, 11,000-tag mention index, and kind
`39002` tag construction past member 1,000

## Why

The relay builds NIP-29 discovery and several authorization decisions
from `get_members()`, but that helper silently returned only the first
1,000 active members. Desktop then counted the truncated kind `39002`
event, while late members could be rejected by roster-scanning member
actions.

Removing the roster cap exposes PostgreSQL's 65,535 bind-parameter
ceiling in mention indexing, so the insert is chunked transactionally to
preserve all-or-nothing indexing.

The existing reconcilers only fill missing discovery events. The
targeted admin option bypasses the separately known 1,000-channel
reconciliation-list ceiling and replaces an existing channel snapshot
using the configured production relay key.

## Attribution

This supersedes and builds on #3166 by @LordMelkor. Thank you for
identifying the roster boundary and contributing the original
complete-roster and mention-index patch. The production roster/query
changes and the two PostgreSQL regressions retain that work's shape;
this PR rebases it onto current `main`, adds relay coverage, and adds
the targeted repair operation requested for rollout.

## Validation

Exact pushed head: `24d02e4f3824150ed84913c9d230e675502e5b12`

- `cargo check -p buzz-db -p buzz-admin`
- `cargo test -p buzz-db
channel::tests::get_members_returns_full_roster_beyond_1000 -- --ignored
--exact --nocapture`
- `cargo test -p buzz-db
feed::tests::insert_mentions_indexes_rosters_past_bind_parameter_cap --
--ignored --exact --nocapture`
- `cargo test -p buzz-relay --lib
handlers::side_effects::tests::group_members_snapshot_keeps_members_past_one_thousand
-- --exact`
- `cargo run -q -p buzz-admin -- reconcile-channels --help`
- mandatory pre-push hook: branch-skew, desktop checks/typecheck/tests,
mobile tests, Rust tests, and desktop Tauri checks all passed on the
pushed head

## Rollout

1. Deploy the relay/backend build.
2. Run `buzz-admin reconcile-channels --channel <general-channel-uuid>`
with `BUZZ_RELAY_PRIVATE_KEY` configured.
3. Verify the replacement kind `39002` roster count matches the active
database membership count.

No schema migration or desktop release is required.

Fixes #3156
Supersedes #3166

---------

Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Co-authored-by: Carl <c7ebe626f000404285d3686e1dc74cc07cc60a9754a150041ba132e14bd3e2ec@buzz.block.builderlab.xyz>
 2026-08-13 14:48:04 -07:00
Kalvin C
8a2c9af2db feat(deletion): add durable whole-community deletion (#4425) 
## Summary

Adds a durable, operator-controlled V1 for deleting an entire Buzz
community without deleting another tenant's data.

The workflow is exposed through `buzz-admin deletions`:

- `sweep` records independent fleet storage-taxonomy observations
- `submit`, `list`, `inspect`, and `approve` manage a deletion request
- `unblock` resumes a fail-closed request after an operator records
remediation identity and reason
- `run` and `drain` execute bounded work

Requests advance through a PostgreSQL-backed state machine and stop at
`retention_pending` after logical deletion has been independently
verified across PostgreSQL, object storage, and Redis.

This PR ships the engine and CLI, not a continuously running worker or
Kubernetes packaging. For V1, a cluster/VM administrator invokes
`/usr/local/bin/buzz-admin` from the existing relay image, for example
with `kubectl exec` or an equivalent container/VM exec path.

## What whole-community V1 removes

For the target community, V1 removes:

- rows from the allowlisted community-scoped PostgreSQL catalog,
including members, profiles, authored events and bodies, DMs, reactions,
mentions, memberships, tokens, workflows, moderation, audit, feedback,
and rate-limit state
- media sidecars and upload-attribution records under
`_meta/<community>/` and `_uploads/<community>/`
- Git repository pointers under `repos/<community>/`
- Redis keys under `buzz:<community>:*`

The community row survives as a permanent tombstone, and deletion
control-plane records remain as evidence of the request, approval,
execution, and result.

## Safety model

Deletion is not a broad `DELETE CASCADE` followed by optimistic cleanup.
The destructive boundaries are durable and fail closed.

### 1. Inventory and approval

- `submit` resolves the target and freezes the schema plus summary-only
storage inventory.
- Approval is bound to the exact request, community, and frozen
inventory digest.
- Unsupported manifest versions, malformed keys inside the target's
owned prefixes, live scoped-table/write-fence coverage drift,
frozen-inventory mismatch, and approval mismatch block execution rather
than guessing. Migration and catalog revision numbers are not
authorization gates; the executor validates the live safety shape
instead.
- Storage inventory is server-side prefix scoped to exactly:
  - `_meta/<community>/`
  - `_uploads/<community>/`
  - `repos/<community>/`
- The deletion path never lists the whole shared bucket and has no
arbitrary per-community object cap. Its listing work is proportional to
the target community's bindings, not total fleet storage.
- Fleet-wide taxonomy sweeps remain independent observability. They
report unknown writer shapes but do not gate deletion submission,
fencing, or destructive progress. Maintainers must add deletion taxonomy
coverage whenever a new community-owned object-key class is introduced;
writer-coverage tests bind the current media and Git writers to that
contract.

### 2. Quiesce, fence, and destructive freeze

- Writes continue through submission, inventory, and approval. They stop
when execution moves the target into `quiescing` and then establishes
the durable fence.
- Already-admitted external effects finish under heartbeated
serving-write leases; the exact admitted lease may renew while the
community is quiescing, but new lease acquisition is rejected. The
executor drains admitted leases before destructive work.
- Invite minting after quiescing begins fails as typed `AccessDenied`
(HTTP 503 at the relay boundary) before an invite can be persisted.
- Database triggers enforce the community write fence across the
complete catalog of community-scoped tables. Startup/readiness and
destructive execution validate that catalog so a newly added but
unfenced table cannot silently escape.
- **Named isolation assumption — fresh write snapshot.** Every writer
transaction that can reach a community-fenced relation must use
PostgreSQL `READ COMMITTED`; each guarded write therefore observes a
statement snapshot no older than acquisition of the community deletion
lock. `REPEATABLE READ` and `SERIALIZABLE` can retain a pre-fence
snapshot and are unsupported for writers. The writer pool refuses
non-`READ COMMITTED` sessions at connection setup, and both SQL fence
functions reject an explicit per-transaction isolation override with
SQLSTATE `25000`. Configuration-delivered bad isolation can surface
through SQLx as a pool-acquire timeout because every `after_connect`
attempt is rejected; the precise `community writes require READ
COMMITTED isolation` reason remains observable when the SQL guard is
reached. Read-only replica transactions are outside this assumption.
- Holding the shared advisory lock until the guarded write executes is a
separate liveness condition: under `READ COMMITTED`, releasing it early
does not permit resurrection because the trigger rechecks the fence, but
it can turn a fleet sweep into a statement-wide SQLSTATE `55000` abort.
- After the fence closes writers, storage is re-enumerated into chunked
side-table rows. Per-prefix counts and digests bind those concrete keys
to the destructive manifest.
- Manifest chunk insertion, update, and deletion are protected after
freeze. This closes the race where an unbound key could otherwise appear
after the manifest was committed.

### 3. Checkpointed destruction

- Target-owned object bindings are deleted from the frozen destructive
manifest in bounded batches with durable progress.
- The concrete key list lives in chunked side-table rows rather than one
request-row JSON value. It supports large communities, resumable
execution, and terminal cleanup.
- Missing objects are accepted as idempotent crash-window outcomes;
malformed ownership, changed evidence, and unexplained target-prefix
drift fail closed.
- PostgreSQL purging remains scoped by `community_id`, including the
guarded NIP-RS hard-delete path discovered with real Desktop kind
`30078` read-state data.
- Redis cleanup explicitly scans and `UNLINK`s only
`buzz:<community_id>:*`. Natural expiry is insufficient because some
keys, including tunnel generation counters used as fencing state, are
deliberately persistent.

### 4. Independent verification

- PostgreSQL logical absence is checked after purge.
- The three target-owned storage prefixes are freshly inventoried again
and must be empty.
- Redis requires two complete empty namespace scans.
- Only after all three stores pass does the request advance through
`logically_verified` to `retention_pending`.

## What V1 deliberately does not erase

### Shared content-addressed storage

Per-community deletion removes bindings, metadata, attribution records,
and Git pointers. It does **not** physically delete fleet-shared CAS
bytes that another community may still reference:

- media blobs and thumbnails
- Git manifests, packs, and indexes (`manifests/`, `packs/`, and `idx/`)

Safe reclamation requires a separate fleet-wide reachability and
retention GC. Unknown keys elsewhere in the shared bucket do not block
one community's deletion; malformed or unrecognized keys inside that
community's three owned prefixes still fail closed.

### External retained copies

The online logical-deletion proof does not erase object
versions/replicas, database backups/WAL, CDN copies, provider retention
copies, or observability exports. Those require their own retention and
purge controls.

### Member-only erasure

This PR erases a whole community. It does not implement the different
operation "erase one npub while preserving the community."

Removing membership or accepting NIP-09 is not member erasure. A
member-only workflow would need to find and selectively remove or redact
authored event content and pubkeys, profile data, DMs, reactions,
mentions, memberships/roles, tokens, workflows/subscriptions, upload
attribution, moderation/audit history, repository attribution, and
identity embedded in tags or JSON. It would also need explicit rules for
ownership transfer, surviving replies and thread metadata, audit-chain
integrity, immutable Git history, and shared-CAS reachability. That
requires a pubkey-level fence and selective graph rewrite; it is a
separate deletion product, not a safe extension of this whole-tenant
worker.

## In scope

- migration `0029_community_deletion.sql`: requests, approvals, leases,
manifest chunks, checkpoints, tombstones, and the universal write-fence
catalog
- durable executor leases, generations, heartbeats, retry/block state,
and resumable stage transitions
- operator-driven `sweep`, `submit`, `list`, `inspect`, `approve`,
`unblock`, `run`, and `drain` commands
- serving-path fences for database writes and external effects across
event ingest, media, Git, workflow, push, invites, mesh/tunnel, and
related paths
- target-prefix-only storage inventory, summary manifests, post-fence
destructive chunks, and bounded batch deletion
- exact community Redis namespace purge and two-pass absence
verification
- cross-community isolation, crash/resume, manifest-integrity,
writer-taxonomy, and schema/migration regressions
- desired-state `schema/schema.sql` support without requiring a SQLx
migration ledger

## Deferred / not covered

- dedicated Helm/chart worker Deployment, service account, secrets,
probes, resources, and network policy
- autonomous `buzz-admin deletions worker` poll loop and worker-only
health server
- least-privilege separation among migration, relay-serving, and
destructive execution roles
- fleet-wide shared-CAS physical GC
- backup/provider/CDN/observability retention completion
- member-only erasure
- provider-native conditional-delete improvements
- a general force-continue escape hatch; permanent safety failures
remain fail closed unless an operator remediates the cause and records
an audited `unblock`

The removed continuous-worker implementation remains deferred; no remote
follow-up branch is claimed by this PR.

## Validation

### Current PR head and repository state

Current pushed head: `359d8402ee15f049768f54156f67b953c7a7e2ed`, rebased
onto `cc9a2f783375e51a6e8d1f2f9d01d5f7e22813d1` (`origin/main` at push
time). The complete PR diff is now 47 files, 9,834 additions, and 517
deletions.

The bespoke source-scanner stack was removed to keep this PR scoped to
community deletion. Tyler/team requested the underlying fenced-write
safety behavior, not `ast-grep`,
`crates/buzz-db/tests/community_fenced_writes.rs`, its 27 fixtures, or
the new `scripts/lints/community_*.yml` rules. Those scanner-specific
files, dependencies, Hermit links, and runner wiring are absent from the
current tree. The production database write fence, startup/destructive
live-catalog validation, and deletion behavior remain.

Source validation on this exact SHA passed:

- `cargo fmt --all -- --check`
- `bash -n scripts/run-tests.sh`
- `cargo nextest run -p buzz-db --all-targets`: 102 passed, 173 skipped,
0 failed
- `cargo nextest run -p buzz-deletion --all-targets`: 10 passed, 9
skipped, 0 failed
- `cargo nextest run -p buzz-admin --all-targets`: 1 passed, 0 failed
- affected-package/all-target Clippy with warnings denied
- lockfile consistency
- Helm 3.16.4 lint and all 44 chart unit tests
- Helm region controls using that fixture: default
`BUZZ_S3_REGION=us-east-1`, explicit `eu-west-2` override, and
blank-region schema rejection

The prior Kubernetes battery below was run against
`928992237358a3294621ac0280830b77155abc04`. It remains useful evidence
for the patch-equivalent production deletion implementation, but it is
**not** claimed as exact-SHA evidence for current head
`359d8402ee15f049768f54156f67b953c7a7e2ed`; the current cleanup removes
only scanner/test/tooling infrastructure. CI restarted for the new head
after the rebase and is pending. Human review remains
`CHANGES_REQUESTED`.

### Prior-head live Kubernetes deletion and safety gates

The full program used one immutable image, real PostgreSQL, Redis,
MinIO, and a three-relay Kubernetes release:

- source: `928992237358a3294621ac0280830b77155abc04` (**prior head**)
- image: `buzz-e2e:sha-928992237358`
- immutable image digest:
`sha256:a1a204f4618ac22d9e210be5e5290645a15d79831ae30b0e44379357c8e4a895`
- evidence root:
`/tmp/buzz-e2e/20260807T033025Z-928992237358-full-gates/`
- evidence-manifest digest:
`82875c5bc9bea7370b796a7aef3457b3a1c8306c84c59e0f7388bbb5ad30e865`

Passed gates at that prior head:

- **Chart/operator region:** default `us-east-1`, explicit nondefault
propagation, blank-region schema rejection, live in-pod environment, and
an in-pod taxonomy sweep over 18 objects with zero unknown.
- **Fenced writers and lifecycle:** open-write/fence ordering;
100-attempt anti-starvation; invite, push matcher, and exhausted-reaper
bystander isolation; non-`READ-COMMITTED` rejection; manifest/tombstone
contracts; eight-failure stage block and audited `unblock`.
- **Destructive lifecycle:** submit → approve → run →
`retention_pending`; PostgreSQL tombstone and Redis/S3 verification
true; zero retries/errors; terminal reruns rejected with exit 5.
- **Fresh 10,001-object crash boundary:** exactly two chunks (10,000 +
1). The executor deleted chunk 0 from MinIO while its PostgreSQL stamp
was row-lock-blocked, was killed with `SIGKILL`, left one object and
both stamps absent, then resumed the same request under generation 2 to
zero objects and terminal state.
- **Independent dead-owner recovery:** a dedicated executor claimed
generation 1, blocked before effects, and was killed through containerd
with `SIGKILL` (no TERM cleanup). The request remained owned and
unreclaimable before lease expiry; a successor claimed generation 2
after 60 seconds and completed with two attempts and zero retries.
- **Three-pod socket isolation:** ordinary NIP-42 and joined
huddle-audio target witnesses on every replica received exact `1008 /
community deleted`; healthy-tenant witnesses on those pods remained
live; deleted-host reconnect returned HTTP 404.
- **Health/provenance:** all replicas independently returned ready and
retained the exact image digest before/after destructive runs and an
audio-enabled rolling restart; PostgreSQL, Redis, and MinIO were healthy
at close.

Instrument corrections were retained as evidence rather than counted as
product failures: a foreground PostgreSQL forward caused an initial
`PoolTimedOut`; Kubernetes pod deletion exercised graceful TERM rather
than dead-owner recovery; shell-background socket witnesses died with
their parent; and the first image build hit the corporate TLS proxy.
Detached forwarding/witnesses, containerd `SIGKILL`, and the configured
internal CA/Artifactory mirror produced the discriminating runs without
weakening product security.

### Prior-head cleanup

For the prior-head Kubernetes run, the Helm release was removed,
namespace absence was verified, run-owned Screen sessions were absent,
and that source worktree remained clean. The evidence manifest was
independently recomputed and every indexed artifact passed `shasum -a
256 -c`. The current `359d8402` source worktree is also clean after the
scanner-only cleanup and push.

---------

Signed-off-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz>
Signed-off-by: npub1dccv64krpcpse5cmkzfeh998cftungyatw3djt8jwdw6g43f7fyqzzmrf7 <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
Signed-off-by: Kalvin Chau <kalvin@block.xyz>
Signed-off-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
Signed-off-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz>
Co-authored-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz>
Co-authored-by: npub1dccv64krpcpse5cmkzfeh998cftungyatw3djt8jwdw6g43f7fyqzzmrf7 <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
Co-authored-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz>
 2026-08-12 09:25:58 -07:00
Wes
af190c93e1 Add private product feedback sidecar (#1857) 
Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Co-authored-by: Pinky <44b8e82baa6e0e254e0208d68f335c283c94e7b78dd1fa10d5a49d3f13dd0435@sprout-oss.stage.blox.sqprod.co>
 2026-07-14 10:41:48 -07:00
Tyler
3292b502aa fix(relay): enable Redis TLS for rediss:// (ElastiCache) (#1417) 
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
 2026-06-30 21:14:43 -04:00
Tyler
14fba21e57 Multi-tenant Buzz relay: community_id as a server-resolved key (comprehensive rewrite) (#1321) 
Signed-off-by: tlongwell-block <109685178+tlongwell-block@users.noreply.github.com>
Signed-off-by: npub1jh9wn95s0472h86ahapupaf7m6kx4v9sx2n0atj2hltcfer8k06s5n3pyf <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@sprout-oss.stage.blox.sqprod.co>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co>
Signed-off-by: npub17jjz49l9jjmhhk7cac63j8yt9z555n9cw8vk7v5jz4vzw4ppld5qgj57cc <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Mari <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Sami <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Max <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Quinn <96f056ad5f2305c8ddf637dc65d048aa4c12d7daeb8867690e34fca46b0ef64c@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Dawn <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Co-authored-by: Sami <sami@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co>
 2026-06-29 12:39:02 -04:00
Tyler
73cc31cc52 chore: remove LLM-slop comments across the codebase (#1277) 
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Max <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Mari <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Quinn <96f056ad5f2305c8ddf637dc65d048aa4c12d7daeb8867690e34fca46b0ef64c@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Sami <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Perci <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
 2026-06-25 11:03:05 -04:00
Will Pfleger
0cee0435f7 feat(relay): add buzz-admin member management CLI with NIP-43 roster publish (#1265) 
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@sprout-oss.stage.blox.sqprod.co>
 2026-06-24 23:13:31 -04:00
tlongwell-block
2300248d3b Add automatic database migrations (#988) 
Signed-off-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co>
Signed-off-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co>
 2026-06-16 08:39:19 -04:00
Will Pfleger
d99ad131f1 refactor: rename sprout backend to buzz (#958) 
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <wpfleger@block.xyz>
Signed-off-by: Will Pfleger <wpfleger@squareup.com>
Signed-off-by: Will Pfleger <wpfleger96@gmail.com>
Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub1fgdl5qqnh3k3f2xkqrvt7cujalhm623x4s7fdjdj5yrtp5fzjl9qrjpucw <4a1bfa0013bc6d14a8d600d8bf6392efefbd2a26ac3c96c9b2a106b0d12297ca@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub16v54tttfqacx9ycvc3k0ut0npj564ahcuajzy6qjvh57ntmsf4uq4806j2 <d32955ad69077062930cc46cfe2df30ca9aaf6f8e76422681265e9e9af704d78@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Will Pfleger <wpfleger96@gmail.com>
 2026-06-10 19:29:51 -04:00