jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-10-27 04:30:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(deletion): add durable whole-community deletion (#4425)

Browse Source 
## Summary

Adds a durable, operator-controlled V1 for deleting an entire Buzz
community without deleting another tenant's data.

The workflow is exposed through `buzz-admin deletions`:

- `sweep` records independent fleet storage-taxonomy observations
- `submit`, `list`, `inspect`, and `approve` manage a deletion request
- `unblock` resumes a fail-closed request after an operator records
remediation identity and reason
- `run` and `drain` execute bounded work

Requests advance through a PostgreSQL-backed state machine and stop at
`retention_pending` after logical deletion has been independently
verified across PostgreSQL, object storage, and Redis.

This PR ships the engine and CLI, not a continuously running worker or
Kubernetes packaging. For V1, a cluster/VM administrator invokes
`/usr/local/bin/buzz-admin` from the existing relay image, for example
with `kubectl exec` or an equivalent container/VM exec path.

## What whole-community V1 removes

For the target community, V1 removes:

- rows from the allowlisted community-scoped PostgreSQL catalog,
including members, profiles, authored events and bodies, DMs, reactions,
mentions, memberships, tokens, workflows, moderation, audit, feedback,
and rate-limit state
- media sidecars and upload-attribution records under
`_meta/<community>/` and `_uploads/<community>/`
- Git repository pointers under `repos/<community>/`
- Redis keys under `buzz:<community>:*`

The community row survives as a permanent tombstone, and deletion
control-plane records remain as evidence of the request, approval,
execution, and result.

## Safety model

Deletion is not a broad `DELETE CASCADE` followed by optimistic cleanup.
The destructive boundaries are durable and fail closed.

### 1. Inventory and approval

- `submit` resolves the target and freezes the schema plus summary-only
storage inventory.
- Approval is bound to the exact request, community, and frozen
inventory digest.
- Unsupported manifest versions, malformed keys inside the target's
owned prefixes, live scoped-table/write-fence coverage drift,
frozen-inventory mismatch, and approval mismatch block execution rather
than guessing. Migration and catalog revision numbers are not
authorization gates; the executor validates the live safety shape
instead.
- Storage inventory is server-side prefix scoped to exactly:
  - `_meta/<community>/`
  - `_uploads/<community>/`
  - `repos/<community>/`
- The deletion path never lists the whole shared bucket and has no
arbitrary per-community object cap. Its listing work is proportional to
the target community's bindings, not total fleet storage.
- Fleet-wide taxonomy sweeps remain independent observability. They
report unknown writer shapes but do not gate deletion submission,
fencing, or destructive progress. Maintainers must add deletion taxonomy
coverage whenever a new community-owned object-key class is introduced;
writer-coverage tests bind the current media and Git writers to that
contract.

### 2. Quiesce, fence, and destructive freeze

- Writes continue through submission, inventory, and approval. They stop
when execution moves the target into `quiescing` and then establishes
the durable fence.
- Already-admitted external effects finish under heartbeated
serving-write leases; the exact admitted lease may renew while the
community is quiescing, but new lease acquisition is rejected. The
executor drains admitted leases before destructive work.
- Invite minting after quiescing begins fails as typed `AccessDenied`
(HTTP 503 at the relay boundary) before an invite can be persisted.
- Database triggers enforce the community write fence across the
complete catalog of community-scoped tables. Startup/readiness and
destructive execution validate that catalog so a newly added but
unfenced table cannot silently escape.
- **Named isolation assumption — fresh write snapshot.** Every writer
transaction that can reach a community-fenced relation must use
PostgreSQL `READ COMMITTED`; each guarded write therefore observes a
statement snapshot no older than acquisition of the community deletion
lock. `REPEATABLE READ` and `SERIALIZABLE` can retain a pre-fence
snapshot and are unsupported for writers. The writer pool refuses
non-`READ COMMITTED` sessions at connection setup, and both SQL fence
functions reject an explicit per-transaction isolation override with
SQLSTATE `25000`. Configuration-delivered bad isolation can surface
through SQLx as a pool-acquire timeout because every `after_connect`
attempt is rejected; the precise `community writes require READ
COMMITTED isolation` reason remains observable when the SQL guard is
reached. Read-only replica transactions are outside this assumption.
- Holding the shared advisory lock until the guarded write executes is a
separate liveness condition: under `READ COMMITTED`, releasing it early
does not permit resurrection because the trigger rechecks the fence, but
it can turn a fleet sweep into a statement-wide SQLSTATE `55000` abort.
- After the fence closes writers, storage is re-enumerated into chunked
side-table rows. Per-prefix counts and digests bind those concrete keys
to the destructive manifest.
- Manifest chunk insertion, update, and deletion are protected after
freeze. This closes the race where an unbound key could otherwise appear
after the manifest was committed.

### 3. Checkpointed destruction

- Target-owned object bindings are deleted from the frozen destructive
manifest in bounded batches with durable progress.
- The concrete key list lives in chunked side-table rows rather than one
request-row JSON value. It supports large communities, resumable
execution, and terminal cleanup.
- Missing objects are accepted as idempotent crash-window outcomes;
malformed ownership, changed evidence, and unexplained target-prefix
drift fail closed.
- PostgreSQL purging remains scoped by `community_id`, including the
guarded NIP-RS hard-delete path discovered with real Desktop kind
`30078` read-state data.
- Redis cleanup explicitly scans and `UNLINK`s only
`buzz:<community_id>:*`. Natural expiry is insufficient because some
keys, including tunnel generation counters used as fencing state, are
deliberately persistent.

### 4. Independent verification

- PostgreSQL logical absence is checked after purge.
- The three target-owned storage prefixes are freshly inventoried again
and must be empty.
- Redis requires two complete empty namespace scans.
- Only after all three stores pass does the request advance through
`logically_verified` to `retention_pending`.

## What V1 deliberately does not erase

### Shared content-addressed storage

Per-community deletion removes bindings, metadata, attribution records,
and Git pointers. It does **not** physically delete fleet-shared CAS
bytes that another community may still reference:

- media blobs and thumbnails
- Git manifests, packs, and indexes (`manifests/`, `packs/`, and `idx/`)

Safe reclamation requires a separate fleet-wide reachability and
retention GC. Unknown keys elsewhere in the shared bucket do not block
one community's deletion; malformed or unrecognized keys inside that
community's three owned prefixes still fail closed.

### External retained copies

The online logical-deletion proof does not erase object
versions/replicas, database backups/WAL, CDN copies, provider retention
copies, or observability exports. Those require their own retention and
purge controls.

### Member-only erasure

This PR erases a whole community. It does not implement the different
operation "erase one npub while preserving the community."

Removing membership or accepting NIP-09 is not member erasure. A
member-only workflow would need to find and selectively remove or redact
authored event content and pubkeys, profile data, DMs, reactions,
mentions, memberships/roles, tokens, workflows/subscriptions, upload
attribution, moderation/audit history, repository attribution, and
identity embedded in tags or JSON. It would also need explicit rules for
ownership transfer, surviving replies and thread metadata, audit-chain
integrity, immutable Git history, and shared-CAS reachability. That
requires a pubkey-level fence and selective graph rewrite; it is a
separate deletion product, not a safe extension of this whole-tenant
worker.

## In scope

- migration `0029_community_deletion.sql`: requests, approvals, leases,
manifest chunks, checkpoints, tombstones, and the universal write-fence
catalog
- durable executor leases, generations, heartbeats, retry/block state,
and resumable stage transitions
- operator-driven `sweep`, `submit`, `list`, `inspect`, `approve`,
`unblock`, `run`, and `drain` commands
- serving-path fences for database writes and external effects across
event ingest, media, Git, workflow, push, invites, mesh/tunnel, and
related paths
- target-prefix-only storage inventory, summary manifests, post-fence
destructive chunks, and bounded batch deletion
- exact community Redis namespace purge and two-pass absence
verification
- cross-community isolation, crash/resume, manifest-integrity,
writer-taxonomy, and schema/migration regressions
- desired-state `schema/schema.sql` support without requiring a SQLx
migration ledger

## Deferred / not covered

- dedicated Helm/chart worker Deployment, service account, secrets,
probes, resources, and network policy
- autonomous `buzz-admin deletions worker` poll loop and worker-only
health server
- least-privilege separation among migration, relay-serving, and
destructive execution roles
- fleet-wide shared-CAS physical GC
- backup/provider/CDN/observability retention completion
- member-only erasure
- provider-native conditional-delete improvements
- a general force-continue escape hatch; permanent safety failures
remain fail closed unless an operator remediates the cause and records
an audited `unblock`

The removed continuous-worker implementation remains deferred; no remote
follow-up branch is claimed by this PR.

## Validation

### Current PR head and repository state

Current pushed head: `359d8402ee15f049768f54156f67b953c7a7e2ed`, rebased
onto `cc9a2f783375e51a6e8d1f2f9d01d5f7e22813d1` (`origin/main` at push
time). The complete PR diff is now 47 files, 9,834 additions, and 517
deletions.

The bespoke source-scanner stack was removed to keep this PR scoped to
community deletion. Tyler/team requested the underlying fenced-write
safety behavior, not `ast-grep`,
`crates/buzz-db/tests/community_fenced_writes.rs`, its 27 fixtures, or
the new `scripts/lints/community_*.yml` rules. Those scanner-specific
files, dependencies, Hermit links, and runner wiring are absent from the
current tree. The production database write fence, startup/destructive
live-catalog validation, and deletion behavior remain.

Source validation on this exact SHA passed:

- `cargo fmt --all -- --check`
- `bash -n scripts/run-tests.sh`
- `cargo nextest run -p buzz-db --all-targets`: 102 passed, 173 skipped,
0 failed
- `cargo nextest run -p buzz-deletion --all-targets`: 10 passed, 9
skipped, 0 failed
- `cargo nextest run -p buzz-admin --all-targets`: 1 passed, 0 failed
- affected-package/all-target Clippy with warnings denied
- lockfile consistency
- Helm 3.16.4 lint and all 44 chart unit tests
- Helm region controls using that fixture: default
`BUZZ_S3_REGION=us-east-1`, explicit `eu-west-2` override, and
blank-region schema rejection

The prior Kubernetes battery below was run against
`928992237358a3294621ac0280830b77155abc04`. It remains useful evidence
for the patch-equivalent production deletion implementation, but it is
**not** claimed as exact-SHA evidence for current head
`359d8402ee15f049768f54156f67b953c7a7e2ed`; the current cleanup removes
only scanner/test/tooling infrastructure. CI restarted for the new head
after the rebase and is pending. Human review remains
`CHANGES_REQUESTED`.

### Prior-head live Kubernetes deletion and safety gates

The full program used one immutable image, real PostgreSQL, Redis,
MinIO, and a three-relay Kubernetes release:

- source: `928992237358a3294621ac0280830b77155abc04` (**prior head**)
- image: `buzz-e2e:sha-928992237358`
- immutable image digest:
`sha256:a1a204f4618ac22d9e210be5e5290645a15d79831ae30b0e44379357c8e4a895`
- evidence root:
`/tmp/buzz-e2e/20260807T033025Z-928992237358-full-gates/`
- evidence-manifest digest:
`82875c5bc9bea7370b796a7aef3457b3a1c8306c84c59e0f7388bbb5ad30e865`

Passed gates at that prior head:

- **Chart/operator region:** default `us-east-1`, explicit nondefault
propagation, blank-region schema rejection, live in-pod environment, and
an in-pod taxonomy sweep over 18 objects with zero unknown.
- **Fenced writers and lifecycle:** open-write/fence ordering;
100-attempt anti-starvation; invite, push matcher, and exhausted-reaper
bystander isolation; non-`READ-COMMITTED` rejection; manifest/tombstone
contracts; eight-failure stage block and audited `unblock`.
- **Destructive lifecycle:** submit → approve → run →
`retention_pending`; PostgreSQL tombstone and Redis/S3 verification
true; zero retries/errors; terminal reruns rejected with exit 5.
- **Fresh 10,001-object crash boundary:** exactly two chunks (10,000 +
1). The executor deleted chunk 0 from MinIO while its PostgreSQL stamp
was row-lock-blocked, was killed with `SIGKILL`, left one object and
both stamps absent, then resumed the same request under generation 2 to
zero objects and terminal state.
- **Independent dead-owner recovery:** a dedicated executor claimed
generation 1, blocked before effects, and was killed through containerd
with `SIGKILL` (no TERM cleanup). The request remained owned and
unreclaimable before lease expiry; a successor claimed generation 2
after 60 seconds and completed with two attempts and zero retries.
- **Three-pod socket isolation:** ordinary NIP-42 and joined
huddle-audio target witnesses on every replica received exact `1008 /
community deleted`; healthy-tenant witnesses on those pods remained
live; deleted-host reconnect returned HTTP 404.
- **Health/provenance:** all replicas independently returned ready and
retained the exact image digest before/after destructive runs and an
audio-enabled rolling restart; PostgreSQL, Redis, and MinIO were healthy
at close.

Instrument corrections were retained as evidence rather than counted as
product failures: a foreground PostgreSQL forward caused an initial
`PoolTimedOut`; Kubernetes pod deletion exercised graceful TERM rather
than dead-owner recovery; shell-background socket witnesses died with
their parent; and the first image build hit the corporate TLS proxy.
Detached forwarding/witnesses, containerd `SIGKILL`, and the configured
internal CA/Artifactory mirror produced the discriminating runs without
weakening product security.

### Prior-head cleanup

For the prior-head Kubernetes run, the Helm release was removed,
namespace absence was verified, run-owned Screen sessions were absent,
and that source worktree remained clean. The evidence manifest was
independently recomputed and every indexed artifact passed `shasum -a
256 -c`. The current `359d8402` source worktree is also clean after the
scanner-only cleanup and push.

---------

Signed-off-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz>
Signed-off-by: npub1dccv64krpcpse5cmkzfeh998cftungyatw3djt8jwdw6g43f7fyqzzmrf7 <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
Signed-off-by: Kalvin Chau <kalvin@block.xyz>
Signed-off-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
Signed-off-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz>
Co-authored-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz>
Co-authored-by: npub1dccv64krpcpse5cmkzfeh998cftungyatw3djt8jwdw6g43f7fyqzzmrf7 <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
Co-authored-by: cid <d9f92a72922bf45c17379a47d64dae84b6020397c2d5a52b5317d512068cd9d3@buzz.block.builderlab.xyz>
This commit is contained in:
Kalvin C
2026-08-12 09:25:58 -07:00
committed by GitHub
parent 63d14a0e95
commit 8a2c9af2db
49 changed files with 11521 additions and 525 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
crates/buzz-admin/Cargo.toml
View File

@@ -13,6 +13,7 @@ path = "src/main.rs"
[dependencies]
buzz-db = { workspace = true }
buzz-deletion = { workspace = true }
buzz-core = { workspace = true }
buzz-auth = { workspace = true }
buzz-pubsub = { workspace = true }

19
crates/buzz-admin/src/deletions.rs Normal file
View File

@@ -0,0 +1,19 @@
//! Thin `buzz-admin deletions` adapter.
pub use buzz_deletion::Command as DeletionsCommand;
/// Delegate to the shared durable deletion engine.
pub async fn run(command: DeletionsCommand) -> anyhow::Result<i32> {
buzz_deletion::run(command).await
}
#[cfg(test)]
mod tests {
use clap::Parser;
#[test]
fn continuous_worker_command_is_not_exposed() {
let command = crate::Cli::try_parse_from(["buzz-admin", "deletions", "worker"]);
assert!(command.is_err());
}
}

8
crates/buzz-admin/src/main.rs
View File

@@ -20,6 +20,8 @@
//! newest timestamp and collide on the bumped second. run.sh serialization is
//! the guard against parallel adds (e.g. `xargs -P`).
mod deletions;
use std::sync::Arc;
use anyhow::Result;
@@ -81,6 +83,11 @@ enum Command {
#[command(subcommand)]
command: ProductFeedbackCommand,
},
/// Durable CLI-only whole-community deletion control plane.
Deletions {
#[command(subcommand)]
command: deletions::DeletionsCommand,
},
/// Emit kind:39000/39002 events for channels missing them.
///
/// Channels created via direct SQL (seed scripts, pre-migration data) won't
@@ -148,6 +155,7 @@ async fn run(cli: Cli) -> Result<i32> {
Command::ProductFeedback {
command: ProductFeedbackCommand::List { limit },
} => cmd_list_product_feedback(limit).await,
Command::Deletions { command } => deletions::run(command).await,
Command::ReconcileChannels { relay_key } => {
reconcile_channels(relay_key).await?;
Ok(0)

1
crates/buzz-db/src/channel.rs
View File

@@ -1506,6 +1506,7 @@ pub async fn reap_expired_ephemeral_channels(pool: &PgPool) -> Result<Vec<Reaped
AND ch.archived_at IS NULL \
AND ch.deleted_at IS NULL \
AND c.archived_at IS NULL \
AND community_write_allowed(ch.community_id) \
RETURNING ch.community_id, c.host, ch.id",
)
.fetch_all(pool)

4707
crates/buzz-db/src/deletion.rs Normal file
View File

File diff suppressed because it is too large Load Diff

19
crates/buzz-db/src/error.rs
View File

@@ -45,6 +45,25 @@ pub enum DbError {
#[error("invalid data: {0}")]
InvalidData(String),
/// A serving write admitted before the lifecycle transition is still live.
/// This is an ordinary retryable drain condition, not a safety violation.
#[error(
"community {community_id} still has {active_count} active serving write lease(s): {operations:?}"
)]
ServingWritesNotDrained {
/// Community whose lifecycle transition must retry.
community_id: uuid::Uuid,
/// Number of currently unexpired serving-write leases.
active_count: i64,
/// Distinct operation categories holding those leases.
operations: Vec<String>,
},
/// A deletion safety invariant is structurally violated and requires
/// operator/code remediation rather than blind retry.
#[error("deletion safety error: {0}")]
DeletionSafety(String),
/// A stored timestamp value could not be interpreted.
#[error("invalid timestamp: {0}")]
InvalidTimestamp(i64),

2
crates/buzz-db/src/event.rs
View File

@@ -1111,7 +1111,7 @@ pub struct ThreadMetadataParams<'a> {
pub broadcast: bool,
}
async fn insert_event_with_thread_metadata_tx(
pub(crate) async fn insert_event_with_thread_metadata_tx(
tx: &mut Transaction<'_, Postgres>,
community_id: CommunityId,
event: &Event,

203
crates/buzz-db/src/lib.rs
View File

@@ -17,6 +17,8 @@ pub mod api_token;
pub mod archived_identities;
/// Channel and membership persistence.
pub mod channel;
/// Durable whole-community deletion lifecycle and PostgreSQL adapter.
pub mod deletion;
/// Direct message channel persistence.
pub mod dm;
/// Database error types.
@@ -651,7 +653,7 @@ impl Db {
/// `buzz.created_at_floor` GUC — this is what makes the replica fence
/// proof hold for every insert path that goes through this pool.
pub async fn new(config: &DbConfig) -> Result<Self> {
let pool = Self::connect_pool(config, &config.database_url, true).await?;
let pool = Self::connect_pool(config, &config.database_url).await?;
let read_max_connections = config
.read_max_connections
.unwrap_or(config.max_connections);
@@ -671,31 +673,39 @@ impl Db {
})
}
/// Connect one pool with the sizing knobs from `config`.
/// Connect the writer pool with all session-level safety premises.
///
/// `arm_floor_guard` sets the `buzz.created_at_floor` session GUC on
/// every connection, arming the deferred commit-time trigger from
/// migration 0021. Writer pools must arm it; replica pools are read-only
/// so the trigger never fires there.
async fn connect_pool(config: &DbConfig, url: &str, arm_floor_guard: bool) -> Result<PgPool> {
let mut options = PgPoolOptions::new()
/// SQLx stores one `after_connect` hook, so the floor guard and transaction
/// isolation assertion must remain in this single closure. Registering a
/// second hook replaces the first and silently disarms the floor trigger.
async fn connect_pool(config: &DbConfig, url: &str) -> Result<PgPool> {
let options = PgPoolOptions::new()
.max_connections(config.max_connections)
.min_connections(config.min_connections)
.acquire_timeout(Duration::from_secs(config.acquire_timeout_secs))
.max_lifetime(Duration::from_secs(config.max_lifetime_secs))
.idle_timeout(Duration::from_secs(config.idle_timeout_secs));
if arm_floor_guard {
options = options.after_connect(|conn, _meta| {
.idle_timeout(Duration::from_secs(config.idle_timeout_secs))
.after_connect(|conn, _meta| {
Box::pin(async move {
// `SET` cannot take bind parameters; `set_config` can.
sqlx::query("SELECT set_config('buzz.created_at_floor', $1, false)")
.bind(replica_fence::CREATED_AT_FLOOR_SECS.to_string())
.execute(conn)
.execute(&mut *conn)
.await?;
let isolation: String = sqlx::query_scalar("SHOW transaction_isolation")
.fetch_one(&mut *conn)
.await?;
if isolation != "read committed" {
return Err(sqlx::Error::Configuration(
format!(
"writer pool requires READ COMMITTED transaction isolation, got {isolation}"
)
.into(),
));
}
Ok(())
})
});
}
Ok(options.connect(url).await?)
}
@@ -720,8 +730,9 @@ impl Db {
/// are dialed only on first acquire; the ~10-minute reaper never tops
/// the pool back up, which is fine — routed reads re-fill it on demand.
///
/// No floor guard: replica sessions are read-only, the trigger never
/// fires there (see [`Db::connect_pool`]).
/// No floor guard or writer-isolation assertion: replica sessions are
/// read-only, so the commit-time trigger from migration 0021 never fires
/// here and the write fence that depends on READ COMMITTED is never reached.
fn connect_read_pool(config: &DbConfig, url: &str, max_connections: u32) -> Result<PgPool> {
Ok(PgPoolOptions::new()
.max_connections(max_connections)
@@ -1021,6 +1032,16 @@ impl Db {
sqlx::query("SELECT 1").execute(&self.pool).await.is_ok()
}
/// Validate the minimum deletion fence catalog required by serving paths.
pub async fn validate_deletion_serving_catalog(&self) -> Result<()> {
self.deletion_store().validate_serving_catalog().await
}
/// Validate the exact live community-deletion tenant catalog for destruction.
pub async fn validate_deletion_catalog(&self) -> Result<()> {
self.deletion_store().validate_catalog().await
}
/// Returns pool utilisation stats for metrics emission.
///
/// `size` — total connections (idle + active)
@@ -1198,6 +1219,11 @@ impl Db {
usage::community_hosts(&self.pool).await
}
/// Return the shared durable whole-community deletion adapter.
pub fn deletion_store(&self) -> deletion::DeletionStore {
deletion::DeletionStore::new(self.pool.clone())
}
/// Begin a database transaction for atomic multi-statement operations.
///
/// Returns a `'static` transaction because `PgPool` is `Arc`-backed internally.
@@ -1221,6 +1247,8 @@ impl Db {
FROM communities
WHERE lower(host) = lower($1)
AND archived_at IS NULL
AND deleted_at IS NULL
AND deletion_state = 'active'
"#,
)
.bind(normalized_host)
@@ -1243,7 +1271,7 @@ impl Db {
#[datastore_span(name = "is_community_active", system = "postgresql")]
pub async fn is_community_active(&self, community_id: CommunityId) -> Result<bool> {
let active = sqlx::query_scalar::<_, bool>(
"SELECT EXISTS(SELECT 1 FROM communities WHERE id = $1 AND archived_at IS NULL)",
"SELECT EXISTS(SELECT 1 FROM communities WHERE id = $1 AND archived_at IS NULL AND deleted_at IS NULL AND deletion_state = 'active')",
)
.bind(community_id.as_uuid())
.fetch_one(&self.pool)
@@ -1331,6 +1359,8 @@ impl Db {
FROM communities
WHERE id = $1
AND archived_at IS NULL
AND deleted_at IS NULL
AND deletion_state = 'active'
"#,
)
.bind(community_id.as_uuid())
@@ -1404,12 +1434,19 @@ impl Db {
INSERT INTO communities (host)
VALUES ($1)
ON CONFLICT (lower(host)) DO UPDATE SET host = communities.host
WHERE communities.deletion_state = 'active'
AND communities.deleted_at IS NULL
RETURNING id, host, (xmax = 0) AS created
"#,
)
.bind(normalized_host)
.fetch_one(&self.pool)
.await?;
.fetch_optional(&self.pool)
.await?
.ok_or_else(|| {
DbError::AccessDenied(format!(
"community host {normalized_host:?} is permanently tombstoned"
))
})?;
let id: Uuid = row.try_get("id")?;
let host: String = row.try_get("host")?;
@@ -1490,6 +1527,8 @@ impl Db {
AND lower(rm.pubkey) = lower($2)
AND rm.role = 'owner'
AND c.archived_at IS NULL
AND c.deletion_state = 'active'
AND c.deleted_at IS NULL
"#,
)
.bind(normalized_host)
@@ -1529,6 +1568,8 @@ impl Db {
AND lower(rm.pubkey) = lower($2)
AND rm.role = 'owner'
AND lower(c.host) <> lower($3)
AND c.deletion_state = 'active'
AND c.deleted_at IS NULL
RETURNING c.id, c.host, c.archived_at"#,
)
.bind(normalized_host)
@@ -1561,6 +1602,8 @@ impl Db {
AND rm.community_id = c.id
AND lower(rm.pubkey) = lower($2)
AND rm.role = 'owner'
AND c.deletion_state = 'active'
AND c.deleted_at IS NULL
RETURNING c.id, c.host"#,
)
.bind(normalized_host)
@@ -1665,6 +1708,49 @@ impl Db {
Ok(result)
}
/// Insert an event while holding and validating an admitted serving-write
/// lease under the community ordering lock through commit.
///
/// External side effects use a durable lease rather than one long-lived DB
/// transaction. Their final database mutation presents that exact lease so
/// it may finish during quiescing without admitting any new serving work.
pub async fn insert_event_with_serving_write_guard(
&self,
lease: &deletion::ServingWriteLease,
event: &nostr::Event,
channel_id: Option<Uuid>,
) -> Result<(StoredEvent, bool)> {
let community_id = lease.community_id;
let kind_u16 = event.kind.as_u16();
let kind_u32 = u32::from(kind_u16);
if kind_u32 == buzz_core::kind::KIND_AUTH {
return Err(DbError::AuthEventRejected);
}
if buzz_core::kind::is_ephemeral(kind_u32) {
return Err(DbError::EphemeralEventRejected(kind_u16));
}
let mut tx = self.pool.begin().await?;
self.deletion_store()
.guard_transaction_with_serving_lease(&mut tx, lease)
.await?;
let result = event::insert_event_with_thread_metadata_tx(
&mut tx,
community_id,
event,
channel_id,
None,
)
.await?;
tx.commit().await?;
if result.1 {
if let Err(e) = insert_mentions(&self.pool, community_id, event, channel_id).await {
tracing::warn!(event_id = %event.id, "Failed to insert mentions: {e}");
}
}
Ok(result)
}
/// Queries events matching the given filter parameters.
///
/// Always reads from the WRITER pool. If the result influences a write
@@ -4086,7 +4172,8 @@ impl Db {
WHERE elem->>0 = 'd' LIMIT 1), \
'' \
) \
WHERE kind BETWEEN 30000 AND 39999 AND d_tag IS NULL",
WHERE kind BETWEEN 30000 AND 39999 AND d_tag IS NULL \
AND community_write_allowed(community_id)",
)
.execute(&self.pool)
.await?;
@@ -8524,6 +8611,76 @@ mod tests {
drop_scratch_db(&admin, pool, &name).await;
}
#[test]
fn writer_pool_safety_hook_is_single_and_composed() {
let source = include_str!("lib.rs");
let connect_pool = source
.split("async fn connect_pool")
.nth(1)
.and_then(|tail| tail.split("const READER_ACQUIRE_TIMEOUT").next())
.expect("connect_pool source block");
assert_eq!(
connect_pool.matches(".after_connect(").count(),
1,
"SQLx replaces after_connect hooks; writer safety must use exactly one"
);
assert!(connect_pool.contains("buzz.created_at_floor"));
assert!(connect_pool.contains("SHOW transaction_isolation"));
assert!(!connect_pool.contains("arm_floor_guard"));
assert!(!connect_pool.contains("_arm_floor_guard"));
assert!(!connect_pool.contains("allow(unused_variables)"));
let reader_doc = source
.split("fn connect_read_pool")
.next()
.and_then(|prefix| prefix.rsplit("/// Connect the read-replica").next())
.expect("reader pool documentation");
assert!(reader_doc.contains("replica sessions are"));
assert!(reader_doc.contains("read-only"));
assert!(!reader_doc.contains("Db::connect_pool"));
}
#[tokio::test]
#[ignore = "requires Postgres"]
async fn writer_pool_rejects_non_read_committed_database_default() {
let admin = PgPool::connect(&admin_url().await)
.await
.expect("connect admin");
let (seed_pool, name) = create_scratch_db(&admin, "writer_isolation").await;
sqlx::query(sqlx::AssertSqlSafe(format!(
"ALTER DATABASE {name} SET default_transaction_isolation = 'repeatable read'"
)))
.execute(&admin)
.await
.expect("set unsafe database default");
seed_pool.close().await;
let base = admin_url().await;
let idx = base.rfind('/').expect("db url has a path segment");
let scratch_url = format!("{}/{}", &base[..idx], name);
let error = Db::new(&DbConfig {
database_url: scratch_url,
max_connections: 1,
min_connections: 1,
acquire_timeout_secs: 1,
..DbConfig::default()
})
.await
.expect_err("writer pool must reject pinned-snapshot database defaults");
assert!(
error.to_string().contains("requires READ COMMITTED")
|| error.to_string().contains("pool timed out"),
"unexpected isolation rejection: {error}"
);
sqlx::query(sqlx::AssertSqlSafe(format!(
"DROP DATABASE {name} WITH (FORCE)"
)))
.execute(&admin)
.await
.expect("drop isolation test database");
}
/// The armed writer pool (`Db::new`) must enforce the floor end-to-end
/// through the public insert APIs, and the session GUC must be verifiably
/// set on pooled connections.
@@ -8563,6 +8720,14 @@ mod tests {
crate::replica_fence::CREATED_AT_FLOOR_SECS.to_string(),
"writer pool must arm the floor guard on every connection"
);
let isolation: String = sqlx::query_scalar("SHOW transaction_isolation")
.fetch_one(&db.pool)
.await
.expect("SHOW writer isolation");
assert_eq!(
isolation, "read committed",
"the same writer after_connect hook must enforce the isolation premise"
);
let now_secs = chrono::Utc::now().timestamp() as u64;
let floor = crate::replica_fence::CREATED_AT_FLOOR_SECS as u64;

847
crates/buzz-db/src/migration.rs
View File

@@ -4,16 +4,37 @@
//! multi-tenant rewrite owns a clean consolidated `0001`; legacy single-tenant
//! cutover/backfill is a separate operator script, not startup migration state.
use sqlx::PgPool;
use std::future::Future;
use sqlx::{Connection, PgConnection, PgPool};
use crate::deletion::SCHEMA_DESTRUCTION_LOCK_KEY;
use crate::Result;
static MIGRATOR: sqlx::migrate::Migrator = sqlx::migrate!("../../migrations");
/// Run all pending Buzz database migrations.
///
/// The entire run holds the exclusive [`SCHEMA_DESTRUCTION_LOCK_KEY`] session
/// lock, serializing schema changes against destructive deletion transactions
/// (which take the shared counterpart while they validate the live catalog
/// and act on it). Every migration statement executes on the same backend
/// that owns the lock — see [`with_exclusive_schema_destruction_lock`] for
/// why that binding, not the explicit unlock, is the safety contract.
/// Migration execution must never bypass this wrapper — a source lint
/// (`migration_execution_cannot_bypass_schema_destruction_lock`) enforces
/// that `MIGRATOR.run` has no other call site.
pub async fn run_migrations(pool: &PgPool) -> Result<()> {
reject_legacy_nip_rs_cardinality_ambiguity(pool).await?;
MIGRATOR.run(pool).await?;
with_exclusive_schema_destruction_lock(pool, |mut conn| async move {
let outcome = run_migrations_locked(&mut conn).await;
(conn, outcome)
})
.await
}
async fn run_migrations_locked(conn: &mut PgConnection) -> Result<()> {
reject_legacy_nip_rs_cardinality_ambiguity(conn).await?;
MIGRATOR.run(&mut *conn).await?;
// The replica-fence proof (see `replica_fence`) requires the commit-time
// `created_at` floor trigger from migration 0021 — correctly shaped — on
// the `events` parent and every partition. `CREATE TABLE .. PARTITION OF`
@@ -21,25 +42,61 @@ pub async fn run_migrations(pool: &PgPool) -> Result<()> {
// PARTITION` or created by an older code path would silently escape the
// guard, so migration fails closed if any is missing. (The fence probe
// re-runs this same check at startup on non-migrating relays.)
crate::replica_fence::verify_floor_guard_catalog(pool).await?;
crate::replica_fence::verify_floor_guard_catalog(&mut *conn).await?;
Ok(())
}
/// Run `op` while holding the exclusive schema/destruction session lock.
///
/// `op` receives ownership of the detached connection that owns the advisory
/// lock and must run every statement on it, handing the same connection back
/// with its outcome. That same-backend lifetime — not the explicit unlock —
/// is the safety contract: PostgreSQL releases a session lock only when its
/// backend finishes, so cancelling this future (dropping the connection while
/// a migration statement is still executing server-side) cannot expose the
/// lock to shared destructive holders before that statement's backend
/// terminates. On completion the lock is explicitly released on the returned
/// connection (success and error alike) and the connection is closed, never
/// returning a locked session to the pool.
pub(crate) async fn with_exclusive_schema_destruction_lock<T, F, Fut>(
pool: &PgPool,
op: F,
) -> Result<T>
where
F: FnOnce(PgConnection) -> Fut,
Fut: Future<Output = (PgConnection, Result<T>)>,
{
let mut lock_conn = pool.acquire().await?.detach();
sqlx::query("SELECT pg_advisory_lock($1)")
.bind(SCHEMA_DESTRUCTION_LOCK_KEY)
.execute(&mut lock_conn)
.await?;
let (mut lock_conn, outcome) = op(lock_conn).await;
let unlock = sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(SCHEMA_DESTRUCTION_LOCK_KEY)
.execute(&mut lock_conn)
.await;
let _ = lock_conn.close().await;
let value = outcome?;
unlock?;
Ok(value)
}
/// Migration 0007 is checksum-frozen and predates exact NIP-RS tag-cardinality
/// enforcement. A populated database still on 0001-0006 must not let 0007
/// irreversibly purge duplicate-tag history. Fail before sqlx starts its
/// migration transaction so an operator can inspect and repair those rows.
async fn reject_legacy_nip_rs_cardinality_ambiguity(pool: &PgPool) -> Result<()> {
async fn reject_legacy_nip_rs_cardinality_ambiguity(conn: &mut PgConnection) -> Result<()> {
let migrations_table: Option<String> =
sqlx::query_scalar("SELECT to_regclass('_sqlx_migrations')::text")
.fetch_one(pool)
.fetch_one(&mut *conn)
.await?;
if migrations_table.is_none() {
return Ok(());
}
let applied: Option<i64> =
sqlx::query_scalar("SELECT max(version) FROM _sqlx_migrations WHERE success")
.fetch_one(pool)
.fetch_one(&mut *conn)
.await?;
if applied.is_none_or(|version| version >= 7) {
return Ok(());
@@ -83,7 +140,7 @@ async fn reject_legacy_nip_rs_cardinality_ambiguity(pool: &PgPool) -> Result<()>
)\
)",
)
.fetch_one(pool)
.fetch_one(conn)
.await?;
if ambiguous {
@@ -348,6 +405,13 @@ mod tests {
"push_gateway_delivery_request_replays",
"product_feedback",
"replica_heartbeat",
"community_deletion_requests",
"community_deletion_approvals",
"community_deletion_checkpoints",
"community_deletion_manifest_keys",
"storage_taxonomy_sweeps",
"community_serving_write_leases",
"community_deletion_executor_heartbeats",
] {
if normalized[insert_pos..].contains(&format!("'{value}'")) {
globals.insert(value.to_owned());
@@ -561,7 +625,7 @@ mod tests {
let mut migrations: Vec<_> = MIGRATOR.iter().collect();
migrations.sort_by_key(|migration| migration.version);
assert_eq!(migrations.len(), 28);
assert_eq!(migrations.len(), 30);
assert_eq!(migrations[0].version, 1);
assert_eq!(&*migrations[0].description, "initial schema");
assert!(migrations[0]
@@ -919,33 +983,59 @@ mod tests {
assert!(heartbeat.contains("epoch"));
assert!(heartbeat.contains("INSERT INTO replica_heartbeat (id) VALUES (1)"));
assert!(heartbeat.contains("_operator_global_tables"));
// Channel-id lookup index (0027): serves the tenant-independent
// `channels` lookups that carry no community_id predicate, which no
// community_id-leading index can satisfy. Covering + partial so the
// planner can go index-only; asserted NOT UNIQUE because `id` alone is
// not unique in this table (the same channel id may exist under more
// than one community), so a unique index would encode a false
// constraint and fail to build on such a database.
// Channel-id lookup index (0027): serves tenant-independent channel lookups.
assert_eq!(migrations[26].version, 27);
let channel_id_index = migrations[26].sql.as_str();
assert!(channel_id_index.contains("idx_channels_id_live"));
assert!(channel_id_index.contains("INCLUDE (community_id)"));
assert!(channel_id_index.contains("WHERE deleted_at IS NULL"));
assert!(
!channel_id_index.contains("CREATE UNIQUE INDEX"),
"channels.id is not unique across communities — index must not be UNIQUE",
);
assert!(
desired_schema.contains("idx_channels_id_live"),
"desired-state schema must carry the channel-id lookup index",
);
assert!(!channel_id_index.contains("CREATE UNIQUE INDEX"));
assert!(desired_schema.contains("idx_channels_id_live"));
// Main owns 0028 for long reaction payloads.
assert_eq!(migrations[27].version, 28);
let long_reactions = migrations[27].sql.as_str();
assert!(
long_reactions.contains("ALTER TABLE reactions ALTER COLUMN emoji TYPE VARCHAR(66)")
);
assert!(desired_schema.contains("emoji VARCHAR(66) NOT NULL"));
// Durable whole-community deletion control plane and universal DB fence.
assert_eq!(migrations[28].version, 29);
let deletion = migrations[28].sql.as_str();
assert!(deletion.contains("CREATE TABLE community_deletion_requests"));
assert!(deletion.contains("CREATE TABLE community_deletion_approvals"));
assert!(deletion.contains("CREATE TABLE community_deletion_checkpoints"));
assert!(deletion.contains("CREATE TABLE community_serving_write_leases"));
assert!(deletion.contains("CREATE TABLE community_deletion_executor_heartbeats"));
assert!(deletion.contains("CREATE FUNCTION community_write_allowed"));
assert!(deletion.contains("LANGUAGE plpgsql VOLATILE"));
assert!(deletion.contains("CREATE FUNCTION assert_community_write_allowed"));
assert!(deletion.contains("current_setting('transaction_isolation') <> 'read committed'"));
assert!(deletion.contains("ERRCODE = 'invalid_transaction_state'"));
assert!(deletion.contains("CREATE FUNCTION enforce_community_write_fence"));
assert!(deletion.contains("CREATE FUNCTION attach_community_write_fence"));
assert!(deletion.contains("community_write_fence_excluded_table"));
assert!(deletion.contains("CREATE FUNCTION enforce_community_tombstone"));
assert!(deletion.contains("community tombstones are permanent"));
assert!(deletion.contains("SET LOCAL lock_timeout = '5s'"));
assert!(deletion.contains("'active', 'quiescing', 'fenced', 'tombstone'"));
assert!(deletion.contains("_operator_global_tables"));
assert!(deletion.contains("'submitted', 'inventoried', 'approved', 'fenced', 'drained'"));
assert!(deletion.contains("UNIQUE (id, community_id, inventory_digest)"));
assert!(deletion.contains("FOREIGN KEY (request_id, community_id, inventory_digest)"));
assert!(deletion.contains("prevent_community_deletion_request_retargeting"));
assert!(deletion.contains("prevent_community_deletion_approval_removal"));
assert!(deletion.contains("retry_stage TEXT CHECK"));
assert!(desired_schema.contains("retry_stage TEXT CHECK"));
// Recovery migration 0030 alters populated tables and must preserve
// the same fail-fast lock behavior as the deletion migration.
assert_eq!(migrations[29].version, 30);
let deletion_recovery = migrations[29].sql.as_str();
assert!(deletion_recovery.contains("SET LOCAL lock_timeout = '5s'"));
}
#[test]
@@ -1091,6 +1181,547 @@ mod tests {
);
}
#[test]
fn migration_execution_cannot_bypass_schema_destruction_lock() {
fn rust_sources(dir: &std::path::Path, files: &mut Vec<std::path::PathBuf>) {
for entry in std::fs::read_dir(dir).expect("read workspace source dir") {
let path = entry.expect("read workspace source entry").path();
if path.is_dir() {
if path.file_name().is_some_and(|name| name == "target") {
continue;
}
rust_sources(&path, files);
} else if path.extension().is_some_and(|ext| ext == "rs") {
files.push(path);
}
}
}
fn count(haystack: &str, needle: &str) -> usize {
haystack.matches(needle).count()
}
// Build the needles so this test's own source never matches them.
let migrate_macro = ["sqlx", "::migrate!"].concat();
let migrator_run = ["MIGRATOR", ".run("].concat();
let manifest_dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR"));
let this_file = manifest_dir.join("src/migration.rs");
let crates_dir = manifest_dir.parent().expect("workspace crates dir");
// The push gateway migrates its own dedicated authority database; it
// never holds relay tenant tables, so it is exempt from the relay
// schema/destruction lock. The community_id check below keeps that
// exemption honest.
let push_gateway_exception = crates_dir.join("buzz-push-gateway/src/postgres.rs");
let push_gateway_migrations = crates_dir.join("buzz-push-gateway/migrations");
for entry in
std::fs::read_dir(&push_gateway_migrations).expect("read push gateway migrations")
{
let path = entry.expect("read push gateway migration entry").path();
let sql = std::fs::read_to_string(&path).expect("read push gateway migration");
assert!(
!sql.to_ascii_lowercase().contains("community_id"),
"{} defines community-scoped data; its migrator would bypass the \
schema/destruction lock and must move under buzz-db migrations",
path.display()
);
}
let mut files = Vec::new();
rust_sources(crates_dir, &mut files);
for path in &files {
let source = std::fs::read_to_string(path).expect("read rust source");
let (macro_hits, run_hits) = (
count(&source, &migrate_macro),
count(&source, &migrator_run),
);
if *path == this_file {
assert_eq!(
(macro_hits, run_hits),
(1, 1),
"migration.rs must embed the migrator once and run it exactly once, \
inside the locked wrapper"
);
} else if *path == push_gateway_exception {
continue;
} else {
assert_eq!(
(macro_hits, run_hits),
(0, 0),
"{} embeds or runs a SQLx migrator outside the schema/destruction \
lock contract; route migration execution through \
buzz_db migration::run_migrations",
path.display()
);
}
}
// Within migration.rs, the single run site must sit inside
// `run_migrations_locked`, and the only public entry point must wrap
// it in the exclusive session lock.
let source = std::fs::read_to_string(&this_file).expect("read migration.rs");
let entry = source
.find("pub async fn run_migrations(")
.expect("public migration entry point");
let locked = source
.find("async fn run_migrations_locked(")
.expect("locked migration body");
let wrapper = source
.find("async fn with_exclusive_schema_destruction_lock")
.expect("exclusive lock wrapper");
let run_site = source.find(&migrator_run).expect("migrator run site");
assert!(
source[entry..locked].contains("with_exclusive_schema_destruction_lock("),
"run_migrations must delegate through the exclusive schema/destruction lock"
);
assert!(
run_site > locked && run_site < wrapper,
"the migrator run site must live inside run_migrations_locked"
);
assert!(
source[wrapper..].contains("pg_advisory_lock($1)")
&& source[wrapper..].contains("pg_advisory_unlock($1)"),
"the lock wrapper must acquire and explicitly release the session lock"
);
}
/// Structural parity between migration 0029's deletion surface and the
/// desired-state bootstrap schema (`schema/schema.sql`).
///
/// Compares parsed statements, not substrings: every deletion control-
/// plane table, function, trigger, and index 0028 creates must exist in
/// schema.sql with an identical normalized definition; every operator-
/// global registry row 0028 inserts must be inserted by schema.sql; the
/// write-fence attachment target sets must be equal; and every column
/// 0028 adds to `communities` must exist in the desired-state
/// `communities` table. A desired-state bootstrap that passes this test
/// cannot silently omit part of the deletion surface the way the
/// pre-parity schema.sql omitted `community_deletion_manifest_keys` (and
/// its immutability trigger) and `storage_taxonomy_sweeps` — booting
/// healthy, then wedging post-fence when the freeze stage first touched
/// the missing relation.
#[test]
fn deletion_surface_parity_between_migration_0029_and_schema_sql() {
use std::collections::BTreeMap;
#[derive(Default)]
struct DeletionSurface {
tables: BTreeMap<String, String>,
functions: BTreeMap<String, String>,
triggers: BTreeMap<String, String>,
indexes: BTreeSet<String>,
registry_rows: BTreeSet<(String, String)>,
fence_attachments: BTreeSet<String>,
communities_added_columns: BTreeSet<String>,
}
fn quoted_strings(statement: &str) -> Vec<String> {
let mut strings = Vec::new();
let mut current: Option<String> = None;
let mut chars = statement.chars().peekable();
while let Some(ch) = chars.next() {
match (&mut current, ch) {
(None, '\'') => current = Some(String::new()),
(Some(literal), '\'') => {
if chars.peek() == Some(&'\'') {
literal.push('\'');
chars.next();
} else {
strings.push(current.take().expect("open literal"));
}
}
(Some(literal), other) => literal.push(other),
(None, _) => {}
}
}
strings
}
fn surface(sql: &str) -> DeletionSurface {
let mut surface = DeletionSurface::default();
for statement in split_sql_statements(sql) {
let normalized = normalize_sql(&statement);
if normalized.starts_with("create table") {
let table = identifier_after_keyword(&statement, "create table")
.expect("table identifier");
surface.tables.insert(table, normalized.clone());
} else if normalized.starts_with("create function")
|| normalized.starts_with("create or replace function")
{
let function = identifier_after_keyword(&statement, "function")
.expect("function identifier");
surface.functions.insert(function, normalized.clone());
} else if normalized.starts_with("create trigger") {
let trigger = identifier_after_keyword(&statement, "create trigger")
.expect("trigger identifier");
surface.triggers.insert(trigger, normalized.clone());
} else if normalized.starts_with("create index")
|| normalized.starts_with("create unique index")
{
surface.indexes.insert(normalized.clone());
} else if normalized.starts_with("insert into _operator_global_tables") {
let literals = quoted_strings(&statement);
assert!(
literals.len().is_multiple_of(2),
"operator-global registry insert must be (table_name, reason) rows"
);
for row in literals.chunks(2) {
surface
.registry_rows
.insert((row[0].clone(), row[1].clone()));
}
} else if normalized.starts_with("alter table communities") {
for added in normalized.split("add column ").skip(1) {
let column = added
.split_whitespace()
.next()
.expect("added column name")
.to_owned();
surface.communities_added_columns.insert(column);
}
}
if let Some(position) = normalized.find("attach_community_write_fence('") {
let target = normalized[position + "attach_community_write_fence('".len()..]
.split('\'')
.next()
.expect("fence attachment target")
.to_owned();
surface.fence_attachments.insert(target);
}
}
surface
}
let migration_0029: &str = MIGRATOR
.iter()
.find(|migration| migration.version == 29)
.expect("embedded migration 0029")
.sql
.as_ref();
let workspace_root = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
.parent()
.and_then(std::path::Path::parent)
.expect("workspace root");
let schema_sql = std::fs::read_to_string(workspace_root.join("schema/schema.sql"))
.expect("read schema/schema.sql");
let migration = surface(migration_0029);
let schema = surface(&schema_sql);
assert_eq!(
migration.tables.len(),
7,
"0029 deletion control plane must define exactly the known tables: {:?}",
migration.tables.keys().collect::<Vec<_>>()
);
assert!(!migration.fence_attachments.is_empty());
assert!(!migration.registry_rows.is_empty());
for (table, definition) in &migration.tables {
let in_schema = schema
.tables
.get(table)
.unwrap_or_else(|| panic!("schema.sql is missing deletion table {table}"));
if table != "community_deletion_requests" {
assert_eq!(
in_schema, definition,
"schema.sql definition of {table} drifted from migration 0029"
);
}
}
for (function, definition) in &migration.functions {
let in_schema = schema
.functions
.get(function)
.unwrap_or_else(|| panic!("schema.sql is missing deletion function {function}"));
if function != "community_write_fence_excluded_table" {
assert_eq!(
in_schema, definition,
"schema.sql definition of {function}() drifted from migration 0029"
);
}
}
for (trigger, definition) in &migration.triggers {
let in_schema = schema
.triggers
.get(trigger)
.unwrap_or_else(|| panic!("schema.sql is missing deletion trigger {trigger}"));
assert_eq!(
in_schema, definition,
"schema.sql definition of trigger {trigger} drifted from migration 0029"
);
}
for index in &migration.indexes {
assert!(
schema.indexes.contains(index),
"schema.sql is missing (or drifted on) deletion index: {index}"
);
}
for row in &migration.registry_rows {
assert!(
schema.registry_rows.contains(row),
"schema.sql is missing operator-global registry row {row:?}"
);
}
let mut expected_fences = migration.fence_attachments.clone();
expected_fences.remove("product_feedback");
expected_fences.remove("rate_limit_violations");
assert_eq!(
expected_fences, schema.fence_attachments,
"write-fence attachment targets differ after recovery policy"
);
// 0029's ALTER TABLE additions are expressed inline by the
// desired-state `communities` definition; require the columns to
// exist there (exact definition equality is impossible across the
// ALTER/inline representations — behavior is pinned by the
// desired-state bootstrap deletion test).
let communities_columns = split_sql_statements(&schema_sql)
.into_iter()
.find_map(|statement| {
let (table, body) = create_table_body(&statement)?;
(table == "communities").then_some(body)
})
.expect("schema.sql defines communities");
let column_names: BTreeSet<String> = communities_columns
.iter()
.filter_map(|definition| column_definition_name(definition))
.collect();
for column in &migration.communities_added_columns {
assert!(
column_names.contains(column),
"schema.sql communities table is missing 0028 column {column}"
);
}
assert!(!migration.communities_added_columns.is_empty());
}
#[tokio::test]
#[ignore = "requires Postgres"]
async fn schema_destruction_lock_excludes_shared_holders_and_releases_on_both_paths() {
let pool = connect_test_pool().await;
async fn assert_exclusive_lock_free(pool: &PgPool) {
let mut probe = pool.acquire().await.expect("acquire lock probe");
let free: bool = sqlx::query_scalar("SELECT pg_try_advisory_lock($1)")
.bind(SCHEMA_DESTRUCTION_LOCK_KEY)
.fetch_one(&mut *probe)
.await
.expect("probe try-lock");
assert!(free, "schema/destruction session lock must be released");
sqlx::query("SELECT pg_advisory_unlock($1)")
.bind(SCHEMA_DESTRUCTION_LOCK_KEY)
.execute(&mut *probe)
.await
.expect("probe unlock");
}
let probe_pool = pool.clone();
with_exclusive_schema_destruction_lock(&pool, move |conn| async move {
// While a migration run is in flight, destructive transactions
// must be unable to take their shared counterpart.
let mut probe = probe_pool.acquire().await.expect("acquire shared probe");
let shared_available: bool =
sqlx::query_scalar("SELECT pg_try_advisory_lock_shared($1)")
.bind(SCHEMA_DESTRUCTION_LOCK_KEY)
.fetch_one(&mut *probe)
.await
.expect("probe shared try-lock");
assert!(
!shared_available,
"exclusive migration lock must exclude shared destructive holders"
);
(conn, Ok(()))
})
.await
.expect("locked migration op");
assert_exclusive_lock_free(&pool).await;
let failed: Result<()> = with_exclusive_schema_destruction_lock(&pool, |conn| async {
(
conn,
Err(crate::DbError::InvalidData(
"forced migration failure".into(),
)),
)
})
.await;
assert!(failed.is_err(), "op failure must propagate");
assert_exclusive_lock_free(&pool).await;
}
/// Cancellation must not release the exclusion contract while migration
/// SQL is still executing server-side.
///
/// The op parks an `ALTER TABLE` behind an ACCESS EXCLUSIVE table lock
/// held by another session, then the whole locked run is aborted. Because
/// the advisory lock lives on the same backend that runs the DDL,
/// dropping the client future cannot release it: the backend keeps the
/// session lock until it finishes the statement and dies on the closed
/// socket. The shared (destructive) counterpart must stay unavailable for
/// that entire interval — and the orphaned DDL really does commit after
/// cancellation, which is exactly the window the lock has to cover.
#[tokio::test]
#[ignore = "requires Postgres"]
async fn cancelled_migration_cannot_expose_shared_lock_while_ddl_backend_lives() {
use std::time::Instant;
use sqlx::AssertSqlSafe;
// Dedicated database: the probe table and the orphaned backend must
// stay invisible to concurrent tests in the shared database.
let base_url = std::env::var("BUZZ_TEST_DATABASE_URL")
.or_else(|_| std::env::var("DATABASE_URL"))
.unwrap_or_else(|_| TEST_DB_URL.to_owned());
let admin = PgPool::connect(&base_url)
.await
.expect("connect admin database");
let probe_db = format!("buzz_lock_cancel_{}", uuid::Uuid::new_v4().simple());
sqlx::query(AssertSqlSafe(format!("CREATE DATABASE {probe_db}")))
.execute(&admin)
.await
.expect("create probe database");
let (base_prefix, _) = base_url.rsplit_once('/').expect("database url has a path");
let pool = PgPool::connect(&format!("{base_prefix}/{probe_db}"))
.await
.expect("connect probe database");
sqlx::query("CREATE TABLE schema_lock_cancel_probe (id int)")
.execute(&pool)
.await
.expect("create probe table");
// Park the migration DDL server-side: the op's ALTER TABLE waits on
// this ACCESS EXCLUSIVE lock, pinning the backend mid-statement.
let mut blocker = pool.begin().await.expect("open blocker transaction");
sqlx::query("LOCK TABLE schema_lock_cancel_probe IN ACCESS EXCLUSIVE MODE")
.execute(&mut *blocker)
.await
.expect("hold probe table lock");
let (pid_tx, pid_rx) = tokio::sync::oneshot::channel::<i32>();
let task_pool = pool.clone();
let locked_run = tokio::spawn(async move {
with_exclusive_schema_destruction_lock(&task_pool, move |mut conn| async move {
let outcome: Result<()> = async {
let pid: i32 = sqlx::query_scalar("SELECT pg_backend_pid()")
.fetch_one(&mut conn)
.await?;
let _ = pid_tx.send(pid);
sqlx::query(
"ALTER TABLE schema_lock_cancel_probe \
ADD COLUMN committed_after_cancel int",
)
.execute(&mut conn)
.await?;
Ok(())
}
.await;
(conn, outcome)
})
.await
});
let ddl_pid = pid_rx.await.expect("locked op reports its backend pid");
let deadline = Instant::now() + std::time::Duration::from_secs(10);
loop {
let waiting: bool = sqlx::query_scalar(
"SELECT EXISTS (SELECT 1 FROM pg_stat_activity \
WHERE pid = $1 AND wait_event_type = 'Lock')",
)
.bind(ddl_pid)
.fetch_one(&pool)
.await
.expect("poll DDL wait state");
if waiting {
break;
}
assert!(
Instant::now() < deadline,
"migration DDL never parked on the table lock"
);
tokio::time::sleep(std::time::Duration::from_millis(25)).await;
}
locked_run.abort();
let joined = locked_run.await;
assert!(
joined.is_err_and(|err| err.is_cancelled()),
"locked migration run must abort mid-statement"
);
// The client future is gone, but the DDL backend is alive: the shared
// destructive lock must remain unavailable for that whole interval.
for _ in 0..20 {
let backend_alive: bool =
sqlx::query_scalar("SELECT EXISTS (SELECT 1 FROM pg_stat_activity WHERE pid = $1)")
.bind(ddl_pid)
.fetch_one(&pool)
.await
.expect("poll DDL backend liveness");
assert!(
backend_alive,
"parked DDL backend must outlive client cancellation"
);
let shared_free: bool = sqlx::query_scalar("SELECT pg_try_advisory_lock_shared($1)")
.bind(SCHEMA_DESTRUCTION_LOCK_KEY)
.fetch_one(&pool)
.await
.expect("probe shared lock");
assert!(
!shared_free,
"cancellation must not expose the shared lock while migration DDL is executing"
);
tokio::time::sleep(std::time::Duration::from_millis(50)).await;
}
// Release the table lock: the orphaned backend finishes the ALTER,
// commits, then exits on the dead socket — only then may shared
// destructive holders enter.
blocker.rollback().await.expect("release probe table lock");
let mut probe = pool.acquire().await.expect("acquire shared-lock probe");
let deadline = Instant::now() + std::time::Duration::from_secs(30);
loop {
let shared_free: bool = sqlx::query_scalar("SELECT pg_try_advisory_lock_shared($1)")
.bind(SCHEMA_DESTRUCTION_LOCK_KEY)
.fetch_one(&mut *probe)
.await
.expect("probe shared lock after backend exit");
if shared_free {
sqlx::query("SELECT pg_advisory_unlock_shared($1)")
.bind(SCHEMA_DESTRUCTION_LOCK_KEY)
.execute(&mut *probe)
.await
.expect("release shared probe lock");
break;
}
assert!(
Instant::now() < deadline,
"shared lock must become available once the DDL backend exits"
);
tokio::time::sleep(std::time::Duration::from_millis(50)).await;
}
drop(probe);
// The cancelled statement committed after the client vanished —
// exactly the interval the same-backend lock covered.
let committed: bool = sqlx::query_scalar(
"SELECT EXISTS (SELECT 1 FROM information_schema.columns \
WHERE table_name = 'schema_lock_cancel_probe' \
AND column_name = 'committed_after_cancel')",
)
.fetch_one(&pool)
.await
.expect("inspect orphaned DDL outcome");
assert!(
committed,
"orphaned migration DDL commits after cancellation; the lock must cover it"
);
pool.close().await;
sqlx::query(AssertSqlSafe(format!(
"DROP DATABASE {probe_db} WITH (FORCE)"
)))
.execute(&admin)
.await
.expect("drop probe database");
}
async fn connect_test_pool() -> PgPool {
let database_url = std::env::var("BUZZ_TEST_DATABASE_URL")
.or_else(|_| std::env::var("DATABASE_URL"))
@@ -1188,7 +1819,15 @@ mod tests {
run_migrations(&pool)
.await
.expect("retry succeeds after operator repair");
assert_eq!(applied_versions(&pool).await.last().copied(), Some(27));
let latest_version = MIGRATOR
.iter()
.map(|migration| migration.version)
.max()
.expect("embedded migrator is non-empty");
assert_eq!(
applied_versions(&pool).await.last().copied(),
Some(latest_version)
);
}
#[tokio::test]
@@ -1310,5 +1949,163 @@ mod tests {
search_expression.contains("ELSE NULL::tsvector"),
"fresh installs must default non-allowlisted kinds to NULL: {search_expression}"
);
let active_a = uuid::Uuid::new_v4();
let active_b = uuid::Uuid::new_v4();
let to_fence = uuid::Uuid::new_v4();
for (community, label) in [
(active_a, "active-a"),
(active_b, "active-b"),
(to_fence, "to-fence"),
] {
sqlx::query("INSERT INTO communities (id, host) VALUES ($1, $2)")
.bind(community)
.bind(format!("late-fence-{label}-{}.example", community.simple()))
.execute(&pool)
.await
.expect("insert late-table test community");
}
sqlx::query(
"CREATE TABLE late_created_scoped (\
community_id UUID NOT NULL, id BIGINT PRIMARY KEY, value TEXT NOT NULL\
)",
)
.execute(&pool)
.await
.expect("create late scoped table");
sqlx::query("SELECT attach_community_write_fence('late_created_scoped'::regclass)")
.execute(&pool)
.await
.expect("attach late create fence");
sqlx::query("CREATE TABLE late_altered_scoped (id BIGINT PRIMARY KEY)")
.execute(&pool)
.await
.expect("create table before late alter");
sqlx::query("ALTER TABLE late_altered_scoped ADD COLUMN community_id UUID NOT NULL")
.execute(&pool)
.await
.expect("add late community id");
sqlx::query("SELECT attach_community_write_fence('late_altered_scoped'::regclass)")
.execute(&pool)
.await
.expect("attach late alter fence");
let attached: Vec<String> = sqlx::query_scalar(
"SELECT c.relname FROM pg_trigger trigger \
JOIN pg_class c ON c.oid = trigger.tgrelid \
JOIN pg_proc procedure ON procedure.oid = trigger.tgfoid \
WHERE c.relname IN ('late_created_scoped', 'late_altered_scoped') \
AND procedure.proname = 'enforce_community_write_fence' \
AND NOT trigger.tgisinternal ORDER BY c.relname",
)
.fetch_all(&pool)
.await
.expect("read late trigger catalog");
assert_eq!(attached, vec!["late_altered_scoped", "late_created_scoped"]);
let malformed_fence_triggers: i64 = sqlx::query_scalar(
"SELECT count(*)::BIGINT FROM pg_trigger trigger \
JOIN pg_class c ON c.oid = trigger.tgrelid \
JOIN pg_proc procedure ON procedure.oid = trigger.tgfoid \
WHERE c.relname IN ('late_created_scoped', 'late_altered_scoped') \
AND procedure.proname = 'enforce_community_write_fence' \
AND NOT trigger.tgisinternal \
AND (trigger.tgenabled <> 'O' OR (trigger.tgtype & 31) <> 31)",
)
.fetch_one(&pool)
.await
.expect("validate late trigger mode and operations");
assert_eq!(malformed_fence_triggers, 0);
sqlx::query(
"INSERT INTO late_created_scoped (community_id, id, value) \
VALUES ($1, 1, 'same'), ($2, 2, 'source-fenced'), \
($1, 3, 'destination-fenced'), ($1, 4, 'opposite-a'), \
($3, 5, 'opposite-b')",
)
.bind(active_a)
.bind(to_fence)
.bind(active_b)
.execute(&pool)
.await
.expect("seed late table while communities active");
sqlx::query("UPDATE late_created_scoped SET value = 'same-ok' WHERE id = 1")
.execute(&pool)
.await
.expect("same-tenant active update");
sqlx::query("UPDATE late_created_scoped SET community_id = $1 WHERE id = 1")
.bind(active_b)
.execute(&pool)
.await
.expect("active-to-active update");
let mut fence_connection = pool.acquire().await.expect("fence connection");
sqlx::query("BEGIN")
.execute(&mut *fence_connection)
.await
.expect("begin direct fence");
sqlx::query(
"SELECT set_config('buzz.deletion_executor_community', $1, true), \
set_config('buzz.deletion_fence_generation', '1', true)",
)
.bind(to_fence.to_string())
.execute(&mut *fence_connection)
.await
.expect("authorize direct fence");
sqlx::query(
"UPDATE communities SET deletion_state = 'fenced', \
deletion_fence_generation = 1, archived_at = now() WHERE id = $1",
)
.bind(to_fence)
.execute(&mut *fence_connection)
.await
.expect("fence test destination");
sqlx::query("COMMIT")
.execute(&mut *fence_connection)
.await
.expect("commit direct fence");
let active_to_fenced =
sqlx::query("UPDATE late_created_scoped SET community_id = $1 WHERE id = 3")
.bind(to_fence)
.execute(&pool)
.await
.expect_err("active to fenced destination must fail");
assert!(active_to_fenced
.to_string()
.contains("community write fenced"));
let fenced_to_active =
sqlx::query("UPDATE late_created_scoped SET community_id = $1 WHERE id = 2")
.bind(active_a)
.execute(&pool)
.await
.expect_err("fenced source to active destination must fail");
assert!(fenced_to_active
.to_string()
.contains("community write fenced"));
let row_locations: Vec<(i64, uuid::Uuid)> = sqlx::query_as(
"SELECT id, community_id FROM late_created_scoped WHERE id IN (2, 3) ORDER BY id",
)
.fetch_all(&pool)
.await
.expect("failed moves preserve row location");
assert_eq!(row_locations, vec![(2, to_fence), (3, active_a)]);
let move_a = sqlx::query("UPDATE late_created_scoped SET community_id = $1 WHERE id = 4")
.bind(active_b)
.execute(&pool);
let move_b = sqlx::query("UPDATE late_created_scoped SET community_id = $1 WHERE id = 5")
.bind(active_a)
.execute(&pool);
tokio::time::timeout(std::time::Duration::from_secs(2), async {
let (a, b) = tokio::join!(move_a, move_b);
a.expect("opposite active move A");
b.expect("opposite active move B");
})
.await
.expect("opposite cross-tenant updates must not deadlock");
sqlx::query("DROP TABLE late_created_scoped, late_altered_scoped")
.execute(&pool)
.await
.expect("drop late-table fixtures");
}
}

126
crates/buzz-db/src/push.rs
View File

@@ -852,6 +852,7 @@ where
WHERE attempts < $3
AND next_attempt_at <= now()
AND (state = 'pending' OR (state = 'matching' AND lease_until < now()))
AND community_write_allowed(community_id)
ORDER BY next_attempt_at, created_at
LIMIT 1
),
@@ -933,7 +934,8 @@ where
pub async fn reap_exhausted_matches(pool: &PgPool) -> Result<u64> {
Ok(sqlx::query(
"DELETE FROM push_match_queue WHERE attempts >= $1 \
AND (state='pending' OR (state='matching' AND lease_until < now()))",
AND (state='pending' OR (state='matching' AND lease_until < now())) \
AND community_write_allowed(community_id)",
)
.bind(MAX_MATCH_ATTEMPTS)
.execute(pool)
@@ -2121,6 +2123,128 @@ mod tests {
);
}
async fn seed_matcher_fixture(
pool: &PgPool,
community: CommunityId,
marker: u8,
attempts: i32,
age_seconds: i64,
) {
let event_id = vec![marker; 32];
sqlx::query(
"INSERT INTO events \
(community_id, id, pubkey, created_at, kind, tags, content, sig) \
VALUES ($1, $2, $3, to_timestamp(1), 9, '[]', '', $4)",
)
.bind(community.as_uuid())
.bind(&event_id)
.bind(vec![marker.saturating_add(20); 32])
.bind(vec![marker.saturating_add(30); 64])
.execute(pool)
.await
.expect("seed source event");
sqlx::query(
"INSERT INTO push_match_queue \
(community_id, event_id, attempts, next_attempt_at) \
VALUES ($1, $2, $3, now() - make_interval(secs => $4))",
)
.bind(community.as_uuid())
.bind(event_id)
.bind(attempts)
.bind(age_seconds)
.execute(pool)
.await
.expect("seed matcher row");
}
async fn quiesce_test_community(pool: &PgPool, community: CommunityId) {
let mut lifecycle = pool.begin().await.expect("begin lifecycle fixture");
sqlx::query(
"SELECT set_config('buzz.deletion_executor_community', $1, true), \
set_config('buzz.deletion_fence_generation', '0', true)",
)
.bind(community.to_string())
.execute(&mut *lifecycle)
.await
.expect("authorize target lifecycle");
sqlx::query("UPDATE communities SET deletion_state = 'quiescing' WHERE id = $1")
.bind(community.as_uuid())
.execute(&mut *lifecycle)
.await
.expect("quiesce target");
lifecycle.commit().await.expect("commit target lifecycle");
}
#[tokio::test]
#[ignore = "requires Postgres"]
async fn matcher_claim_skips_quiescing_tenant_while_active_bystanders_progress() {
let pool = setup_pool().await;
sqlx::query("DELETE FROM push_match_queue WHERE community_write_allowed(community_id)")
.execute(&pool)
.await
.expect("drain active matcher queue");
let active_a = make_community(&pool).await;
let target = make_community(&pool).await;
let active_x = make_community(&pool).await;
seed_matcher_fixture(&pool, active_a, 1, 0, 30).await;
seed_matcher_fixture(&pool, target, 2, 0, 40).await;
seed_matcher_fixture(&pool, active_x, 3, 0, 20).await;
quiesce_test_community(&pool, target).await;
let batch = claim_due_match_batch(&pool, 16, Utc::now() + chrono::Duration::minutes(1))
.await
.expect("claim active bystander")
.expect("active A is claimable despite older target row");
assert_eq!(batch.community, active_a);
let target_state: String =
sqlx::query_scalar("SELECT state FROM push_match_queue WHERE community_id = $1")
.bind(target.as_uuid())
.fetch_one(&pool)
.await
.expect("target row remains attributed");
assert_eq!(target_state, "pending");
}
#[tokio::test]
#[ignore = "requires Postgres"]
async fn exhausted_match_reaper_skips_quiescing_tenant_and_reaps_active_bystanders() {
let pool = setup_pool().await;
sqlx::query("DELETE FROM push_match_queue WHERE community_write_allowed(community_id)")
.execute(&pool)
.await
.expect("drain active matcher queue");
let active_a = make_community(&pool).await;
let target = make_community(&pool).await;
let active_x = make_community(&pool).await;
seed_matcher_fixture(&pool, active_a, 4, MAX_MATCH_ATTEMPTS, 30).await;
seed_matcher_fixture(&pool, target, 5, MAX_MATCH_ATTEMPTS, 40).await;
seed_matcher_fixture(&pool, active_x, 6, MAX_MATCH_ATTEMPTS, 20).await;
quiesce_test_community(&pool, target).await;
assert_eq!(
reap_exhausted_matches(&pool)
.await
.expect("reap active bystanders"),
2
);
let target_remaining: i64 =
sqlx::query_scalar("SELECT count(*) FROM push_match_queue WHERE community_id = $1")
.bind(target.as_uuid())
.fetch_one(&pool)
.await
.expect("target exhausted row remains attributed");
assert_eq!(target_remaining, 1);
for active in [active_a, active_x] {
let active_remaining: i64 =
sqlx::query_scalar("SELECT count(*) FROM push_match_queue WHERE community_id = $1")
.bind(active.as_uuid())
.fetch_one(&pool)
.await
.expect("active bystander is drained");
assert_eq!(active_remaining, 0);
}
}
/// T2b batch contract: one claim returns jobs from exactly ONE community
/// (so downstream lease/membership loads are single statements), the
/// set-wise complete and retry honor the claim fence, and a retried job

209
crates/buzz-db/src/relay_invite.rs
View File

@@ -111,6 +111,14 @@ pub async fn mint_relay_invite(
let now = Utc::now();
let expires_at = now + chrono::Duration::seconds(ttl_secs as i64);
// Mint a v2 opaque invite inside the same lifecycle gate as every other
// community-scoped database write. The trigger remains the final backstop,
// but this typed guard keeps a quiescing community from surfacing as an
// opaque SQLSTATE/HTTP 500 at the API boundary.
let mut tx = pool.begin().await?;
crate::deletion::DeletionStore::new(pool.clone())
.guard_transaction(&mut tx, community)
.await?;
let row = sqlx::query(
"INSERT INTO relay_invites (community_id, token_hash, max_uses, expires_at, created_by) \
VALUES ($1, $2, $3, $4, $5) \
@@ -121,8 +129,9 @@ pub async fn mint_relay_invite(
.bind(max_uses)
.bind(expires_at)
.bind(created_by)
.fetch_one(pool)
.fetch_one(&mut *tx)
.await?;
tx.commit().await?;
let invite_id: uuid::Uuid = row.try_get("id")?;
@@ -167,6 +176,7 @@ pub async fn reap_expired_relay_invites(pool: &PgPool, cutoff: DateTime<Utc>) ->
WHERE (community_id, id) IN (\
SELECT community_id, id FROM relay_invites \
WHERE expires_at < $1 \
AND community_write_allowed(community_id) \
ORDER BY expires_at \
LIMIT $2\
)",
@@ -374,20 +384,53 @@ pub async fn claim_relay_invite(
mod tests {
use super::*;
use crate::relay_members::is_relay_member;
use sha2::Digest;
use sqlx::PgPool;
use uuid::Uuid;
const TEST_DB_URL: &str = "postgres://buzz:buzz_dev@localhost:5432/buzz"; // sadscan:disable np.postgres.1
async fn setup_pool() -> PgPool {
let database_url = std::env::var("BUZZ_TEST_DATABASE_URL")
.or_else(|_| std::env::var("DATABASE_URL"))
.unwrap_or_else(|_| TEST_DB_URL.to_owned());
PgPool::connect(&database_url)
PgPool::connect(&test_database_url())
.await
.expect("connect to test DB")
}
fn test_database_url() -> String {
std::env::var("BUZZ_TEST_DATABASE_URL")
.or_else(|_| std::env::var("DATABASE_URL"))
.unwrap_or_else(|_| TEST_DB_URL.to_owned())
}
async fn create_scratch_database(prefix: &str) -> (PgPool, String, String) {
let admin_url = test_database_url();
let admin = PgPool::connect(&admin_url)
.await
.expect("connect to test database server");
let name = format!("{}_{}", prefix, Uuid::new_v4().simple());
sqlx::query(sqlx::AssertSqlSafe(format!("CREATE DATABASE {name}")))
.execute(&admin)
.await
.expect("create scratch database");
let path_start = admin_url
.rfind('/')
.expect("database URL has a path segment");
let scratch_url = format!("{}/{}", &admin_url[..path_start], name);
(admin, name, scratch_url)
}
async fn drop_scratch_database(admin: PgPool, db: crate::Db, name: &str) {
db.pool.close().await;
drop(db);
sqlx::query(sqlx::AssertSqlSafe(format!(
"DROP DATABASE IF EXISTS {name} WITH (FORCE)"
)))
.execute(&admin)
.await
.expect("drop scratch database");
admin.close().await;
}
async fn make_test_community(pool: &PgPool) -> CommunityId {
let id = Uuid::new_v4();
sqlx::query("INSERT INTO communities (id, host) VALUES ($1, $2)")
@@ -448,6 +491,95 @@ mod tests {
}
}
#[tokio::test]
#[ignore = "requires Postgres"]
async fn mint_after_quiescing_returns_typed_fence_without_persisting() {
let (admin, database_name, database_url) =
create_scratch_database("relay_invite_fence").await;
let db = crate::Db::new(&crate::DbConfig {
database_url,
max_connections: 5,
min_connections: 0,
..crate::DbConfig::default()
})
.await
.expect("connect invite deletion test DB");
db.migrate().await.expect("migrate invite deletion test DB");
let pool = db.pool.clone();
let store = db.deletion_store();
let host = format!("relay-invite-fence-{}.example", Uuid::new_v4().simple());
let community = db
.ensure_configured_community(&host)
.await
.expect("create fenced invite community")
.id;
let request = store
.submit(&host, "owner", None)
.await
.expect("submit deletion request");
let empty_digest = hex::encode(sha2::Sha256::digest([]));
let inventory = crate::deletion::FrozenInventory {
schema: store
.inventory_schema(community)
.await
.expect("inventory schema"),
storage: crate::deletion::StorageManifest {
version: 4,
prefixes: [
format!("_meta/{community}/"),
format!("_uploads/{community}/"),
format!("repos/{community}/"),
]
.into_iter()
.map(|prefix| crate::deletion::PrefixManifest {
prefix,
object_count: 0,
total_bytes: 0,
keys_digest: empty_digest.clone(),
})
.collect(),
},
};
store
.freeze_inventory(request.id, &inventory)
.await
.expect("freeze inventory");
store
.approve(request.id, "owner", None)
.await
.expect("approve deletion");
let claim = store
.claim_specific(
request.id,
"executor",
crate::deletion::DEFAULT_LEASE_DURATION,
)
.await
.expect("claim deletion")
.expect("runnable deletion");
store
.begin_quiescing(&claim.lease)
.await
.expect("begin quiescing");
let error = mint_relay_invite(&pool, community, "owner", 3600, Some(1))
.await
.expect_err("quiescing must reject invite minting");
assert!(matches!(error, crate::error::DbError::AccessDenied(_)));
let invite_count: i64 =
sqlx::query_scalar("SELECT count(*) FROM relay_invites WHERE community_id = $1")
.bind(community.as_uuid())
.fetch_one(&pool)
.await
.expect("count relay invites");
assert_eq!(invite_count, 0, "rejected mint must not persist an invite");
drop(store);
drop(pool);
drop_scratch_database(admin, db, &database_name).await;
}
#[tokio::test]
#[ignore = "requires Postgres"]
async fn bounded_claim_exhausts_and_existing_member_retry_does_not_consume() {
@@ -615,6 +747,73 @@ mod tests {
delete_test_community(&pool, community).await;
}
#[tokio::test]
#[ignore = "requires Postgres"]
async fn retention_sweep_skips_quiescing_tenant_while_active_bystanders_progress() {
let (admin, database_name, database_url) =
create_scratch_database("relay_invite_liveness").await;
let db = crate::Db::new(&crate::DbConfig {
database_url,
max_connections: 5,
min_connections: 0,
..crate::DbConfig::default()
})
.await
.expect("connect invite liveness database");
db.migrate()
.await
.expect("migrate invite liveness database");
let pool = db.pool.clone();
let active_a = make_test_community(&pool).await;
let target = make_test_community(&pool).await;
let active_x = make_test_community(&pool).await;
let cutoff = Utc::now();
for community in [active_a, target, active_x] {
sqlx::query(
"INSERT INTO relay_invites \
(community_id, token_hash, expires_at, created_by) \
VALUES ($1, $2, $3, 'test')",
)
.bind(community.as_uuid())
.bind(sha2::Sha256::digest(community.as_uuid().as_bytes()).as_slice())
.bind(cutoff - chrono::Duration::seconds(1))
.execute(&pool)
.await
.expect("seed expired invite");
}
let mut lifecycle = pool.begin().await.expect("begin lifecycle fixture");
sqlx::query(
"SELECT set_config('buzz.deletion_executor_community', $1, true), \
set_config('buzz.deletion_fence_generation', '0', true)",
)
.bind(target.to_string())
.execute(&mut *lifecycle)
.await
.expect("authorize lifecycle fixture");
sqlx::query("UPDATE communities SET deletion_state = 'quiescing' WHERE id = $1")
.bind(target.as_uuid())
.execute(&mut *lifecycle)
.await
.expect("quiesce target");
lifecycle.commit().await.expect("commit lifecycle fixture");
assert_eq!(
reap_expired_relay_invites(&pool, cutoff)
.await
.expect("reap active bystanders"),
2
);
let remaining: Vec<Uuid> =
sqlx::query_scalar("SELECT community_id FROM relay_invites ORDER BY community_id")
.fetch_all(&pool)
.await
.expect("read remaining invite attribution");
assert_eq!(remaining, vec![*target.as_uuid()]);
drop(pool);
drop_scratch_database(admin, db, &database_name).await;
}
#[tokio::test]
#[ignore = "requires Postgres"]
async fn unlimited_invites_count_each_new_member() {

9
crates/buzz-db/src/replica_fence.rs
View File

@@ -317,8 +317,13 @@ impl ReplicaFence {
///
/// This is a name-and-shape check only; it cannot detect a sabotaged
/// function body. [`verify_floor_guard_behavior`] proves the semantics.
///
/// Generic over the executor so the migration path can run it on the
/// lock-holding connection while the startup probe keeps using the pool.
#[datastore_span(name = "replica_fence_verify_catalog", system = "postgresql")]
pub async fn verify_floor_guard_catalog(pool: &PgPool) -> crate::Result<()> {
pub async fn verify_floor_guard_catalog<'e>(
executor: impl sqlx::PgExecutor<'e>,
) -> crate::Result<()> {
// tgtype bits: 1 = ROW, 2 = BEFORE, 4 = INSERT, 16 = UPDATE, 64 = INSTEAD.
// Required: ROW + INSERT + UPDATE set, BEFORE + INSTEAD clear.
let missing: Vec<String> = sqlx::query_scalar(
@@ -345,7 +350,7 @@ pub async fn verify_floor_guard_catalog(pool: &PgPool) -> crate::Result<()> {
)
"#,
)
.fetch_all(pool)
.fetch_all(executor)
.await?;
if !missing.is_empty() {
return Err(crate::error::DbError::InvalidData(format!(

1
crates/buzz-db/src/workflow.rs
View File

@@ -602,6 +602,7 @@ pub async fn prune_scheduled_workflow_fires_before(
r#"
DELETE FROM scheduled_workflow_fires
WHERE claimed_at < $1
AND community_write_allowed(community_id)
"#,
)
.bind(older_than)

28
crates/buzz-deletion/Cargo.toml Normal file
View File

@@ -0,0 +1,28 @@
[package]
name = "buzz-deletion"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
repository.workspace = true
description = "Durable whole-community deletion engine for Buzz"
[dependencies]
anyhow = { workspace = true }
thiserror = { workspace = true }
buzz-core = { workspace = true }
buzz-db = { workspace = true }
buzz-media = { workspace = true }
chrono = { workspace = true }
clap = { version = "4", features = ["derive"] }
deadpool-redis = { workspace = true }
hex = { workspace = true }
redis = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
tokio = { workspace = true }
tokio-util = { workspace = true }
uuid = { workspace = true }
[dev-dependencies]
sqlx = { workspace = true }

2063
crates/buzz-deletion/src/lib.rs Normal file
View File

File diff suppressed because it is too large Load Diff

255
crates/buzz-media/src/bucket_index.rs
View File

@@ -753,3 +753,258 @@ mod tests {
);
}
}
/// The exact community-scoped listing prefixes owned by one tenant, in
/// ascending key order: media sidecars, upload records, and Git repository
/// pointers. Every tenant-owned binding lives under one of these; shared
/// immutable CAS/thumb/probe data is deliberately outside them (fleet-wide
/// physical GC is a separate retention phase).
pub fn tenant_prefixes(community: Uuid) -> [String; 3] {
[
format!("_meta/{community}/"),
format!("_uploads/{community}/"),
format!("repos/{community}/"),
]
}
/// Whether one bucket key is a tenant-owned binding of `community` in the
/// exact writer taxonomy: a media sidecar, an upload record, or a Git
/// repository pointer. A malformed key under a tenant prefix is NOT owned —
/// deletion fails closed on shapes this binary did not write.
pub fn is_tenant_owned_key(community: Uuid, key: &str) -> bool {
match classify_key(key) {
KeyClass::Sidecar {
community: owner, ..
}
| KeyClass::Auxiliary {
community: owner, ..
} => owner == community,
KeyClass::Unknown => git_pointer_community(key) == Some(community),
KeyClass::Blob { .. } | KeyClass::Thumb { .. } => false,
}
}
/// Whether one bucket key belongs to the fleet's known writer taxonomy:
/// blob/thumb/sidecar/upload shapes, any community's Git pointer, shared Git
/// CAS data, or a `probe/` connectivity key.
pub fn is_known_fleet_key(key: &str) -> bool {
!matches!(classify_key(key), KeyClass::Unknown)
|| git_pointer_community(key).is_some()
|| is_known_git_shared_key(key)
|| key.starts_with("probe/")
}
/// Durable outcome of one fleet-wide taxonomy sweep.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct TaxonomySweepOutcome {
/// Total objects listed.
pub listed_objects: u64,
/// Exact count of keys outside the known writer taxonomy.
pub unknown_object_count: u64,
/// Bounded sample of unknown keys, in listing order.
pub unknown_key_sample: Vec<String>,
}
/// Fold an entire paginated bucket listing into the fleet taxonomy outcome.
///
/// Deleting a tenant while the bucket contains a writer shape this binary
/// does not understand is unsafe — but that is a *fleet* invariant, not a
/// per-request one. This sweep records it once; deletion stages then gate on
/// a recent clean sweep instead of re-listing the whole bucket per request.
/// Same pagination/cap contract as [`fold_bucket_listing`]; memory is
/// bounded by `sample_limit`, never the listing size.
pub async fn sweep_bucket_taxonomy<F, Fut>(
cap: u64,
sample_limit: usize,
mut fetch_page: F,
) -> Result<TaxonomySweepOutcome, SweepError>
where
F: FnMut(Option<String>) -> Fut,
Fut: Future<Output = Result<Page, MediaError>>,
{
let mut outcome = TaxonomySweepOutcome::default();
let mut continuation_token = None;
loop {
let page = fetch_page(continuation_token.take()).await?;
outcome.listed_objects += page.objects.len() as u64;
if outcome.listed_objects > cap {
return Err(SweepError::CapExceeded {
seen: outcome.listed_objects,
cap,
});
}
for (key, _size) in page.objects {
if !is_known_fleet_key(&key) {
outcome.unknown_object_count += 1;
if outcome.unknown_key_sample.len() < sample_limit {
outcome.unknown_key_sample.push(key);
}
}
}
if !page.is_truncated {
break;
}
match page.next_continuation_token {
Some(token) => continuation_token = Some(token),
None => return Err(SweepError::MalformedPage),
}
}
Ok(outcome)
}
fn git_pointer_community(key: &str) -> Option<Uuid> {
let mut parts = key.split('/');
if parts.next()? != "repos" {
return None;
}
let community = parse_canonical_uuid(parts.next()?)?;
let owner = parts.next()?;
let repo = parts.next()?;
let pointer = parts.next()?;
if parts.next().is_some()
|| owner.len() != 64
|| !owner.bytes().all(|byte| byte.is_ascii_hexdigit())
|| repo.is_empty()
|| repo.len() > 64
|| repo.starts_with('.')
|| repo.contains("..")
|| !repo
.bytes()
.all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'_' | b'-'))
|| pointer != "pointer"
{
return None;
}
Some(community)
}
fn is_known_git_shared_key(key: &str) -> bool {
["packs/", "idx/", "manifests/"].iter().any(|prefix| {
key.strip_prefix(prefix).is_some_and(|digest| {
digest.len() == 64 && digest.bytes().all(|byte| byte.is_ascii_hexdigit())
})
})
}
#[cfg(test)]
mod deletion_taxonomy_tests {
use super::*;
#[test]
fn tenant_ownership_is_exact_per_community_and_shape() {
let target = Uuid::from_u128(1);
let other = Uuid::from_u128(2);
let sha = "a".repeat(64);
assert!(is_tenant_owned_key(
target,
&format!("_meta/{target}/{sha}.json")
));
assert!(is_tenant_owned_key(
target,
&format!("_uploads/{target}/{sha}/01ARZ3NDEKTSV4RRFFQ69G5FAV.json")
));
assert!(is_tenant_owned_key(
target,
&format!("repos/{target}/{}/repo/pointer", "b".repeat(64))
));
// Another tenant's bindings and shared CAS are never owned.
assert!(!is_tenant_owned_key(
target,
&format!("_meta/{other}/{sha}.json")
));
assert!(!is_tenant_owned_key(target, &format!("{sha}.png")));
// A malformed key under the tenant's own prefix fails closed.
assert!(!is_tenant_owned_key(
target,
&format!("_meta/{target}/not-a-sidecar")
));
assert!(!is_tenant_owned_key(
target,
&format!("repos/{target}/stray-file")
));
}
#[test]
fn tenant_prefixes_cover_every_owned_shape_and_sort_ascending() {
let community = Uuid::from_u128(7);
let prefixes = tenant_prefixes(community);
assert!(prefixes.windows(2).all(|pair| pair[0] < pair[1]));
let sha = "c".repeat(64);
for key in [
format!("_meta/{community}/{sha}.json"),
format!("_uploads/{community}/{sha}/01ARZ3NDEKTSV4RRFFQ69G5FAV.json"),
format!("repos/{community}/{}/repo/pointer", "d".repeat(64)),
] {
assert!(
prefixes
.iter()
.any(|prefix| key.starts_with(prefix.as_str())),
"owned key {key} must live under a tenant prefix"
);
assert!(is_tenant_owned_key(community, &key));
}
}
#[tokio::test]
async fn taxonomy_sweep_counts_all_unknowns_but_bounds_the_sample() {
let community = Uuid::from_u128(1);
let sha = "a".repeat(64);
let known = vec![
(format!("{sha}.png"), 1),
(format!("{sha}.thumb.jpg"), 1),
(format!("_meta/{community}/{sha}.json"), 1),
(format!("packs/{sha}"), 1),
(
format!("repos/{community}/{}/repo/pointer", "b".repeat(64)),
1,
),
("probe/cas-123.txt".to_string(), 1),
];
let pages = [
Page {
objects: known,
next_continuation_token: Some("next".to_string()),
is_truncated: true,
},
Page {
objects: vec![
("future-format/one".to_string(), 1),
("future-format/two".to_string(), 1),
("future-format/three".to_string(), 1),
],
next_continuation_token: None,
is_truncated: false,
},
];
let outcome = sweep_bucket_taxonomy(100, 2, |token| {
let page = match token.as_deref() {
None => pages[0].clone(),
Some("next") => pages[1].clone(),
other => panic!("unexpected continuation token {other:?}"),
};
async move { Ok(page) }
})
.await
.expect("sweep synthetic listing");
assert_eq!(outcome.listed_objects, 9);
assert_eq!(outcome.unknown_object_count, 3);
assert_eq!(
outcome.unknown_key_sample,
vec!["future-format/one", "future-format/two"]
);
}
#[tokio::test]
async fn taxonomy_sweep_fails_closed_past_the_fleet_cap() {
let result = sweep_bucket_taxonomy(1, 10, |_token| async {
Ok(Page {
objects: vec![("a".to_string(), 1), ("b".to_string(), 1)],
next_continuation_token: None,
is_truncated: false,
})
})
.await;
assert!(matches!(result, Err(SweepError::CapExceeded { .. })));
}
}

24
crates/buzz-media/src/error.rs
View File

@@ -54,6 +54,10 @@ pub enum MediaError {
InsufficientScope,
#[error("relay membership required")]
RelayMembershipRequired,
#[error("community writes are fenced")]
CommunityWriteFenced,
#[error("media service temporarily unavailable")]
ServiceUnavailable,
#[error("token revoked")]
TokenRevoked,
#[error("pubkey mismatch")]
@@ -138,7 +142,10 @@ impl IntoResponse for MediaError {
)
}
Self::InsufficientScope => (StatusCode::FORBIDDEN, self.to_string()),
Self::RelayMembershipRequired => (StatusCode::FORBIDDEN, self.to_string()),
Self::RelayMembershipRequired | Self::CommunityWriteFenced => {
(StatusCode::FORBIDDEN, self.to_string())
}
Self::ServiceUnavailable => (StatusCode::SERVICE_UNAVAILABLE, self.to_string()),
Self::UploadRateLimitExceeded | Self::UploadConcurrencyLimitReached => {
(StatusCode::TOO_MANY_REQUESTS, self.to_string())
}
@@ -164,6 +171,21 @@ impl IntoResponse for MediaError {
mod tests {
use super::*;
#[test]
fn serving_backend_failures_map_to_5xx_but_fences_remain_403() {
for error in [
MediaError::ServiceUnavailable,
MediaError::Internal,
MediaError::StorageError("backend".to_string()),
] {
assert!(error.into_response().status().is_server_error());
}
assert_eq!(
MediaError::CommunityWriteFenced.into_response().status(),
StatusCode::FORBIDDEN
);
}
#[test]
fn unsupported_media_maps_to_415() {
for error in [

7
crates/buzz-media/src/lib.rs
View File

@@ -14,12 +14,13 @@ pub mod upload_record;
pub mod validation;
pub use bucket_index::{
classify_key, fold_bucket_listing, BucketAggregate, BucketSnapshot, CommunityStorage, KeyClass,
Page, SweepError,
classify_key, fold_bucket_listing, is_tenant_owned_key, sweep_bucket_taxonomy, tenant_prefixes,
BucketAggregate, BucketSnapshot, CommunityStorage, KeyClass, Page, SweepError,
TaxonomySweepOutcome,
};
pub use config::{MediaConfig, S3AddressingStyle};
pub use error::MediaError;
pub use storage::{BlobHeadMeta, BlobMeta, ByteStream, MediaStorage};
pub use storage::{BlobHeadMeta, BlobMeta, BulkDeleteOutcome, ByteStream, MediaStorage};
pub use types::BlobDescriptor;
pub use upload::{process_file_upload, process_upload, process_video_upload};
pub use upload_record::{

169
crates/buzz-media/src/storage.rs
View File

@@ -177,6 +177,47 @@ impl MediaStorage {
}
}
/// Detect whether the bucket has ever had versioning enabled.
///
/// rust-s3 exposes no GetBucketVersioning, so this writes and inspects a
/// short-lived fleet probe object instead: versioning-enabled (and
/// versioning-suspended) buckets stamp new writes with a version id.
/// Deletion refuses versioned buckets because bulk deletes without a
/// VersionId would only insert delete markers, not prove logical absence.
pub async fn bucket_versioning_detected(&self) -> Result<bool, MediaError> {
let key = format!("probe/deletion-versioning-{}", uuid::Uuid::new_v4());
self.put(&key, b"buzz deletion versioning probe", "text/plain")
.await?;
let inspected = self.bucket.head_object(&key).await;
let removed = self.bucket.delete_object(&key).await;
let (head, _) = inspected.map_err(|e| MediaError::StorageError(e.to_string()))?;
removed.map_err(|e| MediaError::StorageError(e.to_string()))?;
Ok(head.version_id.is_some())
}
/// Bulk-delete up to one manifest chunk of keys via S3 `DeleteObjects`.
///
/// Never fails on per-key outcomes: they are folded into
/// [`BulkDeleteOutcome`] so the caller owns retry/fail-closed policy.
/// Historical MinIO releases report already-absent keys as
/// `NoSuchKey`/`NoSuchVersion` errors instead of deleted; both map to
/// `already_missing` to keep checkpointed retry idempotent.
pub async fn delete_objects(&self, keys: &[String]) -> Result<BulkDeleteOutcome, MediaError> {
if keys.is_empty() {
return Ok(BulkDeleteOutcome::default());
}
let identifiers = keys
.iter()
.map(|key| s3::serde_types::ObjectIdentifier::new(key.clone()))
.collect::<Vec<_>>();
let result = self
.bucket
.delete_objects(identifiers)
.await
.map_err(|e| MediaError::StorageError(e.to_string()))?;
Ok(fold_bulk_delete_result(result))
}
/// Build the community-scoped sidecar key for a given sha256 (bare hash).
///
/// Raw media bytes remain shared content-addressed CAS (`{sha}.{ext}`), but
@@ -234,6 +275,11 @@ impl MediaStorage {
.map(|m| m.mime_type)
}
/// Probe object-store connectivity and bucket access.
pub async fn ping(&self) -> Result<(), MediaError> {
self.list_page(None, 1).await.map(|_| ())
}
/// One page of a full-bucket listing, for the storage sweep. Wraps
/// rust-s3's manual `list_page` (NOT the auto-paginating `list`, which
/// has no cap) and converts the result into the storage-agnostic
@@ -246,11 +292,28 @@ impl MediaStorage {
&self,
continuation_token: Option<String>,
max_keys: usize,
) -> Result<crate::bucket_index::Page, MediaError> {
self.list_prefix_page("", continuation_token, max_keys)
.await
}
/// One page of a prefix-scoped listing.
///
/// Deletion enumerates the target community's exact key prefixes with
/// this instead of listing the whole fleet bucket: cost stays
/// O(tenant objects) regardless of fleet size. `ListObjectsV2` returns
/// keys in ascending UTF-8 binary order, which callers rely on for
/// streaming key-stream digests.
pub async fn list_prefix_page(
&self,
prefix: &str,
continuation_token: Option<String>,
max_keys: usize,
) -> Result<crate::bucket_index::Page, MediaError> {
let (result, _status) = self
.bucket
.list_page(
String::new(),
prefix.to_string(),
None,
continuation_token,
None,
@@ -269,11 +332,93 @@ impl MediaStorage {
}
}
/// Per-key outcomes of one bulk `DeleteObjects` call.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct BulkDeleteOutcome {
/// Keys the backend reported deleted (S3 reports already-missing keys as
/// deleted too — the API is idempotent by design).
pub deleted: u64,
/// Keys reported absent via legacy MinIO `NoSuchKey`/`NoSuchVersion`
/// per-key errors; equivalent to deleted for retry purposes.
pub already_missing: u64,
/// Keys whose deletion produced a version artifact (delete marker or
/// version id) — evidence of bucket versioning, which deletion must
/// fail closed on.
pub versioned_keys: Vec<String>,
/// Remaining per-key failures as `(key, code, message)`.
pub failed: Vec<(String, String, String)>,
}
fn fold_bulk_delete_result(result: s3::serde_types::DeleteObjectsResult) -> BulkDeleteOutcome {
let mut outcome = BulkDeleteOutcome::default();
for deleted in result.deleted {
if deleted.delete_marker == Some(true)
|| deleted.delete_marker_version_id.is_some()
|| deleted.version_id.is_some()
{
outcome.versioned_keys.push(deleted.key);
} else {
outcome.deleted += 1;
}
}
for error in result.errors {
if error.code == "NoSuchKey" || error.code == "NoSuchVersion" {
outcome.already_missing += 1;
} else {
outcome.failed.push((error.key, error.code, error.message));
}
}
outcome
}
#[cfg(test)]
mod tests {
use super::*;
use std::collections::HashMap;
/// The bulk-delete fold is the retry-idempotence contract: legacy MinIO
/// absent-key errors count as success, version artifacts are surfaced for
/// fail-closed handling, and anything else stays a per-key failure.
#[test]
fn bulk_delete_fold_maps_absent_keys_and_version_artifacts() {
use s3::serde_types::{DeleteError, DeleteObjectsResult, DeletedObject};
let deleted_object = |key: &str, marker: bool| DeletedObject {
key: key.to_string(),
version_id: None,
delete_marker: marker.then_some(true),
delete_marker_version_id: marker.then(|| "v1".to_string()),
};
let delete_error = |key: &str, code: &str, message: &str| DeleteError {
key: key.to_string(),
code: code.to_string(),
message: message.to_string(),
version_id: None,
};
let result = DeleteObjectsResult {
deleted: vec![
deleted_object("plain", false),
deleted_object("marked", true),
],
errors: vec![
delete_error("gone", "NoSuchKey", "absent"),
delete_error("gone-version", "NoSuchVersion", "absent"),
delete_error("denied", "AccessDenied", "nope"),
],
};
let outcome = fold_bulk_delete_result(result);
assert_eq!(outcome.deleted, 1);
assert_eq!(outcome.already_missing, 2);
assert_eq!(outcome.versioned_keys, vec!["marked".to_string()]);
assert_eq!(
outcome.failed,
vec![(
"denied".to_string(),
"AccessDenied".to_string(),
"nope".to_string()
)]
);
}
fn tenant(n: u128) -> TenantContext {
TenantContext::resolved(
CommunityId::from_uuid(uuid::Uuid::from_u128(n)),
@@ -351,6 +496,28 @@ mod tests {
);
}
#[test]
fn tenant_key_writers_are_covered_by_deletion_taxonomy() {
let ctx = tenant(1);
let community = *ctx.community().as_uuid();
let sha = "a".repeat(64);
let event_id = "01ARZ3NDEKTSV4RRFFQ69G5FAV";
let sidecar = MediaStorage::ctx_sidecar_key(&ctx, &sha);
let upload = crate::upload_record::upload_record_key(&ctx, &sha, event_id);
let prefixes = crate::bucket_index::tenant_prefixes(community);
for key in [sidecar, upload] {
assert!(
prefixes.iter().any(|prefix| key.starts_with(prefix)),
"tenant writer key {key} is outside deletion prefixes"
);
assert!(
crate::bucket_index::is_tenant_owned_key(community, &key),
"tenant writer key {key} is not recognized by deletion taxonomy"
);
}
}
#[test]
fn sidecar_keys_are_community_scoped() {
let a = tenant(1);

1
crates/buzz-relay/Cargo.toml
View File

@@ -20,6 +20,7 @@ buzz-core = { workspace = true }
buzz-conformance = { workspace = true }
buzz-db = { workspace = true }
buzz-datastore-tracing = { workspace = true }
buzz-deletion = { workspace = true }
buzz-auth = { workspace = true }
buzz-pubsub = { workspace = true }
buzz-audit = { workspace = true }

11
crates/buzz-relay/src/api/git/manifest.rs
View File

@@ -474,6 +474,17 @@ mod tests {
m.validate().expect("no parent is fine (first push)");
}
#[test]
fn pointer_writer_is_covered_by_deletion_taxonomy() {
let community = CommunityId::from_uuid(uuid::Uuid::from_u128(1));
let owner = "a".repeat(64);
let key = pointer_key(community, &owner, "repo");
let prefixes = buzz_media::tenant_prefixes(*community.as_uuid());
assert!(prefixes.iter().any(|prefix| key.starts_with(prefix)));
assert!(buzz_media::is_tenant_owned_key(*community.as_uuid(), &key));
}
#[test]
fn pointer_key_strips_dot_git() {
let c = CommunityId::from_uuid(uuid::Uuid::from_u128(1));

565
crates/buzz-relay/src/api/git/transport.rs
View File

@@ -1783,6 +1783,21 @@ pub(crate) struct PushContext {
pub repo_handle: HydratedRepo,
}
#[derive(Default)]
struct FinalizePushHooks {
#[cfg(test)]
post_cas_gate: Option<Arc<PostCasGate>>,
#[cfg(test)]
fail_ref_state_insert: bool,
}
#[cfg(test)]
#[derive(Default)]
struct PostCasGate {
reached: tokio::sync::Notify,
resume: tokio::sync::Notify,
}
/// Finalize a push request: CAS-commit the new state into the object
/// store, derive kind:30618 from the committed manifest, and only then
/// build the success response.
@@ -1793,6 +1808,17 @@ pub(crate) struct PushContext {
/// constructor of a push 2xx, so the seam is structural (not by
/// convention).
async fn finalize_push(state: &Arc<AppState>, ctx: PushContext) -> Response {
finalize_push_inner(state, ctx, &FinalizePushHooks::default()).await
}
async fn finalize_push_inner(
state: &Arc<AppState>,
ctx: PushContext,
hooks: &FinalizePushHooks,
) -> Response {
#[cfg(not(test))]
let _ = hooks;
// The push fence, part 0 — **a rejected push publishes nothing.**
//
// `ctx.pack.ok` is false when git aborted the ref updates: either the
@@ -1823,10 +1849,41 @@ async fn finalize_push(state: &Arc<AppState>, ctx: PushContext) -> Response {
return response;
}
// An already-running receive-pack may cross the durable fence after
// request admission. Revalidate immediately before object-store CAS; DB
// trigger fencing alone cannot roll back an S3 pointer mutation.
let serving_write = match buzz_deletion::acquire_serving_write(
&state.db,
ctx.tenant.community(),
"git_publish",
)
.await
{
Ok(guard) => guard,
Err(error) => {
warn!(owner = %ctx.owner, repo = %ctx.repo, %error, "push rejected by community deletion fence");
return (
StatusCode::SERVICE_UNAVAILABLE,
"community writes are fenced",
)
.into_response();
}
};
if let Err(error) = serving_write.verify().await {
warn!(owner = %ctx.owner, repo = %ctx.repo, %error, "push lost community serving lease");
return (
StatusCode::SERVICE_UNAVAILABLE,
"community write lease lost",
)
.into_response();
}
// Step 7 (CAS). The PushContext binds `parent_state` (observed at
// hydrate) to the CAS predicate here — no re-reading of the pointer
// between hydrate and CAS.
let success = match cas_publish(
// between hydrate and CAS. Observe serving-lease loss throughout the
// potentially long upload/CAS operation, not only at its boundaries.
let publish = cas_publish(
&state.git_store,
&ctx.tenant,
ctx.repo_handle.path(),
@@ -1838,72 +1895,87 @@ async fn finalize_push(state: &Arc<AppState>, ctx: PushContext) -> Response {
max_pack_bytes: state.config.git_max_pack_bytes,
max_repo_bytes: state.config.git_max_repo_bytes,
},
)
.await
{
Ok(s) => s,
Err(CasError::Conflict {
winner_manifest_key,
..
}) => {
warn!(
owner = %ctx.owner,
repo = %ctx.repo,
winner = %winner_manifest_key,
"push lost CAS race; tempdir dropped, returning 409"
);
);
let success = match serving_write.protect(publish).await {
Ok(result) => match result {
Ok(s) => s,
Err(CasError::Conflict {
winner_manifest_key,
..
}) => {
warn!(
owner = %ctx.owner,
repo = %ctx.repo,
winner = %winner_manifest_key,
"push lost CAS race; tempdir dropped, returning 409"
);
return (
StatusCode::CONFLICT,
"push superseded by a concurrent writer; pull and retry",
)
.into_response();
}
Err(CasError::ManifestInvalid(e)) => {
// 4xx-class: the workspace produced refs/HEAD/oids the
// manifest validator rejects (unsafe refname, malformed oid,
// empty head, malformed parent). Pre-CAS — no pointer was
// written.
warn!(
owner = %ctx.owner,
repo = %ctx.repo,
error = %e,
"push rejected: manifest validation failed"
);
return (
StatusCode::BAD_REQUEST,
"push produced invalid manifest state",
)
.into_response();
}
Err(CasError::ResourceLimit(e)) => {
warn!(
owner = %ctx.owner,
repo = %ctx.repo,
error = %e,
"push rejected: repo exceeds relay resource limits"
);
return (
StatusCode::PAYLOAD_TOO_LARGE,
"repository exceeds relay resource limits",
)
.into_response();
}
Err(e) => {
// 5xx-class: ManifestReadFailed (parent corruption),
// Backend, PackCapture. The tempdir drops on scope exit; no
// pointer was written (or, on rare ManifestReadFailed during
// winner-fetch, the winner is already installed and the
// loser's data is unrelated).
error!(
owner = %ctx.owner,
repo = %ctx.repo,
error = %e,
"push failed pre-response"
);
return (StatusCode::INTERNAL_SERVER_ERROR, "git backend error").into_response();
}
},
Err(error) => {
warn!(owner = %ctx.owner, repo = %ctx.repo, %error, "push lost community serving lease during CAS publish");
return (
StatusCode::CONFLICT,
"push superseded by a concurrent writer; pull and retry",
StatusCode::SERVICE_UNAVAILABLE,
"community write lease lost",
)
.into_response();
}
Err(CasError::ManifestInvalid(e)) => {
// 4xx-class: the workspace produced refs/HEAD/oids the
// manifest validator rejects (unsafe refname, malformed oid,
// empty head, malformed parent). Pre-CAS — no pointer was
// written.
warn!(
owner = %ctx.owner,
repo = %ctx.repo,
error = %e,
"push rejected: manifest validation failed"
);
return (
StatusCode::BAD_REQUEST,
"push produced invalid manifest state",
)
.into_response();
}
Err(CasError::ResourceLimit(e)) => {
warn!(
owner = %ctx.owner,
repo = %ctx.repo,
error = %e,
"push rejected: repo exceeds relay resource limits"
);
return (
StatusCode::PAYLOAD_TOO_LARGE,
"repository exceeds relay resource limits",
)
.into_response();
}
Err(e) => {
// 5xx-class: ManifestReadFailed (parent corruption),
// Backend, PackCapture. The tempdir drops on scope exit; no
// pointer was written (or, on rare ManifestReadFailed during
// winner-fetch, the winner is already installed and the
// loser's data is unrelated).
error!(
owner = %ctx.owner,
repo = %ctx.repo,
error = %e,
"push failed pre-response"
);
return (StatusCode::INTERNAL_SERVER_ERROR, "git backend error").into_response();
}
};
#[cfg(test)]
if let Some(gate) = &hooks.post_cas_gate {
gate.reached.notify_one();
gate.resume.notified().await;
}
// Derived after CAS: kind:30618 ref-state event over the *committed*
// manifest's refs/head. Spec §Implementation Correspondence:
// "kind:30618 is derived after CAS, never the commit." We emit only
@@ -1927,7 +1999,7 @@ async fn finalize_push(state: &Arc<AppState>, ctx: PushContext) -> Response {
(Some(before), Some(after)) => before != after,
_ => true, // first push (parent None) or impossible-shape after key → publish
};
if manifest_changed {
let publication_result: Result<(), String> = if manifest_changed {
let inputs = RefStateInputs {
repo_id: &ctx.repo_id,
head: &success.manifest.head,
@@ -1938,11 +2010,23 @@ async fn finalize_push(state: &Arc<AppState>, ctx: PushContext) -> Response {
Ok(event) => {
// Relay-signed kind:30618 belongs to the same server-resolved
// tenant as the git request that committed the pointer.
match state
#[cfg(test)]
let insert_result = if hooks.fail_ref_state_insert {
Err(buzz_db::DbError::InvalidData(
"injected kind:30618 insert failure".to_string(),
))
} else {
state
.db
.insert_event_with_serving_write_guard(serving_write.lease(), &event, None)
.await
};
#[cfg(not(test))]
let insert_result = state
.db
.insert_event(ctx.tenant.community(), &event, None)
.await
{
.insert_event_with_serving_write_guard(serving_write.lease(), &event, None)
.await;
match insert_result {
Ok((stored, true)) => {
// Routed through the guarded send path for uniformity;
// the access gate no-ops for this globally-scoped
@@ -1959,6 +2043,7 @@ async fn finalize_push(state: &Arc<AppState>, ctx: PushContext) -> Response {
manifest = %success.manifest_key,
"kind:30618 published (derived after CAS)"
);
Ok(())
}
Ok((_, false)) => {
info!(
@@ -1966,26 +2051,41 @@ async fn finalize_push(state: &Arc<AppState>, ctx: PushContext) -> Response {
repo = %ctx.repo_id,
"kind:30618 deduplicated by relay db"
);
Ok(())
}
Err(e) => {
warn!(
owner = %ctx.owner,
repo = %ctx.repo_id,
error = %e,
"kind:30618 insert failed; push remains durable in object store"
);
}
Err(error) => Err(format!("kind:30618 insert failed: {error}")),
}
}
Err(e) => {
warn!(
owner = %ctx.owner,
repo = %ctx.repo_id,
error = %e,
"kind:30618 build failed; push remains durable in object store"
);
}
Err(error) => Err(format!("kind:30618 build failed: {error}")),
}
} else {
Ok(())
};
// The admitted serving write spans the complete publication attempt. Fence
// acquisition cannot overtake the pointer CAS, durable 30618 insert, or
// local fan-out attempt; only now may the lease be released.
if let Err(error) = serving_write.finish().await {
warn!(owner = %ctx.owner, repo = %ctx.repo, %error, "failed to release community serving lease after push publication");
return (
StatusCode::SERVICE_UNAVAILABLE,
"community write lease lost during publication",
)
.into_response();
}
if let Err(error) = publication_result {
error!(
owner = %ctx.owner,
repo = %ctx.repo_id,
manifest = %success.manifest_key,
%error,
"push pointer committed but kind:30618 publication failed"
);
return (
StatusCode::INTERNAL_SERVER_ERROR,
"push committed but ref-state publication failed; retry",
)
.into_response();
}
// Only now — after CAS commit and (optional) 30618 emission — build
@@ -2014,12 +2114,14 @@ pub fn git_router(state: Arc<AppState>) -> Router {
#[cfg(test)]
mod track_c_tests {
use super::*;
use crate::api::git::hydrate::{hydrate_for_write, HydrationOptions};
use crate::api::git::manifest::Manifest;
use buzz_core::CommunityId;
use nostr::{EventBuilder, Keys, Kind, Tag};
use std::collections::BTreeMap;
use std::io::Write;
use std::process::Output;
use tempfile::TempDir;
fn oid_sha1() -> String {
"cb09a769da1c01f458fa6959d4e8eded38fac8d3".to_string()
@@ -2160,6 +2262,303 @@ mod track_c_tests {
assert!(remote.join("refs/heads/master").exists());
}
async fn run_finalize_git(repo: &Path, args: &[&str]) -> std::process::Output {
let mut command = Command::new("git");
command.current_dir(repo).args(args);
harden_git_env(&mut command);
let output = command.output().await.expect("spawn git");
assert!(
output.status.success(),
"git {args:?}: {}",
String::from_utf8_lossy(&output.stderr)
);
output
}
async fn finalize_test_state() -> (Arc<AppState>, sqlx::PgPool) {
const TEST_DB_URL: &str = "postgres://buzz:buzz_dev@localhost:5432/buzz"; // sadscan:disable np.postgres.1
let mut config = crate::config::Config::from_env().expect("default config loads");
config.require_relay_membership = false;
config.redis_url = "redis://127.0.0.1:1".to_string();
config.database_url = std::env::var("BUZZ_TEST_DATABASE_URL")
.or_else(|_| std::env::var("DATABASE_URL"))
.unwrap_or_else(|_| TEST_DB_URL.to_string());
let pool = sqlx::PgPool::connect(&config.database_url)
.await
.expect("connect test DB");
let db = buzz_db::Db::from_pool(pool.clone());
db.migrate().await.expect("migrate test DB");
let redis_pool = deadpool_redis::Config::from_url(&config.redis_url)
.create_pool(Some(deadpool_redis::Runtime::Tokio1))
.expect("redis pool");
let pubsub = Arc::new(
buzz_pubsub::PubSubManager::new(&config.redis_url, redis_pool.clone())
.await
.expect("pubsub manager"),
);
let audit = buzz_audit::AuditService::new(pool.clone());
let auth = buzz_auth::AuthService::new(config.auth.clone());
let search = buzz_search::SearchService::new(pool.clone());
let workflow_engine = Arc::new(buzz_workflow::WorkflowEngine::new(
db.clone(),
buzz_workflow::WorkflowConfig::default(),
));
let media_storage = buzz_media::MediaStorage::new(&config.media).expect("media storage");
let (state, _audit_shutdown) = AppState::new(
config,
db,
redis_pool,
audit,
pubsub,
auth,
search,
workflow_engine,
Keys::generate(),
media_storage,
);
(Arc::new(state), pool)
}
async fn approved_deletion(
state: &AppState,
host: &str,
) -> (
buzz_db::deletion::DeletionRequest,
buzz_db::deletion::ClaimedDeletion,
) {
use buzz_db::deletion::{
FrozenInventory, KeyStreamDigest, PrefixManifest, StorageManifest,
DEFAULT_LEASE_DURATION,
};
let store = state.db.deletion_store();
let request = store
.submit(host, "git-finalize-test", Some("post-CAS lease regression"))
.await
.expect("submit deletion");
let inventory = FrozenInventory {
schema: store
.inventory_schema(request.community_id)
.await
.expect("schema inventory"),
storage: StorageManifest {
version: 4,
prefixes: buzz_media::tenant_prefixes(*request.community_id.as_uuid())
.into_iter()
.map(|prefix| PrefixManifest {
prefix,
object_count: 0,
total_bytes: 0,
keys_digest: KeyStreamDigest::new().finish().0,
})
.collect(),
},
};
store
.freeze_inventory(request.id, &inventory)
.await
.expect("freeze inventory");
store
.approve(request.id, "git-finalize-test", None)
.await
.expect("approve deletion");
let claim = store
.claim_specific(request.id, "git-finalize-test", DEFAULT_LEASE_DURATION)
.await
.expect("claim deletion")
.expect("won deletion claim");
(request, claim)
}
async fn pushed_context(
state: &AppState,
community: CommunityId,
host: &str,
owner: String,
repo: String,
pusher: nostr::PublicKey,
scratch: &Path,
) -> PushContext {
let tenant = TenantContext::resolved(community, host);
let (hydrated, parent_state) = hydrate_for_write(
&state.git_store,
&tenant,
&owner,
&repo,
HydrationOptions {
pack_cache: &state.git_pack_cache,
scratch_dir: scratch,
max_pack_bytes: 1024 * 1024,
max_repo_bytes: 2 * 1024 * 1024,
},
)
.await
.expect("hydrate empty test repo");
let source = scratch.join("source");
tokio::fs::create_dir(&source)
.await
.expect("source directory");
run_finalize_git(&source, &["init", "--quiet", "--initial-branch=main"]).await;
run_finalize_git(&source, &["config", "user.email", "finalize@test"]).await;
run_finalize_git(&source, &["config", "user.name", "finalize"]).await;
tokio::fs::write(source.join("file.txt"), b"committed\n")
.await
.expect("write source file");
run_finalize_git(&source, &["add", "file.txt"]).await;
run_finalize_git(&source, &["commit", "--quiet", "-m", "committed"]).await;
let remote = hydrated.path().to_str().expect("hydrated path utf8");
run_finalize_git(&source, &["push", "--quiet", remote, "main"]).await;
PushContext {
pack: PackOutput {
stdout: b"push-ok".to_vec(),
ok: true,
},
parent_state,
owner,
repo: repo.clone(),
repo_id: repo,
pusher,
tenant,
repo_handle: hydrated,
}
}
#[tokio::test]
#[ignore = "requires Postgres and MinIO"]
async fn finalize_push_holds_serving_lease_through_post_cas_publication() {
let (state, pool) = finalize_test_state().await;
let host = format!("git-finalize-{}.example", uuid::Uuid::new_v4().simple());
let community = state
.db
.ensure_configured_community(&host)
.await
.expect("create test community")
.id;
let (request, claim) = approved_deletion(&state, &host).await;
let scratch = TempDir::new().expect("scratch");
let owner = format!("owner-{}", uuid::Uuid::new_v4().simple());
let repo = format!("repo-{}", uuid::Uuid::new_v4().simple());
let ctx = pushed_context(
&state,
community,
&host,
owner,
repo.clone(),
Keys::generate().public_key(),
scratch.path(),
)
.await;
let gate = Arc::new(PostCasGate::default());
let hooks = FinalizePushHooks {
post_cas_gate: Some(Arc::clone(&gate)),
fail_ref_state_insert: false,
};
let finalize_state = Arc::clone(&state);
let finalize =
tokio::spawn(async move { finalize_push_inner(&finalize_state, ctx, &hooks).await });
gate.reached.notified().await;
state
.db
.deletion_store()
.begin_quiescing(&claim.lease)
.await
.expect("quiesce after CAS");
let error = state
.db
.deletion_store()
.fence(&claim.lease)
.await
.expect_err("post-CAS serving lease must block fence");
assert!(matches!(
error,
buzz_db::DbError::ServingWritesNotDrained { .. }
));
assert!(!state
.db
.deletion_store()
.is_serving_active(community)
.await
.expect("quiescing rejects new serving work"));
gate.resume.notify_one();
let response = finalize.await.expect("finalize task");
assert_eq!(response.status(), StatusCode::OK);
let mut query = buzz_db::event::EventQuery::for_community(community);
query.kinds = Some(vec![30_618]);
query.d_tag = Some(repo);
let events = state.db.query_events(&query).await.expect("query 30618");
assert_eq!(events.len(), 1, "kind:30618 must be durable before release");
assert!(state
.db
.deletion_store()
.serving_writes_drained(community)
.await
.expect("serving lease released"));
let generation = state
.db
.deletion_store()
.fence(&claim.lease)
.await
.expect("fence after publication");
assert_eq!(generation, 1);
assert_eq!(
state
.db
.deletion_store()
.get(request.id)
.await
.expect("fenced request")
.stage,
buzz_db::deletion::DeletionStage::Fenced
);
drop(state);
pool.close().await;
}
#[tokio::test]
#[ignore = "requires Postgres and MinIO"]
async fn finalize_push_db_failure_after_cas_is_not_success_and_releases_lease() {
let (state, pool) = finalize_test_state().await;
let host = format!(
"git-finalize-fail-{}.example",
uuid::Uuid::new_v4().simple()
);
let community = state
.db
.ensure_configured_community(&host)
.await
.expect("create test community")
.id;
let scratch = TempDir::new().expect("scratch");
let ctx = pushed_context(
&state,
community,
&host,
format!("owner-{}", uuid::Uuid::new_v4().simple()),
format!("repo-{}", uuid::Uuid::new_v4().simple()),
Keys::generate().public_key(),
scratch.path(),
)
.await;
let hooks = FinalizePushHooks {
post_cas_gate: None,
fail_ref_state_insert: true,
};
let response = finalize_push_inner(&state, ctx, &hooks).await;
assert_eq!(response.status(), StatusCode::INTERNAL_SERVER_ERROR);
assert!(state
.db
.deletion_store()
.serving_writes_drained(community)
.await
.expect("serving lease released on failure"));
drop(state);
pool.close().await;
}
/// A gzip-encoded request body is transparently inflated before it
/// reaches the git subprocess. Git's smart-HTTP client gzips the
/// upload-pack/receive-pack request body past a size threshold (fires

31
crates/buzz-relay/src/api/invites.rs
View File

@@ -260,6 +260,19 @@ async fn authenticate(
Ok((tenant, pubkey))
}
fn map_mint_error(error: buzz_db::DbError) -> (StatusCode, Json<Value>) {
match error {
buzz_db::DbError::InvalidData(message) | buzz_db::DbError::DeletionSafety(message) => {
api_error(StatusCode::BAD_REQUEST, &message)
}
buzz_db::DbError::AccessDenied(_) => api_error(
StatusCode::SERVICE_UNAVAILABLE,
"community writes are temporarily unavailable",
),
error => internal_error(&format!("invite mint: {error}")),
}
}
/// Mint an invite code — `POST /api/invites`, NIP-98 signed by an owner/admin.
///
/// Returns the code, its expiry, and a shareable landing-page URL on the
@@ -304,10 +317,7 @@ pub async fn mint_invite(
.db
.mint_relay_invite(tenant.community(), &sender_hex, ttl, max_uses)
.await
.map_err(|error| match error {
buzz_db::DbError::InvalidData(message) => api_error(StatusCode::BAD_REQUEST, &message),
error => internal_error(&format!("invite mint: {error}")),
})?;
.map_err(map_mint_error)?;
// Same TLS-posture logic as nip98_expected_url: wss deployments get an
// https landing page URL, ws dev/test deployments get http.
@@ -895,6 +905,19 @@ mod tests {
}
}
#[test]
fn mint_fence_errors_map_to_temporary_unavailability() {
let (status, body) = super::map_mint_error(buzz_db::DbError::AccessDenied(
"community is write-fenced".to_string(),
));
assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE);
assert_eq!(
body.0.get("error").and_then(Value::as_str),
Some("community writes are temporarily unavailable")
);
}
#[tokio::test]
#[ignore = "requires Postgres"]
async fn malformed_and_unknown_v2_codes_are_forbidden_without_v1_fallback() {

172
crates/buzz-relay/src/api/media.rs
View File

@@ -283,6 +283,19 @@ async fn upload_attribution(
})
}
fn serving_write_error(error: anyhow::Error) -> MediaError {
if buzz_deletion::ServingWriteGuard::acquisition_is_fenced(&error) {
MediaError::CommunityWriteFenced
} else {
MediaError::ServiceUnavailable
}
}
fn serving_lease_lost(error: anyhow::Error) -> MediaError {
tracing::warn!(%error, "media serving-write lease lost");
MediaError::ServiceUnavailable
}
/// PUT `/upload` or the temporary media-only `/media/upload` alias.
///
/// Auth is validated via the [`AuthenticatedUpload`] extractor BEFORE the body
@@ -310,6 +323,11 @@ pub async fn upload_blob(
) -> Result<Json<BlobDescriptor>, MediaError> {
let attribution = upload_attribution(&state, &auth, &headers).await;
let serving_write =
buzz_deletion::acquire_serving_write(&state.db, auth.tenant.community(), "media_upload")
.await
.map_err(serving_write_error)?;
if auth.route_mode == UploadRouteMode::LegacyMedia {
metrics::counter!("buzz_media_legacy_upload_route_total").increment(1);
}
@@ -335,69 +353,86 @@ pub async fn upload_blob(
}
let replay = futures_util::stream::iter(replay_chunks.into_iter().map(Ok)).chain(source);
let mut descriptor = if should_stream_as_video(&sniff) {
// Video path: stream body directly to disk — never fully buffered in RAM.
let content_length = headers
.get("content-length")
.and_then(|v| v.to_str().ok())
.and_then(|v| v.parse::<u64>().ok());
buzz_media::process_video_upload(
&state.media_storage,
&state.config.media,
&auth.tenant,
&auth.auth_event,
replay,
content_length,
attribution,
)
.await?
} else {
// Non-video path: buffer the body (bounded by the larger of the image
// and generic-file caps), then decide image-vs-generic by sniffed MIME.
// Images go through the thumbnailing pipeline; non-media attachments
// (docs, archives, text, data) take the generic file path and are
// served as downloads. Recognized audio/video cannot fall through it.
let max = state
.config
.media
.max_image_bytes
.max(state.config.media.max_file_bytes);
let bytes = axum::body::to_bytes(axum::body::Body::from_stream(replay), max as usize)
.await
.map_err(|_| MediaError::FileTooLarge { size: 0, max })?;
serving_write.verify().await.map_err(serving_lease_lost)?;
let is_image = matches!(
infer::get(&bytes).map(|t| t.mime_type()),
Some("image/jpeg" | "image/png" | "image/gif" | "image/webp")
);
let mut descriptor = serving_write
.protect(async {
Ok(if should_stream_as_video(&sniff) {
// Video path: stream body directly to disk — never fully buffered in RAM.
let content_length = headers
.get("content-length")
.and_then(|v| v.to_str().ok())
.and_then(|v| v.parse::<u64>().ok());
buzz_media::process_video_upload(
&state.media_storage,
&state.config.media,
&auth.tenant,
&auth.auth_event,
replay,
content_length,
attribution,
)
.await?
} else {
// Non-video path: buffer the body (bounded by the larger of the image
// and generic-file caps), then decide image-vs-generic by sniffed MIME.
// Images go through the thumbnailing pipeline; non-media attachments
// (docs, archives, text, data) take the generic file path and are
// served as downloads. Recognized audio/video cannot fall through it.
let max = state
.config
.media
.max_image_bytes
.max(state.config.media.max_file_bytes);
let bytes =
axum::body::to_bytes(axum::body::Body::from_stream(replay), max as usize)
.await
.map_err(|_| MediaError::FileTooLarge { size: 0, max })?;
if is_image {
buzz_media::process_upload(
&state.media_storage,
&state.config.media,
&auth.tenant,
&auth.auth_event,
bytes,
attribution,
)
.await?
} else if auth.route_mode == UploadRouteMode::LegacyMedia {
let mime = infer::get(&bytes)
.map(|kind| kind.mime_type().to_string())
.unwrap_or_else(|| "application/octet-stream".to_string());
return Err(MediaError::DisallowedContentType(mime));
} else {
buzz_media::process_file_upload(
&state.media_storage,
&state.config.media,
&auth.tenant,
&auth.auth_event,
bytes,
attribution,
)
.await?
}
};
let is_image = matches!(
infer::get(&bytes).map(|t| t.mime_type()),
Some("image/jpeg" | "image/png" | "image/gif" | "image/webp")
);
if is_image {
buzz_media::process_upload(
&state.media_storage,
&state.config.media,
&auth.tenant,
&auth.auth_event,
bytes,
attribution,
)
.await?
} else if auth.route_mode == UploadRouteMode::LegacyMedia {
let mime = infer::get(&bytes)
.map(|kind| kind.mime_type().to_string())
.unwrap_or_else(|| "application/octet-stream".to_string());
return Err(MediaError::DisallowedContentType(mime));
} else {
buzz_media::process_file_upload(
&state.media_storage,
&state.config.media,
&auth.tenant,
&auth.auth_event,
bytes,
attribution,
)
.await?
}
})
})
.await
.map_err(|error| {
if buzz_deletion::ServingWriteGuard::is_lease_lost(&error) {
serving_lease_lost(error)
} else {
match error.downcast::<MediaError>() {
Ok(error) => error,
Err(_) => MediaError::Internal,
}
}
})??;
rewrite_descriptor_urls_for_tenant(
&mut descriptor,
@@ -441,6 +476,7 @@ pub async fn upload_blob(
}
}
serving_write.finish().await.map_err(serving_lease_lost)?;
Ok(Json(descriptor))
}
@@ -913,6 +949,20 @@ mod tests {
const VALID_HASH: &str = "abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789";
#[test]
fn serving_write_error_taxonomy_separates_fence_from_backend_failure() {
let fenced = anyhow::Error::from(buzz_db::DbError::AccessDenied("fenced".to_string()));
assert!(matches!(
serving_write_error(fenced),
MediaError::CommunityWriteFenced
));
let backend = anyhow::Error::from(buzz_db::DbError::Sqlx(sqlx::Error::PoolTimedOut));
assert!(matches!(
serving_write_error(backend),
MediaError::ServiceUnavailable
));
}
#[test]
fn upload_routes_distinguish_standard_and_legacy_modes() {
assert_eq!(

105
crates/buzz-relay/src/audio/handler.rs
View File

@@ -25,7 +25,7 @@ use bytes::Bytes;
use futures_util::{SinkExt, StreamExt};
use nostr::{EventBuilder, Kind, Tag};
use serde::Deserialize;
use tokio::sync::{mpsc, OwnedSemaphorePermit, Semaphore};
use tokio::sync::{mpsc, watch, OwnedSemaphorePermit, Semaphore};
use tokio_util::sync::CancellationToken;
use tracing::{debug, error, info, warn};
use uuid::Uuid;
@@ -38,7 +38,7 @@ use buzz_core::StoredEvent;
use buzz_pubsub::EventTopic;
use crate::audio::room::PeerCtrl;
use crate::state::{run_registered_community_connection, AppState};
use crate::state::{run_registered_community_connection, AppState, CommunityConnectionControl};
/// Maximum binary frame size: 4 KB is generous for a single Opus packet.
const MAX_AUDIO_FRAME_BYTES: usize = 4096;
@@ -149,6 +149,7 @@ async fn handle_audio_connection(
_permit: OwnedSemaphorePermit,
) {
let cancel = CancellationToken::new();
let control = CommunityConnectionControl::new(cancel);
let community_id = tenant.community();
let registry = Arc::clone(&state.community_connections);
let check_state = Arc::clone(&state);
@@ -157,9 +158,11 @@ async fn handle_audio_connection(
&registry,
Uuid::new_v4(),
community_id,
cancel.clone(),
control,
move || async move { check_state.db.is_community_active(community_id).await },
move || handle_active_audio_connection(socket, run_state, tenant, channel_id, cancel),
move |control| {
handle_active_audio_connection(socket, run_state, tenant, channel_id, control)
},
)
.await;
}
@@ -169,8 +172,10 @@ async fn handle_active_audio_connection(
state: Arc<AppState>,
tenant: TenantContext,
channel_id: Uuid,
cancel: CancellationToken,
control: CommunityConnectionControl,
) {
let cancel = control.cancellation_token();
let disconnect_reason = control.disconnect_reason();
let (mut ws_send, mut ws_recv) = socket.split();
let challenge = generate_challenge();
@@ -660,7 +665,13 @@ async fn handle_active_audio_connection(
let (ctrl_tx, ctrl_rx) = mpsc::channel::<WsMessage>(8);
let send_cancel = cancel.child_token();
let send_task = tokio::spawn(send_loop(ws_send, data_rx, ctrl_rx, send_cancel));
let send_task = tokio::spawn(send_loop(
ws_send,
data_rx,
ctrl_rx,
send_cancel,
disconnect_reason,
));
let hb_cancel = cancel.clone();
let hb_missed = Arc::clone(&missed_pongs);
@@ -1056,12 +1067,15 @@ async fn recv_loop(
///
/// Control frames (Ping, Pong, Close, control JSON) are drained first on every
/// iteration, so heartbeat pings are never starved by audio backpressure.
async fn send_loop(
mut ws_send: futures_util::stream::SplitSink<WebSocket, WsMessage>,
async fn send_loop<S>(
mut ws_send: S,
mut data_rx: mpsc::Receiver<WsMessage>,
mut ctrl_rx: mpsc::Receiver<WsMessage>,
cancel: CancellationToken,
) {
disconnect_reason: watch::Receiver<Option<crate::state::CommunityDisconnectReason>>,
) where
S: futures_util::Sink<WsMessage> + Unpin,
{
loop {
// Priority: drain all pending control frames before data.
while let Ok(ctrl_msg) = ctrl_rx.try_recv() {
@@ -1073,7 +1087,10 @@ async fn send_loop(
tokio::select! {
biased;
_ = cancel.cancelled() => {
let _ = ws_send.send(WsMessage::Close(None)).await;
let close = disconnect_reason
.borrow()
.map_or(WsMessage::Close(None), |reason| reason.close_message());
let _ = ws_send.send(close).await;
break;
}
Some(ctrl_msg) = ctrl_rx.recv() => {
@@ -1416,6 +1433,74 @@ mod tests {
received
}
#[tokio::test]
async fn audio_send_loop_sends_policy_close_when_community_is_deleted() {
use futures_util::Sink;
struct MockSink {
messages: Arc<Mutex<Vec<WsMessage>>>,
}
impl Sink<WsMessage> for MockSink {
type Error = std::io::Error;
fn poll_ready(
self: std::pin::Pin<&mut Self>,
_cx: &mut std::task::Context<'_>,
) -> std::task::Poll<Result<(), Self::Error>> {
std::task::Poll::Ready(Ok(()))
}
fn start_send(
self: std::pin::Pin<&mut Self>,
item: WsMessage,
) -> Result<(), Self::Error> {
self.messages.lock().expect("mock sink poisoned").push(item);
Ok(())
}
fn poll_flush(
self: std::pin::Pin<&mut Self>,
_cx: &mut std::task::Context<'_>,
) -> std::task::Poll<Result<(), Self::Error>> {
std::task::Poll::Ready(Ok(()))
}
fn poll_close(
self: std::pin::Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
) -> std::task::Poll<Result<(), Self::Error>> {
self.poll_flush(cx)
}
}
let (_data_tx, data_rx) = mpsc::channel(1);
let (_ctrl_tx, ctrl_rx) = mpsc::channel(1);
let cancel = CancellationToken::new();
let control = CommunityConnectionControl::new(cancel.clone());
let disconnect_reason = control.disconnect_reason();
let registry = crate::state::CommunityConnectionRegistry::new();
let community = buzz_core::CommunityId::from_uuid(Uuid::new_v4());
let _guard = registry.register(Uuid::new_v4(), community, control);
assert_eq!(registry.disconnect_community(community), 1);
let messages = Arc::new(Mutex::new(Vec::new()));
let sink = MockSink {
messages: Arc::clone(&messages),
};
send_loop(sink, data_rx, ctrl_rx, cancel, disconnect_reason).await;
let messages = messages.lock().expect("mock sink poisoned");
assert_eq!(messages.len(), 1);
match &messages[0] {
WsMessage::Close(Some(close)) => {
assert_eq!(close.code, axum::extract::ws::close_code::POLICY);
assert_eq!(close.reason.as_str(), "community deleted");
}
other => panic!("expected one 1008 deletion close, got {other:?}"),
}
}
#[tokio::test]
async fn audio_websocket_parser_rejects_oversized_messages_before_handler_reads_them() {
assert!(

170
crates/buzz-relay/src/connection.rs
View File

@@ -8,7 +8,7 @@ use std::time::Duration;
use axum::extract::ws::{Message as WsMessage, WebSocket};
use futures_util::{Sink, SinkExt, StreamExt};
use tokio::sync::{mpsc, Mutex, RwLock};
use tokio::sync::{mpsc, watch, Mutex, RwLock};
use tokio_util::sync::CancellationToken;
use tracing::Instrument as _;
use tracing::{debug, info, trace, warn};
@@ -20,7 +20,10 @@ use nostr::Filter;
use crate::handlers;
use crate::protocol::{ClientMessage, RelayMessage};
use crate::state::{run_registered_community_connection, AppState};
use crate::state::{
run_registered_community_connection, AppState, CommunityConnectionControl,
CommunityDisconnectReason,
};
use buzz_pubsub::EventTopic;
/// Maximum time a new socket may hold a connection slot without completing NIP-42 auth.
@@ -128,6 +131,7 @@ pub async fn handle_connection(
) {
let conn_id = Uuid::new_v4();
let cancel = CancellationToken::new();
let control = CommunityConnectionControl::new(cancel);
let community_id = tenant.community();
let registry = Arc::clone(&state.community_connections);
let check_state = Arc::clone(&state);
@@ -136,9 +140,9 @@ pub async fn handle_connection(
&registry,
conn_id,
community_id,
cancel.clone(),
control,
move || async move { check_state.db.is_community_active(community_id).await },
move || handle_active_connection(socket, run_state, addr, tenant, conn_id, cancel),
move |control| handle_active_connection(socket, run_state, addr, tenant, conn_id, control),
)
.await;
}
@@ -149,8 +153,10 @@ async fn handle_active_connection(
addr: SocketAddr,
tenant: TenantContext,
conn_id: Uuid,
cancel: CancellationToken,
control: CommunityConnectionControl,
) {
let cancel = control.cancellation_token();
let disconnect_reason = control.disconnect_reason();
let permit = match state.conn_semaphore.clone().try_acquire_owned() {
Ok(p) => p,
Err(_) => {
@@ -226,7 +232,14 @@ async fn handle_active_connection(
let (ws_send, ws_recv) = socket.split();
let send_cancel = cancel.child_token();
let send_task = tokio::spawn(send_loop(ws_send, rx, ctrl_rx, restart_rx, send_cancel));
let send_task = tokio::spawn(send_loop(
ws_send,
rx,
ctrl_rx,
restart_rx,
send_cancel,
disconnect_reason,
));
let missed_pongs = Arc::new(AtomicU8::new(0));
let heartbeat_cancel = cancel.clone();
@@ -310,8 +323,17 @@ async fn send_loop(
ctrl_rx: mpsc::Receiver<WsMessage>,
restart_rx: mpsc::Receiver<RestartClose>,
cancel: CancellationToken,
disconnect_reason: watch::Receiver<Option<CommunityDisconnectReason>>,
) {
send_loop_inner(ws_send, data_rx, ctrl_rx, restart_rx, cancel).await;
send_loop_inner(
ws_send,
data_rx,
ctrl_rx,
restart_rx,
cancel,
disconnect_reason,
)
.await;
}
async fn send_loop_inner<S>(
@@ -320,6 +342,7 @@ async fn send_loop_inner<S>(
mut ctrl_rx: mpsc::Receiver<WsMessage>,
mut restart_rx: mpsc::Receiver<RestartClose>,
cancel: CancellationToken,
disconnect_reason: watch::Receiver<Option<CommunityDisconnectReason>>,
) where
S: Sink<WsMessage> + Unpin,
{
@@ -359,7 +382,10 @@ async fn send_loop_inner<S>(
break;
}
}
let _ = ws_send.send(WsMessage::Close(None)).await;
let close = disconnect_reason
.borrow()
.map_or(WsMessage::Close(None), |reason| reason.close_message());
let _ = ws_send.send(close).await;
break;
}
Some(ctrl_msg) = ctrl_rx.recv() => {
@@ -787,6 +813,17 @@ mod tests {
}
}
fn ordinary_disconnect_reason() -> watch::Receiver<Option<CommunityDisconnectReason>> {
let (_tx, rx) = watch::channel(None);
rx
}
fn deleted_community_disconnect_reason() -> watch::Receiver<Option<CommunityDisconnectReason>> {
let (tx, rx) = watch::channel(None);
tx.send_replace(Some(CommunityDisconnectReason::CommunityDeleted));
rx
}
fn text_payloads(messages: &[WsMessage]) -> Vec<String> {
messages
.iter()
@@ -823,7 +860,15 @@ mod tests {
let (sink, state) = MockSink::new(Some(1));
let (_restart_tx, restart_rx) = mpsc::channel(1);
send_loop_inner(sink, data_rx, ctrl_rx, restart_rx, CancellationToken::new()).await;
send_loop_inner(
sink,
data_rx,
ctrl_rx,
restart_rx,
CancellationToken::new(),
ordinary_disconnect_reason(),
)
.await;
let state = state.lock().expect("mock sink poisoned");
assert_eq!(state.flush_count, 1);
@@ -844,7 +889,15 @@ mod tests {
let (sink, state) = MockSink::new(Some(1));
let (_restart_tx, restart_rx) = mpsc::channel(1);
send_loop_inner(sink, data_rx, ctrl_rx, restart_rx, CancellationToken::new()).await;
send_loop_inner(
sink,
data_rx,
ctrl_rx,
restart_rx,
CancellationToken::new(),
ordinary_disconnect_reason(),
)
.await;
let state = state.lock().expect("mock sink poisoned");
assert_eq!(state.flush_count, 1);
@@ -870,7 +923,15 @@ mod tests {
let (sink, state) = MockSink::new(Some(2));
let (_restart_tx, restart_rx) = mpsc::channel(1);
send_loop_inner(sink, data_rx, ctrl_rx, restart_rx, CancellationToken::new()).await;
send_loop_inner(
sink,
data_rx,
ctrl_rx,
restart_rx,
CancellationToken::new(),
ordinary_disconnect_reason(),
)
.await;
let state = state.lock().expect("mock sink poisoned");
assert_eq!(state.flush_count, 2);
@@ -894,7 +955,15 @@ mod tests {
.expect("queue restart close");
let (sink, state) = MockSink::new(None);
send_loop_inner(sink, data_rx, ctrl_rx, restart_rx, CancellationToken::new()).await;
send_loop_inner(
sink,
data_rx,
ctrl_rx,
restart_rx,
CancellationToken::new(),
ordinary_disconnect_reason(),
)
.await;
assert_eq!(flushed_rx.await, Ok(true));
let state = state.lock().expect("mock sink poisoned");
@@ -923,7 +992,15 @@ mod tests {
.expect("queue restart close");
let (sink, state) = MockSink::new(Some(1));
send_loop_inner(sink, data_rx, ctrl_rx, restart_rx, CancellationToken::new()).await;
send_loop_inner(
sink,
data_rx,
ctrl_rx,
restart_rx,
CancellationToken::new(),
ordinary_disconnect_reason(),
)
.await;
assert_eq!(flushed_rx.await, Ok(false));
let state = state.lock().expect("mock sink poisoned");
@@ -931,6 +1008,59 @@ mod tests {
assert_eq!(state.messages.len(), 1, "no fallback close is appended");
}
#[tokio::test]
async fn send_loop_sends_policy_close_when_community_is_deleted() {
let (_data_tx, data_rx) = mpsc::channel(1);
let (_ctrl_tx, ctrl_rx) = mpsc::channel(1);
let (_restart_tx, restart_rx) = mpsc::channel(1);
let cancel = CancellationToken::new();
cancel.cancel();
let (sink, state) = MockSink::new(None);
send_loop_inner(
sink,
data_rx,
ctrl_rx,
restart_rx,
cancel,
deleted_community_disconnect_reason(),
)
.await;
let state = state.lock().expect("mock sink poisoned");
assert_eq!(state.messages.len(), 1);
match &state.messages[0] {
WsMessage::Close(Some(close)) => {
assert_eq!(close.code, axum::extract::ws::close_code::POLICY);
assert_eq!(close.reason.as_str(), "community deleted");
}
other => panic!("expected one 1008 deletion close, got {other:?}"),
}
}
#[tokio::test]
async fn send_loop_sends_bare_close_for_ordinary_cancellation() {
let (_data_tx, data_rx) = mpsc::channel(1);
let (_ctrl_tx, ctrl_rx) = mpsc::channel(1);
let (_restart_tx, restart_rx) = mpsc::channel(1);
let cancel = CancellationToken::new();
cancel.cancel();
let (sink, state) = MockSink::new(None);
send_loop_inner(
sink,
data_rx,
ctrl_rx,
restart_rx,
cancel,
ordinary_disconnect_reason(),
)
.await;
let state = state.lock().expect("mock sink poisoned");
assert_eq!(state.messages.as_slice(), [WsMessage::Close(None)]);
}
#[tokio::test]
async fn send_loop_flushes_queued_control_before_close_on_cancel() {
// A ban disconnect queues its `OK false "blocked: …"` reason frame on
@@ -951,7 +1081,15 @@ mod tests {
let (sink, state) = MockSink::new(None);
let (_restart_tx, restart_rx) = mpsc::channel(1);
send_loop_inner(sink, data_rx, ctrl_rx, restart_rx, cancel).await;
send_loop_inner(
sink,
data_rx,
ctrl_rx,
restart_rx,
cancel,
ordinary_disconnect_reason(),
)
.await;
let state = state.lock().expect("mock sink poisoned");
assert_eq!(
@@ -966,8 +1104,8 @@ mod tests {
other => panic!("expected the ban reason frame first, got {other:?}"),
}
assert!(
matches!(state.messages[1], WsMessage::Close(_)),
"Close is sent only after the reason frame is flushed"
matches!(state.messages[1], WsMessage::Close(None)),
"ordinary cancellation retains the bare Close after the reason frame"
);
}
}

6
crates/buzz-relay/src/handlers/command_executor.rs
View File

@@ -112,6 +112,12 @@ async fn persist_command_event(
.begin_transaction()
.await
.map_err(|e| IngestError::Internal(format!("error: begin transaction: {e}")))?;
buzz_deletion::store(&state.db)
.guard_transaction(&mut tx, tenant.community())
.await
.map_err(|error| {
IngestError::Rejected(format!("restricted: community writes are fenced: {error}"))
})?;
// INSERT with ON CONFLICT DO NOTHING — idempotency guard.
let id_bytes = event.id.as_bytes();

83
crates/buzz-relay/src/handlers/event.rs
View File

@@ -705,16 +705,49 @@ pub async fn handle_event(event: Event, conn: Arc<ConnectionState>, state: Arc<A
));
return;
}
handle_ephemeral_event(
match buzz_deletion::store(&state.db)
.is_serving_active(conn.tenant.community())
.await
{
Ok(true) => {}
Ok(false) => {
reject("restricted");
conn.send(RelayMessage::ok(
&event_id_hex,
false,
"restricted: community writes are fenced",
));
return;
}
Err(error) => {
reject("error");
tracing::warn!(%error, event_id = %event_id_hex, "failed to check ephemeral-event community lifecycle");
conn.send(RelayMessage::ok(
&event_id_hex,
false,
"error: internal server error",
));
return;
}
}
match handle_ephemeral_event(
event,
conn_id,
&event_id_hex,
pubkey_bytes,
auth_pubkey,
conn,
Arc::clone(&conn),
state,
)
.await;
.await
{
Ok(()) => {
conn.send(RelayMessage::ok(&event_id_hex, true, ""));
}
Err(message) => {
reject("invalid");
conn.send(RelayMessage::ok(&event_id_hex, false, &message));
}
}
return;
}
@@ -762,33 +795,19 @@ pub async fn handle_event(event: Event, conn: Arc<ConnectionState>, state: Arc<A
async fn handle_ephemeral_event(
event: Event,
conn_id: uuid::Uuid,
event_id_hex: &str,
pubkey_bytes: Vec<u8>,
auth_pubkey: nostr::PublicKey,
conn: Arc<ConnectionState>,
state: Arc<AppState>,
) {
) -> Result<(), String> {
let event_clone = event.clone();
let event_id = event.id.to_hex();
let verify_result = tokio::task::spawn_blocking(move || verify_event(&event_clone)).await;
match verify_result {
Ok(Ok(())) => {}
Ok(Err(e)) => {
conn.send(RelayMessage::ok(
event_id_hex,
false,
&format!("invalid: {e}"),
));
return;
}
Err(_) => {
conn.send(RelayMessage::ok(
event_id_hex,
false,
"error: internal error",
));
return;
}
Ok(Err(e)) => return Err(format!("invalid: {e}")),
Err(_) => return Err("error: internal error".to_string()),
}
// Special handling for presence events (kind:20001).
@@ -829,18 +848,8 @@ async fn handle_ephemeral_event(
// Check channel membership before publishing other ephemeral events.
if let Some(ch_id) = super::ingest::extract_channel_id(&event) {
if let Err(msg) = super::ingest::check_channel_membership(
&conn.tenant,
&state,
ch_id,
&pubkey_bytes,
None,
)
.await
{
conn.send(RelayMessage::ok(event_id_hex, false, &msg));
return;
}
super::ingest::check_channel_membership(&conn.tenant, &state, ch_id, &pubkey_bytes, None)
.await?;
// Mark as local before Redis publish to prevent double-delivery when
// the event comes back through the Redis subscriber loop.
@@ -854,7 +863,7 @@ async fn handle_ephemeral_event(
state
.local_event_ids
.invalidate(&(conn.tenant.community(), event.id.to_bytes()));
warn!(conn_id = %conn_id, event_id = %event_id_hex, "Ephemeral publish failed: {e}");
warn!(conn_id = %conn_id, event_id = %event_id, "Ephemeral publish failed: {e}");
}
// Direct fan-out to local WS subscribers, through the guarded send path
@@ -882,7 +891,7 @@ async fn handle_ephemeral_event(
state
.local_event_ids
.invalidate(&(conn.tenant.community(), event.id.to_bytes()));
warn!(conn_id = %conn_id, event_id = %event_id_hex, "Ephemeral global publish failed: {e}");
warn!(conn_id = %conn_id, event_id = %event_id, "Ephemeral global publish failed: {e}");
}
// Direct fan-out to local WS subscribers through the guarded send path.
@@ -893,7 +902,7 @@ async fn handle_ephemeral_event(
fan_out_event_to_local_subscribers(&state, conn.tenant.community(), &stored_event).await;
}
conn.send(RelayMessage::ok(event_id_hex, true, ""));
Ok(())
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]

146
crates/buzz-relay/src/handlers/ingest.rs
View File

@@ -299,6 +299,24 @@ pub enum IngestError {
Internal(String),
}
/// Map the durable community write-fence lookup onto the ingest error taxonomy.
///
/// An inactive community is an authorization decision and keeps the exact
/// `restricted:` wire text the ephemeral path uses. A lookup outage is a
/// server fault and fails closed as `error:`/500 — a Postgres blip can
/// neither admit a write past the fence nor read as a client mistake.
fn map_serving_fence_state(active: Result<bool, buzz_db::DbError>) -> Result<(), IngestError> {
match active {
Ok(true) => Ok(()),
Ok(false) => Err(IngestError::Rejected(
"restricted: community writes are fenced".into(),
)),
Err(error) => Err(IngestError::Internal(format!(
"error: checking community write fence: {error}"
))),
}
}
fn map_relay_admin_error(error: super::relay_admin::RelayAdminError) -> IngestError {
use super::relay_admin::RelayAdminError;
match error {
@@ -1931,6 +1949,17 @@ async fn ingest_event_inner(
let kind_u32 = event_kind_u32(&event);
debug!(event_id = %event_id_hex, kind = kind_u32, "ingest_event");
// Durable community write fence: persistent ingest is a DB write the
// deletion engine cannot exclude via serving-write leases (those cover
// external side effects only), so the shared WS/HTTP seam must refuse
// writes once the community leaves the active lifecycle state. Row churn
// inside the remaining race window is swept by the destructive DB stage.
map_serving_fence_state(
buzz_deletion::store(&state.db)
.is_serving_active(tenant.community())
.await,
)?;
if kind_u32 == KIND_AUTH {
return Err(IngestError::Rejected(
"invalid: AUTH events cannot be submitted".into(),
@@ -3171,6 +3200,123 @@ mod tests {
}
}
/// An active community passes the durable write fence untouched.
#[test]
fn serving_fence_active_community_admits_write() {
assert!(map_serving_fence_state(Ok(true)).is_ok());
}
/// A fenced/tombstoned/archived community is an authorization decision:
/// `restricted:` and (via `bridge.rs`) HTTP 400 — with the exact wire text
/// the ephemeral WS path uses, so clients see one refusal vocabulary.
#[test]
fn serving_fence_inactive_community_maps_to_restricted() {
match map_serving_fence_state(Ok(false)) {
Err(IngestError::Rejected(msg)) => {
assert_eq!(msg, "restricted: community writes are fenced");
}
other => panic!("fenced community must map to Rejected, got {other:?}"),
}
}
/// A fence-lookup outage is a server fault and must fail closed as
/// `error:`/500 — a Postgres blip can neither admit a write past the
/// fence nor be reported to an innocent client as a bad request.
#[test]
fn serving_fence_lookup_outage_fails_closed_as_internal() {
let outage = buzz_db::DbError::Sqlx(sqlx::Error::PoolTimedOut);
match map_serving_fence_state(Err(outage)) {
Err(IngestError::Internal(msg)) => {
assert!(
msg.starts_with("error: "),
"fence outages need the `error:` NIP-01 prefix, got {msg:?}"
);
}
other => panic!("fence lookup failure must map to Internal, got {other:?}"),
}
}
/// Production-path regression: the exact predicate `ingest_event_inner`
/// consults must admit writes while a community is active and refuse them
/// once the community deletion lifecycle fences it.
#[tokio::test]
#[ignore = "requires Postgres"]
async fn ingest_write_fence_follows_community_deletion_lifecycle() {
use buzz_db::deletion::{
FrozenInventory, KeyStreamDigest, PrefixManifest, StorageManifest,
DEFAULT_LEASE_DURATION,
};
let url = std::env::var("BUZZ_TEST_DATABASE_URL")
.or_else(|_| std::env::var("DATABASE_URL"))
.unwrap_or_else(|_| "postgres://buzz:buzz_dev@localhost:5432/buzz".to_string()); // sadscan:disable np.postgres.1
let pool = sqlx::PgPool::connect(&url).await.expect("connect test DB");
let db = buzz_db::Db::from_pool(pool);
db.migrate().await.expect("migrate test DB");
let store = buzz_deletion::store(&db);
let host = format!("lane3-fence-{}.example", Uuid::new_v4().simple());
let community = db
.ensure_configured_community(&host)
.await
.expect("community")
.id;
assert!(
map_serving_fence_state(store.is_serving_active(community).await).is_ok(),
"active community must admit persistent ingest"
);
let submitted = store
.submit(
&host,
"test-operator",
Some("lane3 ingest fence regression"),
)
.await
.expect("submit");
let inventory = FrozenInventory {
schema: store
.inventory_schema(community)
.await
.expect("schema inventory"),
storage: StorageManifest {
version: 4,
prefixes: buzz_media::tenant_prefixes(*community.as_uuid())
.into_iter()
.map(|prefix| PrefixManifest {
prefix,
object_count: 0,
total_bytes: 0,
keys_digest: KeyStreamDigest::new().finish().0,
})
.collect(),
},
};
let request = store
.freeze_inventory(submitted.id, &inventory)
.await
.expect("freeze inventory");
store
.approve(request.id, "approver", None)
.await
.expect("approve");
let claim = store
.claim_specific(request.id, "executor", DEFAULT_LEASE_DURATION)
.await
.expect("claim")
.expect("won claim");
store.begin_quiescing(&claim.lease).await.expect("quiesce");
store.fence(&claim.lease).await.expect("fence");
match map_serving_fence_state(store.is_serving_active(community).await) {
Err(IngestError::Rejected(msg)) => {
assert_eq!(msg, "restricted: community writes are fenced");
}
other => panic!("fenced community must refuse persistent ingest, got {other:?}"),
}
}
#[derive(Debug, Default)]
struct VecTracer {
steps: Mutex<Vec<TraceStep>>,

25
crates/buzz-relay/src/main.rs
View File

@@ -201,6 +201,12 @@ async fn main() -> anyhow::Result<()> {
error!("Failed to ensure partitions: {e}");
}
db.validate_deletion_serving_catalog().await.map_err(|e| {
error!("Community deletion serving-fence validation failed: {e}");
anyhow::anyhow!("Community deletion serving fence is unsafe: {e}")
})?;
info!("Community deletion serving fences verified");
// Freshness fence probe: cursor pages route to the replica only for
// history the probe has verified as fully replayed. Deliberately AFTER
// the migration decision: spawn_fence_probe first verifies the
@@ -469,6 +475,7 @@ async fn main() -> anyhow::Result<()> {
if let Some(handle) = buzz_relay::mesh_boot::boot_mesh(
&state.config,
state.redis_pool.clone(),
state.db.clone(),
&state.relay_keypair,
Arc::clone(&state.shutting_down),
)
@@ -1018,6 +1025,24 @@ async fn main() -> anyhow::Result<()> {
metrics::gauge!("buzz_redis_pool_size").set(rs.size as f64);
metrics::gauge!("buzz_redis_pool_max").set(rs.max_size as f64);
metrics::gauge!("buzz_redis_pool_waiting").set(rs.waiting as f64);
let deletion_store = pool_state.db.deletion_store();
match deletion_store.reap_expired_serving_write_leases(1000).await {
Ok(reaped) => metrics::counter!("buzz_deletion_serving_leases_reaped_total")
.increment(reaped),
Err(error) => tracing::warn!(%error, "serving-lease reaper failed"),
}
match deletion_store.serving_lease_stats().await {
Ok(stats) => {
metrics::gauge!("buzz_deletion_serving_leases_active")
.set(stats.active as f64);
metrics::gauge!("buzz_deletion_serving_leases_expired")
.set(stats.expired as f64);
metrics::gauge!("buzz_deletion_serving_leases_dead_tuples")
.set(stats.dead_tuples as f64);
}
Err(error) => tracing::warn!(%error, "serving-lease metrics failed"),
}
}
});
}

11
crates/buzz-relay/src/mesh_boot.rs
View File

@@ -411,6 +411,7 @@ fn advertise_addrs(endpoint: &MeshEndpoint) -> Vec<String> {
pub async fn boot_mesh(
config: &Config,
redis_pool: deadpool_redis::Pool,
db: buzz_db::Db,
relay_keypair: &nostr::Keys,
shutting_down: Arc<AtomicBool>,
) -> anyhow::Result<Option<MeshHandle>> {
@@ -508,7 +509,7 @@ pub async fn boot_mesh(
transport.set_inbound(Box::new(dispatcher.clone()));
Ok(Some(MeshHandle {
directory: SessionDirectory::new(redis_pool),
directory: SessionDirectory::with_db(redis_pool, db),
transport,
membership: membership_arc,
local_runtime_id: runtime_id,
@@ -535,7 +536,13 @@ mod tests {
.create_pool(Some(deadpool_redis::Runtime::Tokio1))
.unwrap();
let keys = nostr::Keys::generate();
let handle = boot_mesh(&config, pool, &keys, Arc::new(AtomicBool::new(false)))
let db = buzz_db::Db::from_pool(
sqlx::postgres::PgPoolOptions::new()
.max_connections(1)
.connect_lazy("postgres://unused:unused@127.0.0.1:1/unused")
.expect("lazy database pool"),
);
let handle = boot_mesh(&config, pool, db, &keys, Arc::new(AtomicBool::new(false)))
.await
.expect("off path is never an error");
assert!(handle.is_none());

35
crates/buzz-relay/src/push_runtime.rs
View File

@@ -418,6 +418,23 @@ async fn deliver_one(
return;
}
};
let serving_write = match buzz_deletion::acquire_serving_write(
&state.db,
outcome.community,
"push_delivery",
)
.await
{
Ok(guard) => guard,
Err(error) => {
warn!(wake=%outcome.id, %error, "push delivery suppressed by community deletion fence");
let _ = state
.db
.fail_push_wake(outcome.community, outcome.id, outcome.claim_id)
.await;
return;
}
};
let Some(url) = state.config.push_gateway_delivery_url.as_ref() else {
return;
};
@@ -429,7 +446,20 @@ async fn deliver_one(
return;
}
};
let response = send_gateway_request(http, url, body, auth).await;
if let Err(error) = serving_write.verify().await {
warn!(wake=%outcome.id, %error, "push serving lease lost before delivery");
return;
}
let response = match serving_write
.protect(send_gateway_request(http, url, body, auth))
.await
{
Ok(response) => response,
Err(error) => {
warn!(wake=%outcome.id, %error, "push serving lease lost during delivery");
return;
}
};
match response {
Ok(r) if r.status().is_success() => match r.json::<DeliveryResponse>().await {
Ok(DeliveryResponse::Accepted) => {
@@ -502,6 +532,9 @@ async fn deliver_one(
.await;
}
}
if let Err(error) = serving_write.finish().await {
warn!(wake=%outcome.id, %error, "failed to release community serving lease after push delivery");
}
}
fn delivery_body(endpoint_grant: &str, request_id: uuid::Uuid, expires_at: i64) -> Vec<u8> {

26
crates/buzz-relay/src/router.rs
View File

@@ -376,22 +376,30 @@ async fn readiness_handler(State(state): State<Arc<AppState>>) -> impl IntoRespo
}
let check = async {
let (pg_ok, redis_ok) = tokio::join!(state.db.ping(), async {
state.redis_pool.get().await.is_ok()
},);
(pg_ok, redis_ok)
let (pg_ok, redis_ok, deletion_catalog_ok) = tokio::join!(
state.db.ping(),
async { state.redis_pool.get().await.is_ok() },
async { state.db.validate_deletion_serving_catalog().await.is_ok() },
);
(pg_ok, redis_ok, deletion_catalog_ok)
};
let (pg_ok, redis_ok) = tokio::time::timeout(Duration::from_secs(2), check)
.await
.unwrap_or((false, false));
let (pg_ok, redis_ok, deletion_catalog_ok) =
tokio::time::timeout(Duration::from_secs(2), check)
.await
.unwrap_or((false, false, false));
if pg_ok && redis_ok {
if pg_ok && redis_ok && deletion_catalog_ok {
(StatusCode::OK, Json(json!({"status": "ready"}))).into_response()
} else {
(
StatusCode::SERVICE_UNAVAILABLE,
Json(json!({"status": "not_ready", "postgres": pg_ok, "redis": redis_ok})),
Json(json!({
"status": "not_ready",
"postgres": pg_ok,
"redis": redis_ok,
"deletion_catalog": deletion_catalog_ok
})),
)
.into_response()
}

131
crates/buzz-relay/src/state.rs
View File

@@ -10,8 +10,7 @@ use axum::body::Bytes;
use axum::extract::ws::{Message as WsMessage, Utf8Bytes as WsUtf8Bytes};
use dashmap::DashMap;
use futures_util::future::join_all;
use tokio::sync::mpsc;
use tokio::sync::Semaphore;
use tokio::sync::{mpsc, watch, Semaphore};
use tokio::task::JoinHandle;
use tokio_util::sync::CancellationToken;
use uuid::Uuid;
@@ -37,6 +36,55 @@ use crate::subscription::SubscriptionRegistry;
pub(crate) type ScopedPubkeyKey = (CommunityId, [u8; 32]);
/// Why a community-bound socket is being asked to stop.
///
/// Only deletion is externally attributed today. Ordinary lifecycle exits keep
/// using cancellation alone and therefore retain the existing bare-close
/// behavior.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum CommunityDisconnectReason {
CommunityDeleted,
}
impl CommunityDisconnectReason {
pub(crate) fn close_message(self) -> WsMessage {
match self {
Self::CommunityDeleted => WsMessage::Close(Some(axum::extract::ws::CloseFrame {
code: axum::extract::ws::close_code::POLICY,
reason: WsUtf8Bytes::from_static("community deleted"),
})),
}
}
}
/// Per-socket lifecycle controls shared by the registry and the writer.
#[derive(Clone)]
pub(crate) struct CommunityConnectionControl {
cancel: CancellationToken,
reason_tx: watch::Sender<Option<CommunityDisconnectReason>>,
}
impl CommunityConnectionControl {
pub(crate) fn new(cancel: CancellationToken) -> Self {
let (reason_tx, _reason_rx) = watch::channel(None);
Self { cancel, reason_tx }
}
pub(crate) fn cancellation_token(&self) -> CancellationToken {
self.cancel.clone()
}
pub(crate) fn disconnect_reason(&self) -> watch::Receiver<Option<CommunityDisconnectReason>> {
self.reason_tx.subscribe()
}
fn disconnect_community(&self) {
self.reason_tx
.send_replace(Some(CommunityDisconnectReason::CommunityDeleted));
self.cancel.cancel();
}
}
/// Leaves headroom under the process-wide drain deadline for a stalled writer.
const RESTART_CLOSE_ACK_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
type SlidingWindowCounter = (u32, Instant);
@@ -68,7 +116,7 @@ struct ConnEntry {
/// registration cancels the token; archival before registration is observed by
/// the revalidation. The returned guard removes the entry on every exit path.
pub struct CommunityConnectionRegistry {
connections: Arc<DashMap<Uuid, (CommunityId, CancellationToken)>>,
connections: Arc<DashMap<Uuid, (CommunityId, CommunityConnectionControl)>>,
}
impl Default for CommunityConnectionRegistry {
@@ -86,26 +134,27 @@ impl CommunityConnectionRegistry {
}
/// Registers one socket and returns a guard that deregisters it on drop.
pub fn register(
pub(crate) fn register(
&self,
connection_id: Uuid,
community_id: CommunityId,
cancel: CancellationToken,
control: CommunityConnectionControl,
) -> CommunityConnectionGuard {
self.connections
.insert(connection_id, (community_id, cancel));
.insert(connection_id, (community_id, control));
CommunityConnectionGuard {
connection_id,
connections: Arc::clone(&self.connections),
}
}
/// Cancels every socket type currently bound to `community_id`.
/// Disconnects every socket type currently bound to `community_id` and
/// attributes the close to community deletion.
pub fn disconnect_community(&self, community_id: CommunityId) -> usize {
let mut closed = 0;
for entry in self.connections.iter() {
if entry.value().0 == community_id {
entry.value().1.cancel();
entry.value().1.disconnect_community();
closed += 1;
}
}
@@ -124,7 +173,7 @@ impl CommunityConnectionRegistry {
/// Removes a socket lifecycle registration on every handler exit path.
pub struct CommunityConnectionGuard {
connection_id: Uuid,
connections: Arc<DashMap<Uuid, (CommunityId, CancellationToken)>>,
connections: Arc<DashMap<Uuid, (CommunityId, CommunityConnectionControl)>>,
}
impl Drop for CommunityConnectionGuard {
@@ -137,20 +186,21 @@ impl Drop for CommunityConnectionGuard {
///
/// The ordering is the archival admission invariant: archive-before-query is
/// observed by the query, while archive-after-registration sees the token.
pub async fn run_registered_community_connection<Check, CheckFuture, Run, RunFuture>(
pub(crate) async fn run_registered_community_connection<Check, CheckFuture, Run, RunFuture>(
registry: &CommunityConnectionRegistry,
connection_id: Uuid,
community_id: CommunityId,
cancel: CancellationToken,
control: CommunityConnectionControl,
check_active: Check,
run: Run,
) where
Check: FnOnce() -> CheckFuture,
CheckFuture: Future<Output = Result<bool, buzz_db::DbError>>,
Run: FnOnce() -> RunFuture,
Run: FnOnce(CommunityConnectionControl) -> RunFuture,
RunFuture: Future<Output = ()>,
{
let _guard = registry.register(connection_id, community_id, cancel.clone());
let cancel = control.cancel.clone();
let _guard = registry.register(connection_id, community_id, control.clone());
if !matches!(check_active().await, Ok(true)) {
cancel.cancel();
return;
@@ -158,7 +208,7 @@ pub async fn run_registered_community_connection<Check, CheckFuture, Run, RunFut
if cancel.is_cancelled() {
return;
}
run().await;
run(control).await;
cancel.cancel();
}
@@ -1676,14 +1726,29 @@ mod tests {
let ordinary_a = CancellationToken::new();
let audio_a = CancellationToken::new();
let ordinary_b = CancellationToken::new();
let _ordinary_a_guard = registry.register(Uuid::new_v4(), community_a, ordinary_a.clone());
let _audio_a_guard = registry.register(Uuid::new_v4(), community_a, audio_a.clone());
let _ordinary_b_guard = registry.register(Uuid::new_v4(), community_b, ordinary_b.clone());
let ordinary_a_control = CommunityConnectionControl::new(ordinary_a.clone());
let audio_a_control = CommunityConnectionControl::new(audio_a.clone());
let ordinary_b_control = CommunityConnectionControl::new(ordinary_b.clone());
let ordinary_a_reason = ordinary_a_control.disconnect_reason();
let audio_a_reason = audio_a_control.disconnect_reason();
let ordinary_b_reason = ordinary_b_control.disconnect_reason();
let _ordinary_a_guard = registry.register(Uuid::new_v4(), community_a, ordinary_a_control);
let _audio_a_guard = registry.register(Uuid::new_v4(), community_a, audio_a_control);
let _ordinary_b_guard = registry.register(Uuid::new_v4(), community_b, ordinary_b_control);
assert_eq!(registry.disconnect_community(community_a), 2);
assert!(ordinary_a.is_cancelled());
assert!(audio_a.is_cancelled());
assert!(!ordinary_b.is_cancelled());
assert_eq!(
*ordinary_a_reason.borrow(),
Some(CommunityDisconnectReason::CommunityDeleted)
);
assert_eq!(
*audio_a_reason.borrow(),
Some(CommunityDisconnectReason::CommunityDeleted)
);
assert_eq!(*ordinary_b_reason.borrow(), None);
}
#[tokio::test]
@@ -1700,9 +1765,9 @@ mod tests {
&registry,
Uuid::new_v4(),
community,
cancel_before.clone(),
CommunityConnectionControl::new(cancel_before.clone()),
|| async { Ok(false) },
move || async move { started_before_run.store(true, Ordering::SeqCst) },
move |_| async move { started_before_run.store(true, Ordering::SeqCst) },
)
.await;
assert!(cancel_before.is_cancelled());
@@ -1722,13 +1787,13 @@ mod tests {
&registry,
Uuid::new_v4(),
community,
cancel_during.clone(),
CommunityConnectionControl::new(cancel_during.clone()),
move || async move {
registered_check.notify_one();
resume_check.notified().await;
Ok(true)
},
move || async move { started_during_run.store(true, Ordering::SeqCst) },
move |_| async move { started_during_run.store(true, Ordering::SeqCst) },
);
tokio::pin!(future);
tokio::select! {
@@ -1751,9 +1816,21 @@ mod tests {
let cancel_a = CancellationToken::new();
let cancel_failed = CancellationToken::new();
let cancel_c = CancellationToken::new();
let _guard_a = registry.register(Uuid::new_v4(), archived_a, cancel_a.clone());
let _guard_failed = registry.register(Uuid::new_v4(), failed, cancel_failed.clone());
let _guard_c = registry.register(Uuid::new_v4(), archived_c, cancel_c.clone());
let _guard_a = registry.register(
Uuid::new_v4(),
archived_a,
CommunityConnectionControl::new(cancel_a.clone()),
);
let _guard_failed = registry.register(
Uuid::new_v4(),
failed,
CommunityConnectionControl::new(cancel_failed.clone()),
);
let _guard_c = registry.register(
Uuid::new_v4(),
archived_c,
CommunityConnectionControl::new(cancel_c.clone()),
);
let (closed, failures) =
revalidate_registered_communities(&registry, |community| async move {
@@ -1784,7 +1861,11 @@ mod tests {
let registry = CommunityConnectionRegistry::new();
let community = CommunityId::from_uuid(Uuid::from_u128(0xa));
let cancel = CancellationToken::new();
let guard = registry.register(Uuid::new_v4(), community, cancel.clone());
let guard = registry.register(
Uuid::new_v4(),
community,
CommunityConnectionControl::new(cancel.clone()),
);
assert_eq!(registry.bound_communities(), HashSet::from([community]));
drop(guard);

103
crates/buzz-relay/src/tunnel/directory.rs
View File

@@ -87,6 +87,7 @@ return {current, known_generation}
pub struct SessionDirectory {
pool: deadpool_redis::Pool,
lease_ttl: Duration,
db: Option<buzz_db::Db>,
}
/// Active session ownership lease read from Redis.
@@ -179,6 +180,9 @@ pub enum DirectoryError {
/// Lease TTL cannot be represented in Redis milliseconds.
#[error("lease ttl must be at least 1ms and fit in i64 milliseconds")]
InvalidLeaseTtl,
/// Durable community deletion fence rejected a Redis mutation.
#[error("community write fenced: {0}")]
CommunityWriteFenced(String),
}
impl SessionDirectory {
@@ -187,9 +191,36 @@ impl SessionDirectory {
Self::with_lease_ttl(pool, DEFAULT_LEASE_TTL)
}
/// Create a serving directory whose Redis mutations use durable,
/// heartbeat-backed community write leases.
pub fn with_db(pool: deadpool_redis::Pool, db: buzz_db::Db) -> Self {
Self {
pool,
lease_ttl: DEFAULT_LEASE_TTL,
db: Some(db),
}
}
/// Create a directory backed by `pool` with an explicit lease TTL.
pub fn with_lease_ttl(pool: deadpool_redis::Pool, lease_ttl: Duration) -> Self {
Self { pool, lease_ttl }
Self {
pool,
lease_ttl,
db: None,
}
}
async fn begin_serving_write(
&self,
community_id: CommunityId,
) -> Result<Option<buzz_deletion::ServingWriteGuard>, DirectoryError> {
match &self.db {
Some(db) => buzz_deletion::acquire_serving_write(db, community_id, "session_directory")
.await
.map(Some)
.map_err(|error| DirectoryError::CommunityWriteFenced(error.to_string())),
None => Ok(None),
}
}
/// Attempt to create/take over the session lease.
@@ -204,10 +235,11 @@ impl SessionDirectory {
owner_runtime_id: RuntimeId,
profile: Profile,
) -> Result<AcquireResult, DirectoryError> {
let serving_write = self.begin_serving_write(community_id).await?;
let keys = SessionKeys::new(community_id, session_id);
let ttl_ms = ttl_ms(self.lease_ttl)?;
let mut conn = self.pool.get().await?;
let (status, value, _known_generation): (String, String, String) =
let mutation = async {
Script::new(ACQUIRE_SCRIPT)
.key(&keys.lease)
.key(&keys.generation)
@@ -215,8 +247,22 @@ impl SessionDirectory {
.arg(profile.as_wire_str())
.arg(ttl_ms)
.invoke_async(&mut *conn)
.await?;
.await
};
let (status, value, _known_generation): (String, String, String) = match &serving_write {
Some(guard) => guard
.protect(mutation)
.await
.map_err(|error| DirectoryError::CommunityWriteFenced(error.to_string()))??,
None => mutation.await?,
};
let lease = parse_lease(community_id, session_id, &value)?;
if let Some(guard) = serving_write {
guard
.finish()
.await
.map_err(|error| DirectoryError::CommunityWriteFenced(error.to_string()))?;
}
match status.as_str() {
"acquired" => Ok(AcquireResult::Acquired(lease)),
"exists" => Ok(AcquireResult::Exists(lease)),
@@ -244,18 +290,34 @@ impl SessionDirectory {
/// Renew a lease only if the current Redis value exactly matches the
/// caller's owner runtime and generation.
pub async fn renew(&self, lease: &SessionLease) -> Result<RenewResult, DirectoryError> {
let serving_write = self.begin_serving_write(lease.community_id).await?;
let keys = SessionKeys::new(lease.community_id, lease.session_id);
let ttl_ms = ttl_ms(self.lease_ttl)?;
let mut conn = self.pool.get().await?;
let (status, value, known_generation): (String, String, String) = Script::new(RENEW_SCRIPT)
.key(&keys.lease)
.key(&keys.generation)
.arg(lease.owner_runtime_id.to_hex())
.arg(lease.generation)
.arg(ttl_ms)
.invoke_async(&mut *conn)
.await?;
let mutation = async {
Script::new(RENEW_SCRIPT)
.key(&keys.lease)
.key(&keys.generation)
.arg(lease.owner_runtime_id.to_hex())
.arg(lease.generation)
.arg(ttl_ms)
.invoke_async(&mut *conn)
.await
};
let (status, value, known_generation): (String, String, String) = match &serving_write {
Some(guard) => guard
.protect(mutation)
.await
.map_err(|error| DirectoryError::CommunityWriteFenced(error.to_string()))??,
None => mutation.await?,
};
let current = parse_optional_lease(lease.community_id, lease.session_id, &value)?;
if let Some(guard) = serving_write {
guard
.finish()
.await
.map_err(|error| DirectoryError::CommunityWriteFenced(error.to_string()))?;
}
match status.as_str() {
"renewed" => Ok(RenewResult::Renewed(
current.expect("renewed returns lease"),
@@ -275,17 +337,32 @@ impl SessionDirectory {
/// Release a lease only if the current Redis value exactly matches the
/// caller's owner runtime and generation.
pub async fn release(&self, lease: &SessionLease) -> Result<ReleaseResult, DirectoryError> {
let serving_write = self.begin_serving_write(lease.community_id).await?;
let keys = SessionKeys::new(lease.community_id, lease.session_id);
let mut conn = self.pool.get().await?;
let (status, value, known_generation): (String, String, String) =
let mutation = async {
Script::new(RELEASE_SCRIPT)
.key(&keys.lease)
.key(&keys.generation)
.arg(lease.owner_runtime_id.to_hex())
.arg(lease.generation)
.invoke_async(&mut *conn)
.await?;
.await
};
let (status, value, known_generation): (String, String, String) = match &serving_write {
Some(guard) => guard
.protect(mutation)
.await
.map_err(|error| DirectoryError::CommunityWriteFenced(error.to_string()))??,
None => mutation.await?,
};
let current = parse_optional_lease(lease.community_id, lease.session_id, &value)?;
if let Some(guard) = serving_write {
guard
.finish()
.await
.map_err(|error| DirectoryError::CommunityWriteFenced(error.to_string()))?;
}
match status.as_str() {
"released" => Ok(ReleaseResult::Released(
current.expect("released returns lease"),

114
crates/buzz-test-client/tests/e2e_relay.rs
View File

@@ -120,7 +120,12 @@ async fn seed_relay_member(host: &str, keys: &Keys, role: &str) {
}
async fn seed_relay_owner(keys: &Keys) {
seed_relay_member("localhost:3000", keys, "owner").await;
seed_relay_member(&relay_authority(), keys, "owner").await;
}
fn relay_authority() -> String {
let url = url::Url::parse(&relay_http_url()).expect("relay HTTP URL");
url[url::Position::BeforeHost..url::Position::AfterPort].to_string()
}
fn http_origin_for_host(host: &str) -> String {
@@ -315,7 +320,7 @@ async fn test_invite_claim_rejects_invalid_code() {
#[ignore]
async fn test_invite_mint_requires_owner_or_admin() {
let member = Keys::generate();
seed_relay_member("localhost:3000", &member, "member").await;
seed_relay_member(&relay_authority(), &member, "member").await;
let response = invite_post(&member, "/api/invites", "{}").await;
assert_eq!(response.status(), reqwest::StatusCode::FORBIDDEN);
@@ -791,10 +796,10 @@ async fn test_auth_event_kind_rejected() {
/// NIP-11 max_subscriptions must be enforced; (limit+1)th REQ gets CLOSED.
///
/// The relay's MAX_SUBSCRIPTIONS is 1024. Opening 1024 subs in a test is slow,
/// so we open a smaller batch and verify the NIP-11 advertised limit matches
/// the actual enforcement constant. The full-limit test is covered by the
/// NIP-11 assertion below (which verifies the advertised value is 1024).
/// This is a protocol-cap test, not an admission-throughput test. Open one REQ
/// at a time and wait out any shared fixed-window quota before retrying a REQ
/// rejected specifically as `rate-limited`, so production admission remains
/// enabled while the test deterministically reaches the independent 1024 cap.
#[tokio::test]
#[ignore]
async fn test_subscription_limit_enforced() {
@@ -802,60 +807,75 @@ async fn test_subscription_limit_enforced() {
let keys = Keys::generate();
let mut client = BuzzTestClient::connect(&url, &keys).await.expect("connect");
// Open 1024 subscriptions (the relay's MAX_SUBSCRIPTIONS).
for i in 0..1024 {
let sid = format!("limit-sub-{i}");
let filter = Filter::new().kind(Kind::Custom(9));
client
.subscribe(&sid, vec![filter])
.await
.expect("subscribe");
// Drain EOSE to avoid buffer buildup.
client
.collect_until_eose(&sid, Duration::from_secs(5))
.await
.expect("EOSE");
let filter = Filter::new().kind(Kind::Custom(49_999));
subscribe_until_eose(&mut client, &sid, filter).await;
}
let overflow_sid = sub_id("overflow");
// Use a kind that no other test writes, so we don't receive stale events.
let filter = Filter::new().kind(Kind::Custom(49999));
client
.subscribe(&overflow_sid, vec![filter])
.await
.expect("send REQ");
// Drain EOSE and stale events from the 100 earlier subscriptions
// until we receive the CLOSED for the overflow subscription.
let msg = loop {
let m = client
.recv_event(Duration::from_secs(5))
let filter = Filter::new().kind(Kind::Custom(49_999));
loop {
client
.subscribe(&overflow_sid, vec![filter.clone()])
.await
.expect("recv CLOSED (or timeout)");
match &m {
RelayMessage::Eose { .. } => continue,
RelayMessage::Event { .. } => continue, // stale event from earlier subs
_ => break m,
}
};
.expect("send overflow REQ");
match msg {
RelayMessage::Closed {
subscription_id,
message,
} => {
assert_eq!(subscription_id, overflow_sid);
assert!(
message.to_lowercase().contains("too many"),
"Expected 'too many' in CLOSED message, got: {message}"
);
match client
.recv_event(Duration::from_secs(6))
.await
.expect("recv overflow CLOSED")
{
RelayMessage::Closed {
subscription_id,
message,
} if subscription_id == overflow_sid && message.starts_with("rate-limited:") => {
tokio::time::sleep(Duration::from_secs(5)).await;
}
RelayMessage::Closed {
subscription_id,
message,
} => {
assert_eq!(subscription_id, overflow_sid);
assert!(
message.to_lowercase().contains("too many"),
"Expected 'too many' in CLOSED message, got: {message}"
);
break;
}
other => panic!("Expected CLOSED for overflow subscription, got {other:?}"),
}
other => panic!("Expected CLOSED for overflow subscription, got {other:?}"),
}
client.disconnect().await.expect("disconnect");
}
async fn subscribe_until_eose(client: &mut BuzzTestClient, sid: &str, filter: Filter) {
loop {
client
.subscribe(sid, vec![filter.clone()])
.await
.expect("subscribe");
match client
.recv_event(Duration::from_secs(6))
.await
.expect("EOSE or rate-limit CLOSED")
{
RelayMessage::Eose { subscription_id } => {
assert_eq!(subscription_id, sid);
return;
}
RelayMessage::Closed {
subscription_id,
message,
} if subscription_id == sid && message.starts_with("rate-limited:") => {
tokio::time::sleep(Duration::from_secs(5)).await;
}
other => panic!("unexpected response while opening {sid}: {other:?}"),
}
}
}
#[tokio::test]
#[ignore]
async fn test_nip11_relay_info() {

1
crates/buzz-workflow/Cargo.toml
View File

@@ -10,6 +10,7 @@ description = "YAML-as-code workflow engine for Buzz"
[dependencies]
buzz-core = { workspace = true }
buzz-db = { workspace = true }
buzz-deletion = { workspace = true }
hex = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }

333
crates/buzz-workflow/src/executor.rs
View File

@@ -526,165 +526,202 @@ pub async fn dispatch_action(
) -> Result<StepResult, WorkflowError> {
use ActionDef::*;
match action {
SendMessage { text, channel } => {
// Look up workflow metadata for destination validation and
// attribution, scoped to the run's community — the same run/workflow
// UUID may exist in another community, so a bare-id lookup could
// load the wrong row and drive a side effect under it.
let wf_run = engine
.db
.get_workflow_run(community_id, run_id)
.await
.map_err(|e| {
WorkflowError::WebhookError(format!(
"SendMessage: failed to load workflow run {run_id}: {e}"
))
})?;
let workflow = engine
.db
.get_workflow(community_id, wf_run.workflow_id)
.await
.map_err(|e| {
WorkflowError::WebhookError(format!(
"SendMessage: failed to load workflow {}: {e}",
wf_run.workflow_id
))
})?;
let channel_id = resolve_send_message_channel(
channel.as_deref(),
&trigger_ctx.channel_id,
workflow.channel_id,
)?;
let owner_pubkey_hex = hex::encode(&workflow.owner_pubkey);
info!(
run_id = %run_id,
step = step_id,
channel = %channel_id,
"SendMessage → {channel_id}: {text}"
);
let event_id = engine
.action_sink()?
.send_message(community_id, &channel_id, text, &owner_pubkey_hex)
.await
.map_err(WorkflowError::from)?;
Ok(StepResult::Completed(serde_json::json!({
"sent": true,
"event_id": event_id,
})))
}
SendDm { to, text: _ } => {
warn!(run_id = %run_id, step = step_id, "SendDm not yet implemented (to={to})");
// TODO (WF-07): emit DM event.
Err(WorkflowError::NotImplemented("SendDm".into()))
}
SetChannelTopic { topic: _ } => {
warn!(run_id = %run_id, step = step_id, "SetChannelTopic not yet implemented");
// TODO (WF-07): update channel topic via DB.
Err(WorkflowError::NotImplemented("SetChannelTopic".into()))
}
AddReaction { emoji } => {
info!(run_id = %run_id, step = step_id, "AddReaction → :{emoji}:");
if trigger_ctx.message_id.is_empty() {
return Err(WorkflowError::InvalidDefinition(
"AddReaction: no trigger.message_id available".into(),
));
}
#[cfg(feature = "reqwest")]
{
let result = add_reaction_impl(&trigger_ctx.message_id, emoji).await?;
Ok(StepResult::Completed(result))
}
#[cfg(not(feature = "reqwest"))]
{
warn!(
run_id = %run_id,
step = step_id,
"AddReaction: reqwest feature not enabled, skipping HTTP call"
);
Ok(StepResult::Completed(
serde_json::json!({ "added": false, "skipped": true }),
// The workflow engine can outlive the serving request that spawned it.
// Revalidate the durable community fence immediately before every external
// side effect (message publish, webhook, delay/resume). A storage failure is
// a denial, never permission to continue.
let serving_write =
buzz_deletion::acquire_serving_write(&engine.db, community_id, "workflow_action")
.await
.map_err(|error| {
WorkflowError::WebhookError(format!(
"community write fence rejected workflow side effect: {error}"
))
}
}
})?;
CallWebhook {
url,
method,
headers,
body,
} => {
let method_str = method.as_deref().unwrap_or("POST");
info!(run_id = %run_id, step = step_id, "CallWebhook → {method_str} {url}");
serving_write.verify().await.map_err(|error| {
WorkflowError::WebhookError(format!("community write lease lost: {error}"))
})?;
#[cfg(feature = "reqwest")]
{
let result = call_webhook_impl(url, method_str, headers, body).await?;
Ok(StepResult::Completed(result))
}
let result = serving_write
.protect(async {
match action {
SendMessage { text, channel } => {
// Look up workflow metadata for destination validation and
// attribution, scoped to the run's community — the same run/workflow
// UUID may exist in another community, so a bare-id lookup could
// load the wrong row and drive a side effect under it.
let wf_run = engine
.db
.get_workflow_run(community_id, run_id)
.await
.map_err(|e| {
WorkflowError::WebhookError(format!(
"SendMessage: failed to load workflow run {run_id}: {e}"
))
})?;
let workflow = engine
.db
.get_workflow(community_id, wf_run.workflow_id)
.await
.map_err(|e| {
WorkflowError::WebhookError(format!(
"SendMessage: failed to load workflow {}: {e}",
wf_run.workflow_id
))
})?;
let channel_id = resolve_send_message_channel(
channel.as_deref(),
&trigger_ctx.channel_id,
workflow.channel_id,
)?;
let owner_pubkey_hex = hex::encode(&workflow.owner_pubkey);
#[cfg(not(feature = "reqwest"))]
{
// reqwest not enabled — log and return placeholder.
warn!(
run_id = %run_id, step = step_id,
"CallWebhook: reqwest feature not enabled, skipping HTTP call"
);
let _ = (headers, body); // suppress unused warnings
Ok(StepResult::Completed(serde_json::json!({
"status": 0,
"body": null,
"skipped": true
})))
}
}
info!(
run_id = %run_id,
step = step_id,
channel = %channel_id,
"SendMessage → {channel_id}: {text}"
);
RequestApproval {
from,
message,
timeout,
} => {
let timeout_str = timeout.as_deref().unwrap_or("24h");
info!(
run_id = %run_id, step = step_id,
"RequestApproval from={from} timeout={timeout_str}: {message}"
);
let event_id = engine
.action_sink()?
.send_message(community_id, &channel_id, text, &owner_pubkey_hex)
.await
.map_err(WorkflowError::from)?;
let token = generate_approval_token(run_id, step_id);
Ok(StepResult::Completed(serde_json::json!({
"sent": true,
"event_id": event_id,
})))
}
// TODO (WF-08): create approval record in DB, emit kind:46010.
// For now, return Suspended with the token so the caller can persist state.
SendDm { to, text: _ } => {
warn!(run_id = %run_id, step = step_id, "SendDm not yet implemented (to={to})");
// TODO (WF-07): emit DM event.
Err(WorkflowError::NotImplemented("SendDm".into()))
}
Ok(StepResult::Suspended {
approval_token: token,
})
}
SetChannelTopic { topic: _ } => {
warn!(run_id = %run_id, step = step_id, "SetChannelTopic not yet implemented");
// TODO (WF-07): update channel topic via DB.
Err(WorkflowError::NotImplemented("SetChannelTopic".into()))
}
Delay { duration } => {
let secs = parse_duration_secs(duration)?;
// Cap delay at 270 seconds (4.5 minutes) — must be less than default_timeout_secs (300s)
// to avoid non-deterministic StepTimeout. Long delays (hours/days)
// should use the scheduled resume pattern (future work: WF-09).
const MAX_DELAY_SECS: u64 = 270;
if secs > MAX_DELAY_SECS {
return Err(WorkflowError::InvalidDefinition(format!(
"delay exceeds maximum of {MAX_DELAY_SECS} seconds (got {secs}s); \
AddReaction { emoji } => {
info!(run_id = %run_id, step = step_id, "AddReaction → :{emoji}:");
if trigger_ctx.message_id.is_empty() {
Err(WorkflowError::InvalidDefinition(
"AddReaction: no trigger.message_id available".into(),
))
} else {
#[cfg(feature = "reqwest")]
{
let result = add_reaction_impl(&trigger_ctx.message_id, emoji).await?;
Ok(StepResult::Completed(result))
}
#[cfg(not(feature = "reqwest"))]
{
warn!(
run_id = %run_id,
step = step_id,
"AddReaction: reqwest feature not enabled, skipping HTTP call"
);
Ok(StepResult::Completed(
serde_json::json!({ "added": false, "skipped": true }),
))
}
}
}
CallWebhook {
url,
method,
headers,
body,
} => {
let method_str = method.as_deref().unwrap_or("POST");
info!(run_id = %run_id, step = step_id, "CallWebhook → {method_str} {url}");
#[cfg(feature = "reqwest")]
{
let result = call_webhook_impl(url, method_str, headers, body).await?;
Ok(StepResult::Completed(result))
}
#[cfg(not(feature = "reqwest"))]
{
// reqwest not enabled — log and return placeholder.
warn!(
run_id = %run_id, step = step_id,
"CallWebhook: reqwest feature not enabled, skipping HTTP call"
);
let _ = (headers, body); // suppress unused warnings
Ok(StepResult::Completed(serde_json::json!({
"status": 0,
"body": null,
"skipped": true
})))
}
}
RequestApproval {
from,
message,
timeout,
} => {
let timeout_str = timeout.as_deref().unwrap_or("24h");
info!(
run_id = %run_id, step = step_id,
"RequestApproval from={from} timeout={timeout_str}: {message}"
);
let token = generate_approval_token(run_id, step_id);
// TODO (WF-08): create approval record in DB, emit kind:46010.
// For now, return Suspended with the token so the caller can persist state.
Ok(StepResult::Suspended {
approval_token: token,
})
}
Delay { duration } => {
let secs = parse_duration_secs(duration)?;
// Cap delay at 270 seconds (4.5 minutes) — must be less than default_timeout_secs (300s)
// to avoid non-deterministic StepTimeout. Long delays (hours/days)
// should use the scheduled resume pattern (future work: WF-09).
const MAX_DELAY_SECS: u64 = 270;
if secs > MAX_DELAY_SECS {
return Err(WorkflowError::InvalidDefinition(format!(
"delay exceeds maximum of {MAX_DELAY_SECS} seconds (got {secs}s); \
use the scheduled resume pattern for long delays"
)));
)));
}
info!(run_id = %run_id, step = step_id, "Delay {duration} ({secs}s)");
tokio::time::sleep(std::time::Duration::from_secs(secs)).await;
Ok(StepResult::Completed(
serde_json::json!({ "slept_secs": secs }),
))
}
}
info!(run_id = %run_id, step = step_id, "Delay {duration} ({secs}s)");
tokio::time::sleep(std::time::Duration::from_secs(secs)).await;
Ok(StepResult::Completed(
serde_json::json!({ "slept_secs": secs }),
))
})
.await
.map_err(|error| {
WorkflowError::WebhookError(format!("community write lease lost: {error}"))
})?;
let release = serving_write.finish().await.map_err(|error| {
WorkflowError::WebhookError(format!("community write lease release failed: {error}"))
});
match result {
Ok(value) => {
release?;
Ok(value)
}
Err(error) => {
let _ = release;
Err(error)
}
}
}