mirror of
https://github.com/block/buzz.git
synced 2026-11-04 00:12:02 +00:00
a56966b777
🤖 ## Summary Clients that show a list of artifacts (tasks, config entries and similar records) had no way to get the complete list and then stay current. They could only page through query results, and a page is not proof that the list is complete. This PR adds **sync**. A client sends one subscription and gets every matching artifact as one consistent snapshot, then a signal that the set is complete. After that it gets the current version of each artifact that changes. When the relay cannot guarantee an exact copy, it closes the subscription with a `resync` reason, so the client never misses a change without knowing it. It also adds a **head lookup**. It returns the current version of named artifacts, deleted ones included, so that "create or restore" flows can retry safely. This implements the Sync section of NIP-AR (the artifact protocol spec), which is the first commit of this PR. ## Details - **Request.** A normal `REQ` whose only filter has `"artifact": "current"` and at least one channel in `#h`. This is the same current-state filter as HTTP `/query`; on a `REQ` it becomes a sync, because a `REQ` already means "stored set, `EOSE`, then live". The client does not page: the relay streams the whole set until `EOSE`, so `limit` and `offset` are refused. Other artifact views (`history`, `head`) stay HTTP-only, as on main. Other clients and other `REQ` shapes do not change. - **Initial set.** The relay first registers for live changes and waits until Redis confirms that registration. Then it reads every matching current version in pages ordered by artifact ID, all inside one read-only `REPEATABLE READ` transaction. So all pages come from the same database snapshot. `EOSE` comes only after a complete read. A failed read, an unreadable channel, or a read that takes too long ends with `CLOSED` and no `EOSE`. - **Live changes.** Fan-out never sends the published event to a sync. It only records which artifact changed. The sync then reads that artifact's current version and sends it, one artifact at a time. So a client never gets an older version after a newer one. Changes that arrive during the initial read are held and sent after `EOSE`. Changes are signalled per channel, so a sync skips an artifact whose `type` or `d` does not match its filter: those never change, so it was never in the set. A board that syncs three record types in the same channels no longer gets each change three times. - **When the relay sends `resync`.** The Redis link drops, reconnects, or skips a message it cannot decode. Local fan-out lags. The connection's send queue is full. A moderator redacts an event in the channel. The reader's channel access changes (remove, leave, metadata edit, channel delete). An access change ends the whole sync, also when the sync names other channels. A sync cannot drop one channel and still claim an exact set. - **Publish retry.** When a Redis publish fails for an event kind that syncs use, the relay retries it in the background until it succeeds. - **Relay info (NIP-11).** The relay now lists `sync` (`max_events: 4096`), adds the `head` mode, and sets `websocket_query_extensions: true`. - **No migration.** Sync reads the existing `artifact_heads` table. - **Next to incremental reads (#8523).** #8523 added a separate HTTP protocol, `artifacts.incremental` / `"artifact":"incremental"`. It keeps a saved store up to date after a disconnect, but it does not prove a complete set. The two protocols use different NIP-11 keys and filter values, so they do not collide. They share one copy of the current-state access rule: `artifact_query::predicates`, which #8523 reads through, now applies the view join and calls `push_scope`, and the sync snapshot and head lookup call `push_scope` too. - **A failed access lookup in fan-out resyncs.** The access gate fails closed when a visibility or membership lookup fails. For a sync, that would drop a change with no signal, so the gate also reports those recipients and each one gets `resync: access check failed`. ### Known cost To find out which redactions remove an artifact that a sync covers, the relay would need an extra database read. So every redaction in a synced channel resyncs that channel's syncs, and so does every channel metadata edit. Each resync reads the set again, up to 4,096 artifacts. This is correct, but it adds load to channels where people often delete chat messages. We can narrow it later if it becomes a problem. ### Related issue None found. The design was reviewed as NIP-AR Sync r28. ### Testing - New Postgres tests: keyset pages see one snapshot, and an artifact created after the snapshot opens is in no page; the head view includes deletions; a private or unknown channel is refused. - New relay end-to-end tests on real Postgres and Redis, using the real `REQ` handler, ingest and fan-out: initial set in order, then `EOSE`, then current versions for an update and a delete; a chat message causes no frame; `CLOSE` stops the sync. Other tests: removing the reader from one channel resyncs the whole sync; an unreadable or unknown channel gets `CLOSED restricted` with no `EOSE`; a node-wide loss resyncs. - Unit tests for the parser, request routing, fan-out diversion, the held-change limit, and access-change signals that must arrive before the access filter. - Lifecycle tests from an adversarial review: every frame is checked against the subscription owner, also when a `CLOSE` or a replacing `REQ` arrives while the frame waits for queue room or while a live head read is in progress. A resync recorded while `EOSE` waits ends the sync with no `EOSE`. The held list refuses a change before it stores it (4096 entries, 4 MiB of held removals). - End to end: a relay container built from this branch, with the buzz-app #1523 browser build and the Work plugin. An edit from another client showed in the open board 4 to 25 ms after the write returned, with no refresh. - `cargo clippy --all-targets -D warnings` is clean for the four changed crates. All `buzz-relay` lib tests pass except `api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo`. That test returns 504 on this machine. It is a mesh networking test, and this PR does not touch that code. --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
89 lines
3.1 KiB
Rust
89 lines
3.1 KiB
Rust
#![deny(unsafe_code)]
|
|
#![warn(missing_docs)]
|
|
//! `buzz-core` — zero-I/O foundation types for the Buzz relay.
|
|
//!
|
|
//! Provides [`StoredEvent`], filter matching, kind constants, and event
|
|
//! verification. All other Buzz crates depend on this one.
|
|
|
|
/// NIP-AM: Agent Turn Metric — payload type and encrypt/decrypt helpers.
|
|
pub mod agent_turn_metric;
|
|
/// Channel and membership enums shared across crates.
|
|
pub mod channel;
|
|
/// NIP-AE Agent Engrams — slug grammar, conversation key, d-tag derivation,
|
|
/// body parse/serialize, envelope build/validate, head selection.
|
|
pub mod engram;
|
|
/// Relay-side error types.
|
|
pub mod error;
|
|
/// Relay-side event wrapper with verification tracking.
|
|
pub mod event;
|
|
/// NIP-01 subscription filter matching.
|
|
pub mod filter;
|
|
/// Git permission types — ref patterns, protection rules, policy evaluation.
|
|
pub mod git_perms;
|
|
/// Shared invite-link contract constants.
|
|
pub mod invite;
|
|
/// Buzz kind number registry — custom event type constants.
|
|
pub mod kind;
|
|
/// Network utilities — SSRF-safe IP classification.
|
|
pub mod network;
|
|
/// NIP-10 thread-marker parsing — shared `root`/`reply` marker resolver.
|
|
pub mod nip10;
|
|
/// Agent observer frame helpers.
|
|
pub mod observer;
|
|
/// NIP-AB device pairing — crypto primitives, message types, and errors.
|
|
pub mod pairing;
|
|
/// Presence status types shared across crates.
|
|
pub mod presence;
|
|
/// NIP-PMA owner-encrypted private managed-agent wire codec.
|
|
pub mod private_managed_agent;
|
|
/// Canonical relay runtime identities.
|
|
pub mod relay;
|
|
/// Tenant identity — the server-resolved community key carried on scoped paths.
|
|
pub mod tenant;
|
|
/// NIP-CW thread-mode normalized newest-first window contract.
|
|
pub mod thread_window;
|
|
/// Schnorr signature and event ID verification.
|
|
pub mod verification;
|
|
|
|
pub use error::VerificationError;
|
|
pub use event::StoredEvent;
|
|
pub use nostr::{Event, EventId, Filter, Keys, Kind, PublicKey};
|
|
pub use presence::PresenceStatus;
|
|
pub use tenant::{normalize_host, CommunityId, TenantContext};
|
|
pub use verification::verify_event;
|
|
|
|
#[cfg(any(test, feature = "test-utils"))]
|
|
/// Test helper utilities for creating events and stored events.
|
|
pub mod test_helpers {
|
|
use crate::StoredEvent;
|
|
use chrono::Utc;
|
|
use nostr::{EventBuilder, Keys, Kind};
|
|
|
|
/// Create a signed test event with the given kind and random keys.
|
|
pub fn make_event(kind: Kind) -> nostr::Event {
|
|
let keys = Keys::generate();
|
|
EventBuilder::new(kind, "test")
|
|
.tags([])
|
|
.sign_with_keys(&keys)
|
|
.expect("sign")
|
|
}
|
|
|
|
/// Create a signed test event with the given keys and kind.
|
|
pub fn make_event_with_keys(keys: &Keys, kind: Kind) -> nostr::Event {
|
|
EventBuilder::new(kind, "test")
|
|
.tags([])
|
|
.sign_with_keys(keys)
|
|
.expect("sign")
|
|
}
|
|
|
|
/// Create a [`StoredEvent`] wrapper around a test event.
|
|
pub fn make_stored_event(kind: Kind, channel_id: Option<uuid::Uuid>) -> StoredEvent {
|
|
StoredEvent::with_received_at(make_event(kind), Utc::now(), channel_id, true)
|
|
}
|
|
}
|
|
|
|
/// NIP-AR channel artifact envelope and limits.
|
|
pub mod artifact;
|
|
/// NIP-AR sync filter parsing and limits.
|
|
pub mod artifact_sync;
|