a57349b237
## Summary Buzz clients need to recover task changes after a disconnect. This adds incremental reads of changed artifacts with database modification timestamps and a 120-second overlap. Each result contains the current revision or an explicit removal. A separate read of known IDs confirms accepted edits and restores access state. Pages use a timestamp and a stable ID to handle equal timestamps. Access grants include older records. Access policy changes require discovery and revalidation of saved IDs. The reader does not load a list of all readable channels. Ordinary artifact writes update their own rows. ### Incremental reads and WebSocket Sync This HTTP protocol maintains an eventually consistent local store. It uses `artifacts.incremental` and `"artifact":"incremental"`. It does not prove a complete set at one database snapshot. Absence never proves deletion or loss of access. [WebSocket Sync](https://github.com/block/buzz/pull/8522) serves a separate use case: a complete set at `EOSE`, followed by live updates. It owns `artifacts.sync`. Both protocols can coexist. The second PR to merge must reuse the shared access helper in `artifact_query.rs` when it resolves the overlapping edits. The gate is `BUZZ_ARTIFACT_INCREMENTAL_ENABLED`. Errors use the `incremental_` prefix. Unlisted open channels require membership or an explicit `#h` scope. Listing changes reset incremental progress. Incremental reads exclude TTL artifacts. Known-ID reads return `remove` for them. The `#h` scope limits candidates before pagination. Initial reads omit deleted heads. Later reads retain deletion removals. ### Why the new timestamps are needed `created_at` comes from the event signer. `received_at` records event arrival. Neither advances when access changes or an event is redacted. Sync needs database timestamps for those changes: | Column | Why sync needs it | | --- | --- | | `artifact_heads.modified_at` | Finds current records that change through an edit, deletion, or redaction. | | `artifact_revisions.modified_at` | Dates retained move records. This lets sync remove a task from its former channel when the reader cannot access its new channel. | | `channel_members.sync_modified_at` | Finds grants, rejoins, and removals. A grant must include older tasks whose own timestamps do not change. | | `communities.artifact_policy_modified_at` | Requests recovery after a relevant channel visibility, TTL, or deletion change. Clients revalidate saved IDs. Ordinary task writes do not update it. | The database sets these timestamps with `clock_timestamp()`. The 120-second overlap covers the accepted delay between timestamp assignment and commit visibility. `BUZZ_ARTIFACT_INCREMENTAL_ENABLED` defaults to off. Apply migrations `0076`–`0078` and upgrade all writers before enablement. The 120-second delay bound is an operating assumption. [Deployment steps](https://github.com/block/buzz/blob/drouleau/artifact-snapshots/docs/artifact-sync-deployment.md) cover index prebuilds and rollback. ### Related issue No duplicate found. Related work: [read permissions](https://github.com/block/buzz/pull/8338), [statement limits](https://github.com/block/buzz/pull/8461), and [atomic channel settings](https://github.com/block/buzz/pull/8516). ### Testing Signed HTTP E2E passes on `f3d6ac852e7ce4e2cc53bd891cef87832a868d72` after CI passes. It includes main through `ef4130da4`. A standalone local relay uses a fresh PostgreSQL 17 database, Redis, and signed NIP-98 HTTP requests. All 78 migrations apply through relay startup. All behavioral writes and reads use HTTP. The tests cover create, edit, delta reads, known-ID recovery, private moves, access grants, removals, rejoins, visibility resets, redaction, and deletion. Unlisted tests cover hidden broad discovery, named scope, known-ID reads, member access, and policy resets. They also cover bootstrap across two pages, overlap replay, and 101 move removals across pages. Decoded cursors contain no private revision IDs. With the incremental gate off, `artifacts.incremental` is absent and a signed incremental request returns `503 incremental_unavailable`. Unsigned requests are rejected. Signed tests also cover TTL exclusion, current-ID removals for TTL artifacts, moves into TTL channels, deleted initial heads, and scoped pages with exactly 100 and 3 upserts. A separate local scale test on `babcdb766` reads 10,050 tasks across 60,002 memberships. It uses seeded fixtures and a second relay state with the same signing key. The initial read takes 8.31 seconds over 101 pages with 4,923-byte events. A read with no changes takes 5.70 milliseconds. The candidate keeps the same database query. These are local measurements, not production limits. Cold pages repeat candidate scans and access checks. Writer load needs validation before fleet enablement. ```sh cargo test -p buzz-relay sync_downloads_ten_thousand_tasks_with_sixty_thousand_memberships -- --ignored --nocapture ``` The behavioral run uses `cargo build -p buzz-relay --bin buzz-relay`, then `cargo run -p buzz-relay --example artifact_incremental_final_oct30_f3`. The temporary harness drives the built relay and is removed after the run. The worktree is clean at the tested SHA. It does not exercise a native client. Git and S3 behavior are outside this validation. The [protocol](https://github.com/block/buzz/blob/drouleau/artifact-snapshots/docs/nips/NIP-AR.md#incremental-reads) defines progress, removals, limits, and recovery. Relay verification covers the signed HTTP path. Native client verification belongs to [the client PR](https://github.com/block/buzz-app/pull/1507). Generated with Codex --------- Signed-off-by: Daniel Rouleau <drouleau@squareup.com>
3.4 KiB
Incremental artifact reads deployment
Incremental artifact reads use a 120-second overlap and database modification
timestamps. BUZZ_ARTIFACT_INCREMENTAL_ENABLED is off by default. Enable it only
after the schema, indexes, and every serving writer are upgraded.
Schema and indexes
Migration 0077 adds columns with constant defaults, then changes the defaults
for future inserts. It does not rewrite existing rows or run a backfill.
Old rows remain at the epoch timestamp. An initial read includes them.
New application writers stamp every head change, grant, rejoin, removal,
redaction, and move source. Changes to channel listing also require recovery. Old application updates do not maintain all these
stamps, so mixed-version serving cannot advertise incremental correctness.
For a populated database, first build the revision lookup index outside a
transaction, before migration 0076:
CREATE INDEX CONCURRENTLY idx_artifact_revisions_artifact
ON public.artifact_revisions (community_id, artifact_id);
Apply 0076_artifact_revision_lookup.sql in a transaction after that prebuild.
Then apply 0077_artifact_sync_timestamps.sql in a
transaction with its lock and statement budgets. The migration is idempotent
for this prebuild step. Then build these indexes outside a transaction:
CREATE INDEX CONCURRENTLY artifact_heads_modified
ON public.artifact_heads (community_id, modified_at, artifact_id);
CREATE INDEX CONCURRENTLY artifact_heads_channel
ON public.artifact_heads (community_id, channel_id, artifact_id);
CREATE INDEX CONCURRENTLY channel_members_sync
ON public.channel_members (community_id, pubkey, sync_modified_at, channel_id);
CREATE INDEX CONCURRENTLY artifact_revisions_moves
ON public.artifact_revisions (community_id, modified_at, event_id)
WHERE source_channel_id IS NOT NULL;
Migration 0078 verifies each index definition and its valid, ready, and live
flags. It refuses a populated table whose index is missing. A fresh empty
database can build the indexes under the migration's short budgets.
Migration 0076 also verifies the exact definition and all three flags.
Remove an invalid index with DROP INDEX CONCURRENTLY before a new build.
After all index builds succeed, run buzz-admin migrate with the target
DATABASE_URL. It verifies and records 0076, reapplies the idempotent 0077,
then verifies and records 0078. Direct SQL application does not record the
migration ledger.
Enable and rollback
Deploy the timestamp writers to every relay instance with the gate off.
Drain the old writers. All instances must use the same configured relay
signing key, because that key authenticates the progress tokens. Then enable
BUZZ_ARTIFACT_INCREMENTAL_ENABLED=true on the serving fleet. NIP-11 advertises
artifacts.incremental only when the gate is enabled. The endpoint refuses incremental
requests while the gate is off.
Before a rollback to an old writer, disable the gate on every serving
instance. Before enabling it again, upgrade every writer and advance
communities.artifact_policy_modified_at for each affected active community
in a reviewed maintenance transaction under community deletion admission.
This forces clients to discover current records and revalidate saved IDs.
No local absence scan is permitted.
The 120-second stamp-to-visibility bound remains an operating assumption. Runtime measurements are tracked separately. This PR does not authorize a deployment or a production maintenance write.