mirror of
https://github.com/block/buzz.git
synced 2026-11-06 17:31:19 +00:00
ff15c347c2
## Problem Migrations `0076`, `0078` and `0081` raise an exception on any table with at least one row unless its index was prebuilt with `CREATE INDEX CONCURRENTLY`. A relay that runs migrations at startup (`BUZZ_AUTO_MIGRATE`, on by default in the Helm chart) and has data in those tables therefore crash-loops on upgrade: ``` migration error: while executing migration 81: error returned from database: prebuild idx_nip_fi_profile_heads_names concurrently; see docs/profile-search.md ``` `0081` (#9019) hits every auto-migrating relay that has published NIP-FI verified profiles. `0076`/`0078` (#8523) hit any relay with artifact data or channel members. Each of these tables is small enough on such relays that a plain index build takes milliseconds. The root cause is `docs/database.md` Rule 6. It told authors that an index on a populated table MUST be built concurrently by an operator ahead of time, so migrations were written to refuse populated tables. With auto-migration that refusal is a crash loop, not a safe stop. ## Fix The migrations are already applied on some databases, so they can't be edited: sqlx would reject the changed checksums. Instead, `run_migrations_locked` builds their missing indexes first. For each entry in `INLINE_INDEX_BUILDS` (`0076`, `0078`, `0081`), it runs migrations through the version before it, then builds any missing index. Each build is a plain `CREATE INDEX` in its own transaction, under the same 1s `lock_timeout` and 5s `statement_timeout` the migrations use. That is the bounded inline build `0049` already does: - **Small populated databases** upgrade the same way empty ones do. - **Large databases** still fail closed. Only a lock or statement timeout (`55P03` / `57014`) gets the "prebuild concurrently" hint, pointing at that migration's own doc. Other errors pass through unchanged. The blocked migration is not recorded. - **Prebuilt indexes** are found with `to_regclass` and skipped before `CREATE INDEX`, so deployments that prebuilt concurrently never request the writer-conflicting lock. - **Invalid prebuilt indexes** are left for the migration to reject, as before. `CONCURRENTLY` is not used in this path. The runner holds the migration advisory lock, and another replica blocked on that lock holds a snapshot that a concurrent build would wait on. ### Rules - `docs/database.md` Rule 6: with `BUZZ_AUTO_MIGRATE` on, every migration must apply its own change on a populated database and must never refuse one because it has rows. A new index is built inline under the migration's budgets. Only deployments that apply migrations out of band (auto-migration off) prebuild concurrently. - Root `AGENTS.md` (step 4) and `migrations/AGENTS.md` repeat the never-refuse rule. - `docs/artifact-sync-deployment.md` and `docs/profile-search.md` describe the startup behavior first and the concurrent prebuild as the out-of-band path. ## Tests - `populated_database_auto_migrates_through_every_migration` (Postgres) migrates to `0075`, seeds artifact rows, and checks four phases: 1. **Lock held:** while a writer lock is held, the `0076` build fails closed. 2. **Index prebuilt:** with the `0076` index prebuilt, the build is skipped. 3. **Partial `0078`:** a blocked `0078` build keeps the builds before it committed. 4. **Verified profile seeded:** with a verified profile head inserted before `0081`, a final startup succeeds. It asserts that the applied versions equal every embedded migration and that all six indexes are valid, ready and live. - `frozen_prebuild_guards_have_inline_builds` (no database) scans `migrations/`. It fails if any migration refuses a populated table (`RAISE EXCEPTION 'prebuild …`) without an inline build, and if a new guard is added. - `migration_execution_cannot_bypass_schema_destruction_lock` allows exactly one production `run_to`, inside `run_migrations_locked`. ## What got simpler - Auto-migrating relays no longer need a manual, operator-only prebuild step for small deployments. That restores the chart README's contract that `helm upgrade` is the whole upgrade procedure. - Rule 6 no longer steers authors toward refusing populated tables, and a test enforces that. - The change adds one table-driven step for three frozen migrations and no new configuration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
72 lines
3.5 KiB
Markdown
72 lines
3.5 KiB
Markdown
# Incremental artifact reads deployment
|
|
|
|
Incremental artifact reads use a 120-second overlap and database modification
|
|
timestamps. The schema, indexes, and every serving writer must support
|
|
incremental reads.
|
|
|
|
## Schema and indexes
|
|
|
|
Migration `0077` adds columns with constant defaults, then changes the defaults
|
|
for future inserts. It does not rewrite existing rows or run a backfill.
|
|
Old rows remain at the epoch timestamp. An initial read includes them.
|
|
New application writers stamp every head change, grant, rejoin, removal,
|
|
redaction, and move source. Changes to channel listing also require recovery. Old application updates do not maintain all these
|
|
stamps, so mixed-version serving cannot advertise incremental correctness.
|
|
|
|
Relay startup migration and `buzz-admin migrate` build the indexes below
|
|
themselves within the migrations' budgets. A deployment too large for those
|
|
budgets runs with auto-migration off; before applying the migrations out of
|
|
band, first build the revision lookup index outside a transaction, before
|
|
migration `0076`:
|
|
|
|
```sql
|
|
CREATE INDEX CONCURRENTLY idx_artifact_revisions_artifact
|
|
ON public.artifact_revisions (community_id, artifact_id);
|
|
```
|
|
|
|
Apply `0076_artifact_revision_lookup.sql` in a transaction after that prebuild.
|
|
Then apply `0077_artifact_sync_timestamps.sql` in a
|
|
transaction with its lock and statement budgets. The migration is idempotent
|
|
for this prebuild step. Then build these indexes outside a transaction:
|
|
|
|
```sql
|
|
CREATE INDEX CONCURRENTLY artifact_heads_modified
|
|
ON public.artifact_heads (community_id, modified_at, artifact_id);
|
|
CREATE INDEX CONCURRENTLY artifact_heads_channel
|
|
ON public.artifact_heads (community_id, channel_id, artifact_id);
|
|
CREATE INDEX CONCURRENTLY channel_members_sync
|
|
ON public.channel_members (community_id, pubkey, sync_modified_at, channel_id);
|
|
CREATE INDEX CONCURRENTLY artifact_revisions_moves
|
|
ON public.artifact_revisions (community_id, modified_at, event_id)
|
|
WHERE source_channel_id IS NOT NULL;
|
|
```
|
|
|
|
Migration `0078` verifies each index definition and its valid, ready, and live
|
|
flags. Applied directly as SQL, it refuses a populated table whose index is
|
|
missing. A fresh empty
|
|
database can build the indexes under the migration's short budgets.
|
|
Migration `0076` also verifies the exact definition and all three flags.
|
|
Remove an invalid index with `DROP INDEX CONCURRENTLY` before a new build.
|
|
|
|
Relay startup migration and `buzz-admin migrate` build any of these indexes
|
|
that are missing before applying `0076` and `0078`. They use the migrations'
|
|
one-second lock and five-second statement budgets, which is enough for a small
|
|
database. Each index builds in its own transaction. A database too large for that
|
|
budget fails closed with an error that points here. The migration whose index
|
|
failed is not recorded, while earlier migrations and indexes stay committed.
|
|
Prebuild the rest concurrently as above, then retry.
|
|
|
|
After all index builds succeed, run `buzz-admin migrate` with the target
|
|
`DATABASE_URL`. It verifies and records `0076`, reapplies the idempotent `0077`,
|
|
then verifies and records `0078`. Direct SQL application does not record the
|
|
migration ledger.
|
|
|
|
## Serving fleet
|
|
|
|
All instances must use the same configured relay signing key, because that key
|
|
authenticates the progress tokens. NIP-11 advertises `artifacts.incremental`.
|
|
Every serving writer must maintain the modification timestamps.
|
|
|
|
The 120-second stamp-to-visibility bound remains an operating assumption.
|
|
Runtime measurements are tracked separately.
|