mirror of
https://github.com/block/buzz.git
synced 2026-11-06 17:31:19 +00:00
ff15c347c2
## Problem Migrations `0076`, `0078` and `0081` raise an exception on any table with at least one row unless its index was prebuilt with `CREATE INDEX CONCURRENTLY`. A relay that runs migrations at startup (`BUZZ_AUTO_MIGRATE`, on by default in the Helm chart) and has data in those tables therefore crash-loops on upgrade: ``` migration error: while executing migration 81: error returned from database: prebuild idx_nip_fi_profile_heads_names concurrently; see docs/profile-search.md ``` `0081` (#9019) hits every auto-migrating relay that has published NIP-FI verified profiles. `0076`/`0078` (#8523) hit any relay with artifact data or channel members. Each of these tables is small enough on such relays that a plain index build takes milliseconds. The root cause is `docs/database.md` Rule 6. It told authors that an index on a populated table MUST be built concurrently by an operator ahead of time, so migrations were written to refuse populated tables. With auto-migration that refusal is a crash loop, not a safe stop. ## Fix The migrations are already applied on some databases, so they can't be edited: sqlx would reject the changed checksums. Instead, `run_migrations_locked` builds their missing indexes first. For each entry in `INLINE_INDEX_BUILDS` (`0076`, `0078`, `0081`), it runs migrations through the version before it, then builds any missing index. Each build is a plain `CREATE INDEX` in its own transaction, under the same 1s `lock_timeout` and 5s `statement_timeout` the migrations use. That is the bounded inline build `0049` already does: - **Small populated databases** upgrade the same way empty ones do. - **Large databases** still fail closed. Only a lock or statement timeout (`55P03` / `57014`) gets the "prebuild concurrently" hint, pointing at that migration's own doc. Other errors pass through unchanged. The blocked migration is not recorded. - **Prebuilt indexes** are found with `to_regclass` and skipped before `CREATE INDEX`, so deployments that prebuilt concurrently never request the writer-conflicting lock. - **Invalid prebuilt indexes** are left for the migration to reject, as before. `CONCURRENTLY` is not used in this path. The runner holds the migration advisory lock, and another replica blocked on that lock holds a snapshot that a concurrent build would wait on. ### Rules - `docs/database.md` Rule 6: with `BUZZ_AUTO_MIGRATE` on, every migration must apply its own change on a populated database and must never refuse one because it has rows. A new index is built inline under the migration's budgets. Only deployments that apply migrations out of band (auto-migration off) prebuild concurrently. - Root `AGENTS.md` (step 4) and `migrations/AGENTS.md` repeat the never-refuse rule. - `docs/artifact-sync-deployment.md` and `docs/profile-search.md` describe the startup behavior first and the concurrent prebuild as the out-of-band path. ## Tests - `populated_database_auto_migrates_through_every_migration` (Postgres) migrates to `0075`, seeds artifact rows, and checks four phases: 1. **Lock held:** while a writer lock is held, the `0076` build fails closed. 2. **Index prebuilt:** with the `0076` index prebuilt, the build is skipped. 3. **Partial `0078`:** a blocked `0078` build keeps the builds before it committed. 4. **Verified profile seeded:** with a verified profile head inserted before `0081`, a final startup succeeds. It asserts that the applied versions equal every embedded migration and that all six indexes are valid, ready and live. - `frozen_prebuild_guards_have_inline_builds` (no database) scans `migrations/`. It fails if any migration refuses a populated table (`RAISE EXCEPTION 'prebuild …`) without an inline build, and if a new guard is added. - `migration_execution_cannot_bypass_schema_destruction_lock` allows exactly one production `run_to`, inside `run_migrations_locked`. ## What got simpler - Auto-migrating relays no longer need a manual, operator-only prebuild step for small deployments. That restores the chart README's contract that `helm upgrade` is the whole upgrade procedure. - Rule 6 no longer steers authors toward refusing populated tables, and a test enforces that. - The change adds one table-driven step for three frozen migrations and no new configuration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>