fix(db): rebuild events_pkey in declared order after pgschema apply (#8525)

## Summary

Re-applying `schema/schema.sql` to a database that `pgschema apply` and
`scripts/reconcile-schema-after-pgschema.sql` already bootstrapped fails
with `cannot drop inherited constraint "events_p2026_01_pkey" of
relation "events_p2026_01" (SQLSTATE 42P16)`. Because of that,
`scripts/start-relay-for-tests.sh` breaks against a reused database.

**Cause:** pgschema 1.7.4 moves a partitioned table's partition key to
the front of its primary key when it creates the table. A
pgschema-bootstrapped `events` therefore gets `PRIMARY KEY (created_at,
community_id, id)`. `schema/tables/public/events.sql` and
`migrations/0001` both declare `(community_id, created_at, id)`. Every
later plan drops and re-adds `events_pkey` and each declared child's
key. This happens even on a database that was never reconciled. The plan
only becomes unexecutable after the reconcile script attaches the
children, because their keys are then inherited. As a side effect,
pgschema-bootstrapped databases also had a differently ordered PK index
than migration-managed ones.

**Fix:** after the partition ATTACH block, the reconcile script now
compares `events_pkey` with the declared `PRIMARY KEY (community_id,
created_at, id)`. If they differ, it rebuilds the key on the parent. It
then asserts that `events` and every partition carry the declared key,
which is the "convergence plus live assertion" shape `docs/database.md`
requires. PostgreSQL recreates each partition's key with the same
`<child>_pkey` names, so a reconciled re-apply now plans nothing. The
block does nothing when the key is already correct. `migrations/0001` is
untouched. Only pgschema-bootstrapped dev and test databases run this
script, so it does not touch the out-of-band partitioned-table
procedure.

`delivery_log` is not affected. Its key `(delivered_at, id)` already
starts with the partition key.

**Existing local databases:** a database reconciled before this change
would still fail its next `pgschema apply`, because the reconcile script
only runs after apply. The repair therefore also lives in
`scripts/repair-events-pkey.sql`. That file does nothing when `events`
doesn't exist yet, and `scripts/start-relay-for-tests.sh` runs it before
apply; it's the only launcher that reuses its database. The reconcile
script keeps a verbatim copy for fresh bootstraps, and a unit test keeps
the two identical.

The rebuild drops the key with `IF EXISTS`, so a database that has lost
its key still converges. It also takes its locks with a 10s
`lock_timeout`. If a relay still holds `events`, it fails with "Stop any
relay using this database, then rerun" instead of hanging.

`start-relay-for-tests.sh` now builds the relay's `DATABASE_URL` from
the `PG*` variables it already exports. The secret scanner flags the
literal URI once the file is touched. The value is unchanged.

pgschema 1.13.3 keeps the declared order and creates children as native
`PARTITION OF`. Upgrading would make this block and the ATTACH blocks
unnecessary, but it changes behavior for every caller, so it is left for
a separate change.

### Related issue

Follow-up to block/buzz#8487, which added the probe test and
`.pgschemaignore`. Now rebased onto `main` after that PR merged.

### Testing

- `pgschema_reapply_keeps_runtime_partitions` now asserts the plan after
the maintenance pass is **empty**, which replaces the `DROP TABLE` and
per-monthly plan checks. It then applies and reconciles a second time.
Without the new block it fails with `ALTER TABLE events DROP CONSTRAINT
events_pkey; ...`. With the block it passes.
- It also includes the optional notes from the #8487 review:
- After maintenance, it checks that both catch-alls are still present
and attached, not only that the zipped bounds differ.
- The first runtime monthly comes from the declared catch-all's lower
bound instead of a hard-coded `p2026_07`.
- The doc comment now states only what the test pins: bounds are not
compared, and the plan is empty.
- I broke the rebuild on purpose by re-adding the key in the wrong
order. The new assertion fails the reconcile script with `events and its
partitions must have PRIMARY KEY (community_id, created_at, id)`.
- CLI runs against local Postgres 17:
- Fresh DB: apply → reconcile → apply → reconcile → plan. The second
apply reports no changes and the plan is empty.
- DB reconciled with the old script: the next apply fails with SQLSTATE
42P16. After running the new reconcile script once, apply reports no
changes and the plan is empty.
- `events_pkey_repair_runs_before_reused_apply_and_matches_reconcile`
checks two things: the reconcile script carries the repair block
verbatim, and `start-relay-for-tests.sh` runs the repair before
`pgschema apply`. It fails if the two copies drift.
- I ran the repair script and then the bootstrap against four local
probe databases: one reconciled with main's script, an empty one, one
applied but never reconciled, and one with no `events` key. All four end
with an empty plan. With another session holding a lock on `events`, the
repair fails after 10s with the hint and leaves the key untouched. It
succeeds once the lock is released.
- `admin_schema_parity_between_desired_state_and_migrations` and the
`runtime::migration` unit tests pass. fmt and clippy are clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

.github/workflows/ci.yml

@@ -53,6 +53,8 @@ jobs:
               - '.config/nextest.toml'
               - 'scripts/postgres-test-*.sh'
               - 'scripts/reconcile-schema-after-pgschema.sql'
+              - 'scripts/repair-events-pkey.sql'
+              - 'scripts/start-relay-for-tests.sh'
               - 'bin/pgschema'
               - 'bin/.pgschema-*.pkg'
               - 'scripts/check-postgres-test-discovery.py'

crates/buzz-db/src/runtime/migration.rs

@@ -1583,6 +1583,32 @@ mod postgres_tests {
         );
     }
 
+    /// The reconcile script repairs `events_pkey` after every apply, and
+    /// `start-relay-for-tests.sh` runs the same repair before apply so a reused
+    /// database reconciled before the repair existed can still be applied.
+    #[test]
+    fn events_pkey_repair_runs_before_reused_apply_and_matches_reconcile() {
+        let repair = include_str!("../../../../scripts/repair-events-pkey.sql");
+        let block = &repair[repair.find("DO $$").expect("repair script has a DO block")..];
+        assert!(
+            include_str!("../../../../scripts/reconcile-schema-after-pgschema.sql").contains(block),
+            "scripts/reconcile-schema-after-pgschema.sql must carry scripts/repair-events-pkey.sql's \
+             DO block verbatim"
+        );
+
+        let launcher = include_str!("../../../../scripts/start-relay-for-tests.sh");
+        let repair_at = launcher
+            .find("scripts/repair-events-pkey.sql")
+            .expect("start-relay-for-tests.sh runs scripts/repair-events-pkey.sql");
+        let apply_at = launcher
+            .find("./bin/pgschema apply")
+            .expect("start-relay-for-tests.sh runs pgschema apply");
+        assert!(
+            repair_at < apply_at,
+            "start-relay-for-tests.sh must repair events_pkey before pgschema apply"
+        );
+    }
+
     /// `.pgschemaignore` hides dated monthlies from pgschema so a re-apply
     /// cannot drop the ones the relay created at runtime. Each monthly
     /// `schema/partitions/` declares must be exempt, or pgschema would stop
@@ -2717,6 +2743,10 @@ mod postgres_tests {
     /// by name, not ordinal, because migrations append via `ALTER TABLE` while
     /// `schema.sql` declares them inline — positions legitimately differ, shapes
     /// must not. Entity-intake constraints are also compared in the live catalog.
+    /// `events` and its partitions must carry the same primary key on both
+    /// probes, so the reconcile script's `events_pkey` rebuild cannot drift from
+    /// the migrations. Partitions are compared as a set of key definitions
+    /// because the two paths create different months.
     ///
     /// Driving the real binary is load-bearing: `pgschema` 1.7.4 discards
     /// per-key `NULLS FIRST`/`NULLS LAST` when it re-emits an index, so a naive
@@ -2789,6 +2819,23 @@ mod postgres_tests {
             .expect("read constraint definitions")
         }
 
+        // Distinct (is the parent, key definition) pairs across events and its
+        // partitions.
+        async fn events_key_defs(pool: &PgPool) -> Vec<(bool, String)> {
+            sqlx::query_as(
+                "SELECT DISTINCT conrelid = 'events'::regclass, pg_get_constraintdef(oid) \
+                 FROM pg_constraint \
+                 WHERE contype = 'p' AND conrelid IN ( \
+                     SELECT 'events'::regclass \
+                     UNION ALL \
+                     SELECT inhrelid FROM pg_inherits WHERE inhparent = 'events'::regclass) \
+                 ORDER BY 1, 2",
+            )
+            .fetch_all(pool)
+            .await
+            .expect("read events key definitions")
+        }
+
         let base_url = std::env::var("BUZZ_TEST_DATABASE_URL")
             .or_else(|_| std::env::var("DATABASE_URL"))
             .unwrap_or_else(|_| TEST_DB_URL.to_owned());
@@ -2809,8 +2856,8 @@ mod postgres_tests {
             .await
             .expect("create migrated probe database");
 
-        // Bootstrap the desired-state probe through the real pgschema binary, the
+        // Bootstrap the desired-state probe the way the test-relay launchers do:
-        // same invocation the test-relay launchers use.
+        // the real pgschema binary, then the post-apply reconcile script.
         let apply = pgschema(&conn, &desired_db, &["apply", "--auto-approve"]);
         assert!(
             apply.status.success(),
@@ -2857,6 +2904,13 @@ mod postgres_tests {
                  migration and schema.sql must both use a representable shape."
             );
         }
+        assert_eq!(
+            events_key_defs(&desired).await,
+            events_key_defs(&migrated).await,
+            "events primary-key parity mismatch: the pgschema-bootstrapped, reconciled \
+             desired state has drifted from the migrations; update the events_pkey block in \
+             scripts/reconcile-schema-after-pgschema.sql and scripts/repair-events-pkey.sql"
+        );
 
         assert_eq!(
             constraints(&desired, "entity_deletion_requests").await,
@@ -2876,12 +2930,19 @@ mod postgres_tests {
         }
     }
 
-    /// Re-applying `schema/schema.sql` to a database a relay has used plans
+    /// Re-applying `schema/schema.sql` to a database bootstrapped the way every
-    /// `DROP TABLE IF EXISTS ... CASCADE` for every monthly the partition
+    /// caller does it (pgschema apply, then the reconcile script) and then used
-    /// manager created at runtime, and their rows go with them.
+    /// by a relay plans nothing. pgschema does not compare partition bounds, so
-    /// `.pgschemaignore` keeps those monthlies out of pgschema's view, and
+    /// the advanced catch-alls do not show up in the plan. Two things keep the
-    /// pgschema does not compare partition bounds, so the advanced catch-all
+    /// rest of it empty:
-    /// is left alone too.
+    ///
+    /// - `.pgschemaignore` hides the monthlies the partition manager created
+    ///   at runtime, which pgschema would otherwise drop, rows included, with
+    ///   `DROP TABLE IF EXISTS ... CASCADE`.
+    /// - The reconcile script rebuilds `events_pkey` in its declared order.
+    ///   pgschema creates it with the partition key first, then plans to drop
+    ///   and re-add it and every declared child's key, which fails once the
+    ///   children are attached.
     #[tokio::test]
     #[ignore = "requires Postgres"]
     async fn pgschema_reapply_keeps_runtime_partitions() {
@@ -2902,24 +2963,27 @@ mod postgres_tests {
             .await
             .expect("create probe database");
 
-        let apply = pgschema(&conn, &database, &["apply", "--auto-approve"]);
-        assert!(
-            apply.status.success(),
-            "pgschema apply failed: {}\n{}",
-            String::from_utf8_lossy(&apply.stdout),
-            String::from_utf8_lossy(&apply.stderr),
-        );
-
         let pool = PgPool::connect(&format!("{base_prefix}/{database}"))
             .await
             .expect("connect probe database");
-        // Every pgschema caller attaches the declared children this way.
+        // The bootstrap every pgschema caller runs; the reconcile script
+        // attaches the declared children.
+        let bootstrap = |round: &str| {
+            let apply = pgschema(&conn, &database, &["apply", "--auto-approve"]);
+            assert!(
+                apply.status.success(),
+                "{round} pgschema apply failed: {}\n{}",
+                String::from_utf8_lossy(&apply.stdout),
+                String::from_utf8_lossy(&apply.stderr),
+            );
             sqlx::raw_sql(include_str!(
                 "../../../../scripts/reconcile-schema-after-pgschema.sql"
             ))
             .execute(&pool)
+        };
+        bootstrap("first")
             .await
-        .expect("reconcile schema after pgschema");
+            .expect("first post-apply reconciliation");
         async fn catch_all_bounds(pool: &PgPool) -> Vec<(String, Option<String>)> {
             sqlx::query_as(
                 "SELECT relname::text, pg_get_expr(relpartbound, oid) FROM pg_class \
@@ -2949,6 +3013,10 @@ mod postgres_tests {
         .await
         .expect("maintain partitions");
         let advanced_bounds = catch_all_bounds(&pool).await;
+        assert!(
+            advanced_bounds.len() == 2 && advanced_bounds.iter().all(|(_, bound)| bound.is_some()),
+            "maintenance dropped or detached a catch-all: {advanced_bounds:?}"
+        );
         for (declared, advanced) in declared_bounds.iter().zip(&advanced_bounds) {
             assert_ne!(declared, advanced, "catch-all did not advance");
         }
@@ -2961,7 +3029,6 @@ mod postgres_tests {
         .fetch_all(&pool)
         .await
         .expect("list partitions");
-        pool.close().await;
         let declared = declared_monthlies();
         let runtime: Vec<_> = children
             .iter()
@@ -2969,9 +3036,18 @@ mod postgres_tests {
                 child.starts_with(&format!("{parent}_p2")) && !declared.contains(child)
             })
             .collect();
-        for parent in ["events", "delivery_log"] {
+        for (catch_all, bound) in &declared_bounds {
-            // The first month the declared catch-all served.
+            // The first month the declared catch-all served, from its
-            let first = format!("{parent}_p2026_07");
+            // `FOR VALUES FROM ('YYYY-MM-01 ...')` lower bound.
+            let parent = catch_all
+                .strip_suffix("_p_future")
+                .expect("catch-all is named <parent>_p_future");
+            let month = bound
+                .as_deref()
+                .and_then(|bound| bound.split_once("FROM ('"))
+                .and_then(|(_, rest)| rest.get(..7))
+                .expect("declared catch-all has a dated lower bound");
+            let first = format!("{parent}_p{}", month.replace('-', "_"));
             assert!(
                 runtime.iter().any(|(_, child)| *child == first),
                 "maintenance did not advance into {first}: {children:?}"
@@ -2986,15 +3062,14 @@ mod postgres_tests {
             String::from_utf8_lossy(&plan.stderr),
         );
         assert!(
-            !plan_sql.contains("DROP TABLE"),
+            plan_sql.trim().is_empty(),
-            "re-applying schema.sql would drop tables:\n{plan_sql}"
+            "re-applying schema.sql to a reconciled database must plan nothing; \
+             runtime monthlies {runtime:?}:\n{plan_sql}"
         );
-        for (_, child) in &runtime {
+        bootstrap("second")
-            assert!(
+            .await
-                !plan_sql.contains(child.as_str()),
+            .expect("second post-apply reconciliation");
-                "re-applying schema.sql would change runtime monthly {child}:\n{plan_sql}"
+        pool.close().await;
-            );
-        }
 
         sqlx::query(AssertSqlSafe(format!(
             "DROP DATABASE {database} WITH (FORCE)"

scripts/reconcile-schema-after-pgschema.sql

@@ -181,6 +181,55 @@ BEGIN
     END IF;
 END $$;
 
+-- pgschema moves the partition key to the front of a partitioned table's
+-- primary key when it creates the table, so events gets
+-- PRIMARY KEY (created_at, community_id, id) instead of the declared
+-- (community_id, created_at, id). Every later plan then drops and re-adds
+-- events_pkey and each declared child's key, which fails once the children are
+-- attached because their keys are inherited. Rebuild the key on the parent in
+-- the declared order; PostgreSQL recreates each partition's key with it.
+-- delivery_log already declares its partition key first. This block is a
+-- verbatim copy of scripts/repair-events-pkey.sql; keep the two identical.
+DO $$
+DECLARE
+    declared CONSTANT text := 'PRIMARY KEY (community_id, created_at, id)';
+BEGIN
+    -- Also runs before pgschema apply, when events may not exist yet.
+    IF to_regclass('events') IS NULL THEN
+        RETURN;
+    END IF;
+
+    IF (
+        SELECT pg_get_constraintdef(oid) FROM pg_constraint
+        WHERE conrelid = 'events'::regclass AND contype = 'p'
+    ) IS DISTINCT FROM declared THEN
+        -- The rebuild locks events and every partition ACCESS EXCLUSIVE. Fail
+        -- fast instead of queueing behind a relay that still uses them.
+        BEGIN
+            PERFORM set_config('lock_timeout', '10s', true);
+            ALTER TABLE events DROP CONSTRAINT IF EXISTS events_pkey;
+            ALTER TABLE events
+                ADD CONSTRAINT events_pkey PRIMARY KEY (community_id, created_at, id);
+        EXCEPTION WHEN lock_not_available THEN
+            RAISE EXCEPTION 'timed out locking events to rebuild events_pkey'
+                USING HINT = 'Stop any relay using this database, then rerun.';
+        END;
+    END IF;
+
+    IF EXISTS (
+        SELECT 1 FROM pg_constraint
+        WHERE contype = 'p'
+          AND conrelid IN (
+              SELECT 'events'::regclass
+              UNION ALL
+              SELECT inhrelid FROM pg_inherits WHERE inhparent = 'events'::regclass
+          )
+          AND pg_get_constraintdef(oid) <> declared
+    ) THEN
+        RAISE EXCEPTION 'events and its partitions must have %', declared;
+    END IF;
+END $$;
+
 -- pgschema drops multi-column CHECK constraints. Restore the direct-action
 -- shape CHECK from schema/schema.sql (and migration 0055) verbatim.
 DO $$

scripts/repair-events-pkey.sql

@@ -0,0 +1,56 @@
+-- Rebuild events_pkey in its declared column order.
+--
+-- pgschema creates events with the partition key moved to the front of its
+-- primary key, and every later plan then tries to drop and re-add the inherited
+-- key on each partition, which fails (SQLSTATE 42P16).
+-- scripts/reconcile-schema-after-pgschema.sql repairs that after every apply
+-- and carries a verbatim copy of the block below. A reused database reconciled
+-- before that repair existed still fails its next apply, before the reconcile
+-- script can run, so scripts/start-relay-for-tests.sh, the one launcher that
+-- reuses its database, runs this file before pgschema apply. It does nothing on
+-- an empty database or one whose key is already in order.
+--
+-- Both copies are temporary. The reconcile block, and the test that keeps the
+-- copies identical, can go once pgschema creates the key in declared order
+-- (1.13.3 does). This file and the launcher step can go once no local database
+-- reconciled before the reconcile block existed remains.
+
+DO $$
+DECLARE
+    declared CONSTANT text := 'PRIMARY KEY (community_id, created_at, id)';
+BEGIN
+    -- Also runs before pgschema apply, when events may not exist yet.
+    IF to_regclass('events') IS NULL THEN
+        RETURN;
+    END IF;
+
+    IF (
+        SELECT pg_get_constraintdef(oid) FROM pg_constraint
+        WHERE conrelid = 'events'::regclass AND contype = 'p'
+    ) IS DISTINCT FROM declared THEN
+        -- The rebuild locks events and every partition ACCESS EXCLUSIVE. Fail
+        -- fast instead of queueing behind a relay that still uses them.
+        BEGIN
+            PERFORM set_config('lock_timeout', '10s', true);
+            ALTER TABLE events DROP CONSTRAINT IF EXISTS events_pkey;
+            ALTER TABLE events
+                ADD CONSTRAINT events_pkey PRIMARY KEY (community_id, created_at, id);
+        EXCEPTION WHEN lock_not_available THEN
+            RAISE EXCEPTION 'timed out locking events to rebuild events_pkey'
+                USING HINT = 'Stop any relay using this database, then rerun.';
+        END;
+    END IF;
+
+    IF EXISTS (
+        SELECT 1 FROM pg_constraint
+        WHERE contype = 'p'
+          AND conrelid IN (
+              SELECT 'events'::regclass
+              UNION ALL
+              SELECT inhrelid FROM pg_inherits WHERE inhparent = 'events'::regclass
+          )
+          AND pg_get_constraintdef(oid) <> declared
+    ) THEN
+        RAISE EXCEPTION 'events and its partitions must have %', declared;
+    END IF;
+END $$;

scripts/start-relay-for-tests.sh

@@ -103,6 +103,10 @@ export PGSCHEMA_PLAN_DB=buzz
 export PGSCHEMA_PLAN_USER=buzz
 export PGSCHEMA_PLAN_PASSWORD=buzz_dev
 
+# This database persists across runs. One reconciled before the events_pkey
+# repair existed would fail apply, so repair it first (see the script).
+docker exec -i -e PGPASSWORD="${PGPASSWORD}" buzz-postgres \
+  psql -U "${PGUSER}" -d "${PGDATABASE}" -v ON_ERROR_STOP=1 < scripts/repair-events-pkey.sql
 ./bin/pgschema apply --file schema/schema.sql --auto-approve
 docker exec -i -e PGPASSWORD="${PGPASSWORD}" buzz-postgres \
   psql -U "${PGUSER}" -d "${PGDATABASE}" -v ON_ERROR_STOP=1 < scripts/reconcile-schema-after-pgschema.sql
@@ -168,7 +172,7 @@ if [[ "${BUZZ_REQUIRE_RELAY_MEMBERSHIP:-}" == "true" ]]; then
 fi
 
 nohup env \
-  DATABASE_URL=postgres://buzz:buzz_dev@localhost:5432/buzz \
+  DATABASE_URL="postgres://${PGUSER}:${PGPASSWORD}@${PGHOST}:${PGPORT}/${PGDATABASE}" \
   REDIS_URL=redis://localhost:6379 \
   RELAY_URL=ws://localhost:3000 \
   BUZZ_BIND_ADDR=0.0.0.0:3000 \