mirror of
https://github.com/block/buzz.git
synced 2026-10-27 12:40:30 +00:00
## Summary Adds `docs/database.md`, which covers two things: - **The target architecture.** `schema/tables/public/` is the declarative source of truth, written within the constraints that out-of-band, online schema tools can apply. Deployments at scale apply schema changes out of band; auto-migration stays supported for smaller and self-hosted deployments, and `migrations/` stays. The database has no triggers, functions, or foreign keys; that behavior lives in the owning application transaction or in durable application work. Data repair runs through the backfill framework in `docs/data-backfills.md`, not through migrations. - **Rules for schema changes made now:** 1. Add no triggers, functions, procedures, rules, or foreign keys, with no exceptions. New tenant-owned tables are fenced by community admission in the application. Startup and deletion validation already tolerate missing fences, but desired-state bootstraps still attach a fence trigger to every `community_id` table, so a new tenant-owned table is blocked until that loop stops (part of the fence-retirement work). The doc includes checklists for adding a tenant-owned table and a provenance-only table. 2. Migrations change schema only. They never mutate existing rows. Until the backfill framework ships, a repair may be a reviewed, idempotent, batched one-off script in `scripts/maintenance/` that takes the community deletion lock and touches only active communities. A repair that writes historical channel events must keep replica routing closed until its WAL has replayed. 3. Update every schema location: `migrations/`, `schema/tables/public/`, the manifest and sibling directories, the pgschema reconcile script where it applies, and both homes of the existing `push_gateway_*` tables (the target is for the gateway to own its tables alone). Never edit a merged migration. Add catalog-level parity tests for what you touch. 4. Every migration must be safe for the relay that is currently deployed, under both supported modes: auto-migration at startup and out-of-band application. Either way the old relay runs against the new schema for a while. Because auto-migration can apply expand and contract in one boot, a contract migration must fail closed when its backfill has not run, and a contract that depends on a writer release (such as dropping a trigger whose work moved into the app) must name that release as a required stop in the release notes, because no row check can prove it. The order is expand, deploy, backfill, then contract. The doc includes a table of changes that break the old relay and the safe form of each. 5. Bound every lock, build `migrations/` indexes concurrently (the declarative schema uses plain `CREATE INDEX`), and keep changes within what out-of-band apply can make online. AGENTS.md (Product Contract), ARCHITECTURE.md (buzz-db), and new path-local `migrations/AGENTS.md`, `schema/AGENTS.md` (a pointer), and `crates/buzz-push-gateway/migrations/AGENTS.md` point to the doc. The path-local files mean agents editing those directories get the rules automatically. AGENTS.md also had a stale claim that migrations are "auto-applied on relay startup". This PR corrects it: `BUZZ_AUTO_MIGRATE` defaults off in the binary and on in the Helm chart, and both modes are supported. ## What got simpler The rules were scattered across `schema/schema.sql` comments, AGENTS.md Common Gotcha #7, the backfill spec, `docs/thread-window-deployment.md`, and test names. They now live in one document, and the rest link to it. No code changes. ## Testing This PR changes documentation only. I checked links and diffs, and every claim in the doc points to code at the base commit: the layout and parity tests in `crates/buzz-db/src/runtime/migration.rs`, `EXPECTED_SCOPED_TABLES` and the serving-catalog check in `crates/buzz-db/src/store/deletion.rs`, and `BUZZ_AUTO_MIGRATE` in `crates/buzz-relay/src/main.rs`. The new `AGENTS.md` files aren't `.sql`, so neither the sqlx migrator nor the schema layout test picks them up; CI's layout, migration-count, and deletion-parity tests pass. At an earlier head, a real relay against isolated PostgreSQL 17 showed that a new unfenced `community_id` table serves normally and makes community deletion report `unknown=<table>`. Since then, the fence requirement at startup was removed on main; the doc now cites `migration_schema_fence_objects_are_optional_for_serving_and_deletion` for that. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
476 lines
30 KiB
Markdown
476 lines
30 KiB
Markdown
# Database: target architecture and schema-change rules
|
|
|
|
This document states where the Buzz PostgreSQL database is going and the rules
|
|
every schema or data change must follow on the way there. Read it before
|
|
changing anything under `migrations/`, `schema/`,
|
|
`scripts/reconcile-schema-after-pgschema.sql`, or a `buzz-db` query whose
|
|
behavior depends on schema shape.
|
|
|
|
The key words **MUST**, **MUST NOT**, **SHOULD**, and **MAY** are to be
|
|
interpreted as described in BCP 14 when they appear in all capitals.
|
|
|
|
## Target architecture
|
|
|
|
PostgreSQL stores data. The application owns behavior.
|
|
|
|
| Concern | Today | Target |
|
|
|---|---|---|
|
|
| Schema source of truth | Two parallel sources: ordered sqlx migrations in `migrations/` and the desired-state manifest in `schema/` | The declarative desired state in `schema/tables/public/` is the source of truth. Deployments that apply schema out of band plan each change from it, with the tool of their choice. `migrations/` stays, because auto-migration stays supported (next row). Each app owns its own migrations and declarative schema: the relay's copies of the push gateway's tables are debt to remove (Rule 4). |
|
|
| Who applies DDL | Startup migrations when `BUZZ_AUTO_MIGRATE` is on (binary default off, Helm chart default on), `buzz-admin migrate`, the push gateway's pre-upgrade Helm Job, and partition creation at relay startup and on a schedule | Both modes stay supported. Deployments at scale, including Block's hosted deployment, apply schema changes out of band through a reviewed workflow before deploying, and their serving relay runs no DDL. Smaller and self-hosted deployments MAY keep auto-migration on. Still open: how the two modes share one schema source, how auto-migration orders a release whose expand, backfill, and contract steps arrive in one upgrade, and how an operator recovers after a contract migration fails at boot (Rule 5). Monthly partition creation is planned to move out of the relay. Changes to partitioned tables (`events`, `delivery_log`) are outside the out-of-band envelope (Rule 6): attaching a partition to a live parent and building an index on a partitioned parent need their own reviewed procedure. |
|
|
| Integrity between tables | Some foreign keys, including `ON DELETE CASCADE` and `ON DELETE SET NULL` | None in the database. The owning application transaction checks references and performs deletes explicitly. |
|
|
| Guards and follow-up writes | Triggers and PL/pgSQL functions (`schema/functions/`, `schema/triggers/`) | None in the database. Guards run in the owning writer's transaction; follow-up work is recorded in the same transaction and processed as durable, retryable application work. |
|
|
| Data repair | Some historical repair embedded in migrations | Backfill definitions in application code, run by the durable framework in [data-backfills.md](data-backfills.md). |
|
|
| Rollout | Effectively schema and code together | Expand, then deploy, then backfill, then contract. Each step is applied and verified on its own. |
|
|
|
|
The directories in `schema/` other than `tables/` hold objects that out-of-band
|
|
apply cannot manage. They are expected to shrink toward zero. A change that
|
|
adds to them moves away from the target.
|
|
|
|
### Why
|
|
|
|
- **Out-of-band apply needs plain tables.** Applying changes online from a
|
|
declarative source works only for plain tables and indexes: no foreign keys,
|
|
and no triggers, comments, non-default relation options, or partitioning on
|
|
the tables it manages. It does not drop tables, and it applies only
|
|
`CREATE TABLE`, `CREATE INDEX`, and bounded `ALTER TABLE` statements. Every
|
|
new trigger, function, or foreign key is another object that must be removed
|
|
before the schema can be applied this way.
|
|
- **At scale, startup should not change the schema.** A relay that migrates at
|
|
boot can hold locks on hot tables during a rolling deploy. A relay that
|
|
starts against an already-compatible schema cannot. Auto-migration stays
|
|
supported for deployments where that risk is acceptable, so every change
|
|
must be safe in both modes (Rule 5).
|
|
- **Behavior in the database is hard to see, test, and roll back.** Rust
|
|
owns the rest of the write path. Logic split between Rust and PL/pgSQL is
|
|
logic with two owners.
|
|
- **Migrations are not a job runner.** A data rewrite inside a migration runs
|
|
unbounded, in one transaction, while holding locks, with no checkpoint, pause,
|
|
or recovery. The backfill framework exists to provide those properties.
|
|
|
|
The move to declarative, out-of-band schema changes is in progress. Until it
|
|
finishes, existing triggers, functions, and foreign keys stay in place as
|
|
backstops. They are removed one at a time, after their application replacement
|
|
has shipped and been verified. They are not removed early, and new ones are not
|
|
added.
|
|
|
|
## Rules for changes now
|
|
|
|
These rules apply to every change, starting now, including changes that touch
|
|
only one table.
|
|
|
|
### 1. No new triggers or functions
|
|
|
|
A change MUST NOT add `CREATE TRIGGER`, `CREATE FUNCTION`, `CREATE PROCEDURE`,
|
|
or `CREATE RULE`. It MUST NOT widen what an existing trigger or function
|
|
does; this document treats any edit to an existing function's body, other
|
|
than dropping or narrowing it, as widening. Put the behavior in the Rust
|
|
transaction that owns the write:
|
|
|
|
- **Validation or guards:** check inside the writer's transaction before
|
|
committing. Tenant writes go through community admission, described below.
|
|
- **Follow-up writes** (counters, queues, materialized state): write them in the
|
|
same transaction, or record a durable work item atomically with the
|
|
originating write and process it idempotently.
|
|
- **Derived values:** compute them in the application.
|
|
|
|
Dropping or narrowing an existing trigger or function is welcome. Follow the
|
|
retirement sequence below.
|
|
|
|
**No exception for the community write fence.** A new table MUST NOT get a
|
|
fence trigger, including by calling the existing helper
|
|
`attach_community_write_fence`. Writers of a new tenant-owned table MUST be
|
|
fenced in the application instead, by community admission. New serving work,
|
|
including fleet sweeps (one community at a time), is admitted by
|
|
`DeletionStore::guard_transaction`, which takes the shared community deletion
|
|
lock and refuses a community that is not `active`. Most writers open their
|
|
transaction with `begin_community_write_transaction` or one of its variants
|
|
(`crates/buzz-db/src/runtime/mod.rs`), which returns an `AdmittedTx`
|
|
(`crates/buzz-db/src/runtime/admitted_tx.rs`). One exception is deliberate:
|
|
a write that finishes under an already-admitted serving lease
|
|
(`insert_event_with_serving_write_guard`, through
|
|
`AdmittedTx::admit_with_serving_lease`) is checked by
|
|
`guard_transaction_with_serving_lease`, which lets that exact unexpired lease
|
|
complete while the community is `quiescing`. Keep it.
|
|
|
|
Helpers that take `&mut AdmittedTx` reject a raw transaction at compile time,
|
|
but crate code can still bypass the type, so the fence triggers remain the
|
|
database backstop until they are dropped (ARCHITECTURE.md § *Supported Writer
|
|
Contract*). A table without a trigger has no database backstop. Every serving
|
|
writer of a fenced table already goes through admission. The remaining
|
|
fence-retirement work drops the fence triggers and functions, and MUST also
|
|
update ARCHITECTURE.md § *Dual enforcement*.
|
|
|
|
**Today this blocks new tenant-owned tables.** Startup and deletion validation
|
|
no longer require a fence on any table; they pass with every universal fence
|
|
object dropped (`migration_schema_fence_objects_are_optional_for_serving_and_deletion`
|
|
in `crates/buzz-db/src/store/deletion.rs`). Two things still matter:
|
|
|
|
- Deletion's catalog check is an exact set. A `community_id` table missing from
|
|
`EXPECTED_SCOPED_TABLES`, fenced or not, fails it
|
|
(`community deletion catalog drift (missing=…, unknown=…)`), and deletion
|
|
fails closed for every community, not only the new table's. Listing the
|
|
table is enough to pass.
|
|
- Desired-state bootstraps run `schema/community_write_fence.sql`, which
|
|
attaches a fence trigger to every `community_id` table not excluded by
|
|
`community_write_fence_excluded_table`. A fresh desired-state database would
|
|
put a trigger on the new table and a migrated one would not, so the two
|
|
schema paths would diverge.
|
|
|
|
So before a feature can add a tenant-owned `community_id` table, the
|
|
desired-state bootstrap must stop attaching fence triggers. That change belongs
|
|
to the fence-retirement work. Land it before the feature, or as the feature's
|
|
first PR. Do not work around it with a trigger, or with an entry in
|
|
`community_write_fence_excluded_table` for a table the community owns.
|
|
|
|
#### Checklist: adding a tenant-owned `community_id` table
|
|
|
|
After the desired-state change above has landed, all in the same PR:
|
|
|
|
- [ ] Every writer of the table goes through community admission
|
|
(`begin_community_write_transaction` or another constructor that returns
|
|
`AdmittedTx`). Nothing writes the table outside it.
|
|
- [ ] No fence trigger: no `attach_community_write_fence` call in the
|
|
migration and no `schema/triggers/` file.
|
|
- [ ] The table in `EXPECTED_SCOPED_TABLES` (`crates/buzz-db/src/store/deletion.rs`).
|
|
- [ ] The table in `PURGE_SCOPED_TABLES` in the same file, ordered child
|
|
before parent. A table missing here keeps its rows after the purge, and
|
|
deletion verification fails closed.
|
|
- [ ] The migration count in `embedded_migrator_contains_consolidated_initial_schema`.
|
|
- [ ] A Postgres-gated test showing that a write to the table is refused once
|
|
deletion of its community has started, and that the purge removes its
|
|
rows.
|
|
- [ ] A run of the Postgres-gated migration-mode tests (see
|
|
[crates/buzz-db/TESTING.md](../crates/buzz-db/TESTING.md)). Most tests
|
|
use desired state, which builds the catalog differently.
|
|
|
|
#### Checklist: adding a provenance-only table
|
|
|
|
A table whose `community_id` only records which community a row came from, and
|
|
whose rows must stay writable while that community is being deleted, is not
|
|
owned by that community and is not fenced. The existing ones (the deletion
|
|
control plane, `product_feedback`, `rate_limit_violations`, and
|
|
`operator_listener_outbox`) predate this rule and are listed in
|
|
`community_write_fence_excluded_table`. A new one MUST name its column
|
|
something other than `community_id`, as
|
|
`relay_admin_actions.report_community_id` does, so the catalog does not treat
|
|
the table as scoped. Adding a case to the exclusion function would widen an
|
|
existing function, which Rule 1 forbids. Renaming the
|
|
column also removes the table from every automatic deletion surface, so rows
|
|
that name a deleted community survive deletion and no check fails, unless the
|
|
PR follows this checklist. All in the same PR:
|
|
|
|
- [ ] Register it as operator-global in all three places: the
|
|
`_operator_global_tables` seed row in the migration, `schema/seeds/`, and
|
|
the hard-coded name list in `operator_global_tables()`
|
|
(`crates/buzz-db/src/runtime/migration.rs`). The lint ignores a registry
|
|
row whose name is not on that list, and without registration the
|
|
`community_id NOT NULL` and key-leads-with-`community_id` lints fail.
|
|
- [ ] Choose and implement its community-deletion disposition in
|
|
`crates/buzz-db/src/store/deletion.rs`:
|
|
- **sever:** set the provenance column to `NULL` in the `purge_postgres`
|
|
transaction, as `product_feedback` and `rate_limit_violations` are;
|
|
- **delete and verify:** add it to `RELAY_ADMIN_TABLES` with a row source,
|
|
as the `relay_admin_*` tables are; or
|
|
- **retain on purpose:** add no deletion code, and state the reason in a
|
|
comment beside the table's migration and in the PR, as
|
|
`operator_listener_outbox` is retained. No check enforces this choice.
|
|
- [ ] A Postgres-gated deletion test that seeds a row for the target community
|
|
and asserts the chosen disposition. For delete and verify, call
|
|
`purge_postgres` and `verify_postgres_logically_deleted`, following
|
|
`purge_postgres_succeeds_with_relay_admin_action_rows`. For sever, call
|
|
`purge_postgres` and assert the column is `NULL`. This is new coverage:
|
|
no existing test drives the sever path through `purge_postgres`.
|
|
|
|
A `DO` block in a migration is not a persisted function. It MAY guard
|
|
idempotent DDL, as migration `0049` does, or make a contract migration refuse
|
|
to run (Rule 5). It MUST NOT be used to run data repair.
|
|
|
|
### 2. No foreign keys
|
|
|
|
A change MUST NOT add `REFERENCES` or `FOREIGN KEY`, including in a new table.
|
|
For a new reference:
|
|
|
|
- Store the referenced key, led by `community_id` for tenant-scoped data.
|
|
- Check that the referenced row exists, in the same community, inside the
|
|
writing transaction. Lock it or otherwise serialize against a concurrent
|
|
delete when a dangling reference would be unsafe.
|
|
- Delete or detach dependent rows explicitly in the transaction that deletes
|
|
the parent. Community deletion MUST cover the new table; see the checklist
|
|
under Rule 1.
|
|
- Make readers tolerate a missing parent. Without a database constraint, a
|
|
dangling row is possible after an unsupported write, and readers MUST NOT
|
|
panic or leak data across communities when they meet one.
|
|
|
|
Removing an existing foreign key requires its replacement checks and explicit
|
|
deletes to ship first, then the constraint drop as a separate contract step.
|
|
|
|
### 3. Backfills are not migrations
|
|
|
|
A migration MUST contain schema changes only. It MUST NOT `UPDATE`, `INSERT`,
|
|
or `DELETE` existing rows, and MUST NOT scan or rewrite a table while holding a
|
|
write-blocking lock. Scans that do not block writes, such as
|
|
`VALIDATE CONSTRAINT` or a bounded concurrent index build, are allowed (Rule 6).
|
|
|
|
- Historical data repair MUST be a backfill definition that conforms to
|
|
[data-backfills.md](data-backfills.md): stable ID, finite upper bound,
|
|
bounded batches, atomic mutation plus checkpoint, and validation.
|
|
- **Until the backfill framework ships**, a repair MAY instead be a one-off
|
|
SQL script in `scripts/maintenance/`, reviewed in its own PR and run by an
|
|
operator. The script MUST be idempotent and safe to rerun, MUST work in
|
|
bounded batches that each commit on their own, and MUST end with a
|
|
validation query whose expected result the PR states. Commit per batch
|
|
from a top-level `DO` block with `COMMIT` (outside any `BEGIN`) or from a
|
|
psql or shell loop; never with `CREATE FUNCTION` or `CREATE PROCEDURE`,
|
|
even a temporary one. Once the framework ships, new repairs MUST use it.
|
|
- A repair script MUST respect community deletion, as
|
|
[ARCHITECTURE.md](../ARCHITECTURE.md) § *Supported Writer Contract*
|
|
requires of operator maintenance. Each batch MUST take
|
|
`pg_advisory_xact_lock_shared(community_deletion_lock_key(<community>))` for
|
|
every community it touches. After taking the lock, in the same
|
|
transaction, it MUST read the community's state and touch only rows whose
|
|
community has `deletion_state = 'active'` and `deleted_at IS NULL`, as
|
|
`DeletionStore::guard_transaction` does. Selecting active communities
|
|
before taking the lock races with deletion.
|
|
- A repair script that writes a channel-bearing `events` row older than the
|
|
floor (inserting one, moving `created_at` backward, or setting `channel_id`
|
|
on an old row) is outside the replica-floor proof: an
|
|
operator session does not set `buzz.created_at_floor`, so the commit-time
|
|
floor guard lets it through (`migrations/0021`). Its PR MUST document a
|
|
reviewed procedure that keeps replica routing closed from before the first
|
|
batch until its WAL has replayed on every replica.
|
|
- A script MUST NOT disable triggers or set `session_replication_role`.
|
|
- The normal write path that maintains the new invariant MUST be fully rolled
|
|
out, with old writers drained, before the backfill starts. The backfill spec
|
|
explains why and what breaks otherwise.
|
|
- Constant seed rows that a new table or registry needs, such as an
|
|
`_operator_global_tables` entry, are schema, not backfill. They MAY ship in a
|
|
migration and MUST also appear in `schema/seeds/`. A seed row the running
|
|
relay reads (today only `replica_heartbeat`) MUST also be converged and
|
|
asserted in `scripts/reconcile-schema-after-pgschema.sql`. Rows read only by
|
|
the static lint, such as `_operator_global_tables`, do not go there.
|
|
|
|
### 4. Change every schema location
|
|
|
|
Until the declarative schema is the only source, one logical change spans
|
|
several files. A PR that changes the schema MUST update all of these that
|
|
apply:
|
|
|
|
| Location | What goes there |
|
|
|---|---|
|
|
| `migrations/NNNN_<name>.sql` | The forward migration for existing databases. Use the next free number; rebase and renumber on a collision. Never edit a merged migration: its checksum changes, and relays that run the migrator fail with sqlx `VersionMismatch`. Fix forward in a new migration. |
|
|
| `schema/tables/public/<table>.sql` | The table's final `CREATE TABLE` and its `CREATE INDEX` statements, one table per file. This is the declarative schema source. |
|
|
| `schema/schema.sql` | An `\i` line for each new file, after the tables it depends on. |
|
|
| `schema/types/`, `schema/partitions/`, `schema/seeds/` | The matching non-table objects. The layout test permits nothing else here. Rule 1 forbids any addition to `functions/` and `triggers/`. |
|
|
| `scripts/reconcile-schema-after-pgschema.sql` | Idempotent convergence, plus a live assertion, for anything `pgschema` does not preserve: seed rows, some storage parameters, partition attachment, and the shapes listed in AGENTS.md *Common Gotchas* #7. |
|
|
| `crates/buzz-push-gateway/migrations/` **and** the relay copies | The six `push_gateway_*` tables have two homes. The gateway applies its own migration history to its own database, through a pre-install/pre-upgrade Helm Job ([push-gateway-deployment.md](push-gateway-deployment.md)). The relay schema also defines them: `migrations/` (`0015`, `0043`, `0045`, `0060`), `schema/tables/public/push_gateway_*.sql`, and `schema/seeds/push_gateway_*.sql`. Until the duplication is removed, a gateway schema change goes in both, as relay `0060` mirrors gateway `0006`. Every rule here applies to both. The target is for the gateway to own its tables alone, in its own migrations and declarative schema; do not add new gateway tables to the relay schema. |
|
|
|
|
A fresh desired-state database and a migrated database MUST converge on the
|
|
same live catalog. The desired-state layout test in
|
|
`crates/buzz-db/src/runtime/migration.rs` checks file structure only. Parity
|
|
tests cover selected surfaces, and the full drift gate does not exist yet. For
|
|
tables and indexes you touch, add or extend a parity assertion that compares
|
|
the live catalog from both bootstrap paths, following
|
|
`admin_schema_parity_between_desired_state_and_migrations`. A string match
|
|
against `schema.sql` does not count.
|
|
|
|
The existing migration lints still apply. Every tenant-scoped table has
|
|
`community_id NOT NULL`, every key on a scoped table leads with `community_id`,
|
|
and operator-global tables are registered in all three places listed in the
|
|
provenance-only checklist under Rule 1.
|
|
|
|
### 5. Every schema change is backward compatible with the running app
|
|
|
|
Buzz supports two ways of applying migrations, and every change MUST be safe
|
|
under both:
|
|
|
|
- **Auto-migration.** The relay binary defaults `BUZZ_AUTO_MIGRATE` to off,
|
|
but the Helm chart defaults `migrate.autoMigrate` to `true`. With it on, the
|
|
first new pod migrates at boot while old pods are still serving.
|
|
- **Out of band.** An operator applies the migrations while the current relay
|
|
keeps serving traffic, then deploys the new relay. Block's hosted
|
|
deployment works this way at scale, with `migrate.autoMigrate=false`.
|
|
|
|
Either way, for a period the **old relay runs against the new schema**. If the
|
|
new relay is rolled back, the old relay runs against the new schema again.
|
|
Every migration MUST be safe for the currently deployed release to read and
|
|
write against.
|
|
|
|
The relay has no general schema-version compatibility check. It verifies only
|
|
specific catalog surfaces, such as the deletion serving fence and the replica
|
|
floor guard, so the ordering below is the main protection. When
|
|
`BUZZ_AUTO_MIGRATE` is on, an older binary refuses a database that has
|
|
migrations it does not know (sqlx `VersionMissing`), so `helm rollback` with
|
|
auto-migration on fails at startup, and so does any old pod that restarts
|
|
mid-rollout after a new pod has migrated. A rollback therefore depends on the
|
|
older binary not running the migrator. Making rollback work under
|
|
auto-migration is part of supporting both modes.
|
|
|
|
One feature therefore usually ships as several PRs, each deployed before the
|
|
next starts:
|
|
|
|
1. **Expand (schema).** Add only things the old relay ignores: nullable columns,
|
|
columns with a constant default, new tables, new indexes. Add nothing the old
|
|
relay could violate.
|
|
2. **Deploy (code).** Ship the relay that writes the new shape on every write
|
|
path and reads both old and new rows correctly.
|
|
3. **Backfill (data).** After the new code is fully rolled out and old writers
|
|
are drained, repair historical rows with a backfill (Rule 3). Readers stay
|
|
correct for both repaired and unrepaired rows throughout.
|
|
4. **Contract (schema, then code).** After the backfill completes and
|
|
validates, add the constraints you wanted from the start: `NOT NULL`,
|
|
`CHECK`, `UNIQUE`. When nothing deployed still reads them, drop the old
|
|
columns, indexes, triggers, or functions. Each tightening is its own
|
|
migration, applied after the release that stops depending on the old shape.
|
|
|
|
Skip a step only when it has nothing to do. For example, a new table with no
|
|
historical data needs no backfill.
|
|
|
|
**Under auto-migration the steps can collapse.** The migrator applies every
|
|
pending migration at one boot, before the new code serves and before any
|
|
backfill or repair script runs. An operator who upgrades across several
|
|
releases therefore applies expand and contract together. How auto-migration
|
|
should order such an upgrade is unresolved. Until it is, a contract depends on
|
|
one or both of these, and each is handled differently:
|
|
|
|
- **Data (a backfill or repair has run).** The contract migration MUST fail
|
|
closed when it has not. `VALIDATE CONSTRAINT` and `SET NOT NULL` already
|
|
fail on unrepaired rows. A contract that drops a column or table, or
|
|
otherwise loses data, MUST first check that the backfill or repair has
|
|
completed, and `RAISE` from a top-level `DO` block when unrepaired rows
|
|
remain. The PR MUST add a Postgres-gated test that applies the expand and
|
|
contract migrations in one run against unrepaired rows and asserts the
|
|
migration errors without dropping data.
|
|
- **Writers (a release that changes how the app writes is fully deployed).**
|
|
Dropping a trigger, function, or foreign key whose work moved into the app,
|
|
and adding a constraint that old writers could violate, depend on a release,
|
|
not on rows. No row check can prove it: the data may already be valid, or
|
|
there may be nothing to backfill, while old writers still serve.
|
|
Auto-migration does not protect this case.
|
|
|
|
For either kind, the release notes MUST name the release to stop at (the one
|
|
that ships the writer) and any repair to run there, before upgrading past the
|
|
contract release. A failed data check stops the relay from starting. That is
|
|
an outage, not data loss, but recovery is not clean: the expand migrations
|
|
before the contract have already committed, so an old pod that restarts hits
|
|
`VersionMissing`.
|
|
|
|
#### What breaks the old relay
|
|
|
|
| Change | Why it breaks | Safe form |
|
|
|---|---|---|
|
|
| Add a `NOT NULL` column without a default | Old `INSERT`s omit it and fail | Add it nullable, deploy writers, backfill, then add the constraint (see below) |
|
|
| Rename a column or table | Old queries reference the old name | Add the new name, dual-write, backfill, move readers, then drop the old name |
|
|
| Change a column type | Old binds and decodes mismatch, and the change may rewrite the table | Add a new column and migrate through the expand/contract sequence |
|
|
| Add `CHECK`, `UNIQUE`, or a narrower index | Old writes violate it, or an old `ON CONFLICT` target no longer matches an index | Deploy writers that satisfy it, backfill, then add it. Add a `CHECK` as `NOT VALID`, then `VALIDATE CONSTRAINT` in a later step. Keep every index an old `ON CONFLICT` names until no deployed code uses it |
|
|
| Add an enum value that new code writes | Old code fails to decode the unknown variant | Deploy readers that accept the value, then deploy writers that produce it |
|
|
| Drop a column, table, index, trigger, function, or FK | Old code reads it or relies on its side effect | Contract step only, after the release that stops relying on it is fully deployed |
|
|
| Add a table with a `community_id` column | The old relay's exact-set deletion catalog check (`EXPECTED_SCOPED_TABLES`) sees an unknown table, so community deletion fails closed until the new release deploys. Serving is unaffected, apart from the auto-migration restart case in Rule 5 | Allowed. For a tenant-owned table, only after the desired-state change in Rule 1. Keep the gap between expand and deploy short, and do not run community deletion during it |
|
|
| `SELECT *` or `RETURNING *` decoded by position | Column order differs between desired-state and migrated databases and changes when columns are added | Select named columns |
|
|
|
|
#### Adding `NOT NULL` to a populated table
|
|
|
|
`ALTER TABLE ... SET NOT NULL` scans the whole table under `ACCESS EXCLUSIVE`.
|
|
Instead, ship three migration files, each its own step. sqlx runs each file in
|
|
a transaction, which `SET LOCAL` needs; pasted into autocommit `psql`,
|
|
`SET LOCAL` only warns and has no effect.
|
|
|
|
```sql
|
|
-- contract migration A
|
|
SET LOCAL lock_timeout = '5s';
|
|
ALTER TABLE t ADD CONSTRAINT t_col_not_null CHECK (col IS NOT NULL) NOT VALID;
|
|
|
|
-- contract migration B (separate step; scans but does not block writes)
|
|
SET LOCAL lock_timeout = '5s';
|
|
ALTER TABLE t VALIDATE CONSTRAINT t_col_not_null;
|
|
|
|
-- contract migration C (PostgreSQL 12+ uses the validated CHECK and skips the scan)
|
|
SET LOCAL lock_timeout = '5s';
|
|
ALTER TABLE t ALTER COLUMN col SET NOT NULL;
|
|
ALTER TABLE t DROP CONSTRAINT t_col_not_null;
|
|
```
|
|
|
|
The desired-state file declares only the final `col ... NOT NULL`, and that
|
|
change lands in the same PR as migration C. Until then, the desired state keeps
|
|
the column nullable. Between A and C, migrated databases carry
|
|
`t_col_not_null` and fresh ones do not, so exclude that constraint from any
|
|
parity assertion on the table for that window.
|
|
|
|
#### Retiring a trigger, function, or foreign key
|
|
|
|
1. Ship application code that does the trigger's work, or the foreign key's
|
|
check and cascade, while the database object is still in place. The two
|
|
paths MUST be idempotent with each other, so the duplicate work is harmless.
|
|
2. After that release is fully deployed, drop the database object in a
|
|
separate migration. Remove it from `schema/` in the same PR. The release
|
|
notes name the step-1 release as a required stop (Rule 5).
|
|
3. Removing a guard requires a falsifiable test showing that the application
|
|
path enforces what the guard enforced.
|
|
|
|
### 6. Bound every lock and statement
|
|
|
|
- Every migration MUST start with `SET LOCAL lock_timeout`, and SHOULD set
|
|
`SET LOCAL statement_timeout` when a statement can scan. A migration that
|
|
cannot get its lock fails and is retried. It MUST NOT queue behind
|
|
traffic on a hot table.
|
|
- In `migrations/`, an index on an existing populated table MUST be built
|
|
with `CREATE INDEX CONCURRENTLY`, because auto-migrating deployments have no
|
|
online schema change tool to build it for them. CONCURRENTLY cannot run
|
|
inside the migration transaction, so follow the prebuild-then-verify
|
|
procedure in [thread-window-deployment.md](thread-window-deployment.md).
|
|
Partitioned parents need per-partition concurrent builds attached to an
|
|
`ON ONLY` parent index.
|
|
- The declarative schema source declares indexes with plain `CREATE INDEX`.
|
|
The tool that applies it out of band decides how to build them online, so
|
|
`schema/` files MUST NOT add `CONCURRENTLY`. The desired-state layout test
|
|
rejects it.
|
|
- A change MUST NOT rewrite or fully scan a large table under a
|
|
write-blocking lock. That includes column type changes, volatile defaults,
|
|
`SET NOT NULL` without a validated check, and changes to the generated
|
|
`events.search_tsv` column. Use the expand/contract forms above.
|
|
- Write changes that out-of-band apply can make online: `CREATE TABLE`,
|
|
`CREATE INDEX`, and bounded `ALTER TABLE` on tables below a size ceiling,
|
|
with short lock and statement budgets. Do not rely on a multi-statement
|
|
change applying atomically. A change outside that envelope needs a
|
|
separately reviewed operational procedure, and the PR MUST say so.
|
|
|
|
### 7. Old and new relays coexist during rollout
|
|
|
|
Rolling deploys run two relay versions against one database. Durable records
|
|
written by the new release, such as backfill rows, outbox items, and new enum
|
|
values, MUST be handled safely by the old one. This means ignoring them,
|
|
leaving them for the new release, or failing closed. The old release MUST NOT
|
|
misinterpret them.
|
|
|
|
## Review checklist
|
|
|
|
A PR that touches the schema or a schema-dependent query states in its
|
|
description:
|
|
|
|
- [ ] Which phase this is (expand, deploy, backfill, or contract), and which
|
|
PRs or releases come before and after it.
|
|
- [ ] That no trigger, function, procedure, rule, or foreign key is added.
|
|
- [ ] That no migration mutates existing rows, and the backfill ID (or,
|
|
until the framework ships, the `scripts/maintenance/` script and how it
|
|
takes the community deletion lock) for any repair.
|
|
- [ ] For a contract migration, how it fails closed when its backfill has not
|
|
run, and which writer release the release notes name as a required stop.
|
|
- [ ] That every location in Rule 4 is updated, with a catalog-level parity
|
|
test for each changed table and index.
|
|
- [ ] That the currently deployed relay works against the new schema, and how
|
|
that was checked.
|
|
- [ ] Lock and statement timeouts, concurrent index builds in `migrations/`,
|
|
and any operator prebuild step.
|
|
|
|
## Related documents
|
|
|
|
- [ARCHITECTURE.md](../ARCHITECTURE.md): *Supported Writer Contract* and the
|
|
`buzz-db` crate.
|
|
- [data-backfills.md](data-backfills.md): the backfill specification.
|
|
- [multi-tenant-conformance.md](multi-tenant-conformance.md): tenant-scoping
|
|
obligations.
|
|
- [thread-window-deployment.md](thread-window-deployment.md): the brownfield
|
|
concurrent-index procedure.
|
|
- [crates/buzz-db/TESTING.md](../crates/buzz-db/TESTING.md): desired-state
|
|
and migration-mode tests.
|
|
- `schema/schema.sql` header: the desired-state layout.
|