buzz/docs/data-backfills.md

Durable data backfills: a specification

draft

Overview

This section is non-normative. It gives the mental model for the requirements that follow.

A Buzz data backfill repairs existing PostgreSQL rows after a compatible schema is available. It may start automatically with the relay or manually while the relay serves traffic. If a worker stops, another worker resumes from committed progress.

Think of each backfill as one durable PostgreSQL row keyed by a stable ID:

1. The application registers the stable ID.
2. Starting the backfill captures a finite upper bound.
3. A worker takes an exclusive claim.
4. Each transaction changes target rows and advances the checkpoint together.
5. Workers repeat bounded batches until they reach the upper bound.
6. Validation proves the result before the row becomes completed.

PostgreSQL is the authority throughout this flow. Queues and timers may wake a worker, but they do not own work or progress. The admin API and client display the PostgreSQL state; they do not keep another copy of the lifecycle.

The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT, and MAY are to be interpreted as described in BCP 14 when they appear in all capitals.

Purpose and scope

This specification covers:

• durable registration and execution of bounded data backfills;
• exclusive claims, crash recovery, retry, pause, and validation;
• atomic target mutations and checkpoint progress;
• independent schema-migration and automatic-backfill controls;
• readiness behavior in automatic and manual modes;
• an authorized, auditable admin API and client; and
• conformance tests at production transaction, startup, and admin seams.

It does not define:

• schema migration mechanics or a generic migration framework;
• a Nostr event kind or other wire protocol;
• a distributed queue as durable state;
• exactly-once worker code outside a PostgreSQL transaction;
• one universal batch size, timeout, or retry schedule;
• a way for a backfill to make an incompatible application schema safe; or
• destructive controls that erase, rewind, or force completion.

Backfill definitions are application code, not runtime-authored migrations.

Core model

Terms

Term	Meaning
Definition	Application code that declares an ID, ordering key, batch bound, mutation, and validation.
Stable ID	The only identity for execution, administration, validation, completion, and readiness.
Upper bound	The inclusive, definition-typed end of the finite source set. It is captured once.
Checkpoint	The greatest source position whose required mutation is durably committed or was deterministically unnecessary.
Claim	Temporary, exclusive PostgreSQL authority for one worker to advance a backfill.
Generation	A monotonically increasing value that fences work from a stale claim.
Validation	Definition-specific proof that work through the upper bound has the required postcondition.
Required automatic backfill	A stable ID that the deployed application includes in automatic startup readiness.

PostgreSQL authority

PostgreSQL MUST be the sole authority for lifecycle, ownership, progress, retry eligibility, diagnostics, validation, and completion. A queue, timer, cache, notification, or in-process registry MAY wake a worker. Loss, delay, duplication, or replay of that signal MUST NOT affect correctness. A worker MUST re-read PostgreSQL before acting.

The row for a stable ID MUST remain compact. It records the lifecycle, immutable bound, monotonic checkpoint, claim and generation, bounded retry and diagnostic state, validation, timestamps, and audit correlation. Bounded definition metadata MAY be stored for diagnostics. Detailed history belongs in the audit facility, not an unbounded per-batch journal. Telemetry is a projection, not another state store.

Stable identity and registration

Each definition MUST declare a stable ID. A stable ID MUST map to exactly one durable row and at most one readiness obligation across builds and deployments. Validated completion MUST be permanent for that ID.

Registration and reconciliation MUST be idempotent upserts by stable ID. Repeated registration MUST NOT create another execution. Registration signals may arrive late, more than once, or out of order; the durable state MUST still converge on one row and one readiness obligation. For registration-owned diagnostic metadata, the last registration wins.

Registration order, revision strings, build metadata, and version metadata MUST NOT select a runnable definition or establish execution precedence. They MUST NOT create another row, checkpoint, completion, or readiness obligation.

Every process allowed to work on an active ID MUST implement the same behavior: the same source set, ordering and checkpoint meaning, mutation, and validation contract. A material change to any of those semantics MUST use a new stable ID. Registering different behavior under an active ID is operator error; the orchestrator MUST NOT compare or choose between definitions.

Rolling deployments

A process MUST reconcile its definitions by stable ID before claiming work. Any process with a compatible local definition MAY claim eligible work, regardless of registration order or diagnostic metadata. A process MUST fail closed if it does not know the definition or cannot safely interpret the row's bound, checkpoint, lifecycle, and completion contract.

Deploying another compatible process does not revoke an existing claim. Its owner MAY commit while the claim and generation remain valid. If the deployment cannot accept that commit, its semantics are materially different and require a new stable ID. Deployment order, binary age, and metadata MUST NOT act as fences.

An exclusive claim authorizes one compatible worker. Generations reject stale owners. These mechanisms MUST NOT establish definition precedence.

Before starting a backfill, deploy and fully roll out the normal write path that maintains the desired postcondition. Operators MUST ensure old writers and in-flight old behavior no longer produce actionable rows, then start the bounded historical repair. This ordering is an operator and developer responsibility, not a condition the backfill framework verifies. If it is violated, the existing checkpoint may have advanced past writes that were never repaired, and no rerun under the same stable ID can revisit them. Operators must run a new repair under a new stable ID, and may need to repeat that process until the normal write path is safe.

Lifecycle and operator actions

One state machine governs automatic and manual execution:

State	Meaning	Next states
pending	Registered but not started; no bound exists.	running
running	Started and eligible for a worker claim, subject to retry backoff.	paused, blocked, failed, validating
paused	Operator intent prevents claims and fences current work.	running, validating
blocked	A prerequisite or data condition needs operator remediation.	running, validating
failed	Execution exhausted bounded retries, or validation failed.	running, validating
validating	Mutation reached the bound and is eligible for validation.	paused, completed, failed, blocked
completed	Validation succeeded. This state is terminal.	none

The supported operator actions are:

Action	Contract
start	From pending, atomically capture the upper bound and enter running before dispatching work.
pause	From running or validating, atomically enter paused, advance the generation, and invalidate the claim.
resume	From paused, enter running, or validating when no admitted work remains.
retry	From blocked or failed, enter running, or validating when no admitted work remains, at the existing bound and checkpoint.
validate	For pending, return current-state. For any other row, use the read-only validation contract below.

Repeated pause and resume requests MUST converge on the requested state. Neither action changes the bound or checkpoint. There is no separate retrying or cancellation state. Retry eligibility and delay are attributes of running; pause is the safe interruption command.

An in-flight batch either commits before pause or fails the generation fence and rolls back. Previously committed batches remain committed.

Retry, blockage, and failure

The engine MUST durably classify an execution error before reporting it as handled. Retryable errors remain running without a claim until bounded backoff permits another claim. Automatic retries MUST be bounded and end in a terminal failed or blocked disposition instead of a hot loop.

An unmet prerequisite or data invariant enters blocked with a bounded, actionable diagnostic. Exhausted execution errors enter failed. Validation failure enters failed, unless it identifies an unmet prerequisite and enters blocked instead.

retry is fresh operator intent. It MAY clear bounded attempt accounting and stale diagnostics. It MUST preserve history, the bound, and the checkpoint, and MUST NOT bypass validation.

Validation and completion

Reaching the upper bound MUST enter validating, not completed. Validation MUST read authoritative PostgreSQL state, cover the postcondition through the bound, and be safe to repeat after a crash.

Only successful validation MAY transition the row to completed. The completion transaction MUST verify the lifecycle and generation so a stale validator cannot complete a paused or retried run. Operators cannot select completed, and completed state MUST remain immutable.

Execution safety

Finite source set and checkpoint

Before the first mutation, start MUST capture an upper bound in the same transaction that initiates execution. The definition MUST state how the bound represents an empty source set. Once captured, the bound MUST NOT be changed or reinterpreted for that stable ID.

The definition MUST use a checkpoint type with a total, stable ordering. The checkpoint MUST be inclusive. Before the first source item commits, the checkpoint MAY be absent; absence means a position before all admitted work, not a worker-supplied text sentinel.

The checkpoint MUST NOT move backward or beyond the upper bound. A worker MUST select positions strictly after the checkpoint and no later than the bound. If the ordering key is not unique, the definition MUST use a stable compound key that cannot ambiguously skip or repeat tied rows.

Rows created beyond the upper bound MUST be handled by normal write paths or a new backfill ID. Application reads and writes MUST remain correct for both repaired and unrepaired rows during execution.

Rationale: the upper bound makes historical repair finite and separates it from live writes.

Bounded batches

Each definition MUST declare a finite maximum amount of work per batch. The production selection and mutation path MUST enforce it. Any work that could grow with the source set MUST have that bound or a stricter finite bound.

Batch transactions and lock-holding intervals MUST have bounded duration. Workers MUST observe cancellation, timeout, and pause intent at transaction and batch boundaries. They MUST NOT start another batch after observing that intent, and an interrupted transaction MUST roll back.

If a work, transaction, lock, or interruption bound cannot be honored, the engine MUST commit neither partial mutations nor a checkpoint. It MUST apply the bounded failure policy instead of widening the batch or continuing without a bound.

Atomic mutation and progress

Every batch MUST commit its target mutations and checkpoint advance in one PostgreSQL transaction. A mutation, checkpoint, ownership, commit, cancellation, or timeout failure MUST roll back both.

Rationale: this transaction keeps committed mutations from being hidden behind an older checkpoint and keeps the checkpoint from claiming rolled-back work. Definitions SHOULD still make mutations idempotent as defense in depth.

Side effects that cannot join the transaction are outside the backfill commit. They MUST NOT be required for the checkpoint to mean complete.

Exclusive claims and generation fencing

A backfill MUST NOT have more than one current claim. Acquisition, renewal, release, and takeover MUST be PostgreSQL state transitions. Each new claim or takeover MUST advance the generation. An admin transition that invalidates in-flight work MUST also advance it.

Every transaction that changes target data or progress, or commits a validation or lifecycle result that can advance execution, MUST verify before commit that:

• the lifecycle permits the operation; and
• the presented generation equals the current generation.

A claim-authorized worker execution commit MUST additionally verify that:

• the presented owner holds the current claim; and
• the claim remains valid.

Operator transitions and diagnostic validation do not require a claim. Operator actions remain lifecycle- and generation-fenced, as do validation or completion results that advance execution.

Failure of any applicable check MUST reject the entire transaction. For a claim-authorized worker, checking only before work starts is insufficient because a paused, expired, or replaced owner may finish after takeover.

Claims MUST have bounded validity and require renewal. Takeover MUST eventually be possible after renewal stops. PostgreSQL time and state decide validity; worker-local clocks do not grant authority.

Release, crash, and takeover

A worker MAY release a claim while the row remains running. A crash leaves either a committed batch or no batch effect. After the abandoned claim expires, another worker advances the generation and resumes strictly after the committed checkpoint.

The old worker may keep computing, but its later transaction MUST fail the generation check. Takeover MUST NOT rewind the checkpoint, recapture the bound, or infer progress from worker memory.

Configuration and readiness

Schema auto-migration and automatic backfills are independent controls. All four configurations are valid:

Schema auto-migration	Automatic backfills	Required behavior
off	off	The relay MUST run no migrations, MUST NOT start pending backfills, and MUST NOT add a backfill readiness gate. Already-started manual work MUST remain recoverable. The installed schema and application MUST serve safely with incomplete data.
off	on	The relay MUST run no migrations. With a compatible installed schema, it MUST start required backfills and remain unready until they reach completed. Missing required schema is a schema-compatibility failure; a backfill MUST NOT replace migration.
on	off	The relay MUST run migrations, but MUST NOT start pending backfills or add a backfill readiness gate. Operators run them manually while the application serves mixed repaired and unrepaired data. Already-started work MUST remain recoverable.
on	on	The relay MUST run migrations, then required backfills, and remain unready until every required backfill reaches completed.

Automatic mode assumes the rollout prerequisite above is already satisfied. A required automatic backfill MUST ship in a release after its prerequisite write path is fully rolled out and old writer behavior is drained. A definition that ships with its prerequisite write-path change MUST NOT auto-start in that release; an operator MUST start it manually only after rollout completion.

In automatic mode, every required pending, running, paused, blocked, failed, or validating row MUST keep the serving gate closed. The relay MUST register the complete set of required stable IDs before evaluating that gate. A restart MUST reconstruct the gate from PostgreSQL; late discovery MUST NOT create a ready interval.

The authorized deployment-admin boundary MUST remain reachable while the ordinary serving readiness gate is closed so operators can inspect and recover incomplete backfills.

In manual mode, incomplete backfills MUST NOT make the relay unready. This is safe only when application reads and writes work correctly before, during, and after the repair. A schema change that cannot serve safely until its backfill finishes does not conform to manual mode.

Turning automatic backfills off prevents automatic initiation of pending rows. It does not abandon running or validating work. Workers MUST recover or take over already-started work after failure without adding a readiness gate; an operator MAY pause it.

Backfill readiness is only one input to the serving decision. Database, cache, deletion-fence, shutdown, and schema-safety checks remain independent.

Admin and client contract

The relay admin API is the external boundary. The client and operator tools MUST project its PostgreSQL-backed state and invoke the lifecycle actions above; they MUST NOT keep another lifecycle or synthesize outcomes.

Read model and honest projection

Authorized list and detail reads MUST expose:

• stable IDs and bounded informational definition metadata;
• lifecycle, bound, checkpoint-derived progress, ownership, retry eligibility, and timestamps;
• bounded execution and validation diagnostics;
• the latest explicit validation disposition, outcome, and time; and
• participation in the automatic readiness gate.

Unknown totals, unavailable diagnostics, and failed or stale reads MUST remain unknown. The UI MUST NOT render them as zero, complete, healthy, or offline. The server MUST derive current status from PostgreSQL on each read. A client MAY cache for presentation, but a failed refresh does not make cached data current.

Control idempotency

The API and client MUST support exactly the lifecycle actions listed above. They MUST be safe under retries, duplicate delivery, concurrent operators, and a response lost after commit. Repeating one logical request MUST return its original result or the converged state without duplicating a transition, recapturing a bound, reusing a generation, or starting another owner. Conflicting requests MUST return a typed conflict with current server state.

In manual mode, start is the durable intent that separates recoverable work from registered pending work that automatic configuration MUST leave alone.

The normal API MUST NOT expose reset, checkpoint rewind, bound changes, record deletion, force completion, unfenced claim release, or generic cancellation. Incompatible work uses a new stable ID. Destructive recovery requires a separate specification.

Explicit validation

validate requests definition-owned, read-only validation against PostgreSQL. It is not a second lifecycle or an operator-selected completion transition.

The API MUST return exactly one of these typed outcomes:

Outcome	Meaning
success	Validation ran and the postcondition held.
failure	Validation ran and rejected the state, with a bounded diagnostic.
current-state	Validation did not run or has no result yet; the response includes the lifecycle and reason without implying success or failure.

A pending row MUST return current-state without running validation. Any other row MAY be validated. When the row is validating and completion preconditions hold, the request MAY drive the same generation-fenced completion transition used by the lifecycle. In every other state, validation is diagnostic and MUST NOT change the lifecycle, bound, checkpoint, generation, or target data.

Revalidating a completed row MUST report a current failure as failure while leaving immutable completion and progress unchanged.

Only one validation invocation per stable ID MAY run at a time. Duplicate delivery of one logical request MUST return its original bounded outcome or a correlated current-state result. Concurrent requests MUST join the active invocation or return current-state. After a lost response, retrying the same request MUST recover the outcome. The client MUST retry or refresh; it MUST NOT infer the outcome.

Diagnostic validation MUST persist only its bounded outcome, request correlation, and audit record. It MUST NOT require or create an execution claim. It MUST remain serialized, operator-authorized, read-only, and unable to change immutable execution state.

Accepted, rejected, coalesced, and completed validation requests MUST be audited with request correlation, actor, stable ID, evaluated lifecycle, and bounded outcome. Audit recording MUST NOT turn validation into a target-data write.

Authorization and audit

Backfills are deployment-wide operational state. Reads and controls MUST require an explicit deployment-operator capability at the relay admin boundary. Community ownership, channel administration, and membership MUST NOT grant this authority. A protected network boundary MAY supply equivalent authority, but the deployment MUST NOT claim individual attribution it does not have.

Every accepted or rejected control attempt MUST be audited with the actor or authority source, stable ID, request correlation, prior and resulting state, claim generation when relevant, time, and a bounded outcome. An accepted state change and its audit record, or durable intent to append that record, MUST commit atomically. An unaudited change is not success.

Diagnostics in the API, audit, and logs MUST be bounded and redacted. They MUST NOT expose raw source rows, credentials, unbounded database errors, or query text. Detailed diagnostic reads SHOULD be observable without logging row data.

Observability

Operators MUST be able to distinguish:

• lifecycle counts and readiness-gating disposition;
• claim acquisition, renewal, loss, takeover, and stale-owner rejection;
• attempted and committed work, checkpoint movement, and lack of progress;
• retryable errors, blockage, exhausted failure, and validation outcomes; and
• execution and validation duration.

Metric labels MUST use bounded vocabularies. Stable IDs, checkpoints, row values, diagnostics, and actor IDs belong in authorized views and structured logs, not labels. Missing telemetry is unknown. Metrics and logs MUST reconcile with PostgreSQL state and MUST NOT authorize transitions.

Conformance tests

Tests MUST exercise production transaction, startup, admin, and client seams. They MUST be falsifiable: removing the relevant bound, atomic write, fence, authorization check, or readiness input MUST fail a test. Test-only lifecycle helpers do not prove conformance.

At minimum, the suite covers:

1. Claim race. Concurrent workers contend for one row. Only one worker owns the current claim and commits under its generation; eventual takeover preserves the checkpoint.
2. Bounded batch. A source set larger than one batch advances only within the declared batch bound before releasing the transaction. Cancellation or timeout rolls back target mutations and checkpoint. Removing production bound enforcement fails the test.
3. Rollback. Failure after mutation but before commit, including checkpoint failure, leaves target data and checkpoint unchanged.
4. Restart. Termination before and after batch commit, and during validation, rebuilds progress from PostgreSQL without recapturing the bound.
5. Stale owner. Pause, expiry, or takeover while a worker is in flight rejects its mutation, checkpoint, validation, and completion at commit.
6. Rolling deployment. Two compatible processes register the same stable ID in either order and retain one row and readiness obligation. Either MAY win an exclusive claim. After takeover, generation fencing rejects the prior owner's writes. A process that does not know the stable ID cannot claim. The test MUST NOT choose by registration order or metadata, or assert definition precedence.
7. Validation. Reaching the bound cannot complete without validation. Failure is durable and repeatable; only success reaches immutable completion.
8. Bounded failure. Persistent errors reach failed or blocked instead of retrying forever. Operator retry preserves the bound and checkpoint.
9. Admin authorization. Unauthorized reads and controls return no protected state and perform no mutation. Community roles grant no authority.
10. Admin idempotency. Duplicate, concurrent, and lost-response requests do not duplicate starts, reuse generations, or recapture bounds. Conflicts return current state.
11. Explicit validation. Start with a completed row and no execution claim. Through the production API and client, run validation, persist and display its current typed outcome, and write its audit record. It creates no claim and does not change the generation, target data, lifecycle, bound, checkpoint, or completion. The API and client also render all typed outcomes for non-pending rows. Unauthorized, duplicate, concurrent, and lost-response requests follow the contract above. Removing serialization, read-only behavior, or an immutable-state guard fails the test.
12. Client projection. The UI preserves unknown and failure states and sends only supported controls.
13. Configuration matrix. All four configurations run through real startup and readiness paths. Automatic mode gates until validation, manual mode does not add the gate, and schema safety remains independent. Automatic mode also exercises authorized inspection and recovery from paused, blocked, and failed through the production admin boundary while ordinary serving remains gated.

PostgreSQL concurrency tests SHOULD use controlled transaction barriers so losing and stale commits are observed rather than inferred from timing.

Component mapping

Responsibility	Buzz component
Stable-ID registration, lifecycle, durable store, claims, fencing, checkpoints, retry, and validation	The new buzz-backfill crate owns the domain and persistence.
Database and transaction primitives for atomic mutation and checkpoint progress	buzz-db provides narrow primitives. It does not own backfill records, policy, or orchestration, and it does not expose its connection pool.
Discovery, automatic execution, configuration, startup ordering, and readiness	buzz-relay composes the engine with startup and serving coordination.
Authorized reads, lifecycle actions, validation, audit, conflicts, and diagnostics	Relay deployment-admin interfaces expose the contract; tools use them instead of accessing tables.
List, detail, progress, diagnostics, readiness, and controls	The desktop/admin client projects relay state and never stores another lifecycle.

This keeps the repository's focused-crate architecture: the relay orchestrates, the backfill crate owns its domain, and the database crate provides general transaction infrastructure. Backfill administration remains an operator admin concern; it does not add a Nostr event or NIP.