Files
buzz/migrations/0069_entity_deletion_approval.sql
ravarora2 4250014bae Run user deletion through saved stage transitions (#8520)
## Summary

An operator can submit, approve, and run a synthetic user-deletion
request through a durable workflow. The actions print messages; they do
not gate real users or delete data. The inventory contains artificial
participant and former-participant communities.

The worker calls one transition function, `get_next_stage(lease,
previous_stage_id)`. It receives a full stage row, passes it to the
handler, then asks again. All branching, community tracking,
initialization, and recovery live behind that function. Existing
community-deletion execution is unchanged.

### Related issue

Builds on merged #8496 and targets `main`. Inventory, separate approval
records, and the dummy executor remain together in this PR.

### Execution design

```text
approve → save approval; create no stages
run → claim request
    get_next_stage(lease, None) → full stage row
    handler(lease, row) → persist status and result
    get_next_stage(lease, Some(row.id)) → next row / Finished / WaitUntil
    repeat, then release ownership
```

- **Approval:** bind an operator's decision to the saved inventory
digest. Approval creates no execution rows. Operator names remain
self-asserted demo input.
- **Initialization:** claiming preserves the parent lifecycle. The first
transition creates `gate_user` and changes `approved` to `running`
atomically. A crash after claim but before initialization can safely
restart.
- **Handler:** read `metadata_json.inputs`, mark the row running, and
save its result or failure. A successful action leaves its row
`completed` and `is_current = true`. This is a durable transition
waiting to be processed.
- **Transition:** lock the parent, validate ownership, reload history
and the handler's saved result, and choose an allowed destination. Clear
the old current flag, save the decision, and insert the successor with
its inputs in one transaction. Terminal transitions complete the parent
instead. An insertion failure leaves the completed old row current.
- **Recovery:** `previous_stage_id` identifies the handler that just
returned; it must belong to the request. A restarted worker passes
`None` and finds the saved current row. If that row is completed,
transition it without repeating its action. If the switch already
committed, return the saved successor. Repeated calls never apply the
old transition twice.
- **Progress:** reconstruct covered communities from saved decisions and
the approved inventory. Each stage records its predecessor ID, so
reconstruction does not depend on timestamp ordering. Missing current
state in a started request, disconnected history, unknown stages, and
malformed results fail without inventing progress.
- **Ownership:** parent leases last 60 seconds and handlers renew them
every ten seconds. Generation checks reject stale writes. Handler and
renewal futures remain independently polled; renewal cannot prevent a
handler holding the parent lock from committing.
- **Step-by-step testing:** `run --next-step` executes one handler,
commits its transition, and exits with the successor pending. Repeat the
same command to advance again; the final step completes the request. The
former `--max-stages` and `--pause-after-handler` flags are removed.
- **Failure:** the handler saves the current row's error and a 30-second
retry deadline. The transition returns `WaitUntil`; the worker exits and
releases its lease. A later invocation retries the same row. No
background scheduler or automatic retry daemon is added.

The user workflow definition in `buzz-deletion/src/user/workflow.rs`
declares allowed destinations and a decision function per stage.
`gate_user` enters the first uncovered community. `purge_db` reads saved
`has_agents`: true selects `purge_agents`; false completes the community
immediately. Community completion selects the next uncovered community,
or finishes the request when all are covered. No full stage plan is
created upfront. The transition transaction contains no stage-name
comparisons or fixed community count.

Stage metadata stores `inputs`, `result`, `transition`, and
`previous_stage_id`. The parent retains inventory, ownership, retry, and
overall status. Approval evidence stays in `entity_deletion_approvals`.
A completed-current row bridges the two transactions; real handlers will
still need idempotent or reconciled actions if a process dies before
saving its result.

### Code organization

`buzz-deletion/src/user/worker.rs` runs the loop and heartbeat. Sibling
`handlers.rs` executes dummy actions. `transition.rs` defines
`UserDeletionTransition.get_next_stage()`, and `workflow.rs` owns the
user stage rules and community-progress reconstruction. The CLI remains
in `user.rs`.

`buzz-db/src/store/entity_deletion/` contains shared workflow
persistence: a small `mod.rs`, request reads/inspection in
`requests.rs`, approval evidence in `approvals.rs`, ownership in
`leases.rs`, and stage outcomes/atomic transitions in `stages.rs`.
User-specific submission, demo inventory, and inventory validation live
separately in `buzz-db/src/store/user_deletion/{requests,inventory}.rs`;
its `mod.rs` only defines and exports the store.

The worker calls the user transition object, which supplies a
synchronous decision function to `EntityDeletionStore.transition_stage`.
The store locks the parent, reads saved state, invokes that function,
checks the returned operation, and commits its writes. The decision
function performs no SQL or external calls. Reading, deciding, and
writing remain in one transaction; there is no stale-state gap. Shared
storage contains no user stage names or community-routing rules.
User-specific approval validation also runs inside the shared approval
transaction.

This reorganization changes no tables, migrations, stage metadata, or
CLI behavior. Requests using the existing saved-result format continue
to work. `--next-step` still runs one handler and commits its transition
before returning.

### Testing

The walkthrough in
[docs/operator-user-deletion-executor.md](https://github.com/block/buzz/blob/rarora/user-deletion-approval/docs/operator-user-deletion-executor.md)
includes commands, SQL inspection, expected rows, and recovery
boundaries.

Submitted and approved a fresh local request and verified zero stage
rows. Ran `run --next-step` five times. Each call completed exactly one
handler, released ownership, and left its successor pending: B DB, B
agents, C DB, C agents, then request completed. A sixth call left saved
state unchanged. Verified both removed flags are rejected.

Ran `--demo-no-agents-in 00000000-0000-0000-0000-00000000000c` and
verified four completed rows, with no C agents stage.

Injected a DB-stage failure, verified immediate retry refusal, waited
for the real deadline, and resumed the same stage row. Observed
heartbeat renewal and rejection of a competing worker, killed the owned
worker process, waited for its actual lease expiry, and resumed the same
unfinished stage with a new generation. Re-running a completed request
left its saved state unchanged. Repeated the CLI walkthrough after the
file reorganization, including runtime branching and a retry after the
real deadline; a completed request created before the reorganization
remained unchanged.

Mutation-tested the ownership and recovery guards against local
PostgreSQL 17. Reclaiming with the same executor ID rejects
old-generation operations. Independently disabling the ownership
generation check, release generation check, saved-transition/result
comparison, or heartbeat-failure handling made the corresponding test
fail. Restored all four protections afterward. The heartbeat test takes
over the request while its handler is already running.

### Schema and rollout

Unmerged migrations 0069–0070 add inventory and execution fields, widen
request lifecycle states, and create operator-global approval and
stage-history tables. They set five-second lock and thirty-second
statement timeouts; constraint validation has its own transaction. No
triggers, functions, foreign keys, or data backfills are added.
Desired-state and migration catalog parity cover all three tables. The
transition redesign adds no further schema changes.

The serving relay does not schedule these requests. Only the explicit
operator CLI runs them, using PostgreSQL. Only `demo:` targets are
accepted. Real account discovery, user-write fencing, owned-community
handling, and actual deletion remain future work. Verify staging with a
new demo label and the documented commands.

Older draft requests do not have the new stage metadata contract.
Preserve them and create a new demo request for this version; they are
not a supported resume format. Use a fresh database if an earlier draft
applied a different migration checksum. Never alter sqlx migration
checksums. With Helm `migrate.autoMigrate=true` (the chart default), a
relay binary that lacks 0069/0070 fails at boot with sqlx
`VersionMissing` after those migrations are applied. Recovery for this
rollout is a roll-forward: deploy a corrected binary that retains the
migrations. Do not delete migration records or change checksums to force
an older binary to boot. Rolling back with auto-migration disabled
requires a separately verified schema-compatibility plan. Demo requests,
stage history, and approvals are intentionally retained; this scaffold
has no cleanup command or retention job. Recheck migration numbering
before merge.

Generated with Codex

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>
2026-10-26 10:44:32 -04:00

61 lines
2.9 KiB
SQL

SET LOCAL lock_timeout = '5s';
SET LOCAL statement_timeout = '30s';
-- Expand the demo intake. Existing submitted rows remain valid and unchanged.
-- A matching resubmission through the new CLI prepares their dummy inventory.
ALTER TABLE entity_deletion_requests
DROP CONSTRAINT entity_deletion_requests_status_check,
ADD CONSTRAINT entity_deletion_requests_status_check
CHECK (status IN ('submitted', 'inventoried', 'approved', 'running', 'retry_wait', 'completed')) NOT VALID,
ADD COLUMN inventory_manifest JSONB,
ADD COLUMN inventory_digest TEXT,
ADD COLUMN lease_owner TEXT,
ADD COLUMN lease_generation BIGINT NOT NULL DEFAULT 0,
ADD COLUMN lease_expires_at TIMESTAMPTZ,
ADD COLUMN heartbeat_at TIMESTAMPTZ,
ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0,
ADD COLUMN next_attempt_at TIMESTAMPTZ,
ADD COLUMN last_error TEXT,
ADD COLUMN completed_at TIMESTAMPTZ;
-- Validate in 0070 so the scan does not retain this transaction's DDL lock.
-- Operator-global workflow history. Retained after community deletion as
-- orchestration evidence; target_community_id is provenance, not tenant ownership.
CREATE TABLE entity_deletion_stage_history (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
request_id UUID NOT NULL,
target_community_id UUID,
stage TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending' CHECK (status IN ('pending', 'running', 'failed', 'completed')),
is_current BOOLEAN NOT NULL DEFAULT true,
started_at TIMESTAMPTZ,
finished_at TIMESTAMPTZ,
metadata_json JSONB NOT NULL DEFAULT '{}'::jsonb,
last_error TEXT,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE UNIQUE INDEX entity_deletion_one_current_stage
ON entity_deletion_stage_history (request_id) WHERE is_current = true;
CREATE INDEX entity_deletion_stage_history_request_idx
ON entity_deletion_stage_history (request_id);
INSERT INTO _operator_global_tables (table_name, reason)
VALUES ('entity_deletion_stage_history', 'Operator-global deletion workflow history retained for recovery and audit')
ON CONFLICT (table_name) DO NOTHING;
-- Approval evidence is retained after community deletion for operator audit.
-- Each operator approves an inventory once; retries preserve the original row.
CREATE TABLE entity_deletion_approvals (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
request_id UUID NOT NULL,
inventory_digest TEXT NOT NULL CHECK (inventory_digest ~ '^[0-9a-f]{64}$'),
approved_by TEXT NOT NULL CHECK (approved_by = btrim(approved_by) AND octet_length(approved_by) >= 1 AND octet_length(approved_by) <= 256),
approved_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (request_id, inventory_digest, approved_by)
);
INSERT INTO _operator_global_tables (table_name, reason)
VALUES ('entity_deletion_approvals', 'Operator-global deletion approval history retained for audit')
ON CONFLICT (table_name) DO NOTHING;