Files
buzz/docs/operator-user-deletion-intake.md
ravarora2 4250014bae Run user deletion through saved stage transitions (#8520)
## Summary

An operator can submit, approve, and run a synthetic user-deletion
request through a durable workflow. The actions print messages; they do
not gate real users or delete data. The inventory contains artificial
participant and former-participant communities.

The worker calls one transition function, `get_next_stage(lease,
previous_stage_id)`. It receives a full stage row, passes it to the
handler, then asks again. All branching, community tracking,
initialization, and recovery live behind that function. Existing
community-deletion execution is unchanged.

### Related issue

Builds on merged #8496 and targets `main`. Inventory, separate approval
records, and the dummy executor remain together in this PR.

### Execution design

```text
approve → save approval; create no stages
run → claim request
    get_next_stage(lease, None) → full stage row
    handler(lease, row) → persist status and result
    get_next_stage(lease, Some(row.id)) → next row / Finished / WaitUntil
    repeat, then release ownership
```

- **Approval:** bind an operator's decision to the saved inventory
digest. Approval creates no execution rows. Operator names remain
self-asserted demo input.
- **Initialization:** claiming preserves the parent lifecycle. The first
transition creates `gate_user` and changes `approved` to `running`
atomically. A crash after claim but before initialization can safely
restart.
- **Handler:** read `metadata_json.inputs`, mark the row running, and
save its result or failure. A successful action leaves its row
`completed` and `is_current = true`. This is a durable transition
waiting to be processed.
- **Transition:** lock the parent, validate ownership, reload history
and the handler's saved result, and choose an allowed destination. Clear
the old current flag, save the decision, and insert the successor with
its inputs in one transaction. Terminal transitions complete the parent
instead. An insertion failure leaves the completed old row current.
- **Recovery:** `previous_stage_id` identifies the handler that just
returned; it must belong to the request. A restarted worker passes
`None` and finds the saved current row. If that row is completed,
transition it without repeating its action. If the switch already
committed, return the saved successor. Repeated calls never apply the
old transition twice.
- **Progress:** reconstruct covered communities from saved decisions and
the approved inventory. Each stage records its predecessor ID, so
reconstruction does not depend on timestamp ordering. Missing current
state in a started request, disconnected history, unknown stages, and
malformed results fail without inventing progress.
- **Ownership:** parent leases last 60 seconds and handlers renew them
every ten seconds. Generation checks reject stale writes. Handler and
renewal futures remain independently polled; renewal cannot prevent a
handler holding the parent lock from committing.
- **Step-by-step testing:** `run --next-step` executes one handler,
commits its transition, and exits with the successor pending. Repeat the
same command to advance again; the final step completes the request. The
former `--max-stages` and `--pause-after-handler` flags are removed.
- **Failure:** the handler saves the current row's error and a 30-second
retry deadline. The transition returns `WaitUntil`; the worker exits and
releases its lease. A later invocation retries the same row. No
background scheduler or automatic retry daemon is added.

The user workflow definition in `buzz-deletion/src/user/workflow.rs`
declares allowed destinations and a decision function per stage.
`gate_user` enters the first uncovered community. `purge_db` reads saved
`has_agents`: true selects `purge_agents`; false completes the community
immediately. Community completion selects the next uncovered community,
or finishes the request when all are covered. No full stage plan is
created upfront. The transition transaction contains no stage-name
comparisons or fixed community count.

Stage metadata stores `inputs`, `result`, `transition`, and
`previous_stage_id`. The parent retains inventory, ownership, retry, and
overall status. Approval evidence stays in `entity_deletion_approvals`.
A completed-current row bridges the two transactions; real handlers will
still need idempotent or reconciled actions if a process dies before
saving its result.

### Code organization

`buzz-deletion/src/user/worker.rs` runs the loop and heartbeat. Sibling
`handlers.rs` executes dummy actions. `transition.rs` defines
`UserDeletionTransition.get_next_stage()`, and `workflow.rs` owns the
user stage rules and community-progress reconstruction. The CLI remains
in `user.rs`.

`buzz-db/src/store/entity_deletion/` contains shared workflow
persistence: a small `mod.rs`, request reads/inspection in
`requests.rs`, approval evidence in `approvals.rs`, ownership in
`leases.rs`, and stage outcomes/atomic transitions in `stages.rs`.
User-specific submission, demo inventory, and inventory validation live
separately in `buzz-db/src/store/user_deletion/{requests,inventory}.rs`;
its `mod.rs` only defines and exports the store.

The worker calls the user transition object, which supplies a
synchronous decision function to `EntityDeletionStore.transition_stage`.
The store locks the parent, reads saved state, invokes that function,
checks the returned operation, and commits its writes. The decision
function performs no SQL or external calls. Reading, deciding, and
writing remain in one transaction; there is no stale-state gap. Shared
storage contains no user stage names or community-routing rules.
User-specific approval validation also runs inside the shared approval
transaction.

This reorganization changes no tables, migrations, stage metadata, or
CLI behavior. Requests using the existing saved-result format continue
to work. `--next-step` still runs one handler and commits its transition
before returning.

### Testing

The walkthrough in
[docs/operator-user-deletion-executor.md](https://github.com/block/buzz/blob/rarora/user-deletion-approval/docs/operator-user-deletion-executor.md)
includes commands, SQL inspection, expected rows, and recovery
boundaries.

Submitted and approved a fresh local request and verified zero stage
rows. Ran `run --next-step` five times. Each call completed exactly one
handler, released ownership, and left its successor pending: B DB, B
agents, C DB, C agents, then request completed. A sixth call left saved
state unchanged. Verified both removed flags are rejected.

Ran `--demo-no-agents-in 00000000-0000-0000-0000-00000000000c` and
verified four completed rows, with no C agents stage.

Injected a DB-stage failure, verified immediate retry refusal, waited
for the real deadline, and resumed the same stage row. Observed
heartbeat renewal and rejection of a competing worker, killed the owned
worker process, waited for its actual lease expiry, and resumed the same
unfinished stage with a new generation. Re-running a completed request
left its saved state unchanged. Repeated the CLI walkthrough after the
file reorganization, including runtime branching and a retry after the
real deadline; a completed request created before the reorganization
remained unchanged.

Mutation-tested the ownership and recovery guards against local
PostgreSQL 17. Reclaiming with the same executor ID rejects
old-generation operations. Independently disabling the ownership
generation check, release generation check, saved-transition/result
comparison, or heartbeat-failure handling made the corresponding test
fail. Restored all four protections afterward. The heartbeat test takes
over the request while its handler is already running.

### Schema and rollout

Unmerged migrations 0069–0070 add inventory and execution fields, widen
request lifecycle states, and create operator-global approval and
stage-history tables. They set five-second lock and thirty-second
statement timeouts; constraint validation has its own transaction. No
triggers, functions, foreign keys, or data backfills are added.
Desired-state and migration catalog parity cover all three tables. The
transition redesign adds no further schema changes.

The serving relay does not schedule these requests. Only the explicit
operator CLI runs them, using PostgreSQL. Only `demo:` targets are
accepted. Real account discovery, user-write fencing, owned-community
handling, and actual deletion remain future work. Verify staging with a
new demo label and the documented commands.

Older draft requests do not have the new stage metadata contract.
Preserve them and create a new demo request for this version; they are
not a supported resume format. Use a fresh database if an earlier draft
applied a different migration checksum. Never alter sqlx migration
checksums. With Helm `migrate.autoMigrate=true` (the chart default), a
relay binary that lacks 0069/0070 fails at boot with sqlx
`VersionMissing` after those migrations are applied. Recovery for this
rollout is a roll-forward: deploy a corrected binary that retains the
migrations. Do not delete migration records or change checksums to force
an older binary to boot. Rolling back with auto-migration disabled
requires a separately verified schema-compatibility plan. Demo requests,
stage history, and approvals are intentionally retained; this scaffold
has no cleanup command or retention job. Recheck migration numbering
before merge.

Generated with Codex

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>
2026-10-26 10:44:32 -04:00

7.6 KiB

Prepare and approve a synthetic user-deletion request

Submit saves a request and a fixed dummy inventory: community B (participant), then community C (former participant). These are artificial UUIDs, not discovered communities. Approve saves authorization without creating stages. The run command initializes execution through get_next_stage, dispatches dummy handlers, and asks the transition function for the next saved row. No user or community data is deleted. Only demo: labels are accepted.

Prepare a local database

From the repository root, activate Hermit and start a separate PostgreSQL container. These are disposable local credentials.

. ./bin/activate-hermit
docker run --detach --name buzz-user-deletion-intake \
  --publish 127.0.0.1:55433:5432 \
  --env POSTGRES_USER=buzz \
  --env POSTGRES_PASSWORD=buzz-local-demo \
  --env POSTGRES_DB=buzz postgres:17-alpine
export DATABASE_URL=postgres://buzz:buzz-local-demo@127.0.0.1:55433/buzz
cargo build -p buzz-admin
# Wait for "accepting connections" before migrating.
docker exec buzz-user-deletion-intake pg_isready -U buzz -d buzz
./target/debug/buzz-admin migrate

Use an unused port and container name if these are occupied. The commands below need only PostgreSQL; a relay, Redis, and object storage are unnecessary.

Submit and see the row

./target/debug/buzz-admin user-deletions submit \
  --target demo:alice --requested-by operator-a

The command prints the saved row as JSON. It contains:

Field Value
id New UUID; copy this for inspection
entity_type user
target demo:alice
status inventoried
requested_by operator-a
reason null, unless supplied with --reason
created_at Submission time
updated_at Same time; no execution has started
inventory_manifest Ordered dummy communities and their actions
inventory_digest SHA-256 identifying the saved inventory

List recent requests, then inspect the UUID returned above:

./target/debug/buzz-admin user-deletions list --limit 10
REQUEST_ID=<returned-uuid>
./target/debug/buzz-admin user-deletions inspect "$REQUEST_ID"

To see the PostgreSQL row directly:

docker exec buzz-user-deletion-intake psql -X -U buzz -d buzz -c \
  "SELECT id, entity_type, target, status, requested_by, created_at
   FROM entity_deletion_requests ORDER BY created_at DESC;"

Exactly one row is saved in entity_deletion_requests. Repeating the same target, operator, and reason returns that row without changing its timestamps. Reusing the target with a different operator or reason fails and preserves the original row. Use a new demo: label for a new request.

Approve without starting execution

Inspect the inventory before approving. Copy its inventory_digest into the command so approval is bound to what you inspected:

INVENTORY_DIGEST=<digest-from-inspect>
./target/debug/buzz-admin user-deletions approve "$REQUEST_ID" \
  --approved-by operator-b --inventory-digest "$INVENTORY_DIGEST"
./target/debug/buzz-admin user-deletions inspect "$REQUEST_ID"

The parent now has status: approved. Approval evidence lives in the separate entity_deletion_approvals table. Each row has id, request_id, inventory_digest, approved_by, and approved_at. Inspection returns these rows in an approvals array (empty before approval), ordered by time and ID. The parent keeps the inventory; approval rows bind each operator's decision to its digest without overwriting earlier decisions.

stage_history is empty after approval. The first run creates the initial stage. Inspection reads parent, approvals, and history from one consistent snapshot. No worker is started by approval.

docker exec buzz-user-deletion-intake psql -X -U buzz -d buzz -c \
  "SELECT id, status, inventory_digest
   FROM entity_deletion_requests ORDER BY created_at DESC;"
docker exec buzz-user-deletion-intake psql -X -U buzz -d buzz -c \
  "SELECT id, request_id, approved_by, approved_at, inventory_digest
   FROM entity_deletion_approvals ORDER BY approved_at, id;"
docker exec buzz-user-deletion-intake psql -X -U buzz -d buzz -c \
  "SELECT request_id, stage, status, is_current, target_community_id,
          started_at, finished_at, metadata_json
   FROM entity_deletion_stage_history;"

The first approval inserts an approval row and changes the parent status in one transaction. A failure rolls both back. Repeating the same operator and digest preserves the approval ID and timestamp. Another operator adds a new approval row, with its own time, without creating another stage or changing the earlier approval:

./target/debug/buzz-admin user-deletions approve "$REQUEST_ID" \
  --approved-by operator-c --inventory-digest "$INVENTORY_DIGEST"
./target/debug/buzz-admin user-deletions inspect "$REQUEST_ID"

There are now two approvals and still no stage rows. The unique key (request_id, inventory_digest, approved_by) makes retries idempotent. A digest that does not match the saved inventory is rejected. This slice has no inventory replacement or later approval gates yet; those policies can be added without replacing these historical rows. A partial unique index prevents two current stages for one request. Executor retries update the same stage row.

Operator names are self-asserted demo input. This is not authenticated approval for real deletion. Owned communities and actual deletion handlers belong to later changes. Continue with the executor walkthrough to run the five dummy stages, inspect transitions, and try failure recovery.

Upgrade and rollout

Apply after the intake schema from PR #8496. Migrations 0069 and 0070 add inventory and execution fields, widen the lifecycle constraint, and create the operator-global stage-history and approval tables. Constraint validation uses a separate transaction so it does not retain the schema-change transaction's write-blocking lock. Both migrations set a five-second lock timeout and a thirty-second statement timeout. There are no new triggers, functions, foreign keys, or data backfills.

Existing submitted requests remain unchanged by migration. Resubmit with the same target, operator, and reason to prepare their dummy inventory; the request ID and original creation time stay unchanged. The previous CLI can still insert and read requests against the expanded schema. The serving relay does not automatically execute these requests; an operator runs the CLI worker explicitly. Stage history and approvals intentionally survive community deletion as operator recovery/audit evidence.

Verify on a test environment with the submit, inspect, approve, and SQL commands above using a new demo: target. Rollback has the usual sqlx limitation: an older binary running auto-migration refuses unknown migrations (VersionMissing). Do not roll back to an older auto-migrating relay without the deployment's reviewed rollback procedure; the additive schema itself can remain in place.

The unmerged draft migration 0069 changed when approvals moved to their own table, and execution fields were added to requests. If you tested an earlier draft, use a fresh disposable local database; do not edit its sqlx checksums or erase its saved demo requests.

This executor revision does not change SQL schema. Earlier draft requests whose stage metadata predates saved inputs/results/predecessor IDs are not a supported resume format. Keep their records intact and submit a new demo: label to test this version. Approval no longer creates a gate row; it is initialized by the first get_next_stage call under worker ownership.