## Summary An operator can submit, approve, and run a synthetic user-deletion request through a durable workflow. The actions print messages; they do not gate real users or delete data. The inventory contains artificial participant and former-participant communities. The worker calls one transition function, `get_next_stage(lease, previous_stage_id)`. It receives a full stage row, passes it to the handler, then asks again. All branching, community tracking, initialization, and recovery live behind that function. Existing community-deletion execution is unchanged. ### Related issue Builds on merged #8496 and targets `main`. Inventory, separate approval records, and the dummy executor remain together in this PR. ### Execution design ```text approve → save approval; create no stages run → claim request get_next_stage(lease, None) → full stage row handler(lease, row) → persist status and result get_next_stage(lease, Some(row.id)) → next row / Finished / WaitUntil repeat, then release ownership ``` - **Approval:** bind an operator's decision to the saved inventory digest. Approval creates no execution rows. Operator names remain self-asserted demo input. - **Initialization:** claiming preserves the parent lifecycle. The first transition creates `gate_user` and changes `approved` to `running` atomically. A crash after claim but before initialization can safely restart. - **Handler:** read `metadata_json.inputs`, mark the row running, and save its result or failure. A successful action leaves its row `completed` and `is_current = true`. This is a durable transition waiting to be processed. - **Transition:** lock the parent, validate ownership, reload history and the handler's saved result, and choose an allowed destination. Clear the old current flag, save the decision, and insert the successor with its inputs in one transaction. Terminal transitions complete the parent instead. An insertion failure leaves the completed old row current. - **Recovery:** `previous_stage_id` identifies the handler that just returned; it must belong to the request. A restarted worker passes `None` and finds the saved current row. If that row is completed, transition it without repeating its action. If the switch already committed, return the saved successor. Repeated calls never apply the old transition twice. - **Progress:** reconstruct covered communities from saved decisions and the approved inventory. Each stage records its predecessor ID, so reconstruction does not depend on timestamp ordering. Missing current state in a started request, disconnected history, unknown stages, and malformed results fail without inventing progress. - **Ownership:** parent leases last 60 seconds and handlers renew them every ten seconds. Generation checks reject stale writes. Handler and renewal futures remain independently polled; renewal cannot prevent a handler holding the parent lock from committing. - **Step-by-step testing:** `run --next-step` executes one handler, commits its transition, and exits with the successor pending. Repeat the same command to advance again; the final step completes the request. The former `--max-stages` and `--pause-after-handler` flags are removed. - **Failure:** the handler saves the current row's error and a 30-second retry deadline. The transition returns `WaitUntil`; the worker exits and releases its lease. A later invocation retries the same row. No background scheduler or automatic retry daemon is added. The user workflow definition in `buzz-deletion/src/user/workflow.rs` declares allowed destinations and a decision function per stage. `gate_user` enters the first uncovered community. `purge_db` reads saved `has_agents`: true selects `purge_agents`; false completes the community immediately. Community completion selects the next uncovered community, or finishes the request when all are covered. No full stage plan is created upfront. The transition transaction contains no stage-name comparisons or fixed community count. Stage metadata stores `inputs`, `result`, `transition`, and `previous_stage_id`. The parent retains inventory, ownership, retry, and overall status. Approval evidence stays in `entity_deletion_approvals`. A completed-current row bridges the two transactions; real handlers will still need idempotent or reconciled actions if a process dies before saving its result. ### Code organization `buzz-deletion/src/user/worker.rs` runs the loop and heartbeat. Sibling `handlers.rs` executes dummy actions. `transition.rs` defines `UserDeletionTransition.get_next_stage()`, and `workflow.rs` owns the user stage rules and community-progress reconstruction. The CLI remains in `user.rs`. `buzz-db/src/store/entity_deletion/` contains shared workflow persistence: a small `mod.rs`, request reads/inspection in `requests.rs`, approval evidence in `approvals.rs`, ownership in `leases.rs`, and stage outcomes/atomic transitions in `stages.rs`. User-specific submission, demo inventory, and inventory validation live separately in `buzz-db/src/store/user_deletion/{requests,inventory}.rs`; its `mod.rs` only defines and exports the store. The worker calls the user transition object, which supplies a synchronous decision function to `EntityDeletionStore.transition_stage`. The store locks the parent, reads saved state, invokes that function, checks the returned operation, and commits its writes. The decision function performs no SQL or external calls. Reading, deciding, and writing remain in one transaction; there is no stale-state gap. Shared storage contains no user stage names or community-routing rules. User-specific approval validation also runs inside the shared approval transaction. This reorganization changes no tables, migrations, stage metadata, or CLI behavior. Requests using the existing saved-result format continue to work. `--next-step` still runs one handler and commits its transition before returning. ### Testing The walkthrough in [docs/operator-user-deletion-executor.md](https://github.com/block/buzz/blob/rarora/user-deletion-approval/docs/operator-user-deletion-executor.md) includes commands, SQL inspection, expected rows, and recovery boundaries. Submitted and approved a fresh local request and verified zero stage rows. Ran `run --next-step` five times. Each call completed exactly one handler, released ownership, and left its successor pending: B DB, B agents, C DB, C agents, then request completed. A sixth call left saved state unchanged. Verified both removed flags are rejected. Ran `--demo-no-agents-in 00000000-0000-0000-0000-00000000000c` and verified four completed rows, with no C agents stage. Injected a DB-stage failure, verified immediate retry refusal, waited for the real deadline, and resumed the same stage row. Observed heartbeat renewal and rejection of a competing worker, killed the owned worker process, waited for its actual lease expiry, and resumed the same unfinished stage with a new generation. Re-running a completed request left its saved state unchanged. Repeated the CLI walkthrough after the file reorganization, including runtime branching and a retry after the real deadline; a completed request created before the reorganization remained unchanged. Mutation-tested the ownership and recovery guards against local PostgreSQL 17. Reclaiming with the same executor ID rejects old-generation operations. Independently disabling the ownership generation check, release generation check, saved-transition/result comparison, or heartbeat-failure handling made the corresponding test fail. Restored all four protections afterward. The heartbeat test takes over the request while its handler is already running. ### Schema and rollout Unmerged migrations 0069–0070 add inventory and execution fields, widen request lifecycle states, and create operator-global approval and stage-history tables. They set five-second lock and thirty-second statement timeouts; constraint validation has its own transaction. No triggers, functions, foreign keys, or data backfills are added. Desired-state and migration catalog parity cover all three tables. The transition redesign adds no further schema changes. The serving relay does not schedule these requests. Only the explicit operator CLI runs them, using PostgreSQL. Only `demo:` targets are accepted. Real account discovery, user-write fencing, owned-community handling, and actual deletion remain future work. Verify staging with a new demo label and the documented commands. Older draft requests do not have the new stage metadata contract. Preserve them and create a new demo request for this version; they are not a supported resume format. Use a fresh database if an earlier draft applied a different migration checksum. Never alter sqlx migration checksums. With Helm `migrate.autoMigrate=true` (the chart default), a relay binary that lacks 0069/0070 fails at boot with sqlx `VersionMissing` after those migrations are applied. Recovery for this rollout is a roll-forward: deploy a corrected binary that retains the migrations. Do not delete migration records or change checksums to force an older binary to boot. Rolling back with auto-migration disabled requires a separately verified schema-compatibility plan. Demo requests, stage history, and approvals are intentionally retained; this scaffold has no cleanup command or retention job. Recheck migration numbering before merge. Generated with Codex --------- Signed-off-by: Ravneet Arora <rarora@squareup.com>
7.6 KiB
Prepare and approve a synthetic user-deletion request
Submit saves a request and a fixed dummy inventory: community B (participant),
then community C (former participant). These are artificial UUIDs, not discovered
communities. Approve saves authorization without creating stages. The
run command initializes execution through get_next_stage, dispatches dummy
handlers, and asks the transition function for the next saved row. No user or
community data is deleted. Only demo: labels are accepted.
Prepare a local database
From the repository root, activate Hermit and start a separate PostgreSQL container. These are disposable local credentials.
. ./bin/activate-hermit
docker run --detach --name buzz-user-deletion-intake \
--publish 127.0.0.1:55433:5432 \
--env POSTGRES_USER=buzz \
--env POSTGRES_PASSWORD=buzz-local-demo \
--env POSTGRES_DB=buzz postgres:17-alpine
export DATABASE_URL=postgres://buzz:buzz-local-demo@127.0.0.1:55433/buzz
cargo build -p buzz-admin
# Wait for "accepting connections" before migrating.
docker exec buzz-user-deletion-intake pg_isready -U buzz -d buzz
./target/debug/buzz-admin migrate
Use an unused port and container name if these are occupied. The commands below need only PostgreSQL; a relay, Redis, and object storage are unnecessary.
Submit and see the row
./target/debug/buzz-admin user-deletions submit \
--target demo:alice --requested-by operator-a
The command prints the saved row as JSON. It contains:
| Field | Value |
|---|---|
id |
New UUID; copy this for inspection |
entity_type |
user |
target |
demo:alice |
status |
inventoried |
requested_by |
operator-a |
reason |
null, unless supplied with --reason |
created_at |
Submission time |
updated_at |
Same time; no execution has started |
inventory_manifest |
Ordered dummy communities and their actions |
inventory_digest |
SHA-256 identifying the saved inventory |
List recent requests, then inspect the UUID returned above:
./target/debug/buzz-admin user-deletions list --limit 10
REQUEST_ID=<returned-uuid>
./target/debug/buzz-admin user-deletions inspect "$REQUEST_ID"
To see the PostgreSQL row directly:
docker exec buzz-user-deletion-intake psql -X -U buzz -d buzz -c \
"SELECT id, entity_type, target, status, requested_by, created_at
FROM entity_deletion_requests ORDER BY created_at DESC;"
Exactly one row is saved in entity_deletion_requests. Repeating the same
target, operator, and reason returns that row without changing its timestamps.
Reusing the target with a different operator or reason fails and preserves the
original row. Use a new demo: label for a new request.
Approve without starting execution
Inspect the inventory before approving. Copy its inventory_digest into the
command so approval is bound to what you inspected:
INVENTORY_DIGEST=<digest-from-inspect>
./target/debug/buzz-admin user-deletions approve "$REQUEST_ID" \
--approved-by operator-b --inventory-digest "$INVENTORY_DIGEST"
./target/debug/buzz-admin user-deletions inspect "$REQUEST_ID"
The parent now has status: approved. Approval evidence lives in the separate
entity_deletion_approvals table. Each row has id, request_id,
inventory_digest, approved_by, and approved_at. Inspection returns these
rows in an approvals array (empty before approval), ordered by time and ID.
The parent keeps the inventory; approval rows bind each operator's decision to
its digest without overwriting earlier decisions.
stage_history is empty after approval. The first run creates the initial
stage. Inspection reads parent, approvals, and history from one consistent
snapshot. No worker is started by approval.
docker exec buzz-user-deletion-intake psql -X -U buzz -d buzz -c \
"SELECT id, status, inventory_digest
FROM entity_deletion_requests ORDER BY created_at DESC;"
docker exec buzz-user-deletion-intake psql -X -U buzz -d buzz -c \
"SELECT id, request_id, approved_by, approved_at, inventory_digest
FROM entity_deletion_approvals ORDER BY approved_at, id;"
docker exec buzz-user-deletion-intake psql -X -U buzz -d buzz -c \
"SELECT request_id, stage, status, is_current, target_community_id,
started_at, finished_at, metadata_json
FROM entity_deletion_stage_history;"
The first approval inserts an approval row and changes the parent status in one transaction. A failure rolls both back. Repeating the same operator and digest preserves the approval ID and timestamp. Another operator adds a new approval row, with its own time, without creating another stage or changing the earlier approval:
./target/debug/buzz-admin user-deletions approve "$REQUEST_ID" \
--approved-by operator-c --inventory-digest "$INVENTORY_DIGEST"
./target/debug/buzz-admin user-deletions inspect "$REQUEST_ID"
There are now two approvals and still no stage rows. The unique
key (request_id, inventory_digest, approved_by) makes retries idempotent.
A digest that does not match the saved inventory is rejected. This slice has no
inventory replacement or later approval gates yet; those policies can be added
without replacing these historical rows. A partial unique index prevents two
current stages for one request. Executor retries update the same stage row.
Operator names are self-asserted demo input. This is not authenticated approval for real deletion. Owned communities and actual deletion handlers belong to later changes. Continue with the executor walkthrough to run the five dummy stages, inspect transitions, and try failure recovery.
Upgrade and rollout
Apply after the intake schema from PR #8496. Migrations 0069 and 0070 add inventory and execution fields, widen the lifecycle constraint, and create the operator-global stage-history and approval tables. Constraint validation uses a separate transaction so it does not retain the schema-change transaction's write-blocking lock. Both migrations set a five-second lock timeout and a thirty-second statement timeout. There are no new triggers, functions, foreign keys, or data backfills.
Existing submitted requests remain unchanged by migration. Resubmit with the
same target, operator, and reason to prepare their dummy inventory; the request
ID and original creation time stay unchanged. The previous CLI can still insert
and read requests against the expanded schema. The serving relay does not
automatically execute these requests; an operator runs the CLI worker explicitly.
Stage history and approvals intentionally survive
community deletion as operator recovery/audit evidence.
Verify on a test environment with the submit, inspect, approve, and SQL commands
above using a new demo: target. Rollback has the usual sqlx limitation: an older
binary running auto-migration refuses unknown migrations (VersionMissing).
Do not roll back to an older auto-migrating relay without the deployment's
reviewed rollback procedure; the additive schema itself can remain in place.
The unmerged draft migration 0069 changed when approvals moved to their own table, and execution fields were added to requests. If you tested an earlier draft, use a fresh disposable local database; do not edit its sqlx checksums or erase its saved demo requests.
This executor revision does not change SQL schema. Earlier draft requests whose
stage metadata predates saved inputs/results/predecessor IDs are not a supported
resume format. Keep their records intact and submit a new demo: label to test
this version. Approval no longer creates a gate row; it is initialized by the
first get_next_stage call under worker ownership.