jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-10-27 04:30:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

Run user deletion through saved stage transitions (#8520)

Browse Source 
## Summary

An operator can submit, approve, and run a synthetic user-deletion
request through a durable workflow. The actions print messages; they do
not gate real users or delete data. The inventory contains artificial
participant and former-participant communities.

The worker calls one transition function, `get_next_stage(lease,
previous_stage_id)`. It receives a full stage row, passes it to the
handler, then asks again. All branching, community tracking,
initialization, and recovery live behind that function. Existing
community-deletion execution is unchanged.

### Related issue

Builds on merged #8496 and targets `main`. Inventory, separate approval
records, and the dummy executor remain together in this PR.

### Execution design

```text
approve → save approval; create no stages
run → claim request
    get_next_stage(lease, None) → full stage row
    handler(lease, row) → persist status and result
    get_next_stage(lease, Some(row.id)) → next row / Finished / WaitUntil
    repeat, then release ownership
```

- **Approval:** bind an operator's decision to the saved inventory
digest. Approval creates no execution rows. Operator names remain
self-asserted demo input.
- **Initialization:** claiming preserves the parent lifecycle. The first
transition creates `gate_user` and changes `approved` to `running`
atomically. A crash after claim but before initialization can safely
restart.
- **Handler:** read `metadata_json.inputs`, mark the row running, and
save its result or failure. A successful action leaves its row
`completed` and `is_current = true`. This is a durable transition
waiting to be processed.
- **Transition:** lock the parent, validate ownership, reload history
and the handler's saved result, and choose an allowed destination. Clear
the old current flag, save the decision, and insert the successor with
its inputs in one transaction. Terminal transitions complete the parent
instead. An insertion failure leaves the completed old row current.
- **Recovery:** `previous_stage_id` identifies the handler that just
returned; it must belong to the request. A restarted worker passes
`None` and finds the saved current row. If that row is completed,
transition it without repeating its action. If the switch already
committed, return the saved successor. Repeated calls never apply the
old transition twice.
- **Progress:** reconstruct covered communities from saved decisions and
the approved inventory. Each stage records its predecessor ID, so
reconstruction does not depend on timestamp ordering. Missing current
state in a started request, disconnected history, unknown stages, and
malformed results fail without inventing progress.
- **Ownership:** parent leases last 60 seconds and handlers renew them
every ten seconds. Generation checks reject stale writes. Handler and
renewal futures remain independently polled; renewal cannot prevent a
handler holding the parent lock from committing.
- **Step-by-step testing:** `run --next-step` executes one handler,
commits its transition, and exits with the successor pending. Repeat the
same command to advance again; the final step completes the request. The
former `--max-stages` and `--pause-after-handler` flags are removed.
- **Failure:** the handler saves the current row's error and a 30-second
retry deadline. The transition returns `WaitUntil`; the worker exits and
releases its lease. A later invocation retries the same row. No
background scheduler or automatic retry daemon is added.

The user workflow definition in `buzz-deletion/src/user/workflow.rs`
declares allowed destinations and a decision function per stage.
`gate_user` enters the first uncovered community. `purge_db` reads saved
`has_agents`: true selects `purge_agents`; false completes the community
immediately. Community completion selects the next uncovered community,
or finishes the request when all are covered. No full stage plan is
created upfront. The transition transaction contains no stage-name
comparisons or fixed community count.

Stage metadata stores `inputs`, `result`, `transition`, and
`previous_stage_id`. The parent retains inventory, ownership, retry, and
overall status. Approval evidence stays in `entity_deletion_approvals`.
A completed-current row bridges the two transactions; real handlers will
still need idempotent or reconciled actions if a process dies before
saving its result.

### Code organization

`buzz-deletion/src/user/worker.rs` runs the loop and heartbeat. Sibling
`handlers.rs` executes dummy actions. `transition.rs` defines
`UserDeletionTransition.get_next_stage()`, and `workflow.rs` owns the
user stage rules and community-progress reconstruction. The CLI remains
in `user.rs`.

`buzz-db/src/store/entity_deletion/` contains shared workflow
persistence: a small `mod.rs`, request reads/inspection in
`requests.rs`, approval evidence in `approvals.rs`, ownership in
`leases.rs`, and stage outcomes/atomic transitions in `stages.rs`.
User-specific submission, demo inventory, and inventory validation live
separately in `buzz-db/src/store/user_deletion/{requests,inventory}.rs`;
its `mod.rs` only defines and exports the store.

The worker calls the user transition object, which supplies a
synchronous decision function to `EntityDeletionStore.transition_stage`.
The store locks the parent, reads saved state, invokes that function,
checks the returned operation, and commits its writes. The decision
function performs no SQL or external calls. Reading, deciding, and
writing remain in one transaction; there is no stale-state gap. Shared
storage contains no user stage names or community-routing rules.
User-specific approval validation also runs inside the shared approval
transaction.

This reorganization changes no tables, migrations, stage metadata, or
CLI behavior. Requests using the existing saved-result format continue
to work. `--next-step` still runs one handler and commits its transition
before returning.

### Testing

The walkthrough in
[docs/operator-user-deletion-executor.md](https://github.com/block/buzz/blob/rarora/user-deletion-approval/docs/operator-user-deletion-executor.md)
includes commands, SQL inspection, expected rows, and recovery
boundaries.

Submitted and approved a fresh local request and verified zero stage
rows. Ran `run --next-step` five times. Each call completed exactly one
handler, released ownership, and left its successor pending: B DB, B
agents, C DB, C agents, then request completed. A sixth call left saved
state unchanged. Verified both removed flags are rejected.

Ran `--demo-no-agents-in 00000000-0000-0000-0000-00000000000c` and
verified four completed rows, with no C agents stage.

Injected a DB-stage failure, verified immediate retry refusal, waited
for the real deadline, and resumed the same stage row. Observed
heartbeat renewal and rejection of a competing worker, killed the owned
worker process, waited for its actual lease expiry, and resumed the same
unfinished stage with a new generation. Re-running a completed request
left its saved state unchanged. Repeated the CLI walkthrough after the
file reorganization, including runtime branching and a retry after the
real deadline; a completed request created before the reorganization
remained unchanged.

Mutation-tested the ownership and recovery guards against local
PostgreSQL 17. Reclaiming with the same executor ID rejects
old-generation operations. Independently disabling the ownership
generation check, release generation check, saved-transition/result
comparison, or heartbeat-failure handling made the corresponding test
fail. Restored all four protections afterward. The heartbeat test takes
over the request while its handler is already running.

### Schema and rollout

Unmerged migrations 0069–0070 add inventory and execution fields, widen
request lifecycle states, and create operator-global approval and
stage-history tables. They set five-second lock and thirty-second
statement timeouts; constraint validation has its own transaction. No
triggers, functions, foreign keys, or data backfills are added.
Desired-state and migration catalog parity cover all three tables. The
transition redesign adds no further schema changes.

The serving relay does not schedule these requests. Only the explicit
operator CLI runs them, using PostgreSQL. Only `demo:` targets are
accepted. Real account discovery, user-write fencing, owned-community
handling, and actual deletion remain future work. Verify staging with a
new demo label and the documented commands.

Older draft requests do not have the new stage metadata contract.
Preserve them and create a new demo request for this version; they are
not a supported resume format. Use a fresh database if an earlier draft
applied a different migration checksum. Never alter sqlx migration
checksums. With Helm `migrate.autoMigrate=true` (the chart default), a
relay binary that lacks 0069/0070 fails at boot with sqlx
`VersionMissing` after those migrations are applied. Recovery for this
rollout is a roll-forward: deploy a corrected binary that retains the
migrations. Do not delete migration records or change checksums to force
an older binary to boot. Rolling back with auto-migration disabled
requires a separately verified schema-compatibility plan. Demo requests,
stage history, and approvals are intentionally retained; this scaffold
has no cleanup command or retention job. Recheck migration numbering
before merge.

Generated with Codex

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>
This commit is contained in:
ravarora2
2026-10-26 10:44:32 -04:00
committed by GitHub
parent 7ce422417b
commit 4250014bae
34 changed files with 3732 additions and 170 deletions
Download Patch File Download Diff File Expand all files Collapse all files

304
docs/operator-user-deletion-executor.md Normal file
View File

@@ -0,0 +1,304 @@
# Run and recover a dummy user-deletion workflow
This workflow tests orchestration using real PostgreSQL rows and dummy actions.
It never gates a real account or deletes user data. Submit still creates an
artificial inventory: B (participant), then C (former participant). Only `demo:`
targets are accepted. Owned communities and production fencing are future work.
## Code map: worker, policy, and persistence
The worker and user-specific routing live in `buzz-deletion`. PostgreSQL
transactions live in `buzz-db`. Both are libraries in the same CLI process;
there is no additional service or database-side workflow function.
| File | Responsibility |
| --- | --- |
| `buzz-deletion/src/user.rs` | Parse command arguments, call the user/shared stores or worker, print results |
| `buzz-deletion/src/user/worker.rs` | Claim, heartbeat, call transition, dispatch handler, repeat, release |
| `buzz-deletion/src/user/handlers.rs` | Dummy actions; save start/result/failure through the shared store |
| `buzz-deletion/src/user/transition.rs` | `UserDeletionTransition.get_next_stage`: recover or choose the next saved position |
| `buzz-deletion/src/user/workflow.rs` | Allowed user stages, result-based branching, community-progress reconstruction |
| `buzz-db/src/store/entity_deletion/mod.rs` | Shared store definition and exports; no stage policy |
| `buzz-db/src/store/entity_deletion/requests.rs` | Read/list/inspect requests and calculate/verify inventory fingerprints |
| `buzz-db/src/store/entity_deletion/approvals.rs` | Persist approval evidence and parent status together |
| `buzz-db/src/store/entity_deletion/leases.rs` | Claim, heartbeat, release, and owner/generation checks |
| `buzz-db/src/store/entity_deletion/stages.rs` | Save handler outcomes and commit stage transitions atomically |
| `buzz-db/src/store/user_deletion/mod.rs` | User-specific store definition and exports |
| `buzz-db/src/store/user_deletion/requests.rs` | Submit user requests and validate user inventory during approval |
| `buzz-db/src/store/user_deletion/inventory.rs` | Demo target checks, B/C inventory generation, and user inventory validation |
The common store does not know the names `gate_user`, `purge_db`, or
`purge_agents`. `UserDeletionTransition` supplies a synchronous decision function
to `EntityDeletionStore.transition_stage`. The store locks the parent and loads
history, invokes that function while still holding the lock, checks the returned
operation against the current lifecycle, applies it, and commits. The decision
function performs no SQL or external work. Returning an error rolls back the
transaction. This avoids a gap between reading state and saving a decision.
The operations are initialize, resume, advance, and already finished. An advance
includes the expected completed stage ID, the decision to retain in history,
and the successor's name, community, and inputs (or no successor for completion).
The store enforces ownership and current-stage preconditions; the user policy
ensures that the chosen destination follows the user workflow rules.
User submission and inventory interpretation are deliberately separate from
common workflow storage, even though they use the same tables. User approval
runs its inventory validator inside the common approval transaction. Future
real PostgreSQL deletion operations belong in the user-specific database folder;
the current handlers still perform no real deletion. Existing community deletion
continues using its existing implementation.
## The execution contract
```text
approve: save approval, create no stages
run: claim parent request
get_next_stage(lease, None) → full stage row
handler(lease, row) → persist status and result
get_next_stage(lease, Some(row.id)) → full next row, Finished, or WaitUntil
repeat; release ownership on return
```
The worker owns claiming, heartbeats, shutdown, dispatch, and a bounded execution
budget. It does not inspect handler results, reconstruct community progress, or
choose branches. `get_next_stage` owns all those decisions. Specific rules live
in `user/workflow.rs`; the database transition transaction contains no
stage-name comparisons or two-community requirement.
The handler gets the whole row. `metadata_json.inputs` holds the target and
community input. The handler marks the row running and either saves success
under `metadata_json.result` or records a failure and retry deadline. Returning
success from the handler means its outcome was saved, including a retryable
failure. Failure to save propagates out of the worker.
### Durable state
| Record | State |
| --- | --- |
| `entity_deletion_requests` | Inventory/digest, overall lifecycle, lease owner/generation/expiry, heartbeat, retry deadline/error/count, completion time |
| `entity_deletion_approvals` | Operator, time, approved inventory digest; retained separately |
| `entity_deletion_stage_history` | Stage/community, current position, execution status, timestamps, inputs, result, transition decision, predecessor ID |
`is_current` means the workflow's durable position. It does not mean a handler
is presently running. In particular, **completed + current is valid**: the
handler finished, but the transition still needs to be committed.
The handler's success transaction sets `completed`, saves its result and finish
time, and leaves `is_current = true`. The next transition transaction checks the
lease, reloads that result, chooses an allowed branch, clears the old current
flag, saves its decision, and inserts the successor with its inputs. If there
is no successor, it records request completion instead. Those transition writes
commit together; an insertion failure leaves the completed old row current.
Every inserted stage stores `metadata_json.previous_stage_id` (null for the
initial stage). Reconstruction follows this chain, not timestamps or UUID sort
order. It checks inputs against the approved inventory, validates saved
transition decisions, and builds the set of completed communities. A persisted
`community_complete` decision covers a community even when a runtime branch
skipped a stage. Unknown stages, disconnected history, and missing results fail
without inventing progress. Reads are bounded to 10,000 reached stages and
inventories to 1,000 communities; these are resource limits, not a demo shape.
### Starting, branching, and finishing
Claiming saves ownership while preserving the lifecycle status. Thus a crash
between claim and initialization leaves an `approved` request with no stages.
The first `get_next_stage` creates the configured initial stage and sets the
parent to `running` in one transaction. A previously started request with no
current row and no terminal outcome is an error, not permission to create a
second gate.
The version-1 workflow rules are:
| Stage | Allowed outcome | Runtime choice |
| --- | --- | --- |
| `gate_user` | Enter communities | Select the first uncovered community |
| `purge_db` | `purge_agents` or community complete | Read saved boolean `result.has_agents` |
| `purge_agents` | Community complete | Mark this community covered |
After a community completes, select the first uncovered community in inventory
order. After all are covered, finish the request. Missing `has_agents` is an
error, not an implicit skip. The inventory's version selects the currently
supported rules; future versions must retain the rules needed by active requests.
No full execution plan is created upfront.
`previous_stage_id` is bookkeeping for uninterrupted execution. It must belong
to this request. After restart pass `None`; the database current row is the
source of truth. An old non-current stage is never transitioned twice.
| Restart point | Recovery |
| --- | --- |
| Before the first transition | Initialize an approved request with empty history |
| During a handler | Resume the unfinished current row after lease takeover |
| After a saved failure | Wait for the deadline, then retry the same row |
| After saved success, before transition | Process the completed current row's saved result; do not rerun its handler |
| After transition commit | Return the saved successor row |
| After terminal commit | A rerun returns success without changing saved state |
A lease lasts 60 seconds and handlers renew it every ten seconds. Handler and
renewal futures are polled independently in the same task, so a renewal waiting
for the handler's parent lock cannot stop the handler from committing. No
detached renewal task survives cancellation. Generation checks reject stale
workers' status, result, transition, heartbeat, and release writes.
This is one request at a time through an explicit CLI command. There is no relay
scheduler or automatic retry daemon. Real handlers will need idempotent or
reconciled external actions: a crash after an external action but before saving
its result may repeat that handler.
## Prepare a local test
Follow [the intake guide](operator-user-deletion-intake.md) to build the binary
and migrate a fresh disposable database. The commands below use its container
`buzz-user-deletion-intake` and database `buzz`. Keep all commands in one shell.
These commands need PostgreSQL only, not a relay, Redis, or storage credentials.
```bash
TARGET="demo:workflow-$(date +%s)"
REQUEST_JSON=$(./target/debug/buzz-admin user-deletions submit \
--target "$TARGET" --requested-by operator-a)
REQUEST_ID=$(printf '%s\n' "$REQUEST_JSON" | jq -r '.id')
INVENTORY_DIGEST=$(printf '%s\n' "$REQUEST_JSON" | jq -r '.inventory_digest')
./target/debug/buzz-admin user-deletions approve "$REQUEST_ID" \
--approved-by operator-b --inventory-digest "$INVENTORY_DIGEST"
```
Expected after approval: parent `approved`, one approval, **zero stage rows**.
Define a helper to inspect the actual tables after each command:
```bash
show_rows() {
docker exec buzz-user-deletion-intake psql -X -U buzz -d buzz -c "
SELECT target, status, lease_owner, lease_generation, lease_expires_at,
heartbeat_at, retry_count, next_attempt_at, last_error, completed_at
FROM entity_deletion_requests WHERE id = '$REQUEST_ID';
SELECT id, stage, target_community_id, status, is_current,
started_at, finished_at, metadata_json, last_error
FROM entity_deletion_stage_history WHERE request_id = '$REQUEST_ID'
ORDER BY started_at NULLS LAST, id;
SELECT approved_by, approved_at, inventory_digest
FROM entity_deletion_approvals WHERE request_id = '$REQUEST_ID'
ORDER BY approved_at, id;"
}
show_rows
```
## Advance one stage at a time
Run the same command each time, then inspect the tables:
```bash
./target/debug/buzz-admin user-deletions run "$REQUEST_ID" --next-step
show_rows
```
Each invocation executes one handler, processes its saved result through
`get_next_stage`, and releases ownership. It leaves the successor pending and
current. The final invocation marks the request completed instead.
Starting with a newly approved request, expect:
| Invocation | Handler completed | Current stage after the command |
| --- | --- | --- |
| 1 | `gate_user` | B `purge_db`, pending |
| 2 | B `purge_db` | B `purge_agents`, pending |
| 3 | B `purge_agents` | C `purge_db`, pending |
| 4 | C `purge_db` | C `purge_agents`, pending |
| 5 | C `purge_agents` | None; request completed |
All earlier rows remain completed and non-current. Every command releases the
worker lease. Re-running a completed request is read-only. If a handler fails,
the same row remains failed and current; the command exits 1 with a saved retry
deadline instead of advancing past the failure.
To finish the remaining workflow in one command, omit `--next-step`:
```bash
./target/debug/buzz-admin user-deletions run "$REQUEST_ID"
show_rows
```
Ordinary runs have an internal budget of 100 handler calls per invocation.
There are no `--max-stages` or `--pause-after-handler` options. A crash can still
leave a completed current row before its transition; the next run processes
that saved result before executing the next unfinished handler.
## Test runtime branching
Submit and approve a new request using the first block. Then:
```bash
./target/debug/buzz-admin user-deletions run "$REQUEST_ID" \
--executor-id worker-a \
--demo-no-agents-in 00000000-0000-0000-0000-00000000000c
show_rows
```
Expected: four completed rows: gate, B DB, B agents, C DB. C's DB result contains
`has_agents: false`; its saved transition is `community_complete`. There is no
C agents row. The flag simulates a runtime observation in the dummy handler; it
does not change the saved inventory or transition map. It may be repeated.
To inspect branching step by step, submit and approve another request. Run once
with `--next-step` to finish the gate, then run with
`--next-step --demo-no-agents-in 00000000-0000-0000-0000-00000000000b`.
B DB completes and transitions directly to C DB. Resume without the flag: the
saved B decision remains unchanged, and no B agents row appears.
## Test failure and retry
On a new approved request:
```bash
./target/debug/buzz-admin user-deletions run "$REQUEST_ID" \
--executor-id worker-a --fail-stage purge_db
echo "Exit code: $?"
show_rows
```
Expected exit 1. Gate has transitioned; B DB is failed and current. The parent
is `retry_wait`, with error, retry count 1, and `next_attempt_at` 30 seconds later.
The handler saved this outcome; `get_next_stage` returned `WaitUntil`, and the
worker released ownership. An immediate retry is refused.
After the saved deadline:
```bash
./target/debug/buzz-admin user-deletions run "$REQUEST_ID" --executor-id worker-b
show_rows
```
The same B DB row resumes with its original first start time. The request
finishes. Successful transition clears the parent retry state.
## Observe heartbeat and process interruption
On a new approved request:
```bash
./target/debug/buzz-admin user-deletions run "$REQUEST_ID" \
--executor-id worker-a --stage-delay-ms 60000
```
Inspect from another terminal using the same database/request. Heartbeat and
expiry advance every ten seconds. A competing worker is refused. Ctrl-C stops
execution and releases ownership; the interrupted row remains current. After
an abrupt process kill, wait for the saved lease expiry before running another
worker. It resumes the same unfinished row with a higher generation.
## Rollout, recovery, and retained demo data
Migrations 0069–0070 are additive. The serving relay does not schedule these
requests; only an operator explicitly running the demo CLI executes them.
The Helm chart defaults `migrate.autoMigrate=true`. After these migrations are
applied, rolling back to a binary that does not contain them fails at startup
with sqlx `VersionMissing`. For this rollout, recover by rolling forward to a
corrected binary that retains migrations 0069–0070. Do not remove migration
records or change their checksums to force an older binary to start. An older
binary deployed with auto-migration disabled requires a separately verified
schema-compatibility plan; it is not the recovery procedure assumed here.
Demo request rows, approval evidence, and stage history are deliberately retained
for inspection. This scaffold has no cleanup command or automatic retention job.