Correct the deletion drain runbook's operational claims

The runbook told operators to target `cronjob/<release>-buzz-deletion-drain`.
`buzz.fullname` collapses to the release name when it already contains the
chart name, so the documented name is wrong for the documented install:
`helm install buzz ...` renders `buzz-deletion-drain`. Discover the CronJob by
its component label instead, and describe the name rule rather than a single
guessed spelling.

Three more corrections:

`activeDeadlineSeconds` was presented as if a timed-out run were just another
retry. It is not recorded as one — shutdown releases the claim without
recording a retry, and only the object-store drain resumes mid-stage — so a
deadline landing repeatedly inside a non-resumable stage loops forever with a
rising `attempts`, a flat `retry_count`, and no block. Document how to spot
that from both the request and Kubernetes, how to size the deadline, and how
to recover.

`terminationGracePeriodSeconds` was presented as a clean handoff. Document
that a pod still working at the end of the window is SIGKILLed holding its
lease, and that recovery is lease expiry plus reclaim under a new generation.

An empty `serviceAccountName` was described as a neutral default. It inherits
the relay's service account; `automountServiceAccountToken: false` hides the
projected token but does not detach cloud IAM bindings resolved through the
node metadata path. Recommend a dedicated pre-created account when the
executor's IAM blast radius should be smaller than the relay's.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: tornquist <tornquist@squareup.com>

deploy/charts/buzz/values.yaml

@@ -402,7 +402,9 @@ operatorJobs:
     successfulJobsHistoryLimit: 1
     failedJobsHistoryLimit: 3
     terminationGracePeriodSeconds: 30
-    serviceAccountName: ""       # defaults to the main Buzz service account
+    # Empty inherits the relay service account, and with it any cloud IAM
+    # binding on that account. Name a dedicated account to narrow the executor.
+    serviceAccountName: ""
     podLabels: {}
     podAnnotations:
       sidecar.istio.io/inject: "false"

docs/operator-community-deletion.md

@@ -15,8 +15,7 @@ replaced.
 ## Enablement
 
 The CronJob is disabled by default. Production deployments should use an
-existing Secret and a dedicated service account when their cluster policy
-supports one:
+existing Secret and a dedicated, pre-created service account:
 
 ```yaml
 secrets:
@@ -52,6 +51,16 @@ The pod also disables service-account token automounting and Kubernetes service
 link environment injection because the executor does not call the Kubernetes
 API or discover cluster Services.
 
+Leaving `operatorJobs.deletionDrain.serviceAccountName` empty falls back to the
+relay's own service account. `automountServiceAccountToken: false` only
+suppresses the projected token inside the pod; it does not detach the identity.
+Cloud IAM bindings attached to that service account — IRSA on EKS, Workload
+Identity on GKE — are resolved by the node/metadata path and still apply, so the
+drain pod inherits the relay's cloud permissions. Create a dedicated service
+account with only the object-store permissions listed below and name it
+explicitly if you want the executor's IAM blast radius to be smaller than the
+relay's.
+
 The S3 principal needs the relay's normal object permissions plus bucket-level
 `s3:ListBucketVersions` and object-level `s3:DeleteObjectVersion` for every
 tenant-owned prefix. This also applies to never-versioned buckets because S3
@@ -66,14 +75,60 @@ reports their objects with the `null` version id.
    has version-list and exact-version delete permissions.
 3. Enable the CronJob and inspect its rendered command and environment before
    rollout.
-4. Start one staffed manual run with
-   `kubectl create job --from=cronjob/<release>-buzz-deletion-drain <job-name>`.
-5. Follow pod logs and re-run `buzz-admin deletions inspect <request-id>` to
+4. Locate the rendered CronJob by label rather than by guessing its name:
+
+   ```sh
+   kubectl get cronjob -n <namespace> \
+     -l app.kubernetes.io/component=deletion-drain,app.kubernetes.io/instance=<release>
+   ```
+
+   The name is `<fullname>-deletion-drain`. `buzz.fullname` collapses to the
+   release name when the release name already contains the chart name, so
+   `helm install buzz ...` renders `buzz-deletion-drain`, not
+   `buzz-buzz-deletion-drain`.
+5. Start one staffed manual run with
+   `kubectl create job --from=cronjob/<cronjob-name> <job-name>`.
+6. Follow pod logs and re-run `buzz-admin deletions inspect <request-id>` to
    verify lease, checkpoint, retry, blocked, and terminal state.
-6. If a run fails or times out, fix the recorded dependency or permission
+7. If a run fails or times out, fix the recorded dependency or permission
    failure. Do not add Kubernetes retries: the next scheduled drain consults the
    durable retry/checkpoint state and resumes only when the store allows it.
 
+## Deadlines, termination, and the retry budget
+
+`activeDeadlineSeconds` is a Kubernetes-side limit, and the deletion store does
+not learn why a pod went away. Two consequences matter when reading state:
+
+- A `DeadlineExceeded` Job is not recorded as a deletion retry. Shutdown
+  releases the claim without recording one, so `retry_count` and `blocked_at`
+  do not advance. Only the object-store drain checkpoints per manifest chunk
+  and resumes mid-stage; every other stage restarts from its beginning on the
+  next run. A deadline that keeps landing inside one of those non-resumable
+  stages therefore repeats indefinitely: each run increments `attempts` and
+  burns the window again while the retry budget never moves and the request is
+  never blocked.
+- Diagnose this from both sides. `buzz-admin deletions inspect <request-id>`
+  shows a rising `attempts` with a flat `retry_count` and no `last_error`;
+  Kubernetes holds the reason. The chart labels the CronJob and the drain pods,
+  but not the generated Jobs, so find the attempts with
+  `kubectl get pods -n <namespace> -l app.kubernetes.io/component=deletion-drain`
+  and read the condition with `kubectl describe job <job-name>`.
+
+Size `activeDeadlineSeconds` for the longest single stage this community will
+run, not for the average run. To recover, either raise the deadline and let the
+schedule pick the request back up, or take one staffed run with
+`buzz-admin deletions run <request-id>` outside the CronJob's deadline.
+
+`terminationGracePeriodSeconds` is a best-effort window, not a guarantee. The
+drain command handles `SIGTERM` and releases its lease cleanly when it wins the
+race, but a pod that is still working when the grace period expires is
+`SIGKILL`ed with the lease still held. Nothing is lost: the durable lease simply
+expires (60s by default, heartbeated every 10s) and the next run reclaims the
+request with a fresh lease generation, which fences any straggler write from the
+killed process. Expect up to roughly a lease duration of delay before the
+request is runnable again; do not raise the grace period expecting a clean
+handoff.
+
 The current owner self-serve relay admission records an owner-origin request at
 `submitted` and intentionally performs no inventory or approval synchronously.
 The deletion engine rejects `submitted` and `inventoried` requests at its