diff --git a/deploy/charts/buzz/values.yaml b/deploy/charts/buzz/values.yaml index f43925346e..d51d37b4bb 100644 --- a/deploy/charts/buzz/values.yaml +++ b/deploy/charts/buzz/values.yaml @@ -402,7 +402,9 @@ operatorJobs: successfulJobsHistoryLimit: 1 failedJobsHistoryLimit: 3 terminationGracePeriodSeconds: 30 - serviceAccountName: "" # defaults to the main Buzz service account + # Empty inherits the relay service account, and with it any cloud IAM + # binding on that account. Name a dedicated account to narrow the executor. + serviceAccountName: "" podLabels: {} podAnnotations: sidecar.istio.io/inject: "false" diff --git a/docs/operator-community-deletion.md b/docs/operator-community-deletion.md index ec2c20f000..d7fd69b3a2 100644 --- a/docs/operator-community-deletion.md +++ b/docs/operator-community-deletion.md @@ -15,8 +15,7 @@ replaced. ## Enablement The CronJob is disabled by default. Production deployments should use an -existing Secret and a dedicated service account when their cluster policy -supports one: +existing Secret and a dedicated, pre-created service account: ```yaml secrets: @@ -52,6 +51,16 @@ The pod also disables service-account token automounting and Kubernetes service link environment injection because the executor does not call the Kubernetes API or discover cluster Services. +Leaving `operatorJobs.deletionDrain.serviceAccountName` empty falls back to the +relay's own service account. `automountServiceAccountToken: false` only +suppresses the projected token inside the pod; it does not detach the identity. +Cloud IAM bindings attached to that service account — IRSA on EKS, Workload +Identity on GKE — are resolved by the node/metadata path and still apply, so the +drain pod inherits the relay's cloud permissions. Create a dedicated service +account with only the object-store permissions listed below and name it +explicitly if you want the executor's IAM blast radius to be smaller than the +relay's. + The S3 principal needs the relay's normal object permissions plus bucket-level `s3:ListBucketVersions` and object-level `s3:DeleteObjectVersion` for every tenant-owned prefix. This also applies to never-versioned buckets because S3 @@ -66,14 +75,60 @@ reports their objects with the `null` version id. has version-list and exact-version delete permissions. 3. Enable the CronJob and inspect its rendered command and environment before rollout. -4. Start one staffed manual run with - `kubectl create job --from=cronjob/-buzz-deletion-drain `. -5. Follow pod logs and re-run `buzz-admin deletions inspect ` to +4. Locate the rendered CronJob by label rather than by guessing its name: + + ```sh + kubectl get cronjob -n \ + -l app.kubernetes.io/component=deletion-drain,app.kubernetes.io/instance= + ``` + + The name is `-deletion-drain`. `buzz.fullname` collapses to the + release name when the release name already contains the chart name, so + `helm install buzz ...` renders `buzz-deletion-drain`, not + `buzz-buzz-deletion-drain`. +5. Start one staffed manual run with + `kubectl create job --from=cronjob/ `. +6. Follow pod logs and re-run `buzz-admin deletions inspect ` to verify lease, checkpoint, retry, blocked, and terminal state. -6. If a run fails or times out, fix the recorded dependency or permission +7. If a run fails or times out, fix the recorded dependency or permission failure. Do not add Kubernetes retries: the next scheduled drain consults the durable retry/checkpoint state and resumes only when the store allows it. +## Deadlines, termination, and the retry budget + +`activeDeadlineSeconds` is a Kubernetes-side limit, and the deletion store does +not learn why a pod went away. Two consequences matter when reading state: + +- A `DeadlineExceeded` Job is not recorded as a deletion retry. Shutdown + releases the claim without recording one, so `retry_count` and `blocked_at` + do not advance. Only the object-store drain checkpoints per manifest chunk + and resumes mid-stage; every other stage restarts from its beginning on the + next run. A deadline that keeps landing inside one of those non-resumable + stages therefore repeats indefinitely: each run increments `attempts` and + burns the window again while the retry budget never moves and the request is + never blocked. +- Diagnose this from both sides. `buzz-admin deletions inspect ` + shows a rising `attempts` with a flat `retry_count` and no `last_error`; + Kubernetes holds the reason. The chart labels the CronJob and the drain pods, + but not the generated Jobs, so find the attempts with + `kubectl get pods -n -l app.kubernetes.io/component=deletion-drain` + and read the condition with `kubectl describe job `. + +Size `activeDeadlineSeconds` for the longest single stage this community will +run, not for the average run. To recover, either raise the deadline and let the +schedule pick the request back up, or take one staffed run with +`buzz-admin deletions run ` outside the CronJob's deadline. + +`terminationGracePeriodSeconds` is a best-effort window, not a guarantee. The +drain command handles `SIGTERM` and releases its lease cleanly when it wins the +race, but a pod that is still working when the grace period expires is +`SIGKILL`ed with the lease still held. Nothing is lost: the durable lease simply +expires (60s by default, heartbeated every 10s) and the next run reclaims the +request with a fresh lease generation, which fences any straggler write from the +killed process. Expect up to roughly a lease duration of delay before the +request is runnable again; do not raise the grace period expecting a clean +handoff. + The current owner self-serve relay admission records an owner-origin request at `submitted` and intentionally performs no inventory or approval synchronously. The deletion engine rejects `submitted` and `inventoried` requests at its