From e56f4d4c3aaf2099effe421ba15e76cc4a83caa4 Mon Sep 17 00:00:00 2001 From: tornquist Date: Fri, 25 Sep 2026 18:03:39 +0000 Subject: [PATCH] Add typed deletion drain operator CronJob Co-authored-by: Codex Signed-off-by: tornquist --- ARCHITECTURE.md | 9 + deploy/charts/buzz/Chart.yaml | 4 +- deploy/charts/buzz/README.md | 22 +- .../charts/buzz/templates/_operator-jobs.tpl | 86 +++++++ deploy/charts/buzz/templates/_validate.tpl | 7 + .../templates/deletion-drain-cronjob.yaml | 4 + .../buzz/tests/deletion_drain_test.yaml | 214 ++++++++++++++++++ deploy/charts/buzz/values.schema.json | 28 +++ deploy/charts/buzz/values.yaml | 23 ++ docs/operator-community-deletion.md | 87 +++++++ 10 files changed, 481 insertions(+), 3 deletions(-) create mode 100644 deploy/charts/buzz/templates/_operator-jobs.tpl create mode 100644 deploy/charts/buzz/templates/deletion-drain-cronjob.yaml create mode 100644 deploy/charts/buzz/tests/deletion_drain_test.yaml create mode 100644 docs/operator-community-deletion.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 8aecbd61cc..d30b9c9a2f 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -750,10 +750,19 @@ Subcommands: | `remove-member` | Remove a pubkey from the relay membership list (`--pubkey`, optional `--role` guard); publishes kind:13534 roster | | `list-members` | List all relay members | | `generate-key` | Generate a new Nostr keypair (for bootstrapping) | +| `deletions` | Submit, inspect, approve, abort, unblock, run, or drain durable whole-community deletion requests | +| `storage-snapshot` | Run one isolated S3 accounting scan and publish its complete Postgres snapshot | | `reconcile-channels` | Emit kind:39000/39002 discovery events for channels missing them (idempotent) | The `buzz-admin` binary is shipped in the relay Docker image (`/usr/local/bin/buzz-admin`) and is the recommended way to manage relay membership in production. Use `./run.sh add-member`, `./run.sh remove-member`, and `./run.sh list-members` in Docker Compose deployments. +Kubernetes deployments may schedule the typed one-shot +`buzz-admin deletions drain` command directly. The pod owns its bounded +Postgres/Redis clients and S3 client; it does not call relay HTTP. Durable +requests, leases, retry timing, and checkpoints in Postgres are the handoff and +execution authority, so Kubernetes uses `Forbid` concurrency and zero Job +retries rather than introducing a second retry system. + --- ### buzz-test-client — Integration Test Harness diff --git a/deploy/charts/buzz/Chart.yaml b/deploy/charts/buzz/Chart.yaml index e26a2815ff..55d9db5488 100644 --- a/deploy/charts/buzz/Chart.yaml +++ b/deploy/charts/buzz/Chart.yaml @@ -7,7 +7,7 @@ description: | PostgreSQL and Redis. Configurable for single-node evaluation (subcharts on) and HA production (external services, existingSecret). type: application -version: 0.1.9 +version: 0.1.10 appVersion: "0.1.0" home: https://github.com/block/buzz sources: @@ -24,7 +24,7 @@ maintainers: annotations: artifacthub.io/changes: | - kind: added - description: Optional isolated storage-accounting CronJob with durable relay snapshots. + description: Optional typed deletion-drain operator CronJob using durable database leases. artifacthub.io/license: Apache-2.0 # Optional eval-only subcharts. Production deploys disable both and point diff --git a/deploy/charts/buzz/README.md b/deploy/charts/buzz/README.md index f5075fd19d..1bdff7592e 100644 --- a/deploy/charts/buzz/README.md +++ b/deploy/charts/buzz/README.md @@ -12,7 +12,7 @@ This chart has two operating profiles selected by values: ## Quickstart (eval only) ```sh -helm install buzz oci://ghcr.io/block/buzz/charts/buzz --version 0.1.8 \ +helm install buzz oci://ghcr.io/block/buzz/charts/buzz --version 0.1.10 \ --create-namespace --namespace buzz \ --set quickstart=true \ --set postgresql.enabled=true \ @@ -116,6 +116,26 @@ is still parsed strictly, but reachability and addressing errors surface on the first storage operation. `/_readiness` tests no external dependency in either case — see the readiness contract below. +## Community deletion operator job + +`operatorJobs.deletionDrain` is a disabled-by-default, typed CronJob for +`/usr/local/bin/buzz-admin deletions drain`. It runs inside the relay image with +bounded Job lifetime/history, `concurrencyPolicy: Forbid`, `backoffLimit: 0`, +and no relay HTTP call. Postgres deletion requests, leases, retries, and +checkpoints remain the execution authority. + +The pod receives only `DATABASE_URL`, `REDIS_URL`, and required S3 +configuration/credential variables. It does not receive the relay private key, +git-hook secret, relay URL, service-account token, service links, or a generic +environment registry. Schedule, +deadline, history, termination grace, resources, service account, pod labels, +and pod annotations are independently configurable under +`operatorJobs.deletionDrain`. + +See [`docs/operator-community-deletion.md`](../../../docs/operator-community-deletion.md) +for enablement, permissions, the staffed first-run procedure, failure recovery, +and the current explicit approval/alerting boundaries. + ### Early-startup telemetry contract `buzz_process_lifecycle` JSON records are the authoritative history for the diff --git a/deploy/charts/buzz/templates/_operator-jobs.tpl b/deploy/charts/buzz/templates/_operator-jobs.tpl new file mode 100644 index 0000000000..aec91d52ff --- /dev/null +++ b/deploy/charts/buzz/templates/_operator-jobs.tpl @@ -0,0 +1,86 @@ +{{/* Closed rendering foundation for typed Buzz operator jobs. */}} + +{{- define "buzz.operatorCronJob" -}} +{{- $root := .root -}} +{{- if eq .type "deletionDrain" -}} +{{- $job := $root.Values.operatorJobs.deletionDrain -}} +apiVersion: batch/v1 +kind: CronJob +metadata: + name: {{ include "buzz.fullname" $root }}-deletion-drain + labels: + {{- include "buzz.labels" $root | nindent 4 }} + app.kubernetes.io/component: deletion-drain +spec: + schedule: {{ $job.schedule | quote }} + concurrencyPolicy: Forbid + successfulJobsHistoryLimit: {{ $job.successfulJobsHistoryLimit }} + failedJobsHistoryLimit: {{ $job.failedJobsHistoryLimit }} + jobTemplate: + spec: + activeDeadlineSeconds: {{ $job.activeDeadlineSeconds }} + backoffLimit: 0 + template: + metadata: + labels: + {{- include "buzz.selectorLabels" $root | nindent 12 }} + app.kubernetes.io/component: deletion-drain + {{- with $job.podLabels }} + {{- toYaml . | nindent 12 }} + {{- end }} + annotations: + {{- toYaml $job.podAnnotations | nindent 12 }} + spec: + restartPolicy: Never + terminationGracePeriodSeconds: {{ $job.terminationGracePeriodSeconds }} + serviceAccountName: {{ default (include "buzz.serviceAccountName" $root) $job.serviceAccountName }} + automountServiceAccountToken: false + enableServiceLinks: false + securityContext: + {{- toYaml $root.Values.relay.securityContext | nindent 12 }} + {{- with $root.Values.image.pullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 12 }} + {{- end }} + containers: + - name: deletion-drain + image: {{ include "buzz.image" $root }} + imagePullPolicy: {{ $root.Values.image.pullPolicy }} + securityContext: + {{- omit $root.Values.relay.containerSecurityContext "readOnlyRootFilesystem" | toYaml | nindent 16 }} + readOnlyRootFilesystem: true + command: ["/usr/local/bin/buzz-admin"] + args: ["deletions", "drain"] + env: + - { name: BUZZ_S3_ENDPOINT, value: {{ required "s3.endpoint is required when operatorJobs.deletionDrain.enabled=true" (include "buzz.s3Endpoint" $root) | quote }} } + - { name: BUZZ_S3_BUCKET, value: {{ required "s3.bucket is required when operatorJobs.deletionDrain.enabled=true" $root.Values.s3.bucket | quote }} } + - { name: BUZZ_S3_REGION, value: {{ $root.Values.s3.region | quote }} } + - { name: BUZZ_S3_ADDRESSING_STYLE, value: {{ $root.Values.s3.addressingStyle | quote }} } + - name: DATABASE_URL + valueFrom: + secretKeyRef: + name: {{ include "buzz.envSecretName" $root }} + key: DATABASE_URL + - name: REDIS_URL + valueFrom: + secretKeyRef: + name: {{ include "buzz.envSecretName" $root }} + key: REDIS_URL + - name: BUZZ_S3_ACCESS_KEY + valueFrom: + secretKeyRef: + name: {{ include "buzz.envSecretName" $root }} + key: BUZZ_S3_ACCESS_KEY + optional: true + - name: BUZZ_S3_SECRET_KEY + valueFrom: + secretKeyRef: + name: {{ include "buzz.envSecretName" $root }} + key: BUZZ_S3_SECRET_KEY + optional: true + resources: + {{- toYaml $job.resources | nindent 16 }} +{{- else -}} +{{- fail (printf "unsupported typed operator job %q" .type) -}} +{{- end -}} +{{- end -}} diff --git a/deploy/charts/buzz/templates/_validate.tpl b/deploy/charts/buzz/templates/_validate.tpl index aa7f7ac13c..8d055f825e 100644 --- a/deploy/charts/buzz/templates/_validate.tpl +++ b/deploy/charts/buzz/templates/_validate.tpl @@ -18,6 +18,13 @@ surface at template time regardless of which manifest helm renders first. {{- end -}} {{- end -}} +{{/* The deletion executor always uses Redis for tenant-scoped invalidation. */}} +{{- if .Values.operatorJobs.deletionDrain.enabled -}} + {{- if and (not .Values.redis.enabled) (not .Values.externalRedis.url) (not .Values.secrets.existingSecret) -}} + {{- fail "operatorJobs.deletionDrain requires Redis. Enable redis.enabled=true, set externalRedis.url, or provide secrets.existingSecret with key REDIS_URL." -}} + {{- end -}} +{{- end -}} + {{/* Multiple replicas do NOT require ReadWriteMany git storage. Git ref/object state is object-store-backed: every read and write hydrates diff --git a/deploy/charts/buzz/templates/deletion-drain-cronjob.yaml b/deploy/charts/buzz/templates/deletion-drain-cronjob.yaml new file mode 100644 index 0000000000..84acc56590 --- /dev/null +++ b/deploy/charts/buzz/templates/deletion-drain-cronjob.yaml @@ -0,0 +1,4 @@ +{{- include "buzz.validate" . -}} +{{- if .Values.operatorJobs.deletionDrain.enabled }} +{{- include "buzz.operatorCronJob" (dict "root" . "type" "deletionDrain") }} +{{- end }} diff --git a/deploy/charts/buzz/tests/deletion_drain_test.yaml b/deploy/charts/buzz/tests/deletion_drain_test.yaml new file mode 100644 index 0000000000..18e0d1a9ef --- /dev/null +++ b/deploy/charts/buzz/tests/deletion_drain_test.yaml @@ -0,0 +1,214 @@ +suite: typed deletion drain operator job +templates: + - templates/deletion-drain-cronjob.yaml +tests: + - it: renders no deletion executor by default + set: + relayUrl: wss://buzz.example.com + ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000" + externalPostgresql.url: postgres://u:p@h:5432/d + externalRedis.url: redis://h:6379 + s3.endpoint: https://s3.example.com + s3.accessKey: test + s3.secretKey: test + asserts: + - hasDocuments: + count: 0 + + - it: renders the bounded typed drain command with isolated pod controls + set: + relayUrl: wss://buzz.example.com + ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000" + externalPostgresql.url: postgres://u:p@h:5432/d + externalRedis.url: redis://h:6379 + s3.endpoint: https://s3.example.com + s3.bucket: buzz-media-example + s3.region: us-west-2 + s3.addressingStyle: virtual + s3.accessKey: test + s3.secretKey: test + operatorJobs.deletionDrain: + enabled: true + schedule: "*/7 * * * *" + activeDeadlineSeconds: 900 + successfulJobsHistoryLimit: 2 + failedJobsHistoryLimit: 4 + terminationGracePeriodSeconds: 45 + serviceAccountName: buzz-deletion-drain + podLabels: + tags.datadoghq.com/service: buzz-deletion-drain + podAnnotations: + sidecar.istio.io/inject: "false" + example.com/operator-job: deletion-drain + resources: + requests: + cpu: 250m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi + asserts: + - hasDocuments: + count: 1 + - isAPIVersion: + of: batch/v1 + - isKind: + of: CronJob + - equal: + path: metadata.name + value: RELEASE-NAME-buzz-deletion-drain + - equal: + path: metadata.labels["app.kubernetes.io/component"] + value: deletion-drain + - equal: + path: spec.schedule + value: "*/7 * * * *" + - equal: + path: spec.concurrencyPolicy + value: Forbid + - equal: + path: spec.successfulJobsHistoryLimit + value: 2 + - equal: + path: spec.failedJobsHistoryLimit + value: 4 + - equal: + path: spec.jobTemplate.spec.activeDeadlineSeconds + value: 900 + - equal: + path: spec.jobTemplate.spec.backoffLimit + value: 0 + - equal: + path: spec.jobTemplate.spec.template.spec.restartPolicy + value: Never + - equal: + path: spec.jobTemplate.spec.template.spec.terminationGracePeriodSeconds + value: 45 + - equal: + path: spec.jobTemplate.spec.template.spec.serviceAccountName + value: buzz-deletion-drain + - equal: + path: spec.jobTemplate.spec.template.spec.automountServiceAccountToken + value: false + - equal: + path: spec.jobTemplate.spec.template.spec.enableServiceLinks + value: false + - equal: + path: spec.jobTemplate.spec.template.metadata.labels["tags.datadoghq.com/service"] + value: buzz-deletion-drain + - equal: + path: spec.jobTemplate.spec.template.metadata.annotations["example.com/operator-job"] + value: deletion-drain + - equal: + path: spec.jobTemplate.spec.template.spec.containers[0].securityContext.readOnlyRootFilesystem + value: true + - equal: + path: spec.jobTemplate.spec.template.spec.containers[0].command + value: ["/usr/local/bin/buzz-admin"] + - equal: + path: spec.jobTemplate.spec.template.spec.containers[0].args + value: ["deletions", "drain"] + - equal: + path: spec.jobTemplate.spec.template.spec.containers[0].resources.requests.cpu + value: 250m + - equal: + path: spec.jobTemplate.spec.template.spec.containers[0].resources.limits.memory + value: 1Gi + + - it: exposes only database redis and deletion S3 configuration + set: + relayUrl: wss://buzz.example.com + ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000" + externalPostgresql.url: postgres://u:p@h:5432/d + externalRedis.url: redis://h:6379 + secrets.existingSecret: buzz-operator-secrets + s3.endpoint: https://s3.example.com + s3.bucket: buzz-media-example + s3.region: us-west-2 + s3.addressingStyle: virtual + operatorJobs.deletionDrain.enabled: true + asserts: + - lengthEqual: + path: spec.jobTemplate.spec.template.spec.containers[0].env + count: 8 + - notExists: + path: spec.jobTemplate.spec.template.spec.containers[0].envFrom + - contains: + path: spec.jobTemplate.spec.template.spec.containers[0].env + content: + name: DATABASE_URL + valueFrom: + secretKeyRef: + name: buzz-operator-secrets + key: DATABASE_URL + - contains: + path: spec.jobTemplate.spec.template.spec.containers[0].env + content: + name: REDIS_URL + valueFrom: + secretKeyRef: + name: buzz-operator-secrets + key: REDIS_URL + - contains: + path: spec.jobTemplate.spec.template.spec.containers[0].env + content: + name: BUZZ_S3_ENDPOINT + value: https://s3.example.com + - contains: + path: spec.jobTemplate.spec.template.spec.containers[0].env + content: + name: BUZZ_S3_BUCKET + value: buzz-media-example + - contains: + path: spec.jobTemplate.spec.template.spec.containers[0].env + content: + name: BUZZ_S3_REGION + value: us-west-2 + - contains: + path: spec.jobTemplate.spec.template.spec.containers[0].env + content: + name: BUZZ_S3_ADDRESSING_STYLE + value: virtual + - contains: + path: spec.jobTemplate.spec.template.spec.containers[0].env + content: + name: BUZZ_S3_ACCESS_KEY + valueFrom: + secretKeyRef: + name: buzz-operator-secrets + key: BUZZ_S3_ACCESS_KEY + optional: true + - contains: + path: spec.jobTemplate.spec.template.spec.containers[0].env + content: + name: BUZZ_S3_SECRET_KEY + valueFrom: + secretKeyRef: + name: buzz-operator-secrets + key: BUZZ_S3_SECRET_KEY + optional: true + - notContains: + path: spec.jobTemplate.spec.template.spec.containers[0].env + content: + name: BUZZ_RELAY_PRIVATE_KEY + - notContains: + path: spec.jobTemplate.spec.template.spec.containers[0].env + content: + name: BUZZ_GIT_HOOK_HMAC_SECRET + - notContains: + path: spec.jobTemplate.spec.template.spec.containers[0].env + content: + name: RELAY_URL + + - it: rejects enablement without a Redis source + set: + relayUrl: wss://buzz.example.com + ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000" + externalPostgresql.url: postgres://u:p@h:5432/d + s3.endpoint: https://s3.example.com + s3.accessKey: test + s3.secretKey: test + operatorJobs.deletionDrain.enabled: true + asserts: + - failedTemplate: + errorPattern: "operatorJobs.deletionDrain requires Redis" diff --git a/deploy/charts/buzz/values.schema.json b/deploy/charts/buzz/values.schema.json index ea5db6398d..309b2f5c37 100644 --- a/deploy/charts/buzz/values.schema.json +++ b/deploy/charts/buzz/values.schema.json @@ -234,6 +234,34 @@ "resources": { "type": "object" } } }, + "operatorJobs": { + "type": "object", + "additionalProperties": false, + "properties": { + "deletionDrain": { + "type": "object", + "additionalProperties": false, + "properties": { + "enabled": { "type": "boolean" }, + "schedule": { "type": "string", "minLength": 1 }, + "activeDeadlineSeconds": { "type": "integer", "minimum": 1, "maximum": 86400 }, + "successfulJobsHistoryLimit": { "type": "integer", "minimum": 0, "maximum": 10 }, + "failedJobsHistoryLimit": { "type": "integer", "minimum": 0, "maximum": 10 }, + "terminationGracePeriodSeconds": { "type": "integer", "minimum": 1, "maximum": 300 }, + "serviceAccountName": { "type": "string" }, + "podLabels": { + "type": "object", + "additionalProperties": { "type": "string" } + }, + "podAnnotations": { + "type": "object", + "additionalProperties": { "type": "string" } + }, + "resources": { "type": "object" } + } + } + } + }, "minio": { "type": "object", "additionalProperties": false, diff --git a/deploy/charts/buzz/values.yaml b/deploy/charts/buzz/values.yaml index 496c32e979..f43925346e 100644 --- a/deploy/charts/buzz/values.yaml +++ b/deploy/charts/buzz/values.yaml @@ -391,6 +391,29 @@ storageAccounting: cpu: "1" memory: 20Gi +# Typed, one-shot operator jobs. The chart intentionally exposes no generic +# command or environment registry: each job has a reviewed executable and +# least-privilege environment contract. +operatorJobs: + deletionDrain: + enabled: false + schedule: "*/5 * * * *" + activeDeadlineSeconds: 3600 + successfulJobsHistoryLimit: 1 + failedJobsHistoryLimit: 3 + terminationGracePeriodSeconds: 30 + serviceAccountName: "" # defaults to the main Buzz service account + podLabels: {} + podAnnotations: + sidecar.istio.io/inject: "false" + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi + # In-cluster MinIO for the quickstart profile only. Production deploys leave # this disabled and use s3.* (or secrets.existingSecret) against managed S3. minio: diff --git a/docs/operator-community-deletion.md b/docs/operator-community-deletion.md new file mode 100644 index 0000000000..ec2c20f000 --- /dev/null +++ b/docs/operator-community-deletion.md @@ -0,0 +1,87 @@ +# Community Deletion Operator Job + +Buzz executes whole-community deletion through the typed, one-shot +`/usr/local/bin/buzz-admin deletions drain` command. The Helm chart can schedule +that command as a Kubernetes CronJob; it does not call relay HTTP and it does +not add another queue or retry service. + +Postgres remains the handoff and source of truth. A run claims only requests +that the deletion store considers runnable, heartbeats the existing lease, and +resumes from durable checkpoints. `concurrencyPolicy: Forbid` prevents scheduled +pod overlap, `backoffLimit: 0` prevents Kubernetes Job retries, and the deletion +store remains authoritative when a pod exits, reaches its deadline, or is +replaced. + +## Enablement + +The CronJob is disabled by default. Production deployments should use an +existing Secret and a dedicated service account when their cluster policy +supports one: + +```yaml +secrets: + existingSecret: buzz-operator-secrets + +operatorJobs: + deletionDrain: + enabled: true + schedule: "*/5 * * * *" + activeDeadlineSeconds: 3600 + terminationGracePeriodSeconds: 30 + serviceAccountName: buzz-deletion-drain + podLabels: + tags.datadoghq.com/service: buzz-deletion-drain + podAnnotations: + sidecar.istio.io/inject: "false" + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi +``` + +Set `s3.endpoint`, `s3.bucket`, `s3.region`, and `s3.addressingStyle` in chart +values. The selected Secret must contain `DATABASE_URL` and `REDIS_URL`; it may +contain `BUZZ_S3_ACCESS_KEY` and `BUZZ_S3_SECRET_KEY` when the object store uses +static credentials. The pod receives only those connection values and the four +non-secret S3 settings. It does not receive `BUZZ_RELAY_PRIVATE_KEY`, +`BUZZ_GIT_HOOK_HMAC_SECRET`, `RELAY_URL`, or the full Secret through `envFrom`. +The pod also disables service-account token automounting and Kubernetes service +link environment injection because the executor does not call the Kubernetes +API or discover cluster Services. + +The S3 principal needs the relay's normal object permissions plus bucket-level +`s3:ListBucketVersions` and object-level `s3:DeleteObjectVersion` for every +tenant-owned prefix. This also applies to never-versioned buckets because S3 +reports their objects with the `null` version id. + +## Runbook + +1. Confirm database migrations are current and the deletion request has crossed + the explicit inventory and approval boundary with + `buzz-admin deletions inspect `. +2. Confirm the selected Secret contains the required keys and the S3 principal + has version-list and exact-version delete permissions. +3. Enable the CronJob and inspect its rendered command and environment before + rollout. +4. Start one staffed manual run with + `kubectl create job --from=cronjob/-buzz-deletion-drain `. +5. Follow pod logs and re-run `buzz-admin deletions inspect ` to + verify lease, checkpoint, retry, blocked, and terminal state. +6. If a run fails or times out, fix the recorded dependency or permission + failure. Do not add Kubernetes retries: the next scheduled drain consults the + durable retry/checkpoint state and resumes only when the store allows it. + +The current owner self-serve relay admission records an owner-origin request at +`submitted` and intentionally performs no inventory or approval synchronously. +The deletion engine rejects `submitted` and `inventoried` requests at its +explicit approval boundary. Automating the privileged inventory/approval step +is therefore a separate control-plane slice; enabling this CronJob alone does +not make a newly accepted owner request destructive. + +The chart has no existing PrometheusRule or provider-neutral CronJob alert +integration. Operators must alert on failed/missed Jobs and long-running active +Jobs in their deployment platform. Adding a chart-native alert abstraction is +debt, not part of this job contract.