jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-10-28 21:11:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add typed deletion drain operator CronJob

Browse Source 
Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: tornquist <tornquist@squareup.com>
This commit is contained in:
tornquist
2026-09-25 18:03:39 +00:00
committed by Luke Tornquist
parent 15d44dccee
commit e56f4d4c3a
10 changed files with 481 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
ARCHITECTURE.md
View File

@@ -750,10 +750,19 @@ Subcommands:
| `remove-member` | Remove a pubkey from the relay membership list (`--pubkey`, optional `--role` guard); publishes kind:13534 roster |
| `list-members` | List all relay members |
| `generate-key` | Generate a new Nostr keypair (for bootstrapping) |
| `deletions` | Submit, inspect, approve, abort, unblock, run, or drain durable whole-community deletion requests |
| `storage-snapshot` | Run one isolated S3 accounting scan and publish its complete Postgres snapshot |
| `reconcile-channels` | Emit kind:39000/39002 discovery events for channels missing them (idempotent) |
The `buzz-admin` binary is shipped in the relay Docker image (`/usr/local/bin/buzz-admin`) and is the recommended way to manage relay membership in production. Use `./run.sh add-member`, `./run.sh remove-member`, and `./run.sh list-members` in Docker Compose deployments.
Kubernetes deployments may schedule the typed one-shot
`buzz-admin deletions drain` command directly. The pod owns its bounded
Postgres/Redis clients and S3 client; it does not call relay HTTP. Durable
requests, leases, retry timing, and checkpoints in Postgres are the handoff and
execution authority, so Kubernetes uses `Forbid` concurrency and zero Job
retries rather than introducing a second retry system.
---
### buzz-test-client — Integration Test Harness

4
deploy/charts/buzz/Chart.yaml
View File

@@ -7,7 +7,7 @@ description: |
PostgreSQL and Redis. Configurable for single-node evaluation
(subcharts on) and HA production (external services, existingSecret).
type: application
version: 0.1.9
version: 0.1.10
appVersion: "0.1.0"
home: https://github.com/block/buzz
sources:
@@ -24,7 +24,7 @@ maintainers:
annotations:
artifacthub.io/changes: |
- kind: added
description: Optional isolated storage-accounting CronJob with durable relay snapshots.
description: Optional typed deletion-drain operator CronJob using durable database leases.
artifacthub.io/license: Apache-2.0
# Optional eval-only subcharts. Production deploys disable both and point

22
deploy/charts/buzz/README.md
View File

@@ -12,7 +12,7 @@ This chart has two operating profiles selected by values:
## Quickstart (eval only)
```sh
helm install buzz oci://ghcr.io/block/buzz/charts/buzz --version 0.1.8 \
helm install buzz oci://ghcr.io/block/buzz/charts/buzz --version 0.1.10 \
--create-namespace --namespace buzz \
--set quickstart=true \
--set postgresql.enabled=true \
@@ -116,6 +116,26 @@ is still parsed strictly, but reachability and addressing errors surface on the
first storage operation. `/_readiness` tests no external dependency in either
case — see the readiness contract below.
## Community deletion operator job
`operatorJobs.deletionDrain` is a disabled-by-default, typed CronJob for
`/usr/local/bin/buzz-admin deletions drain`. It runs inside the relay image with
bounded Job lifetime/history, `concurrencyPolicy: Forbid`, `backoffLimit: 0`,
and no relay HTTP call. Postgres deletion requests, leases, retries, and
checkpoints remain the execution authority.
The pod receives only `DATABASE_URL`, `REDIS_URL`, and required S3
configuration/credential variables. It does not receive the relay private key,
git-hook secret, relay URL, service-account token, service links, or a generic
environment registry. Schedule,
deadline, history, termination grace, resources, service account, pod labels,
and pod annotations are independently configurable under
`operatorJobs.deletionDrain`.
See [`docs/operator-community-deletion.md`](../../../docs/operator-community-deletion.md)
for enablement, permissions, the staffed first-run procedure, failure recovery,
and the current explicit approval/alerting boundaries.
### Early-startup telemetry contract
`buzz_process_lifecycle` JSON records are the authoritative history for the

86
deploy/charts/buzz/templates/_operator-jobs.tpl Normal file
View File

@@ -0,0 +1,86 @@
{{/* Closed rendering foundation for typed Buzz operator jobs. */}}
{{- define "buzz.operatorCronJob" -}}
{{- $root := .root -}}
{{- if eq .type "deletionDrain" -}}
{{- $job := $root.Values.operatorJobs.deletionDrain -}}
apiVersion: batch/v1
kind: CronJob
metadata:
name: {{ include "buzz.fullname" $root }}-deletion-drain
labels:
{{- include "buzz.labels" $root | nindent 4 }}
app.kubernetes.io/component: deletion-drain
spec:
schedule: {{ $job.schedule | quote }}
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: {{ $job.successfulJobsHistoryLimit }}
failedJobsHistoryLimit: {{ $job.failedJobsHistoryLimit }}
jobTemplate:
spec:
activeDeadlineSeconds: {{ $job.activeDeadlineSeconds }}
backoffLimit: 0
template:
metadata:
labels:
{{- include "buzz.selectorLabels" $root | nindent 12 }}
app.kubernetes.io/component: deletion-drain
{{- with $job.podLabels }}
{{- toYaml . | nindent 12 }}
{{- end }}
annotations:
{{- toYaml $job.podAnnotations | nindent 12 }}
spec:
restartPolicy: Never
terminationGracePeriodSeconds: {{ $job.terminationGracePeriodSeconds }}
serviceAccountName: {{ default (include "buzz.serviceAccountName" $root) $job.serviceAccountName }}
automountServiceAccountToken: false
enableServiceLinks: false
securityContext:
{{- toYaml $root.Values.relay.securityContext | nindent 12 }}
{{- with $root.Values.image.pullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 12 }}
{{- end }}
containers:
- name: deletion-drain
image: {{ include "buzz.image" $root }}
imagePullPolicy: {{ $root.Values.image.pullPolicy }}
securityContext:
{{- omit $root.Values.relay.containerSecurityContext "readOnlyRootFilesystem" | toYaml | nindent 16 }}
readOnlyRootFilesystem: true
command: ["/usr/local/bin/buzz-admin"]
args: ["deletions", "drain"]
env:
- { name: BUZZ_S3_ENDPOINT, value: {{ required "s3.endpoint is required when operatorJobs.deletionDrain.enabled=true" (include "buzz.s3Endpoint" $root) | quote }} }
- { name: BUZZ_S3_BUCKET, value: {{ required "s3.bucket is required when operatorJobs.deletionDrain.enabled=true" $root.Values.s3.bucket | quote }} }
- { name: BUZZ_S3_REGION, value: {{ $root.Values.s3.region | quote }} }
- { name: BUZZ_S3_ADDRESSING_STYLE, value: {{ $root.Values.s3.addressingStyle | quote }} }
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: {{ include "buzz.envSecretName" $root }}
key: DATABASE_URL
- name: REDIS_URL
valueFrom:
secretKeyRef:
name: {{ include "buzz.envSecretName" $root }}
key: REDIS_URL
- name: BUZZ_S3_ACCESS_KEY
valueFrom:
secretKeyRef:
name: {{ include "buzz.envSecretName" $root }}
key: BUZZ_S3_ACCESS_KEY
optional: true
- name: BUZZ_S3_SECRET_KEY
valueFrom:
secretKeyRef:
name: {{ include "buzz.envSecretName" $root }}
key: BUZZ_S3_SECRET_KEY
optional: true
resources:
{{- toYaml $job.resources | nindent 16 }}
{{- else -}}
{{- fail (printf "unsupported typed operator job %q" .type) -}}
{{- end -}}
{{- end -}}

7
deploy/charts/buzz/templates/_validate.tpl
View File

@@ -18,6 +18,13 @@ surface at template time regardless of which manifest helm renders first.
{{- end -}}
{{- end -}}
{{/* The deletion executor always uses Redis for tenant-scoped invalidation. */}}
{{- if .Values.operatorJobs.deletionDrain.enabled -}}
{{- if and (not .Values.redis.enabled) (not .Values.externalRedis.url) (not .Values.secrets.existingSecret) -}}
{{- fail "operatorJobs.deletionDrain requires Redis. Enable redis.enabled=true, set externalRedis.url, or provide secrets.existingSecret with key REDIS_URL." -}}
{{- end -}}
{{- end -}}
{{/* Multiple replicas do NOT require ReadWriteMany git storage.
Git ref/object state is object-store-backed: every read and write hydrates

4
deploy/charts/buzz/templates/deletion-drain-cronjob.yaml Normal file
View File

@@ -0,0 +1,4 @@
{{- include "buzz.validate" . -}}
{{- if .Values.operatorJobs.deletionDrain.enabled }}
{{- include "buzz.operatorCronJob" (dict "root" . "type" "deletionDrain") }}
{{- end }}

214
deploy/charts/buzz/tests/deletion_drain_test.yaml Normal file
View File

@@ -0,0 +1,214 @@
suite: typed deletion drain operator job
templates:
- templates/deletion-drain-cronjob.yaml
tests:
- it: renders no deletion executor by default
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
externalRedis.url: redis://h:6379
s3.endpoint: https://s3.example.com
s3.accessKey: test
s3.secretKey: test
asserts:
- hasDocuments:
count: 0
- it: renders the bounded typed drain command with isolated pod controls
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
externalRedis.url: redis://h:6379
s3.endpoint: https://s3.example.com
s3.bucket: buzz-media-example
s3.region: us-west-2
s3.addressingStyle: virtual
s3.accessKey: test
s3.secretKey: test
operatorJobs.deletionDrain:
enabled: true
schedule: "*/7 * * * *"
activeDeadlineSeconds: 900
successfulJobsHistoryLimit: 2
failedJobsHistoryLimit: 4
terminationGracePeriodSeconds: 45
serviceAccountName: buzz-deletion-drain
podLabels:
tags.datadoghq.com/service: buzz-deletion-drain
podAnnotations:
sidecar.istio.io/inject: "false"
example.com/operator-job: deletion-drain
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
asserts:
- hasDocuments:
count: 1
- isAPIVersion:
of: batch/v1
- isKind:
of: CronJob
- equal:
path: metadata.name
value: RELEASE-NAME-buzz-deletion-drain
- equal:
path: metadata.labels["app.kubernetes.io/component"]
value: deletion-drain
- equal:
path: spec.schedule
value: "*/7 * * * *"
- equal:
path: spec.concurrencyPolicy
value: Forbid
- equal:
path: spec.successfulJobsHistoryLimit
value: 2
- equal:
path: spec.failedJobsHistoryLimit
value: 4
- equal:
path: spec.jobTemplate.spec.activeDeadlineSeconds
value: 900
- equal:
path: spec.jobTemplate.spec.backoffLimit
value: 0
- equal:
path: spec.jobTemplate.spec.template.spec.restartPolicy
value: Never
- equal:
path: spec.jobTemplate.spec.template.spec.terminationGracePeriodSeconds
value: 45
- equal:
path: spec.jobTemplate.spec.template.spec.serviceAccountName
value: buzz-deletion-drain
- equal:
path: spec.jobTemplate.spec.template.spec.automountServiceAccountToken
value: false
- equal:
path: spec.jobTemplate.spec.template.spec.enableServiceLinks
value: false
- equal:
path: spec.jobTemplate.spec.template.metadata.labels["tags.datadoghq.com/service"]
value: buzz-deletion-drain
- equal:
path: spec.jobTemplate.spec.template.metadata.annotations["example.com/operator-job"]
value: deletion-drain
- equal:
path: spec.jobTemplate.spec.template.spec.containers[0].securityContext.readOnlyRootFilesystem
value: true
- equal:
path: spec.jobTemplate.spec.template.spec.containers[0].command
value: ["/usr/local/bin/buzz-admin"]
- equal:
path: spec.jobTemplate.spec.template.spec.containers[0].args
value: ["deletions", "drain"]
- equal:
path: spec.jobTemplate.spec.template.spec.containers[0].resources.requests.cpu
value: 250m
- equal:
path: spec.jobTemplate.spec.template.spec.containers[0].resources.limits.memory
value: 1Gi
- it: exposes only database redis and deletion S3 configuration
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
externalRedis.url: redis://h:6379
secrets.existingSecret: buzz-operator-secrets
s3.endpoint: https://s3.example.com
s3.bucket: buzz-media-example
s3.region: us-west-2
s3.addressingStyle: virtual
operatorJobs.deletionDrain.enabled: true
asserts:
- lengthEqual:
path: spec.jobTemplate.spec.template.spec.containers[0].env
count: 8
- notExists:
path: spec.jobTemplate.spec.template.spec.containers[0].envFrom
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: DATABASE_URL
valueFrom:
secretKeyRef:
name: buzz-operator-secrets
key: DATABASE_URL
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: REDIS_URL
valueFrom:
secretKeyRef:
name: buzz-operator-secrets
key: REDIS_URL
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_S3_ENDPOINT
value: https://s3.example.com
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_S3_BUCKET
value: buzz-media-example
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_S3_REGION
value: us-west-2
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_S3_ADDRESSING_STYLE
value: virtual
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_S3_ACCESS_KEY
valueFrom:
secretKeyRef:
name: buzz-operator-secrets
key: BUZZ_S3_ACCESS_KEY
optional: true
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_S3_SECRET_KEY
valueFrom:
secretKeyRef:
name: buzz-operator-secrets
key: BUZZ_S3_SECRET_KEY
optional: true
- notContains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_RELAY_PRIVATE_KEY
- notContains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_GIT_HOOK_HMAC_SECRET
- notContains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: RELAY_URL
- it: rejects enablement without a Redis source
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
s3.endpoint: https://s3.example.com
s3.accessKey: test
s3.secretKey: test
operatorJobs.deletionDrain.enabled: true
asserts:
- failedTemplate:
errorPattern: "operatorJobs.deletionDrain requires Redis"

28
deploy/charts/buzz/values.schema.json
View File

@@ -234,6 +234,34 @@
"resources": { "type": "object" }
}
},
"operatorJobs": {
"type": "object",
"additionalProperties": false,
"properties": {
"deletionDrain": {
"type": "object",
"additionalProperties": false,
"properties": {
"enabled": { "type": "boolean" },
"schedule": { "type": "string", "minLength": 1 },
"activeDeadlineSeconds": { "type": "integer", "minimum": 1, "maximum": 86400 },
"successfulJobsHistoryLimit": { "type": "integer", "minimum": 0, "maximum": 10 },
"failedJobsHistoryLimit": { "type": "integer", "minimum": 0, "maximum": 10 },
"terminationGracePeriodSeconds": { "type": "integer", "minimum": 1, "maximum": 300 },
"serviceAccountName": { "type": "string" },
"podLabels": {
"type": "object",
"additionalProperties": { "type": "string" }
},
"podAnnotations": {
"type": "object",
"additionalProperties": { "type": "string" }
},
"resources": { "type": "object" }
}
}
}
},
"minio": {
"type": "object",
"additionalProperties": false,

23
deploy/charts/buzz/values.yaml
View File

@@ -391,6 +391,29 @@ storageAccounting:
cpu: "1"
memory: 20Gi
# Typed, one-shot operator jobs. The chart intentionally exposes no generic
# command or environment registry: each job has a reviewed executable and
# least-privilege environment contract.
operatorJobs:
deletionDrain:
enabled: false
schedule: "*/5 * * * *"
activeDeadlineSeconds: 3600
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
terminationGracePeriodSeconds: 30
serviceAccountName: "" # defaults to the main Buzz service account
podLabels: {}
podAnnotations:
sidecar.istio.io/inject: "false"
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
# In-cluster MinIO for the quickstart profile only. Production deploys leave
# this disabled and use s3.* (or secrets.existingSecret) against managed S3.
minio:

87
docs/operator-community-deletion.md Normal file
View File

@@ -0,0 +1,87 @@
# Community Deletion Operator Job
Buzz executes whole-community deletion through the typed, one-shot
`/usr/local/bin/buzz-admin deletions drain` command. The Helm chart can schedule
that command as a Kubernetes CronJob; it does not call relay HTTP and it does
not add another queue or retry service.
Postgres remains the handoff and source of truth. A run claims only requests
that the deletion store considers runnable, heartbeats the existing lease, and
resumes from durable checkpoints. `concurrencyPolicy: Forbid` prevents scheduled
pod overlap, `backoffLimit: 0` prevents Kubernetes Job retries, and the deletion
store remains authoritative when a pod exits, reaches its deadline, or is
replaced.
## Enablement
The CronJob is disabled by default. Production deployments should use an
existing Secret and a dedicated service account when their cluster policy
supports one:
```yaml
secrets:
existingSecret: buzz-operator-secrets
operatorJobs:
deletionDrain:
enabled: true
schedule: "*/5 * * * *"
activeDeadlineSeconds: 3600
terminationGracePeriodSeconds: 30
serviceAccountName: buzz-deletion-drain
podLabels:
tags.datadoghq.com/service: buzz-deletion-drain
podAnnotations:
sidecar.istio.io/inject: "false"
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
```
Set `s3.endpoint`, `s3.bucket`, `s3.region`, and `s3.addressingStyle` in chart
values. The selected Secret must contain `DATABASE_URL` and `REDIS_URL`; it may
contain `BUZZ_S3_ACCESS_KEY` and `BUZZ_S3_SECRET_KEY` when the object store uses
static credentials. The pod receives only those connection values and the four
non-secret S3 settings. It does not receive `BUZZ_RELAY_PRIVATE_KEY`,
`BUZZ_GIT_HOOK_HMAC_SECRET`, `RELAY_URL`, or the full Secret through `envFrom`.
The pod also disables service-account token automounting and Kubernetes service
link environment injection because the executor does not call the Kubernetes
API or discover cluster Services.
The S3 principal needs the relay's normal object permissions plus bucket-level
`s3:ListBucketVersions` and object-level `s3:DeleteObjectVersion` for every
tenant-owned prefix. This also applies to never-versioned buckets because S3
reports their objects with the `null` version id.
## Runbook
1. Confirm database migrations are current and the deletion request has crossed
the explicit inventory and approval boundary with
`buzz-admin deletions inspect <request-id>`.
2. Confirm the selected Secret contains the required keys and the S3 principal
has version-list and exact-version delete permissions.
3. Enable the CronJob and inspect its rendered command and environment before
rollout.
4. Start one staffed manual run with
`kubectl create job --from=cronjob/<release>-buzz-deletion-drain <job-name>`.
5. Follow pod logs and re-run `buzz-admin deletions inspect <request-id>` to
verify lease, checkpoint, retry, blocked, and terminal state.
6. If a run fails or times out, fix the recorded dependency or permission
failure. Do not add Kubernetes retries: the next scheduled drain consults the
durable retry/checkpoint state and resumes only when the store allows it.
The current owner self-serve relay admission records an owner-origin request at
`submitted` and intentionally performs no inventory or approval synchronously.
The deletion engine rejects `submitted` and `inventoried` requests at its
explicit approval boundary. Automating the privileged inventory/approval step
is therefore a separate control-plane slice; enabling this CronJob alone does
not make a newly accepted owner request destructive.
The chart has no existing PrometheusRule or provider-neutral CronJob alert
integration. Operators must alert on failed/missed Jobs and long-running active
Jobs in their deployment platform. Adding a chart-native alert abstraction is
debt, not part of this job contract.