jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-10-27 04:30:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

Schedule the deletion drain safely (#7827)

Browse Source 
## Why

The deletion engine already exposes a one-shot `buzz-admin deletions
drain` command, but operators have no safe way to schedule it. A generic
workload wrapper could expose arbitrary commands or inherit relay
credentials, which would add a second control plane and widen the blast
radius.

## What

This PR adds an opt-in Kubernetes CronJob for the typed deletion drain
command. It is stacked on #7818, which adds owner-request admission.

## How

The chart keeps the job disabled by default. The CronJob uses `Forbid`,
zero Kubernetes retries, bounded runtime and history, and only the
database, Redis, and object-store settings required by the executor.
PostgreSQL leases, checkpoints, and retry timing remain the sole durable
retry authority.

The closed `_operator-jobs.tpl` helper centralizes safe CronJob
mechanics without creating an arbitrary command or environment registry.
This leaves the touched system simpler than a general job framework
would.

## Risk

The deployment blast radius is low because the job is disabled by
default. When enabled, it can run destructive deletion work that the
database has already approved, so the chart limits concurrency, retries,
credentials, and execution time.

## Testing

Rendered every `deploy/charts/buzz/ci/*.yaml` fixture with `helm
template`; all passed at `e68cb753bb6673aa1baa60046969bdb2d4738b62`.

Ran the full independent Desktop, Tauri, and web lanes; all passed at
`e68cb753bb6673aa1baa60046969bdb2d4738b62`. Flutter package resolution
made no progress within three minutes, so the mobile lane is not claimed
green.

## Bigger picture

A later stack layer must inventory and approve admitted owner requests.
Until then, this scheduler cannot make a new owner request destructive.

Generated with Codex

---------

Signed-off-by: tornquist <tornquist@squareup.com>
Signed-off-by: Codex <noreply@openai.com>
Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
This commit is contained in:
Luke Tornquist
2026-09-28 18:23:42 -04:00
committed by GitHub
parent 662014d544
commit 6918dd3a83
16 changed files with 713 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
ARCHITECTURE.md
View File

@@ -750,10 +750,19 @@ Subcommands:
| `remove-member` | Remove a pubkey from the relay membership list (`--pubkey`, optional `--role` guard); publishes kind:13534 roster |
| `list-members` | List all relay members |
| `generate-key` | Generate a new Nostr keypair (for bootstrapping) |
| `deletions` | Submit, inspect, approve, abort, unblock, run, or drain durable whole-community deletion requests |
| `storage-snapshot` | Run one isolated S3 accounting scan and publish its complete Postgres snapshot |
| `reconcile-channels` | Emit kind:39000/39002 discovery events for channels missing them (idempotent) |
The `buzz-admin` binary is shipped in the relay Docker image (`/usr/local/bin/buzz-admin`) and is the recommended way to manage relay membership in production. Use `./run.sh add-member`, `./run.sh remove-member`, and `./run.sh list-members` in Docker Compose deployments.
Kubernetes deployments may schedule the typed one-shot
`buzz-admin deletions drain` command directly. The pod owns its bounded
Postgres/Redis clients and S3 client; it does not call relay HTTP. Durable
requests, leases, retry timing, and checkpoints in Postgres are the handoff and
execution authority, so Kubernetes uses `Forbid` concurrency and zero Job
retries rather than introducing a second retry system.
---
### buzz-test-client — Integration Test Harness

4
deploy/charts/buzz/Chart.yaml
View File

@@ -7,7 +7,7 @@ description: |
PostgreSQL and Redis. Configurable for single-node evaluation
(subcharts on) and HA production (external services, existingSecret).
type: application
version: 0.1.9
version: 0.1.10
appVersion: "0.1.0"
home: https://github.com/block/buzz
sources:
@@ -24,7 +24,7 @@ maintainers:
annotations:
artifacthub.io/changes: |
- kind: added
description: Optional isolated storage-accounting CronJob with durable relay snapshots.
description: Optional typed deletion-drain operator CronJob using durable database leases.
artifacthub.io/license: Apache-2.0
# Optional eval-only subcharts. Production deploys disable both and point

24
deploy/charts/buzz/README.md
View File

@@ -12,7 +12,7 @@ This chart has two operating profiles selected by values:
## Quickstart (eval only)
```sh
helm install buzz oci://ghcr.io/block/buzz/charts/buzz --version 0.1.8 \
helm install buzz oci://ghcr.io/block/buzz/charts/buzz --version 0.1.10 \
--create-namespace --namespace buzz \
--set quickstart=true \
--set postgresql.enabled=true \
@@ -116,6 +116,28 @@ is still parsed strictly, but reachability and addressing errors surface on the
first storage operation. `/_readiness` tests no external dependency in either
case — see the readiness contract below.
## Community deletion operator job
`operatorJobs.deletionDrain` is a disabled-by-default, typed CronJob for
`/usr/local/bin/buzz-admin deletions drain`. It runs inside the relay image with
bounded Job lifetime/history, `concurrencyPolicy: Forbid`, `backoffLimit: 0`,
and no relay HTTP call. Postgres deletion requests, leases, retries, and
checkpoints remain the execution authority.
The pod receives only `DATABASE_URL`, `REDIS_URL`, and required S3
configuration/credential variables. It does not receive the relay private key,
git-hook secret, relay URL, service links, or a generic environment registry.
The chart disables the ordinary Kubernetes API service-account token mount;
platform workload-identity admission may still inject its own projected token
and provider environment variables. Schedule,
deadline, history, termination grace, resources, service account, pod labels,
and pod annotations are independently configurable under
`operatorJobs.deletionDrain`.
See [`docs/operator-community-deletion.md`](../../../docs/operator-community-deletion.md)
for enablement, permissions, the staffed first-run procedure, failure recovery,
and the current explicit approval/alerting boundaries.
### Early-startup telemetry contract
`buzz_process_lifecycle` JSON records are the authoritative history for the

7
deploy/charts/buzz/templates/_helpers.tpl
View File

@@ -17,6 +17,13 @@
{{- end -}}
{{- end -}}
{{/* Kubernetes CronJob names are limited to 52 characters. */}}
{{- define "buzz.cronJobName" -}}
{{- $maxBaseLength := sub 51 (len .suffix) | int -}}
{{- $base := include "buzz.fullname" .root | trunc $maxBaseLength | trimSuffix "-" -}}
{{- printf "%s-%s" $base .suffix -}}
{{- end -}}
{{- define "buzz.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}

91
deploy/charts/buzz/templates/_operator-jobs.tpl Normal file
View File

@@ -0,0 +1,91 @@
{{/* Closed rendering foundation for typed Buzz operator jobs. */}}
{{- define "buzz.operatorCronJob" -}}
{{- $root := .root -}}
{{- if eq .type "deletionDrain" -}}
{{- $job := $root.Values.operatorJobs.deletionDrain -}}
{{- range $label := list "app.kubernetes.io/name" "app.kubernetes.io/instance" "app.kubernetes.io/component" -}}
{{- if hasKey $job.podLabels $label -}}
{{- fail (printf "operatorJobs.deletionDrain.podLabels may not set chart-owned label %q" $label) -}}
{{- end -}}
{{- end -}}
apiVersion: batch/v1
kind: CronJob
metadata:
name: {{ include "buzz.cronJobName" (dict "root" $root "suffix" "deletion-drain") }}
labels:
{{- include "buzz.labels" $root | nindent 4 }}
app.kubernetes.io/component: deletion-drain
spec:
schedule: {{ $job.schedule | quote }}
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: {{ $job.successfulJobsHistoryLimit }}
failedJobsHistoryLimit: {{ $job.failedJobsHistoryLimit }}
jobTemplate:
spec:
activeDeadlineSeconds: {{ $job.activeDeadlineSeconds }}
backoffLimit: 0
template:
metadata:
labels:
{{- include "buzz.selectorLabels" $root | nindent 12 }}
app.kubernetes.io/component: deletion-drain
{{- with $job.podLabels }}
{{- toYaml . | nindent 12 }}
{{- end }}
annotations:
{{- toYaml $job.podAnnotations | nindent 12 }}
spec:
restartPolicy: Never
terminationGracePeriodSeconds: {{ $job.terminationGracePeriodSeconds }}
serviceAccountName: {{ default (include "buzz.serviceAccountName" $root) $job.serviceAccountName }}
automountServiceAccountToken: false
enableServiceLinks: false
securityContext:
{{- toYaml $root.Values.relay.securityContext | nindent 12 }}
{{- with $root.Values.image.pullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 12 }}
{{- end }}
containers:
- name: deletion-drain
image: {{ include "buzz.image" $root }}
imagePullPolicy: {{ $root.Values.image.pullPolicy }}
securityContext:
{{- omit $root.Values.relay.containerSecurityContext "readOnlyRootFilesystem" | toYaml | nindent 16 }}
readOnlyRootFilesystem: true
command: ["/usr/local/bin/buzz-admin"]
args: ["deletions", "drain"]
env:
- { name: BUZZ_S3_ENDPOINT, value: {{ required "s3.endpoint is required when operatorJobs.deletionDrain.enabled=true" (include "buzz.s3Endpoint" $root) | quote }} }
- { name: BUZZ_S3_BUCKET, value: {{ required "s3.bucket is required when operatorJobs.deletionDrain.enabled=true" $root.Values.s3.bucket | quote }} }
- { name: BUZZ_S3_REGION, value: {{ $root.Values.s3.region | quote }} }
- { name: BUZZ_S3_ADDRESSING_STYLE, value: {{ $root.Values.s3.addressingStyle | quote }} }
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: {{ include "buzz.envSecretName" $root }}
key: DATABASE_URL
- name: REDIS_URL
valueFrom:
secretKeyRef:
name: {{ include "buzz.envSecretName" $root }}
key: REDIS_URL
- name: BUZZ_S3_ACCESS_KEY
valueFrom:
secretKeyRef:
name: {{ include "buzz.envSecretName" $root }}
key: BUZZ_S3_ACCESS_KEY
optional: true
- name: BUZZ_S3_SECRET_KEY
valueFrom:
secretKeyRef:
name: {{ include "buzz.envSecretName" $root }}
key: BUZZ_S3_SECRET_KEY
optional: true
resources:
{{- toYaml $job.resources | nindent 16 }}
{{- else -}}
{{- fail (printf "unsupported typed operator job %q" .type) -}}
{{- end -}}
{{- end -}}

7
deploy/charts/buzz/templates/_validate.tpl
View File

@@ -18,6 +18,13 @@ surface at template time regardless of which manifest helm renders first.
{{- end -}}
{{- end -}}
{{/* The deletion executor always uses Redis for tenant-scoped invalidation. */}}
{{- if .Values.operatorJobs.deletionDrain.enabled -}}
{{- if and (not .Values.redis.enabled) (not .Values.externalRedis.url) (not .Values.secrets.existingSecret) -}}
{{- fail "operatorJobs.deletionDrain requires Redis. Enable redis.enabled=true, set externalRedis.url, or provide secrets.existingSecret with key REDIS_URL." -}}
{{- end -}}
{{- end -}}
{{/* Multiple replicas do NOT require ReadWriteMany git storage.
Git ref/object state is object-store-backed: every read and write hydrates

4
deploy/charts/buzz/templates/deletion-drain-cronjob.yaml Normal file
View File

@@ -0,0 +1,4 @@
{{- include "buzz.validate" . -}}
{{- if .Values.operatorJobs.deletionDrain.enabled }}
{{- include "buzz.operatorCronJob" (dict "root" . "type" "deletionDrain") }}
{{- end }}

2
deploy/charts/buzz/templates/storage-accounting-cronjob.yaml
View File

@@ -2,7 +2,7 @@
apiVersion: batch/v1
kind: CronJob
metadata:
name: {{ include "buzz.fullname" . }}-storage-accounting
name: {{ include "buzz.cronJobName" (dict "root" . "suffix" "storage-accounting") }}
labels:
{{- include "buzz.labels" . | nindent 4 }}
app.kubernetes.io/component: storage-accounting

328
deploy/charts/buzz/tests/deletion_drain_test.yaml Normal file
View File

@@ -0,0 +1,328 @@
suite: typed deletion drain operator job
templates:
- templates/deletion-drain-cronjob.yaml
tests:
- it: renders no deletion executor by default
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
externalRedis.url: redis://h:6379
s3.endpoint: https://s3.example.com
s3.accessKey: test
s3.secretKey: test
asserts:
- hasDocuments:
count: 0
- it: renders the bounded typed drain command with isolated pod controls
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
externalRedis.url: redis://h:6379
s3.endpoint: https://s3.example.com
s3.bucket: buzz-media-example
s3.region: us-west-2
s3.addressingStyle: virtual
s3.accessKey: test
s3.secretKey: test
operatorJobs.deletionDrain:
enabled: true
schedule: "*/7 * * * *"
activeDeadlineSeconds: 900
successfulJobsHistoryLimit: 2
failedJobsHistoryLimit: 4
terminationGracePeriodSeconds: 45
serviceAccountName: buzz-deletion-drain
podLabels:
tags.datadoghq.com/service: buzz-deletion-drain
podAnnotations:
example.com/operator-job: deletion-drain
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
asserts:
- hasDocuments:
count: 1
- isAPIVersion:
of: batch/v1
- isKind:
of: CronJob
- equal:
path: metadata.name
value: RELEASE-NAME-buzz-deletion-drain
- equal:
path: metadata.labels["app.kubernetes.io/component"]
value: deletion-drain
- equal:
path: spec.schedule
value: "*/7 * * * *"
- equal:
path: spec.concurrencyPolicy
value: Forbid
- equal:
path: spec.successfulJobsHistoryLimit
value: 2
- equal:
path: spec.failedJobsHistoryLimit
value: 4
- equal:
path: spec.jobTemplate.spec.activeDeadlineSeconds
value: 900
- equal:
path: spec.jobTemplate.spec.backoffLimit
value: 0
- equal:
path: spec.jobTemplate.spec.template.spec.restartPolicy
value: Never
- equal:
path: spec.jobTemplate.spec.template.spec.terminationGracePeriodSeconds
value: 45
- equal:
path: spec.jobTemplate.spec.template.spec.serviceAccountName
value: buzz-deletion-drain
- equal:
path: spec.jobTemplate.spec.template.spec.automountServiceAccountToken
value: false
- equal:
path: spec.jobTemplate.spec.template.spec.enableServiceLinks
value: false
- equal:
path: spec.jobTemplate.spec.template.metadata.labels["tags.datadoghq.com/service"]
value: buzz-deletion-drain
- equal:
path: spec.jobTemplate.spec.template.metadata.labels["app.kubernetes.io/name"]
value: buzz
- equal:
path: spec.jobTemplate.spec.template.metadata.labels["app.kubernetes.io/instance"]
value: RELEASE-NAME
- equal:
path: spec.jobTemplate.spec.template.metadata.labels["app.kubernetes.io/component"]
value: deletion-drain
- equal:
path: spec.jobTemplate.spec.template.metadata.annotations["sidecar.istio.io/inject"]
value: "false"
- equal:
path: spec.jobTemplate.spec.template.metadata.annotations["example.com/operator-job"]
value: deletion-drain
- equal:
path: spec.jobTemplate.spec.template.spec.containers[0].securityContext.readOnlyRootFilesystem
value: true
- equal:
path: spec.jobTemplate.spec.template.spec.containers[0].command
value: ["/usr/local/bin/buzz-admin"]
- equal:
path: spec.jobTemplate.spec.template.spec.containers[0].args
value: ["deletions", "drain"]
- equal:
path: spec.jobTemplate.spec.template.spec.containers[0].resources.requests.cpu
value: 250m
- equal:
path: spec.jobTemplate.spec.template.spec.containers[0].resources.limits.memory
value: 1Gi
- it: exposes only database redis and deletion S3 configuration
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
externalRedis.url: redis://h:6379
secrets.existingSecret: buzz-operator-secrets
s3.endpoint: https://s3.example.com
s3.bucket: buzz-media-example
s3.region: us-west-2
s3.addressingStyle: virtual
operatorJobs.deletionDrain.enabled: true
asserts:
- lengthEqual:
path: spec.jobTemplate.spec.template.spec.containers[0].env
count: 8
- notExists:
path: spec.jobTemplate.spec.template.spec.containers[0].envFrom
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: DATABASE_URL
valueFrom:
secretKeyRef:
name: buzz-operator-secrets
key: DATABASE_URL
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: REDIS_URL
valueFrom:
secretKeyRef:
name: buzz-operator-secrets
key: REDIS_URL
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_S3_ENDPOINT
value: https://s3.example.com
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_S3_BUCKET
value: buzz-media-example
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_S3_REGION
value: us-west-2
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_S3_ADDRESSING_STYLE
value: virtual
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_S3_ACCESS_KEY
valueFrom:
secretKeyRef:
name: buzz-operator-secrets
key: BUZZ_S3_ACCESS_KEY
optional: true
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_S3_SECRET_KEY
valueFrom:
secretKeyRef:
name: buzz-operator-secrets
key: BUZZ_S3_SECRET_KEY
optional: true
- notContains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_RELAY_PRIVATE_KEY
- notContains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_GIT_HOOK_HMAC_SECRET
- notContains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: RELAY_URL
- it: rejects the chart-owned name pod label with an explicit guard error
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
externalRedis.url: redis://h:6379
s3.endpoint: https://s3.example.com
s3.bucket: buzz-media-example
s3.accessKey: test
s3.secretKey: test
operatorJobs.deletionDrain:
enabled: true
podLabels:
app.kubernetes.io/name: relay
asserts:
- failedTemplate:
errorPattern: 'operatorJobs.deletionDrain.podLabels may not set chart-owned label "app.kubernetes.io/name"'
- it: rejects the chart-owned instance pod label with an explicit guard error
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
externalRedis.url: redis://h:6379
s3.endpoint: https://s3.example.com
s3.bucket: buzz-media-example
s3.accessKey: test
s3.secretKey: test
operatorJobs.deletionDrain:
enabled: true
podLabels:
app.kubernetes.io/instance: another-release
asserts:
- failedTemplate:
errorPattern: 'operatorJobs.deletionDrain.podLabels may not set chart-owned label "app.kubernetes.io/instance"'
- it: rejects the chart-owned component pod label with an explicit guard error
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
externalRedis.url: redis://h:6379
s3.endpoint: https://s3.example.com
s3.bucket: buzz-media-example
s3.accessKey: test
s3.secretKey: test
operatorJobs.deletionDrain:
enabled: true
podLabels:
app.kubernetes.io/component: relay
asserts:
- failedTemplate:
errorPattern: 'operatorJobs.deletionDrain.podLabels may not set chart-owned label "app.kubernetes.io/component"'
- it: rejects existing-secret enablement without an explicit S3 endpoint
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
secrets.existingSecret: buzz-operator-secrets
externalPostgresql.url: postgres://u:p@h:5432/d
externalRedis.url: redis://h:6379
s3.bucket: buzz-media-example
operatorJobs.deletionDrain.enabled: true
asserts:
- failedTemplate:
errorPattern: "s3.endpoint is required when operatorJobs.deletionDrain.enabled=true"
- it: rejects unknown deletion drain keys through the values schema
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
externalRedis.url: redis://h:6379
s3.endpoint: https://s3.example.com
s3.bucket: buzz-media-example
s3.accessKey: test
s3.secretKey: test
operatorJobs.deletionDrain:
enabled: true
arbitraryCommand: ["unsafe"]
asserts:
- failedTemplate:
errorPattern: "Additional property arbitraryCommand is not allowed"
- it: rejects enablement without a Redis source
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
s3.endpoint: https://s3.example.com
s3.accessKey: test
s3.secretKey: test
operatorJobs.deletionDrain.enabled: true
asserts:
- failedTemplate:
errorPattern: "operatorJobs.deletionDrain requires Redis"
- it: preserves the deletion suffix while bounding a long release and name override
release:
name: very-long-release-name-for-operator-jobs
set:
nameOverride: custom-buzz-name
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
externalRedis.url: redis://h:6379
s3.endpoint: https://s3.example.com
s3.bucket: buzz-media-example
s3.accessKey: test
s3.secretKey: test
operatorJobs.deletionDrain.enabled: true
asserts:
- equal:
path: metadata.name
value: very-long-release-name-for-operator-j-deletion-drain

11
deploy/charts/buzz/tests/fixtures/deletion-drain-bundled-values.yaml vendored Normal file
View File

@@ -0,0 +1,11 @@
relayUrl: wss://buzz.example.com
ownerPubkey: "abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789"
postgresql:
enabled: true
redis:
enabled: true
minio:
enabled: true
operatorJobs:
deletionDrain:
enabled: true

3
deploy/charts/buzz/tests/fixtures/production-existing-secret-values.yaml vendored
View File

@@ -8,6 +8,9 @@ externalPostgresql:
url: "postgres://buzz:pw@postgres.example.com:5432/buzz"
externalRedis:
url: "redis://:pw@redis.example.com:6379"
operatorJobs:
deletionDrain:
enabled: true
s3:
endpoint: "https://s3.us-east-1.amazonaws.com"
bucket: "buzz-media"

22
deploy/charts/buzz/tests/storage_accounting_test.yaml
View File

@@ -53,6 +53,10 @@ tests:
path: kind
value: CronJob
template: templates/storage-accounting-cronjob.yaml
- equal:
path: metadata.name
value: RELEASE-NAME-buzz-storage-accounting
template: templates/storage-accounting-cronjob.yaml
- equal:
path: spec.concurrencyPolicy
value: Forbid
@@ -143,3 +147,21 @@ tests:
content:
name: BUZZ_GIT_HOOK_HMAC_SECRET
template: templates/storage-accounting-cronjob.yaml
- it: preserves the storage suffix without a trailing separator after truncation
set:
fullnameOverride: storage-accounting-fullname-that-needs-truncation-at-the-separator
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
externalRedis.url: redis://h:6379
s3.endpoint: https://s3.example.com
s3.bucket: buzz-media-example
s3.accessKey: test
s3.secretKey: test
storageAccounting.enabled: true
asserts:
- equal:
path: metadata.name
value: storage-accounting-fullname-that-storage-accounting
template: templates/storage-accounting-cronjob.yaml

28
deploy/charts/buzz/values.schema.json
View File

@@ -234,6 +234,34 @@
"resources": { "type": "object" }
}
},
"operatorJobs": {
"type": "object",
"additionalProperties": false,
"properties": {
"deletionDrain": {
"type": "object",
"additionalProperties": false,
"properties": {
"enabled": { "type": "boolean" },
"schedule": { "type": "string", "minLength": 1 },
"activeDeadlineSeconds": { "type": "integer", "minimum": 1, "maximum": 86400 },
"successfulJobsHistoryLimit": { "type": "integer", "minimum": 0, "maximum": 10 },
"failedJobsHistoryLimit": { "type": "integer", "minimum": 0, "maximum": 10 },
"terminationGracePeriodSeconds": { "type": "integer", "minimum": 1, "maximum": 300 },
"serviceAccountName": { "type": "string" },
"podLabels": {
"type": "object",
"additionalProperties": { "type": "string" }
},
"podAnnotations": {
"type": "object",
"additionalProperties": { "type": "string" }
},
"resources": { "type": "object" }
}
}
}
},
"minio": {
"type": "object",
"additionalProperties": false,

25
deploy/charts/buzz/values.yaml
View File

@@ -391,6 +391,31 @@ storageAccounting:
cpu: "1"
memory: 20Gi
# Typed, one-shot operator jobs. The chart intentionally exposes no generic
# command or environment registry: each job has a reviewed executable and
# least-privilege environment contract.
operatorJobs:
deletionDrain:
enabled: false
schedule: "*/5 * * * *"
activeDeadlineSeconds: 3600
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
terminationGracePeriodSeconds: 30
# Empty inherits the relay service account, and with it any cloud IAM
# binding on that account. Name a dedicated account to narrow the executor.
serviceAccountName: ""
podLabels: {}
podAnnotations:
sidecar.istio.io/inject: "false"
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
# In-cluster MinIO for the quickstart profile only. Production deploys leave
# this disabled and use s3.* (or secrets.existingSecret) against managed S3.
minio:

9
desktop/tests/e2e/sidebar-snapshot.spec.ts
View File

@@ -487,6 +487,7 @@ test("mismatched not-modified hash falls back to a full list", async ({
page,
}) => {
await seedSnapshot(page, { hash: "persisted-stale-hash" });
await trackSnapshotRows(page);
await installMockBridge(page, {
channelsReadDelayMs: READ_DELAY_MS,
channelsNotModifiedResponses: 1,
@@ -494,9 +495,11 @@ test("mismatched not-modified hash falls back to a full list", async ({
await page.goto("/");
const snapshotRows = page.locator('[data-channel-id^="snapshot-"]');
await expect(snapshotRows).toHaveCount(FULL_SNAPSHOT.length, {
timeout: 500,
});
// Record the transient boot frame independently of test-runner scheduling;
// cold-boot coverage separately enforces the snapshot paint deadline.
await expect
.poll(() => getTrackedSnapshotRows(page))
.toEqual(FULL_SNAPSHOT.map((channel) => channel.id));
await expect
.poll(() => getChannelsPayloads(page))
.toEqual([{ knownHash: "persisted-stale-hash" }, { knownHash: null }]);

146
docs/operator-community-deletion.md Normal file
View File

@@ -0,0 +1,146 @@
# Community Deletion Operator Job
Buzz executes whole-community deletion through the typed, one-shot
`/usr/local/bin/buzz-admin deletions drain` command. The Helm chart can schedule
that command as a Kubernetes CronJob; it does not call relay HTTP and it does
not add another queue or retry service.
Postgres remains the handoff and source of truth. A run claims only requests
that the deletion store considers runnable, heartbeats the existing lease, and
resumes from durable checkpoints. `concurrencyPolicy: Forbid` prevents scheduled
pod overlap, `backoffLimit: 0` prevents Kubernetes Job retries, and the deletion
store remains authoritative when a pod exits, reaches its deadline, or is
replaced.
## Enablement
The CronJob is disabled by default. Production deployments should use an
existing Secret and a dedicated, pre-created service account:
```yaml
secrets:
existingSecret: buzz-operator-secrets
operatorJobs:
deletionDrain:
enabled: true
schedule: "*/5 * * * *"
activeDeadlineSeconds: 3600
terminationGracePeriodSeconds: 30
serviceAccountName: buzz-deletion-drain
podLabels:
tags.datadoghq.com/service: buzz-deletion-drain
podAnnotations:
sidecar.istio.io/inject: "false"
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
```
Set `s3.endpoint`, `s3.bucket`, `s3.region`, and `s3.addressingStyle` in chart
values. The selected Secret must contain `DATABASE_URL` and `REDIS_URL`; it may
contain `BUZZ_S3_ACCESS_KEY` and `BUZZ_S3_SECRET_KEY` when the object store uses
static credentials. The pod receives only those connection values and the four
non-secret S3 settings. It does not receive `BUZZ_RELAY_PRIVATE_KEY`,
`BUZZ_GIT_HOOK_HMAC_SECRET`, `RELAY_URL`, or the full Secret through `envFrom`.
The pod also disables service-account token automounting and Kubernetes service
link environment injection because the executor does not call the Kubernetes
API or discover cluster Services. This disables the ordinary Kubernetes API
token mount, not credentials injected by a platform workload-identity
mechanism.
Leaving `operatorJobs.deletionDrain.serviceAccountName` empty falls back to the
relay's own service account, including cloud IAM attached through the platform's
workload-identity mechanism. For example, the EKS IRSA admission webhook can
inject its projected web-identity token and AWS environment variables despite
`automountServiceAccountToken: false`; other platforms provide their own
identity mechanism. The drain pod therefore inherits the relay's cloud role by
default. Create a dedicated service account with only the object-store
permissions listed below and name it explicitly if you want the executor's IAM
blast radius to be smaller than the relay's.
The S3 principal needs the relay's normal object permissions plus bucket-level
`s3:ListBucketVersions` and object-level `s3:DeleteObjectVersion` for every
tenant-owned prefix. This also applies to never-versioned buckets because S3
reports their objects with the `null` version id.
## Runbook
1. Confirm database migrations are current and the deletion request has crossed
the explicit inventory and approval boundary with
`buzz-admin deletions inspect <request-id>`.
2. Confirm the selected Secret contains the required keys and the S3 principal
has version-list and exact-version delete permissions.
3. Enable the CronJob and inspect its rendered command and environment before
rollout.
4. Locate the rendered CronJob by label rather than by guessing its name:
```sh
kubectl get cronjob -n <namespace> \
-l app.kubernetes.io/component=deletion-drain,app.kubernetes.io/instance=<release>
```
The name is `<bounded-fullname>-deletion-drain`. The chart truncates only the
fullname portion when needed so the suffix remains stable within Kubernetes'
52-character CronJob name limit. `buzz.fullname` collapses to the release
name when the release name already contains the chart name, so
`helm install buzz ...` renders `buzz-deletion-drain`, not
`buzz-buzz-deletion-drain`.
5. Start one staffed manual run with
`kubectl create job --from=cronjob/<cronjob-name> <job-name>`.
6. Follow pod logs and re-run `buzz-admin deletions inspect <request-id>` to
verify lease, checkpoint, retry, blocked, and terminal state.
7. If a run fails or times out, fix the recorded dependency or permission
failure. Do not add Kubernetes retries: the next scheduled drain consults the
durable retry/checkpoint state and resumes only when the store allows it.
## Deadlines, termination, and the retry budget
`activeDeadlineSeconds` is a Kubernetes-side limit, and the deletion store does
not learn why a pod went away. Two consequences matter when reading state:
- A `DeadlineExceeded` Job is not recorded as a deletion retry. Shutdown
releases the claim without recording one, so `retry_count` and `blocked_at`
do not advance. Only the object-store drain checkpoints per manifest chunk
and resumes mid-stage; every other stage restarts from its beginning on the
next run. A deadline that keeps landing inside one of those non-resumable
stages therefore repeats indefinitely: each run increments `attempts` and
burns the window again while the retry budget never moves and the request is
never blocked.
- Diagnose this from both sides. `buzz-admin deletions inspect <request-id>`
shows a rising `attempts` with a flat `retry_count` and no `last_error`;
Kubernetes holds the reason. The chart labels the CronJob and the drain pods,
but not the generated Jobs, so find the attempts with
`kubectl get pods -n <namespace> -l app.kubernetes.io/component=deletion-drain`
and read the condition with `kubectl describe job <job-name>`.
Size `activeDeadlineSeconds` for the longest single stage this community will
run, not for the average run. To recover, either raise the deadline and let the
schedule pick the request back up, or take one staffed run with
`buzz-admin deletions run <request-id>` outside the CronJob's deadline.
`terminationGracePeriodSeconds` is a best-effort window, not a guarantee. The
drain command handles `SIGTERM` and releases its lease cleanly when it wins the
race, but a pod that is still working when the grace period expires is
`SIGKILL`ed with the lease still held. Nothing is lost: the durable lease simply
expires (60s by default, heartbeated every 10s) and the next run reclaims the
request with a fresh lease generation, which fences any straggler write from the
killed process. Expect up to roughly a lease duration of delay before the
request is runnable again; do not raise the grace period expecting a clean
handoff.
The current owner self-serve relay admission records an owner-origin request at
`submitted` and intentionally performs no inventory or approval synchronously.
The deletion engine rejects `submitted` and `inventoried` requests at its
explicit approval boundary. Automating the privileged inventory/approval step
is therefore a separate control-plane slice; enabling this CronJob alone does
not make a newly accepted owner request destructive.
The chart has no existing PrometheusRule or provider-neutral CronJob alert
integration. Operators must alert on failed/missed Jobs and long-running active
Jobs in their deployment platform. Adding a chart-native alert abstraction is
debt, not part of this job contract.