jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-10-27 12:40:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Isolate S3 storage metrics from the relay (#7543)

Browse Source 
## Why

**This PR moves the S3 storage scan out of the serving relay and into a
separate batch job.** The job saves a complete result in PostgreSQL. The
relay reads that result, exports the existing storage totals, and
reports the snapshot's freshness and database-read health.

```text
Today:         relay -> scan S3 -> calculate totals -> Datadog metrics
After rollout: CronJob -> scan S3 -> PostgreSQL snapshot -> relay -> Datadog metrics
```

The code in this PR is the new pipeline. It adds the worker, database
snapshot, relay read mode, and Helm CronJob template. It does **not**
enable the pipeline in production. The chart keeps the CronJob disabled,
and the relay keeps its current inline mode by default.
[squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244)
will set the production limits, enable the job, and switch the relay to
the saved snapshot after this PR merges.

The relay currently lists the media bucket once an hour and calculates
storage totals inside the serving process. The production bucket now
exceeds the relay's one million object cap, so each sweep fails and the
last good per-community values remain stale. The Datadog dashboard can
no longer show current storage use by community.

Raising that cap inside the serving relay would move a larger,
memory-heavy scan into the process that owns live WebSocket connections.
The scan retains state for unique blob hashes and community bindings, so
its memory use grows with the bucket. A large scan could raise relay
memory use, trigger restarts, and drop user connections.

The goal is to restore current metrics without putting a
production-scale bucket scan in the relay's failure domain. No media
objects move or change. Only the process that calculates the totals and
the handoff to the relay change.

## What

- Add `buzz-admin storage-snapshot`, a run-once worker that scans S3 and
saves one complete accounting snapshot in PostgreSQL.
- Add `external` relay mode to read the saved snapshot. Keep the current
`inline` mode as the default, and add `off` as a fail-closed option.
- Add an opt-in Helm CronJob for the worker. The 10 million object cap
exists only on this Job.
- Keep the current fleet and per-community gauges, and add snapshot age,
generation duration, configured cap, cap utilization, and database-load
health gauges.
- Add an additive migration for the latest complete snapshot and its run
metadata.

## Overall plan

1. Merge this PR to add the worker, database handoff, relay modes, and
disabled-by-default chart support. This step does not change production
behavior.
2. Publish the relay image and Helm chart `0.1.9` from `main`.
3. Update
[squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244)
to pin those final artifacts and rerender its manifests.
4. Use the infrastructure PR to deploy the CronJob with a 10 million
object cap, a one-hour deadline, and a 20 GiB memory limit.
5. Start one manual Job during a staffed window and verify its snapshot,
duration, memory use, relay stability, and Datadog output.
6. Keep the daily schedule and `external` relay mode only after that
gate passes, then add alerts for stale snapshots, snapshot-load failure,
failed jobs, cap pressure, duration, and worker memory.

The code and infrastructure remain separate so this PR can ship the
mechanism without silently enabling a production-scale scan. The
infrastructure PR owns the production settings and rollout gate.

## How

### Complete snapshot handoff

The worker takes a deployment-wide PostgreSQL advisory lock before it
lists S3. This prevents scheduled and manual workers from overlapping,
even if Kubernetes starts both.

The final database replacement runs through that same lock-owning
session. If the session dies during the scan, publication fails instead
of letting the stale worker overwrite its successor. A worker that no
longer owns the lock cannot publish through another pooled connection.

The worker reads S3 in pages of 1,000 objects and folds each page into
physical totals, logical per-community usage, and anomaly counts. It
stops before folding a page that would exceed `--max-objects` and emits
structured progress every 100,000 objects.

The worker replaces the singleton database row only after the full scan
succeeds. A listing error, malformed page, cap breach, serialization
error, lost lock session, or database error cannot publish a partial
snapshot. The last completed snapshot stays available. Each stored
result includes its completion time, duration, cap, and image or source
revision.

PostgreSQL is the handoff between the batch worker and the relay. The
worker writes one complete deployment-wide snapshot, and the relay reads
that snapshot on its normal leader metrics tick. No new network service
or worker endpoint is required.

### Relay behavior

`BUZZ_STORAGE_METRICS=inline` keeps the existing in-process sweep and
its one million object default. `external` loads the newest completed
worker snapshot on each leader metrics tick. `off` emits no
storage-family metrics. An unknown value fails closed to `off`.

External mode republishes the existing fleet, per-community, and anomaly
metric names, so those current queries remain valid. Snapshot handoff
health uses a separate family: `buzz_storage_snapshot_age_seconds`,
`buzz_storage_snapshot_duration_seconds`,
`buzz_storage_snapshot_max_objects`,
`buzz_storage_snapshot_cap_utilization`, and
`buzz_storage_snapshot_load_ok`.

The `buzz_storage_sweep_*` gauges remain inline-only because they
describe a scan attempted by the relay process. External mode does not
synthesize `sweep_ok=1` from the last successful database row: an old
row proves only that an earlier worker completed. Kubernetes CronJob/Job
status is authoritative for the latest worker result, including non-zero
exit, deadline expiry, and OOM termination.

The relay keeps the last complete snapshot available when a worker run
fails. Snapshot age shows that no replacement arrived on schedule, while
`buzz_storage_snapshot_load_ok` distinguishes staleness from a missing
row, database read failure, or invalid payload. If a community
disappears, changes host, or leaves the configured emission scope, the
relay zeroes the old labeled series instead of leaving stale values
behind.

Per-community bytes are logical referenced media usage. If two
communities reference the same blob, each community receives the blob's
bytes. Fleet physical totals still count each S3 object once. This PR
preserves those existing metric semantics.

### Deployment boundary

The CronJob is disabled by default. When enabled, it uses
`concurrencyPolicy: Forbid`, no retry, a configurable active deadline,
bounded history, explicit CPU and memory resources, a read-only root
filesystem, and no Istio sidecar by default.

The worker receives only `DATABASE_URL`, S3 settings, optional S3
credential references, and the code revision. It does not receive the
relay private key, Git hook HMAC secret, Redis URL, or the full relay
secret as environment variables. Operators may reuse the Buzz service
account or select a dedicated one.

### Non-goals

- This PR does not enable the worker or `external` mode in production.
- This PR does not change media uploads, object ownership, retention, or
deletion.
- This PR does not change the names or meaning of the existing fleet,
per-community, or anomaly storage gauges.
- This PR does not add a lease service or allow multiple workers to
publish concurrently.

## Risk

The application and chart defaults do not enable the new worker or raise
the serving relay's cap. The database change is an additive
deployment-global singleton table. The relay's default remains `inline`,
so merging this PR alone does not change the active metrics path.

The main rollout risk is worker memory use because the fold retains
state for distinct blob hashes and community bindings. The first
production-sized run must confirm peak memory and duration before the
schedule is trusted. The 10 million cap and one-hour deadline bound the
first deployment, but they do not replace measurement.

External mode also needs stale-snapshot, snapshot-load, and job-failure
monitors because the relay intentionally keeps publishing the last
complete result after a failed job or read error. This behavior avoids
gaps and partial data, while snapshot age and load health tell operators
when the result or PostgreSQL handoff needs attention. Job status
remains necessary for immediate failures and termination reasons that a
killed process cannot record itself.

## Testing

No production S3 scan was run from this PR.

Automated regressions verify that external mode keeps stale last-good
totals visible without emitting inline attempt-health gauges, reports
database-load failure while retaining the cache, and leaves inline
telemetry unchanged. Mutation checks also prove those tests fail if
external mode is routed through inline emission or a failed load is
reported as healthy.

After deployment, create one manual Job from the CronJob during a
staffed window. Require a completed Job, exactly one new snapshot, fewer
objects than the configured cap, acceptable duration and peak memory, no
relay restart regression, restored Datadog points, and no OpenMetrics
sample truncation.

## Next steps

1. Complete review and merge this PR.
2. Publish chart `0.1.9` and the final main-branch relay image.
3. Replace the provisional pins in
[squareup/builderbot-platform-core-infrastructure#244](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/244),
rerender, and require clean CI.
4. Run the staffed manual Job gate and record object count, peak memory,
duration, snapshot revision, and relay stability.
5. Confirm the relay reads the new snapshot in `external` mode and the
existing `block.buzz_relay.*` storage series update again.
6. Enable the daily 03:17 UTC schedule and add dashboard panels and
monitors for snapshot age, snapshot-load health, Job failures, cap
pressure, duration, and worker memory.

If production must roll back to an older relay image, set
`BUZZ_STORAGE_METRICS=off` first. Older images do not understand
`external` and would resume the failing inline scan.

## Bigger picture

This is the application half of the storage-metrics repair. The paired
infrastructure PR owns the production resource limits, schedule,
artifact pins, secret boundary, and monitored rollout. Keeping those
changes separate lets this PR remain safe by default while still
providing a complete path to current per-community S3 totals.

Related issue: N/A. No matching issue or pull request exists in
`block/buzz`.

Generated with Codex

---------

Signed-off-by: Ravneet Arora <rarora@squareup.com>
Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
ravarora2
2026-09-17 10:34:31 -04:00
committed by GitHub
parent 4ab4f78608
commit 6b1e45d2f6
19 changed files with 1169 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
deploy/charts/buzz/Chart.yaml
View File

@@ -7,7 +7,7 @@ description: |
PostgreSQL and Redis. Configurable for single-node evaluation
(subcharts on) and HA production (external services, existingSecret).
type: application
version: 0.1.8
version: 0.1.9
appVersion: "0.1.0"
home: https://github.com/block/buzz
sources:
@@ -24,7 +24,7 @@ maintainers:
annotations:
artifacthub.io/changes: |
- kind: added
description: Optional immutable relay image digest pinning with backwards-compatible tag fallback.
description: Optional isolated storage-accounting CronJob with durable relay snapshots.
artifacthub.io/license: Apache-2.0
# Optional eval-only subcharts. Production deploys disable both and point

8
deploy/charts/buzz/templates/_helpers.tpl
View File

@@ -61,6 +61,14 @@ app.kubernetes.io/component: relay
{{- end -}}
{{- end -}}
{{- define "buzz.imageRevision" -}}
{{- if .Values.image.digest -}}
{{- .Values.image.digest -}}
{{- else -}}
{{- default .Chart.AppVersion .Values.image.tag -}}
{{- end -}}
{{- end -}}
{{/*
Name of the chart-managed Secret holding relay-identity material and any
chart-composed connection strings.

1
deploy/charts/buzz/templates/deployment.yaml
View File

@@ -172,6 +172,7 @@ spec:
- { name: BUZZ_S3_BUCKET, value: {{ .Values.s3.bucket | quote }} }
- { name: BUZZ_S3_REGION, value: {{ .Values.s3.region | quote }} }
- { name: BUZZ_S3_ADDRESSING_STYLE, value: {{ .Values.s3.addressingStyle | quote }} }
- { name: BUZZ_STORAGE_METRICS, value: {{ .Values.storageAccounting.relayMode | quote }} }
# ── Secrets (from chart-managed or existing) ─────────────
- name: BUZZ_RELAY_PRIVATE_KEY

74
deploy/charts/buzz/templates/storage-accounting-cronjob.yaml Normal file
View File

@@ -0,0 +1,74 @@
{{- if .Values.storageAccounting.enabled }}
apiVersion: batch/v1
kind: CronJob
metadata:
name: {{ include "buzz.fullname" . }}-storage-accounting
labels:
{{- include "buzz.labels" . | nindent 4 }}
app.kubernetes.io/component: storage-accounting
spec:
schedule: {{ .Values.storageAccounting.schedule | quote }}
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: {{ .Values.storageAccounting.successfulJobsHistoryLimit }}
failedJobsHistoryLimit: {{ .Values.storageAccounting.failedJobsHistoryLimit }}
jobTemplate:
spec:
activeDeadlineSeconds: {{ .Values.storageAccounting.activeDeadlineSeconds }}
backoffLimit: 0
template:
metadata:
labels:
{{- include "buzz.selectorLabels" . | nindent 12 }}
app.kubernetes.io/component: storage-accounting
{{- with .Values.storageAccounting.podLabels }}
{{- toYaml . | nindent 12 }}
{{- end }}
annotations:
{{- toYaml .Values.storageAccounting.podAnnotations | nindent 12 }}
spec:
restartPolicy: Never
serviceAccountName: {{ default (include "buzz.serviceAccountName" .) .Values.storageAccounting.serviceAccountName }}
securityContext:
{{- toYaml .Values.relay.securityContext | nindent 12 }}
{{- with .Values.image.pullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 12 }}
{{- end }}
containers:
- name: storage-accounting
image: {{ include "buzz.image" . }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
securityContext:
{{- omit .Values.relay.containerSecurityContext "readOnlyRootFilesystem" | toYaml | nindent 16 }}
readOnlyRootFilesystem: true
command: ["/usr/local/bin/buzz-admin"]
args: ["storage-snapshot", "--max-objects", {{ .Values.storageAccounting.maxObjects | int64 | quote }}]
env:
{{- $s3Endpoint := include "buzz.s3Endpoint" . }}
{{- if $s3Endpoint }}
- { name: BUZZ_S3_ENDPOINT, value: {{ $s3Endpoint | quote }} }
{{- end }}
- { name: BUZZ_S3_BUCKET, value: {{ .Values.s3.bucket | quote }} }
- { name: BUZZ_S3_REGION, value: {{ .Values.s3.region | quote }} }
- { name: BUZZ_S3_ADDRESSING_STYLE, value: {{ .Values.s3.addressingStyle | quote }} }
- { name: BUZZ_STORAGE_SNAPSHOT_CODE_SHA, value: {{ include "buzz.imageRevision" . | quote }} }
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: {{ include "buzz.envSecretName" . }}
key: DATABASE_URL
- name: BUZZ_S3_ACCESS_KEY
valueFrom:
secretKeyRef:
name: {{ include "buzz.envSecretName" . }}
key: BUZZ_S3_ACCESS_KEY
optional: true
- name: BUZZ_S3_SECRET_KEY
valueFrom:
secretKeyRef:
name: {{ include "buzz.envSecretName" . }}
key: BUZZ_S3_SECRET_KEY
optional: true
resources:
{{- toYaml .Values.storageAccounting.resources | nindent 16 }}
{{- end }}

139
deploy/charts/buzz/tests/storage_accounting_test.yaml Normal file
View File

@@ -0,0 +1,139 @@
suite: isolated storage accounting
templates:
- templates/storage-accounting-cronjob.yaml
- templates/deployment.yaml
- templates/secret-chart.yaml
tests:
- it: renders no worker by default
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
externalRedis.url: redis://h:6379
s3.endpoint: https://s3.example.com
s3.accessKey: test
s3.secretKey: test
asserts:
- hasDocuments:
count: 0
template: templates/storage-accounting-cronjob.yaml
- it: renders a non-overlapping worker with an isolated cap
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
externalRedis.url: redis://h:6379
s3.endpoint: https://s3.example.com
s3.bucket: buzz-media-example
s3.region: us-west-2
s3.addressingStyle: virtual
s3.accessKey: test
s3.secretKey: test
storageAccounting:
enabled: true
relayMode: external
maxObjects: 10000000
activeDeadlineSeconds: 3600
serviceAccountName: buzz-storage-accounting
podLabels:
tags.datadoghq.com/env: production
tags.datadoghq.com/service: buzz-storage-accounting
tags.datadoghq.com/version: test-sha
asserts:
- hasDocuments:
count: 1
template: templates/storage-accounting-cronjob.yaml
- equal:
path: kind
value: CronJob
template: templates/storage-accounting-cronjob.yaml
- equal:
path: spec.concurrencyPolicy
value: Forbid
template: templates/storage-accounting-cronjob.yaml
- equal:
path: spec.jobTemplate.spec.backoffLimit
value: 0
template: templates/storage-accounting-cronjob.yaml
- equal:
path: spec.jobTemplate.spec.activeDeadlineSeconds
value: 3600
template: templates/storage-accounting-cronjob.yaml
- equal:
path: spec.jobTemplate.spec.template.spec.restartPolicy
value: Never
template: templates/storage-accounting-cronjob.yaml
- equal:
path: spec.jobTemplate.spec.template.spec.serviceAccountName
value: buzz-storage-accounting
template: templates/storage-accounting-cronjob.yaml
- equal:
path: spec.jobTemplate.spec.template.spec.containers[0].securityContext.readOnlyRootFilesystem
value: true
template: templates/storage-accounting-cronjob.yaml
- equal:
path: spec.jobTemplate.spec.template.metadata.annotations["sidecar.istio.io/inject"]
value: "false"
template: templates/storage-accounting-cronjob.yaml
- equal:
path: spec.jobTemplate.spec.template.metadata.labels["tags.datadoghq.com/service"]
value: buzz-storage-accounting
template: templates/storage-accounting-cronjob.yaml
- equal:
path: spec.jobTemplate.spec.template.metadata.labels["tags.datadoghq.com/env"]
value: production
template: templates/storage-accounting-cronjob.yaml
- equal:
path: spec.jobTemplate.spec.template.metadata.labels["tags.datadoghq.com/version"]
value: test-sha
template: templates/storage-accounting-cronjob.yaml
- equal:
path: spec.jobTemplate.spec.template.spec.containers[0].args
value: ["storage-snapshot", "--max-objects", "10000000"]
template: templates/storage-accounting-cronjob.yaml
- equal:
path: spec.jobTemplate.spec.template.spec.containers[0].resources.requests.memory
value: 20Gi
template: templates/storage-accounting-cronjob.yaml
- equal:
path: spec.jobTemplate.spec.template.spec.containers[0].resources.limits.memory
value: 20Gi
template: templates/storage-accounting-cronjob.yaml
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_STORAGE_SNAPSHOT_CODE_SHA
value: 0.1.0
template: templates/storage-accounting-cronjob.yaml
- contains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_S3_BUCKET
value: buzz-media-example
template: templates/storage-accounting-cronjob.yaml
- lengthEqual:
path: spec.jobTemplate.spec.template.spec.containers[0].env
count: 8
template: templates/storage-accounting-cronjob.yaml
- contains:
path: spec.template.spec.containers[0].env
content:
name: BUZZ_STORAGE_METRICS
value: external
template: templates/deployment.yaml
- notContains:
path: spec.template.spec.containers[0].env
content:
name: BUZZ_STORAGE_SWEEP_MAX_OBJECTS
value: "10000000"
template: templates/deployment.yaml
- notContains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_RELAY_PRIVATE_KEY
template: templates/storage-accounting-cronjob.yaml
- notContains:
path: spec.jobTemplate.spec.template.spec.containers[0].env
content:
name: BUZZ_GIT_HOOK_HMAC_SECRET
template: templates/storage-accounting-cronjob.yaml

17
deploy/charts/buzz/values.schema.json
View File

@@ -217,6 +217,23 @@
"secretKey": { "type": "string" }
}
},
"storageAccounting": {
"type": "object",
"additionalProperties": false,
"properties": {
"enabled": { "type": "boolean" },
"schedule": { "type": "string", "minLength": 1 },
"maxObjects": { "type": "integer", "minimum": 1 },
"activeDeadlineSeconds": { "type": "integer", "minimum": 1 },
"successfulJobsHistoryLimit": { "type": "integer", "minimum": 0 },
"failedJobsHistoryLimit": { "type": "integer", "minimum": 0 },
"relayMode": { "type": "string", "enum": ["inline", "external", "off"] },
"serviceAccountName": { "type": "string" },
"podLabels": { "type": "object" },
"podAnnotations": { "type": "object" },
"resources": { "type": "object" }
}
},
"minio": {
"type": "object",
"additionalProperties": false,

47
deploy/charts/buzz/values.yaml
View File

@@ -328,13 +328,20 @@ externalRedis:
# MinIO Deployment, creates the bucket via a post-install Job, and composes
# the endpoint + autogenerated credentials automatically.
#
# Storage metrics (hourly bucket sweep, BUZZ_STORAGE_METRICS — see env docs):
# the credentials above must additionally grant `s3:ListBucket` on the bucket
# ARN itself (bucket-level; distinct from the object-level GetObject/
# PutObject/DeleteObject perms already required for media). Without it the
# first sweep fails AccessDenied and buzz_storage_sweep_ok stays 0 — no other
# media functionality is affected. Set BUZZ_STORAGE_METRICS=off to disable
# the sweep entirely on a deployment that can't grant it.
# Storage metrics are controlled by storageAccounting.relayMode below:
# - inline (default) keeps the relay-local hourly bucket sweep.
# - external reads the latest complete snapshot written by the CronJob.
# - off disables storage metrics in the relay.
# storageAccounting.enabled controls the CronJob independently, so operators
# can stage the worker before switching the relay to external mode. Inline and
# worker scans require `s3:ListBucket` on the bucket ARN itself (bucket-level;
# distinct from the object-level GetObject/PutObject/DeleteObject permissions
# already required for media). Without it the scan fails AccessDenied and no
# other media functionality is affected. External mode emits
# buzz_storage_snapshot_{age_seconds,duration_seconds,max_objects,
# cap_utilization,load_ok}; it does not infer worker-attempt health from the
# last successful snapshot. Monitor CronJob/Job status for worker failures,
# including deadline expiry and OOM termination.
#
# Whole-community deletion (`buzz-admin deletions ...`) permanently removes
# tenant-owned object versions through the v5 deletion path. In addition to the
@@ -344,8 +351,8 @@ externalRedis:
# never-versioned buckets, which S3 exposes through `ListObjectVersions` with the
# `null` version id.
#
# Note: buzz_storage_sweep_failures is a process-local gauge — on leader
# failover it resets to the new leader's local count, not a global total.
# Note: inline-only buzz_storage_sweep_failures is a process-local gauge — on
# leader failover it resets to the new leader's local count, not a global total.
# Note: on a failed sweep attempt, the next retry fires on the next usage tick
# (default 300 s BUZZ_USAGE_METRICS_INTERVAL_SECS), not at sweep-interval
# cadence — so a permanently missing s3:ListBucket yields one cheap LIST call
@@ -363,6 +370,28 @@ s3:
accessKey: ""
secretKey: ""
# Isolated S3 accounting worker. Disabled by default, so chart upgrades retain
# the existing relay-local sweep until an operator enables and sizes the job.
storageAccounting:
enabled: false
schedule: "0 3 * * *"
maxObjects: 10000000
activeDeadlineSeconds: 3600
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
relayMode: inline # inline | external | off
serviceAccountName: "" # defaults to the main Buzz service account
podLabels: {}
podAnnotations:
sidecar.istio.io/inject: "false"
resources:
requests:
cpu: "1"
memory: 20Gi
limits:
cpu: "1"
memory: 20Gi
# In-cluster MinIO for the quickstart profile only. Production deploys leave
# this disabled and use s3.* (or secrets.existingSecret) against managed S3.
minio: