mirror of
https://github.com/block/buzz.git
synced 2026-11-01 23:11:58 +00:00
e14fff74d0
## Problem On SIGTERM the relay sends every live WebSocket a **1012 Service Restart** close frame via `ConnectionManager::drain_all()` — all in the same instant (`main.rs` shutdown task → `state.rs::drain_all`). On a pod holding thousands of sessions, that makes every client reconnect simultaneously: the thundering-herd reconnect behind the DB pool-timeout bursts observed on each rolling deploy. Client-side jitter can't fix this — the desktop client *resets* its backoff to base on a 1012 and reconnects with only ±25% jitter (`relayClientSession.ts`), so the spread has to come from the server. ## Change Add `BUZZ_DRAIN_JITTER_MS` (default `0` = unchanged behavior). The two paths are kept **deliberately separate** so the default is byte-for-byte the previously shipped shutdown: - **Jitter off (`0`/unset, the default):** the original synchronous, all-at-once `drain_all()` runs unchanged — queue the 1012 on each connection's control channel, cancel, return. No new machinery on the default path. - **Jitter on (`> 0`):** a separate async `drain_all_jittered(jitter_ms)` spreads each connection's restart close over an independent uniform delay in **`[1, jitter_ms]`**. Each delayed close travels a dedicated `RestartClose` channel; the writer flushes the 1012 frame and **acknowledges the flush over a oneshot**, so drain waits for confirmed delivery (up to `RESTART_CLOSE_ACK_TIMEOUT` = 5s) rather than assuming it, falling back to cancellation if the channel is full/closed or the ack times out. The drain future is **owned and awaited** by the shutdown task, and the 30s hard-drain backstop is aborted only after a clean drain — so a clean roll exits `0`. The two methods can be unified and the old one dropped later once the jittered path is proven for all cases. - **`config.rs`** — `drain_jitter_ms`: non-negative parse, clamped to `MAX_DRAIN_JITTER_MS` = **20s** (leaving 10s of the 30s budget for flush). Junk fails loudly at startup; **empty/whitespace-only is treated as unset (jitter off)** so a `BUZZ_DRAIN_JITTER_MS=""` kill switch does not crashloop the relay (matches the sibling env vars in this file). - **`state.rs`** — `drain_all()` (unchanged synchronous default) + `drain_all_jittered()` (jittered + flush-ack). Both set the sticky `draining` flag before the first await. A registration that lands mid-shutdown always self-signals via the **immediate** control-frame + cancel path — jitter smears already-established sockets, not late arrivals. - **`main.rs`** — shutdown task dispatches: `drain_jitter_ms == 0` → `drain_all()`, else `drain_all_jittered(...).await`. ## Safety - **Default off is the currently-committed path.** With jitter unset/0 the shutdown runs the original synchronous `drain_all()` — no restart channel, no ack wait. Safe to deploy dark and dial up. - **Shutdown-boundary race preserved.** Sticky flag set before any await; a late registration self-signals its close with no jitter. - **Owned + backstopped.** The jittered drain future is awaited; the 30s hard-drain `process::exit(1)` remains the ceiling. `MAX_DRAIN_JITTER_MS` (20s) + `RESTART_CLOSE_ACK_TIMEOUT` (5s) = 25s, inside the 30s budget; 5s pre-sleep + 25s = 30s against `terminationGracePeriodSeconds: 60`. ## Known behavior to note (not a blocker, flagged from review) On a **successful** flush the jittered path deliberately does not cancel the connection token — teardown then depends on the client echoing our Close, or on process exit. Compliant clients echo; a silent client rides to the 30s hard exit. The default (jitter-off) path cancels deterministically as before. ## Tests - `config::tests::drain_jitter_defaults_off_and_rejects_junk` — default off, `20000`, clamp `60000`→`20000`, explicit `0`, junk `"soon"` fails, **empty `""` and whitespace-only treated as off**. - `state::tests::drain_all_is_immediate` — default path queues frame + cancels synchronously. - `state::tests::drain_all_sends_restart_close_and_cancels_every_conn`, `drain_all_full_control_buffer_still_cancels`, `register_after_drain_self_signals_restart_close_and_cancel`. - `state::tests::drain_all_jittered_defers_close_until_within_jitter_window` (paused time). - `state::tests::drain_all_jittered_waits_for_writer_acknowledgement_without_cancelling`. - `state::tests::drain_all_jittered_cancels_when_restart_channel_is_full_or_closed`. - `state::tests::drain_all_jittered_cancels_when_flush_ack_times_out` (paused time — the 5s ack-timeout fallback). Validation at `46c690940`: `cargo fmt -p buzz-relay --check`, `cargo clippy -p buzz-relay --all-targets -- -D warnings`, and the drain/config unit suite all clean. Local live SIGTERM test with a real relay process + 200 NIP-42-authenticated sockets — see the PR comment for the before/after distribution and exit codes. ## Rollout Ship with default `0`, then set `BUZZ_DRAIN_JITTER_MS` (e.g. 10000–20000) on bb-block first, watch the roll-window pool-timeout metric, then bb-public. `""` is a safe kill switch. Complements the preStop `sleep` (stops routing before close). --------- Signed-off-by: npub1srl70fhzyu3fsnahl06vw2czvqc2w3ds37hyzvjnk8ve8f03ngcqg9le2w <80ffe7a6e22722984fb7fbf4c72b026030a745b08fae413253b1d993a5f19a30@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: Brad Seiler <seiler@squareup.com> Co-authored-by: npub1srl70fhzyu3fsnahl06vw2czvqc2w3ds37hyzvjnk8ve8f03ngcqg9le2w <80ffe7a6e22722984fb7fbf4c72b026030a745b08fae413253b1d993a5f19a30@buzz.block.builderlab.xyz> Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
303 lines
11 KiB
JSON
303 lines
11 KiB
JSON
{
|
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
"title": "Buzz Helm chart values",
|
|
"description": "Schema for values.yaml. Catches misconfiguration at `helm install` time before _validate.tpl runtime fails. Both layers are intentional: schema rejects malformed inputs; templates reject inconsistent combinations.",
|
|
"type": "object",
|
|
"additionalProperties": true,
|
|
"properties": {
|
|
"quickstart": {
|
|
"type": "boolean",
|
|
"description": "Master toggle for evaluation profile (enables postgresql + redis subcharts and chart-side autogen). Not GitOps-safe."
|
|
},
|
|
"image": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"repository": { "type": "string", "minLength": 1 },
|
|
"tag": { "type": "string" },
|
|
"pullPolicy": { "type": "string", "enum": ["Always", "IfNotPresent", "Never"] },
|
|
"pullSecrets": {
|
|
"type": "array",
|
|
"items": { "type": "object", "required": ["name"], "properties": { "name": { "type": "string" } } }
|
|
}
|
|
},
|
|
"required": ["repository", "pullPolicy"]
|
|
},
|
|
"replicaCount": {
|
|
"type": "integer",
|
|
"minimum": 1,
|
|
"description": "Replica count for the relay Deployment. replicaCount > 1 requires Redis (for buzz-pubsub) — enforced by _validate.tpl. Git storage does NOT need ReadWriteMany: git state is object-store-backed and repo names live in Postgres, so ReadWriteOnce is fine per replica."
|
|
},
|
|
"relayUrl": {
|
|
"type": "string",
|
|
"pattern": "^(wss?://.+)?$",
|
|
"description": "Public wss:// URL clients connect to. Required (validated by _validate.tpl). Drives RELAY_URL, default mediaBaseUrl, default ingress host."
|
|
},
|
|
"mediaBaseUrl": {
|
|
"type": "string",
|
|
"pattern": "^(https?://.+)?$"
|
|
},
|
|
"ownerPubkey": {
|
|
"type": "string",
|
|
"pattern": "^([0-9a-f]{64})?$",
|
|
"description": "64-char lowercase hex Nostr pubkey of the relay operator. Required when relay.requireRelayMembership=true."
|
|
},
|
|
"secrets": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"existingSecret": { "type": "string", "description": "Name of an externally managed Secret. Production / GitOps path." },
|
|
"relayPrivateKey": { "type": "string" },
|
|
"gitHookHmacSecret": { "type": "string" }
|
|
}
|
|
},
|
|
"relay": {
|
|
"type": "object",
|
|
"additionalProperties": true,
|
|
"properties": {
|
|
"bindAddr": { "type": "string", "minLength": 1 },
|
|
"maxConnections": { "type": "integer", "minimum": 1 },
|
|
"maxConcurrentHandlers": { "type": "integer", "minimum": 1 },
|
|
"sendBuffer": { "type": "integer", "minimum": 1 },
|
|
"drainJitterMs": { "type": "integer", "minimum": 0 },
|
|
"requireAuthToken": { "type": "boolean" },
|
|
"requireRelayMembership": { "type": "boolean" },
|
|
"requireMediaGetAuth": { "type": "boolean" },
|
|
"allowNipOaAuth": { "type": "boolean" },
|
|
"huddleAudioAvailable": {
|
|
"type": ["boolean", "null"],
|
|
"description": "Whether relay huddle audio is available. null means chart default: true for one replica, false for replicaCount > 1. Explicit true with replicaCount > 1 means the operator accepts/owns external multi-pod audio/SFU behavior."
|
|
},
|
|
"pubkeyAllowlist": { "type": "boolean" },
|
|
"corsOrigins": {
|
|
"type": "array",
|
|
"items": { "type": "string" }
|
|
},
|
|
"ephemeralTtlOverride": { "type": "integer", "minimum": 0 },
|
|
"command": {
|
|
"type": "array",
|
|
"items": { "type": "string" },
|
|
"description": "Optional relay container entrypoint override. Empty preserves the image default."
|
|
},
|
|
"args": {
|
|
"type": "array",
|
|
"items": { "type": "string" },
|
|
"description": "Optional relay container arguments override. Empty preserves the image default."
|
|
},
|
|
"extraVolumeMounts": {
|
|
"type": "array",
|
|
"items": { "type": "object" },
|
|
"description": "Raw Kubernetes volumeMount fragments appended to the relay container."
|
|
}
|
|
}
|
|
},
|
|
"extraInitContainers": {
|
|
"type": "array",
|
|
"items": { "type": "object" },
|
|
"description": "Raw Kubernetes init-container fragments appended to the relay Pod."
|
|
},
|
|
"extraVolumes": {
|
|
"type": "array",
|
|
"items": { "type": "object" },
|
|
"description": "Raw Kubernetes volume fragments appended to the relay Pod."
|
|
},
|
|
"service": {
|
|
"type": "object",
|
|
"additionalProperties": true,
|
|
"properties": {
|
|
"type": { "type": "string", "enum": ["ClusterIP", "NodePort", "LoadBalancer"] },
|
|
"port": { "type": "integer", "minimum": 1, "maximum": 65535 },
|
|
"healthPort": { "type": "integer", "minimum": 1, "maximum": 65535 },
|
|
"metricsPort": { "type": "integer", "minimum": 1, "maximum": 65535 }
|
|
}
|
|
},
|
|
"serviceAccount": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"create": { "type": "boolean" },
|
|
"name": { "type": "string" },
|
|
"annotations": { "type": "object" }
|
|
}
|
|
},
|
|
"podDisruptionBudget": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"enabled": { "type": "boolean" },
|
|
"minAvailable": { "oneOf": [{ "type": "integer" }, { "type": "string" }] },
|
|
"maxUnavailable": { "oneOf": [{ "type": "integer" }, { "type": "string" }] }
|
|
}
|
|
},
|
|
"ingress": {
|
|
"type": "object",
|
|
"additionalProperties": true,
|
|
"properties": {
|
|
"enabled": { "type": "boolean" },
|
|
"className": { "type": "string" },
|
|
"annotations": { "type": "object" },
|
|
"hosts": { "type": "array" },
|
|
"tls": { "type": "array" }
|
|
}
|
|
},
|
|
"httproute": {
|
|
"type": "object",
|
|
"additionalProperties": true,
|
|
"properties": {
|
|
"enabled": { "type": "boolean" },
|
|
"parentRefs": { "type": "array" },
|
|
"hostnames": { "type": "array", "items": { "type": "string" } },
|
|
"rules": { "type": "array" }
|
|
}
|
|
},
|
|
"persistence": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"git": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"enabled": { "type": "boolean" },
|
|
"mountPath": { "type": "string", "minLength": 1 },
|
|
"storageClass": { "type": "string" },
|
|
"accessMode": { "type": "string", "enum": ["ReadWriteOnce", "ReadWriteMany", "ReadOnlyMany", "ReadWriteOncePod"] },
|
|
"size": { "type": "string", "pattern": "^[0-9]+(\\.[0-9]+)?(E|P|T|G|M|K|Ei|Pi|Ti|Gi|Mi|Ki)?$" },
|
|
"annotations": { "type": "object" },
|
|
"existingClaim": { "type": "string" }
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"postgresql": {
|
|
"type": "object",
|
|
"additionalProperties": true,
|
|
"properties": { "enabled": { "type": "boolean" } }
|
|
},
|
|
"externalPostgresql": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"url": { "type": "string", "pattern": "^(postgres(ql)?://.+)?$" }
|
|
}
|
|
},
|
|
"redis": {
|
|
"type": "object",
|
|
"additionalProperties": true,
|
|
"properties": { "enabled": { "type": "boolean" } }
|
|
},
|
|
"externalRedis": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"url": { "type": "string", "pattern": "^(rediss?://.+)?$" }
|
|
}
|
|
},
|
|
"s3": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"endpoint": { "type": "string", "pattern": "^(https?://.+)?$" },
|
|
"bucket": { "type": "string", "minLength": 1 },
|
|
"region": {
|
|
"type": "string",
|
|
"description": "Optional S3 region used for SigV4 signing. When empty, BUZZ_S3_REGION is omitted so the relay can use AWS_REGION or its own default."
|
|
},
|
|
"addressingStyle": {
|
|
"type": "string",
|
|
"enum": ["path", "virtual"],
|
|
"description": "S3 URL style shared by media and Git/CAS clients. Defaults to path for bundled MinIO compatibility."
|
|
},
|
|
"accessKey": { "type": "string" },
|
|
"secretKey": { "type": "string" }
|
|
}
|
|
},
|
|
"minio": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"enabled": { "type": "boolean" },
|
|
"image": { "type": "string", "minLength": 1 },
|
|
"mcImage": { "type": "string", "minLength": 1 },
|
|
"persistence": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"enabled": { "type": "boolean" },
|
|
"size": { "type": "string", "pattern": "^[0-9]+(\\.[0-9]+)?(E|P|T|G|M|K|Ei|Pi|Ti|Gi|Mi|Ki)?$" }
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"git": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"maxPackBytes": { "type": "integer", "minimum": 1 },
|
|
"packCachePath": { "type": "string", "minLength": 1 },
|
|
"packCacheMaxBytes": { "type": "integer", "minimum": 0 },
|
|
"packCacheMaxConcurrentPopulations": { "type": "integer", "minimum": 1 },
|
|
"packCacheVolumeSize": { "type": "string", "pattern": "^[0-9]+(\\.[0-9]+)?(E|P|T|G|M|K|Ei|Pi|Ti|Gi|Mi|Ki)?$" },
|
|
"maxReposPerPubkey": { "type": "integer", "minimum": 1 },
|
|
"maxConcurrentOps": { "type": "integer", "minimum": 1 }
|
|
}
|
|
},
|
|
"migrate": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"autoMigrate": { "type": "boolean" },
|
|
"preUpgradeJob": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"enabled": { "type": "boolean" },
|
|
"resources": { "type": "object" },
|
|
"backoffLimit": { "type": "integer", "minimum": 0 },
|
|
"activeDeadlineSeconds": { "type": "integer", "minimum": 1 }
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"serviceMonitor": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"enabled": { "type": "boolean" },
|
|
"namespace": { "type": "string" },
|
|
"interval": { "type": "string" },
|
|
"scrapeTimeout": { "type": "string" },
|
|
"labels": { "type": "object" }
|
|
}
|
|
},
|
|
"pairingRelay": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"enabled": { "type": "boolean" },
|
|
"url": {
|
|
"type": "string",
|
|
"pattern": "^(wss?://.+)?$",
|
|
"description": "Public WebSocket URL advertised to clients for NIP-AB device pairing."
|
|
},
|
|
"replicaCount": { "type": "integer", "minimum": 1 },
|
|
"service": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"type": { "type": "string", "enum": ["ClusterIP", "NodePort", "LoadBalancer"] },
|
|
"port": { "type": "integer", "minimum": 1, "maximum": 65535 },
|
|
"annotations": { "type": "object" }
|
|
}
|
|
},
|
|
"podAnnotations": { "type": "object" },
|
|
"podLabels": { "type": "object" },
|
|
"resources": { "type": "object" }
|
|
}
|
|
},
|
|
"extraManifests": {
|
|
"type": "array"
|
|
}
|
|
}
|
|
}
|