diff --git a/deploy/charts/buzz/Chart.yaml b/deploy/charts/buzz/Chart.yaml index 49a6fafd19..fcd4582253 100644 --- a/deploy/charts/buzz/Chart.yaml +++ b/deploy/charts/buzz/Chart.yaml @@ -7,7 +7,7 @@ description: | PostgreSQL and Redis. Configurable for single-node evaluation (subcharts on) and HA production (external services, existingSecret). type: application -version: 0.1.8 +version: 0.1.9 appVersion: "0.1.0" home: https://github.com/block/buzz sources: @@ -24,7 +24,7 @@ maintainers: annotations: artifacthub.io/changes: | - kind: added - description: Optional immutable relay image digest pinning with backwards-compatible tag fallback. + description: Opt-in fixed-replica Argo Rollouts with explicit handover counts and manual canary pauses. artifacthub.io/license: Apache-2.0 # Optional eval-only subcharts. Production deploys disable both and point diff --git a/deploy/charts/buzz/README.md b/deploy/charts/buzz/README.md index 91a2b59ea3..aa7985bab9 100644 --- a/deploy/charts/buzz/README.md +++ b/deploy/charts/buzz/README.md @@ -52,6 +52,7 @@ security boundaries. See: +- [`docs/argo-rollouts.md`](docs/argo-rollouts.md) — optional manual canary updates and Deployment handover - [`examples/argocd-app.yaml`](examples/argocd-app.yaml) — ArgoCD Application - [`examples/flux-helmrelease.yaml`](examples/flux-helmrelease.yaml) — Flux HelmRelease v2 - [`examples/secret-sample.yaml`](examples/secret-sample.yaml) — Secret schema diff --git a/deploy/charts/buzz/docs/argo-rollouts.md b/deploy/charts/buzz/docs/argo-rollouts.md new file mode 100644 index 0000000000..d114e4e3e1 --- /dev/null +++ b/deploy/charts/buzz/docs/argo-rollouts.md @@ -0,0 +1,114 @@ +# Optional Argo Rollouts + +The chart uses a Kubernetes Deployment by default. Setting `rollout.enabled: true` +adds an Argo `Rollout` in the same namespace. Install the Argo Rollouts controller +and its CRDs separately before enabling it. This configuration targets the +`argoproj.io/v1alpha1` API used by Argo Rollouts v1.8.3. + +The Deployment still contains the image, environment variables, probes, volumes, +and other pod settings. The Rollout reads those settings through `workloadRef` +and creates **new pods of its own**. It does not adopt the Deployment's pods. +Both sets use the existing Service and PodDisruptionBudget selectors. + +This first version supports fixed replica counts and manual promotion. It does +not add health analyses, automatic abort decisions, traffic routing, or HPA +support. `autoscaling.enabled: true` is rejected while Rollouts are enabled. +Existing installations with Rollouts disabled retain their Deployment/HPA path. + +## First handover: two replicas + +Keep the relay image, secrets, and pod settings unchanged throughout handover. +Use an immutable image digest. Confirm that both existing pods are healthy and +that the cluster has room for one extra pod. Redis must be configured for the +overlapping pods; all pods must use the same external database, Redis, and storage. + +Start with this addition to your existing production values: + +```yaml +replicaCount: 2 +autoscaling: + enabled: false +rollout: + enabled: true + deploymentReplicas: 2 + replicas: 0 + minReadySeconds: 60 + progressDeadlineSeconds: 900 + weights: [50, 100] +``` + +The two replica fields are intentionally required. Enabling the feature alone +fails rendering instead of silently scaling down the Deployment. `replicaCount` +remains the desired fleet size; the two explicit counts divide that fleet +between controllers during the handover. + +Apply **one row at a time** through your normal Helm or GitOps process: + +| Stage | `rollout.deploymentReplicas` | `rollout.replicas` | Action | +|---|---:|---:|---| +| Prepare | 2 | 0 | Create the Rollout with no requested pods. | +| Add first pod | 2 | 1 | Wait for the new Rollout pod to be healthy. | +| Drain first old pod | 1 | 1 | Wait for client reconnections and recovery. | +| Add second pod | 1 | 2 | Wait for the second Rollout pod to be healthy. | +| Finish | 0 | 2 | Drain the last Deployment pod and check recovery. | + +Before every drain, verify the replacement has been Ready for at least 60 seconds, +is serving through the Service, and the fleet has recovered: connections, +authenticated sessions, subscriptions, error rates, and database load. Before +each next row, wait for the previous change to finish, including pod termination. +If recovery is incomplete, stop and investigate. Desired counts alone do not prove +healthy capacity. The chart checks only that the desired counts total two or +three and that neither controller requests more than two. + +Argo skips canary steps when creating the first revision. **The pauses do not +protect this handover.** The explicit replica changes and checks above do. +`workloadRef.scaleDown: never` ensures Argo does not also scale the Deployment; +your values remain its replica source of truth. + +Keep the zero-replica Deployment after handover. Argo still needs its pod +configuration. Argo CD continues applying both the Deployment and the Rollout. +There is no switch that tells Argo CD to stop managing the Deployment. + +## Later image updates + +Leave the counts at Deployment `0`, Rollout `2`, and change the usual `image.tag` +or `image.digest`. Argo observes the changed Deployment template and manages the +new revision. With `[50, 100]`, the generated steps are: + +```yaml +- pause: {} +- setWeight: 50 +- pause: {} +- setWeight: 100 +- pause: {} +``` + +Each pause requires an operator to inspect health and promote one step using the +Argo Rollouts UI or `kubectl argo rollouts promote `. Do not use +`--full` to bypass the checks. The controller allows one surge pod and requests +zero unavailable pods; this does not guarantee every client has reconnected. +For larger fleets, choose smaller weights before enabling the feature. + +The weights approximate the share of Rollout pods running the new revision. +They do not split WebSocket traffic by percentage or move existing connections. +Pauses also do not stop ordinary scaling, pod replacement after failures, or a +direct change to the replica counts. + +## Returning to a Deployment + +Do not disable Rollouts while they still own serving pods: removing the Rollout +can delete its pods before the Deployment has replaced them. Freeze image and +pod settings, then reverse the table above. Add and verify one Deployment pod +before draining a Rollout pod. Once the Deployment has two healthy pods and the +Rollout has zero remaining pods, set `rollout.enabled: false` and remove the +unused rollout overrides. Leave `replicaCount: 2`. + +## What this does not prove + +This chart is a prerequisite for a staged migration, not an automatic recovery +system. Enabling it with nonzero replicas affects real pods and traffic. It is +not a report-only or shadow mode. Observation-only health analyses and enforced +recovery checks require a separate change and validation before activation. + +See the upstream [migration guide](https://argo-rollouts.readthedocs.io/en/stable/migrating/) +and [canary behavior](https://argo-rollouts.readthedocs.io/en/stable/features/canary/). diff --git a/deploy/charts/buzz/templates/NOTES.txt b/deploy/charts/buzz/templates/NOTES.txt index a0dd96a1a4..6347fc0603 100644 --- a/deploy/charts/buzz/templates/NOTES.txt +++ b/deploy/charts/buzz/templates/NOTES.txt @@ -90,4 +90,11 @@ ────────────────────────────────────────────────────────────────────────────── kubectl -n {{ .Release.Namespace }} get pods -l app.kubernetes.io/instance={{ .Release.Name }} kubectl -n {{ .Release.Namespace }} logs -l app.kubernetes.io/instance={{ .Release.Name }} --tail=200 +{{- $rollout := .Values.rollout | default dict }} +{{- if $rollout.enabled }} + kubectl -n {{ .Release.Namespace }} get deployment,rollout {{ include "buzz.fullname" . }} + Argo Rollouts is enabled. The Deployment supplies the pod template. + Updates pause for manual promotion. See docs/argo-rollouts.md before handover. +{{- else }} kubectl -n {{ .Release.Namespace }} rollout status deployment/{{ include "buzz.fullname" . }} +{{- end }} diff --git a/deploy/charts/buzz/templates/_rollout.tpl b/deploy/charts/buzz/templates/_rollout.tpl new file mode 100644 index 0000000000..929cd36a22 --- /dev/null +++ b/deploy/charts/buzz/templates/_rollout.tpl @@ -0,0 +1,38 @@ +{{/* Missing rollout values must remain compatible with --reuse-values. */}} +{{- define "buzz.validateRollout" -}} +{{- $rollout := .Values.rollout | default dict -}} +{{- if $rollout.enabled -}} + {{- if .Values.autoscaling.enabled -}} + {{- fail "rollout.enabled requires autoscaling.enabled=false; this chart currently supports fixed-replica Rollouts only" -}} + {{- end -}} + {{- if or (not (hasKey $rollout "deploymentReplicas")) (eq $rollout.deploymentReplicas nil) (not (hasKey $rollout "replicas")) (eq $rollout.replicas nil) -}} + {{- fail "rollout.enabled requires explicit rollout.deploymentReplicas and rollout.replicas; start with replicaCount and 0 for an existing Deployment" -}} + {{- end -}} + {{- $deployment := int $rollout.deploymentReplicas -}} + {{- $replicas := int $rollout.replicas -}} + {{- $target := int .Values.replicaCount -}} + {{- if or (lt $deployment 0) (lt $replicas 0) (gt $deployment $target) (gt $replicas $target) (lt (add $deployment $replicas) $target) (gt (add $deployment $replicas) (add $target 1)) -}} + {{- fail "rollout replica counts must each be between 0 and replicaCount, and their sum must be replicaCount or replicaCount + 1" -}} + {{- end -}} + {{- if and (not .Values.redis.enabled) (not .Values.externalRedis.url) (not .Values.secrets.existingSecret) -}} + {{- fail "rollout.enabled requires Redis for overlapping relay pods; configure redis.enabled, externalRedis.url, or secrets.existingSecret with REDIS_URL" -}} + {{- end -}} + {{- if not $rollout.weights -}} + {{- fail "rollout.weights must contain increasing percentages ending at 100" -}} + {{- end -}} + {{- if le (int ($rollout.progressDeadlineSeconds | default 900)) (int ($rollout.minReadySeconds | default 60)) -}} + {{- fail "rollout.progressDeadlineSeconds must exceed rollout.minReadySeconds" -}} + {{- end -}} + {{- $previous := 0 -}} + {{- range $rollout.weights -}} + {{- $weight := int . -}} + {{- if or (le $weight $previous) (gt $weight 100) -}} + {{- fail "rollout.weights must contain increasing percentages ending at 100" -}} + {{- end -}} + {{- $previous = $weight -}} + {{- end -}} + {{- if ne $previous 100 -}} + {{- fail "rollout.weights must contain increasing percentages ending at 100" -}} + {{- end -}} +{{- end -}} +{{- end -}} diff --git a/deploy/charts/buzz/templates/_validate.tpl b/deploy/charts/buzz/templates/_validate.tpl index aa7f7ac13c..1b95ae543d 100644 --- a/deploy/charts/buzz/templates/_validate.tpl +++ b/deploy/charts/buzz/templates/_validate.tpl @@ -5,6 +5,8 @@ surface at template time regardless of which manifest helm renders first. {{- define "buzz.validate" -}} +{{- include "buzz.validateRollout" . -}} + {{/* relayUrl is required */}} {{- if not .Values.relayUrl -}} {{- fail "relayUrl is required: set --set relayUrl=wss://your.domain" -}} diff --git a/deploy/charts/buzz/templates/deployment.yaml b/deploy/charts/buzz/templates/deployment.yaml index 319ec7f159..c6421d6500 100644 --- a/deploy/charts/buzz/templates/deployment.yaml +++ b/deploy/charts/buzz/templates/deployment.yaml @@ -1,4 +1,5 @@ {{- include "buzz.validate" . -}} +{{- $rollout := .Values.rollout | default dict -}} apiVersion: apps/v1 kind: Deployment metadata: @@ -6,7 +7,9 @@ metadata: labels: {{- include "buzz.labels" . | nindent 4 }} spec: - {{- if not .Values.autoscaling.enabled }} + {{- if $rollout.enabled }} + replicas: {{ $rollout.deploymentReplicas }} + {{- else if not .Values.autoscaling.enabled }} replicas: {{ .Values.replicaCount }} {{- end }} strategy: diff --git a/deploy/charts/buzz/templates/rollout.yaml b/deploy/charts/buzz/templates/rollout.yaml new file mode 100644 index 0000000000..68f524bfc0 --- /dev/null +++ b/deploy/charts/buzz/templates/rollout.yaml @@ -0,0 +1,33 @@ +{{- include "buzz.validate" . -}} +{{- $rollout := .Values.rollout | default dict -}} +{{- if $rollout.enabled }} +apiVersion: argoproj.io/v1alpha1 +kind: Rollout +metadata: + name: {{ include "buzz.fullname" . }} + labels: + {{- include "buzz.labels" . | nindent 4 }} +spec: + replicas: {{ $rollout.replicas }} + minReadySeconds: {{ $rollout.minReadySeconds | default 60 }} + progressDeadlineSeconds: {{ $rollout.progressDeadlineSeconds | default 900 }} + progressDeadlineAbort: false + selector: + matchLabels: + {{- include "buzz.relaySelectorLabels" . | nindent 6 }} + workloadRef: + apiVersion: apps/v1 + kind: Deployment + name: {{ include "buzz.fullname" . }} + scaleDown: never + strategy: + canary: + maxSurge: 1 + maxUnavailable: 0 + steps: + - pause: {} + {{- range $rollout.weights }} + - setWeight: {{ . }} + - pause: {} + {{- end }} +{{- end }} diff --git a/deploy/charts/buzz/tests/fixtures/rollout-values.yaml b/deploy/charts/buzz/tests/fixtures/rollout-values.yaml new file mode 100644 index 0000000000..66bb62cf27 --- /dev/null +++ b/deploy/charts/buzz/tests/fixtures/rollout-values.yaml @@ -0,0 +1,9 @@ +relayUrl: wss://buzz.example.com +ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000" +secrets: + existingSecret: buzz-secrets +replicaCount: 2 +rollout: + enabled: true + deploymentReplicas: 0 + replicas: 2 diff --git a/deploy/charts/buzz/tests/rollout_test.yaml b/deploy/charts/buzz/tests/rollout_test.yaml new file mode 100644 index 0000000000..67bcc2639e --- /dev/null +++ b/deploy/charts/buzz/tests/rollout_test.yaml @@ -0,0 +1,173 @@ +suite: opt-in relay Rollout +templates: + - templates/rollout.yaml + - templates/deployment.yaml + - templates/service.yaml + - templates/pdb.yaml + - templates/secret-chart.yaml +set: + relayUrl: wss://buzz.example.com + ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000" + secrets.existingSecret: buzz-secrets + replicaCount: 2 +tests: + - it: tolerates missing rollout values from an older release + set: + rollout: null + asserts: + - hasDocuments: + count: 0 + template: templates/rollout.yaml + - equal: + path: spec.replicas + value: 2 + template: templates/deployment.yaml + + - it: creates no Rollout by default + template: templates/rollout.yaml + asserts: + - hasDocuments: + count: 0 + + - it: keeps Deployment replicas and rolling update strategy by default + template: templates/deployment.yaml + asserts: + - equal: + path: spec.replicas + value: 2 + - equal: + path: spec.strategy + value: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 + + - it: permits preparing a zero-pod Rollout without draining the Deployment + set: + rollout.enabled: true + rollout.deploymentReplicas: 2 + rollout.replicas: 0 + asserts: + - equal: + path: spec.replicas + value: 0 + template: templates/rollout.yaml + - equal: + path: spec.replicas + value: 2 + template: templates/deployment.yaml + + - it: permits one extra pod during handover + set: + rollout.enabled: true + rollout.deploymentReplicas: 2 + rollout.replicas: 1 + asserts: + - equal: + path: spec.replicas + value: 1 + template: templates/rollout.yaml + - equal: + path: spec.replicas + value: 2 + template: templates/deployment.yaml + + - it: retains the Deployment pod configuration at zero replicas + template: templates/deployment.yaml + set: + rollout.enabled: true + rollout.deploymentReplicas: 0 + rollout.replicas: 2 + image.repository: ghcr.io/example/buzz + image.tag: test-image + asserts: + - equal: + path: spec.replicas + value: 0 + - equal: + path: spec.template.spec.containers[0].image + value: ghcr.io/example/buzz:test-image + - exists: + path: spec.template.spec.containers[0].readinessProbe + + - it: references the same Deployment name with manual canary pauses + template: templates/rollout.yaml + set: + fullnameOverride: custom-relay + rollout.enabled: true + rollout.deploymentReplicas: 0 + rollout.replicas: 2 + asserts: + - isAPIVersion: + of: argoproj.io/v1alpha1 + - isKind: + of: Rollout + - equal: + path: metadata.name + value: custom-relay + - equal: + path: spec.workloadRef + value: + apiVersion: apps/v1 + kind: Deployment + name: custom-relay + scaleDown: never + - notExists: + path: spec.template + - equal: + path: spec.minReadySeconds + value: 60 + - equal: + path: spec.progressDeadlineSeconds + value: 900 + - equal: + path: spec.progressDeadlineAbort + value: false + - equal: + path: spec.strategy.canary + value: + maxSurge: 1 + maxUnavailable: 0 + steps: + - pause: {} + - setWeight: 50 + - pause: {} + - setWeight: 100 + - pause: {} + - equal: + path: spec.selector.matchLabels + value: + app.kubernetes.io/name: buzz + app.kubernetes.io/instance: RELEASE-NAME + app.kubernetes.io/component: relay + + - it: keeps the Service selecting both sets of relay pods + template: templates/service.yaml + set: + rollout.enabled: true + rollout.deploymentReplicas: 1 + rollout.replicas: 1 + asserts: + - equal: + path: spec.selector + value: + app.kubernetes.io/name: buzz + app.kubernetes.io/instance: RELEASE-NAME + app.kubernetes.io/component: relay + + - it: keeps the PodDisruptionBudget selecting both sets of relay pods + template: templates/pdb.yaml + set: + rollout.enabled: true + rollout.deploymentReplicas: 1 + rollout.replicas: 2 + podDisruptionBudget.enabled: true + asserts: + - equal: + path: spec.selector.matchLabels + value: + app.kubernetes.io/name: buzz + app.kubernetes.io/instance: RELEASE-NAME + app.kubernetes.io/component: relay + diff --git a/deploy/charts/buzz/tests/rollout_validation_test.yaml b/deploy/charts/buzz/tests/rollout_validation_test.yaml new file mode 100644 index 0000000000..281b48c8f7 --- /dev/null +++ b/deploy/charts/buzz/tests/rollout_validation_test.yaml @@ -0,0 +1,89 @@ +suite: Rollout validation +templates: + - templates/rollout.yaml +set: + relayUrl: wss://buzz.example.com + ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000" + secrets.existingSecret: buzz-secrets + replicaCount: 2 +tests: + - it: rejects enabling without explicit handover counts + template: templates/rollout.yaml + set: + rollout.enabled: true + asserts: + - failedTemplate: + errorPattern: requires explicit rollout.deploymentReplicas and rollout.replicas + + - it: rejects simultaneous autoscaling and Rollout ownership + template: templates/rollout.yaml + set: + rollout.enabled: true + autoscaling.enabled: true + asserts: + - failedTemplate: + errorPattern: requires autoscaling.enabled=false + + - it: rejects requested capacity below the fleet target + template: templates/rollout.yaml + set: + rollout.enabled: true + rollout.deploymentReplicas: 0 + rollout.replicas: 1 + asserts: + - failedTemplate: + errorPattern: their sum must be replicaCount or replicaCount \+ 1 + + - it: rejects doubling the fleet during handover + template: templates/rollout.yaml + set: + rollout.enabled: true + rollout.deploymentReplicas: 2 + rollout.replicas: 2 + asserts: + - failedTemplate: + errorPattern: their sum must be replicaCount or replicaCount \+ 1 + + - it: rejects fractional replica counts + template: templates/rollout.yaml + set: + rollout.enabled: true + rollout.deploymentReplicas: 1.5 + rollout.replicas: 1 + asserts: + - failedTemplate: + errorPattern: rollout.deploymentReplicas + + - it: rejects decreasing weights + template: templates/rollout.yaml + set: + rollout.enabled: true + rollout.deploymentReplicas: 0 + rollout.replicas: 2 + rollout.weights: [50, 25, 100] + asserts: + - failedTemplate: + errorPattern: increasing percentages ending at 100 + + - it: rejects weights that never finish the update + template: templates/rollout.yaml + set: + rollout.enabled: true + rollout.deploymentReplicas: 0 + rollout.replicas: 2 + rollout.weights: [50] + asserts: + - failedTemplate: + errorPattern: increasing percentages ending at 100 + + - it: rejects a deadline shorter than the readiness window + template: templates/rollout.yaml + set: + rollout.enabled: true + rollout.deploymentReplicas: 0 + rollout.replicas: 2 + rollout.minReadySeconds: 120 + rollout.progressDeadlineSeconds: 60 + asserts: + - failedTemplate: + errorPattern: must exceed rollout.minReadySeconds diff --git a/deploy/charts/buzz/values.schema.json b/deploy/charts/buzz/values.schema.json index aaab848fd3..8400bacf95 100644 --- a/deploy/charts/buzz/values.schema.json +++ b/deploy/charts/buzz/values.schema.json @@ -31,7 +31,23 @@ "replicaCount": { "type": "integer", "minimum": 1, - "description": "Replica count for the relay Deployment. replicaCount > 1 requires Redis (for buzz-pubsub) — enforced by _validate.tpl. Git storage does NOT need ReadWriteMany: git state is object-store-backed and repo names live in Postgres, so ReadWriteOnce is fine per replica." + "description": "Replica count for the relay Deployment, or fleet target when rollout.enabled. replicaCount > 1 requires Redis (for buzz-pubsub) — enforced by _validate.tpl. Git storage does NOT need ReadWriteMany: git state is object-store-backed and repo names live in Postgres, so ReadWriteOnce is fine per replica." + }, + "rollout": { + "type": ["object", "null"], + "additionalProperties": false, + "description": "Opt-in fixed-replica Argo Rollouts. Explicit counts control the handover; the existing Deployment remains the pod-template source.", + "properties": { + "enabled": { "type": "boolean" }, + "deploymentReplicas": { "type": ["integer", "null"], "minimum": 0 }, + "replicas": { "type": ["integer", "null"], "minimum": 0 }, + "minReadySeconds": { "type": "integer", "minimum": 1 }, + "progressDeadlineSeconds": { "type": "integer", "minimum": 1 }, + "weights": { + "type": "array", + "items": { "type": "integer", "minimum": 1, "maximum": 100 } + } + } }, "relayUrl": { "type": "string", diff --git a/deploy/charts/buzz/values.yaml b/deploy/charts/buzz/values.yaml index addfa4f5b3..26ef527c02 100644 --- a/deploy/charts/buzz/values.yaml +++ b/deploy/charts/buzz/values.yaml @@ -35,8 +35,26 @@ image: # object-store-backed (each request hydrates an ephemeral repo from S3; writer # serialization is the object-store pointer CAS), and repo-name uniqueness lives # in Postgres. Each replica can use its own ReadWriteOnce volume (or none). +# With rollout.enabled, this is the fleet target; rollout's two replica fields +# explicitly divide that target between the Deployment and Rollout. replicaCount: 1 +# ── Argo Rollouts (opt-in, fixed replicas only) ──────────────────────────────── +# Requires a separately installed Argo Rollouts controller and CRDs. +# The Deployment remains the pod-template source via workloadRef. Set BOTH +# counts explicitly; enabling this flag alone cannot scale down the Deployment. +# For a two-pod handover, see docs/argo-rollouts.md before changing these values. +rollout: + enabled: false + deploymentReplicas: null + replicas: null + minReadySeconds: 60 + progressDeadlineSeconds: 900 + # Increasing percentages ending at 100. Each weight has an indefinite pause, + # with another pause before the first weight. Initial creation skips steps. + # [50, 100] is intended for two replicas; larger fleets need smaller steps. + weights: [50, 100] + # ── Autoscaling ────────────────────────────────────────────────────────────── # Requires Metrics Server for CPU. Optional WebSocket scaling additionally # requires a custom-metrics adapter exposing its pod-level Prometheus gauge.