Add opt-in Argo Rollouts to the Buzz chart

Signed-off-by: Ravneet Arora <rarora@squareup.com>

deploy/charts/buzz/Chart.yaml

@@ -7,7 +7,7 @@ description: |
   PostgreSQL and Redis. Configurable for single-node evaluation
   (subcharts on) and HA production (external services, existingSecret).
 type: application
-version: 0.1.8
+version: 0.1.9
 appVersion: "0.1.0"
 home: https://github.com/block/buzz
 sources:
@@ -24,7 +24,7 @@ maintainers:
 annotations:
   artifacthub.io/changes: |
     - kind: added
-      description: Optional immutable relay image digest pinning with backwards-compatible tag fallback.
+      description: Opt-in fixed-replica Argo Rollouts with explicit handover counts and manual canary pauses.
   artifacthub.io/license: Apache-2.0
 
 # Optional eval-only subcharts. Production deploys disable both and point

deploy/charts/buzz/README.md

@@ -52,6 +52,7 @@ security boundaries.
 
 See:
 
+- [`docs/argo-rollouts.md`](docs/argo-rollouts.md) — optional manual canary updates and Deployment handover
 - [`examples/argocd-app.yaml`](examples/argocd-app.yaml) — ArgoCD Application
 - [`examples/flux-helmrelease.yaml`](examples/flux-helmrelease.yaml) — Flux HelmRelease v2
 - [`examples/secret-sample.yaml`](examples/secret-sample.yaml) — Secret schema

deploy/charts/buzz/docs/argo-rollouts.md

@@ -0,0 +1,114 @@
+# Optional Argo Rollouts
+
+The chart uses a Kubernetes Deployment by default. Setting `rollout.enabled: true`
+adds an Argo `Rollout` in the same namespace. Install the Argo Rollouts controller
+and its CRDs separately before enabling it. This configuration targets the
+`argoproj.io/v1alpha1` API used by Argo Rollouts v1.8.3.
+
+The Deployment still contains the image, environment variables, probes, volumes,
+and other pod settings. The Rollout reads those settings through `workloadRef`
+and creates **new pods of its own**. It does not adopt the Deployment's pods.
+Both sets use the existing Service and PodDisruptionBudget selectors.
+
+This first version supports fixed replica counts and manual promotion. It does
+not add health analyses, automatic abort decisions, traffic routing, or HPA
+support. `autoscaling.enabled: true` is rejected while Rollouts are enabled.
+Existing installations with Rollouts disabled retain their Deployment/HPA path.
+
+## First handover: two replicas
+
+Keep the relay image, secrets, and pod settings unchanged throughout handover.
+Use an immutable image digest. Confirm that both existing pods are healthy and
+that the cluster has room for one extra pod. Redis must be configured for the
+overlapping pods; all pods must use the same external database, Redis, and storage.
+
+Start with this addition to your existing production values:
+
+```yaml
+replicaCount: 2
+autoscaling:
+  enabled: false
+rollout:
+  enabled: true
+  deploymentReplicas: 2
+  replicas: 0
+  minReadySeconds: 60
+  progressDeadlineSeconds: 900
+  weights: [50, 100]
+```
+
+The two replica fields are intentionally required. Enabling the feature alone
+fails rendering instead of silently scaling down the Deployment. `replicaCount`
+remains the desired fleet size; the two explicit counts divide that fleet
+between controllers during the handover.
+
+Apply **one row at a time** through your normal Helm or GitOps process:
+
+| Stage | `rollout.deploymentReplicas` | `rollout.replicas` | Action |
+|---|---:|---:|---|
+| Prepare | 2 | 0 | Create the Rollout with no requested pods. |
+| Add first pod | 2 | 1 | Wait for the new Rollout pod to be healthy. |
+| Drain first old pod | 1 | 1 | Wait for client reconnections and recovery. |
+| Add second pod | 1 | 2 | Wait for the second Rollout pod to be healthy. |
+| Finish | 0 | 2 | Drain the last Deployment pod and check recovery. |
+
+Before every drain, verify the replacement has been Ready for at least 60 seconds,
+is serving through the Service, and the fleet has recovered: connections,
+authenticated sessions, subscriptions, error rates, and database load. Before
+each next row, wait for the previous change to finish, including pod termination.
+If recovery is incomplete, stop and investigate. Desired counts alone do not prove
+healthy capacity. The chart checks only that the desired counts total two or
+three and that neither controller requests more than two.
+
+Argo skips canary steps when creating the first revision. **The pauses do not
+protect this handover.** The explicit replica changes and checks above do.
+`workloadRef.scaleDown: never` ensures Argo does not also scale the Deployment;
+your values remain its replica source of truth.
+
+Keep the zero-replica Deployment after handover. Argo still needs its pod
+configuration. Argo CD continues applying both the Deployment and the Rollout.
+There is no switch that tells Argo CD to stop managing the Deployment.
+
+## Later image updates
+
+Leave the counts at Deployment `0`, Rollout `2`, and change the usual `image.tag`
+or `image.digest`. Argo observes the changed Deployment template and manages the
+new revision. With `[50, 100]`, the generated steps are:
+
+```yaml
+- pause: {}
+- setWeight: 50
+- pause: {}
+- setWeight: 100
+- pause: {}
+```
+
+Each pause requires an operator to inspect health and promote one step using the
+Argo Rollouts UI or `kubectl argo rollouts promote <release-name>`. Do not use
+`--full` to bypass the checks. The controller allows one surge pod and requests
+zero unavailable pods; this does not guarantee every client has reconnected.
+For larger fleets, choose smaller weights before enabling the feature.
+
+The weights approximate the share of Rollout pods running the new revision.
+They do not split WebSocket traffic by percentage or move existing connections.
+Pauses also do not stop ordinary scaling, pod replacement after failures, or a
+direct change to the replica counts.
+
+## Returning to a Deployment
+
+Do not disable Rollouts while they still own serving pods: removing the Rollout
+can delete its pods before the Deployment has replaced them. Freeze image and
+pod settings, then reverse the table above. Add and verify one Deployment pod
+before draining a Rollout pod. Once the Deployment has two healthy pods and the
+Rollout has zero remaining pods, set `rollout.enabled: false` and remove the
+unused rollout overrides. Leave `replicaCount: 2`.
+
+## What this does not prove
+
+This chart is a prerequisite for a staged migration, not an automatic recovery
+system. Enabling it with nonzero replicas affects real pods and traffic. It is
+not a report-only or shadow mode. Observation-only health analyses and enforced
+recovery checks require a separate change and validation before activation.
+
+See the upstream [migration guide](https://argo-rollouts.readthedocs.io/en/stable/migrating/)
+and [canary behavior](https://argo-rollouts.readthedocs.io/en/stable/features/canary/).

deploy/charts/buzz/templates/NOTES.txt

@@ -90,4 +90,11 @@
 ──────────────────────────────────────────────────────────────────────────────
   kubectl -n {{ .Release.Namespace }} get pods -l app.kubernetes.io/instance={{ .Release.Name }}
   kubectl -n {{ .Release.Namespace }} logs -l app.kubernetes.io/instance={{ .Release.Name }} --tail=200
+{{- $rollout := .Values.rollout | default dict }}
+{{- if $rollout.enabled }}
+  kubectl -n {{ .Release.Namespace }} get deployment,rollout {{ include "buzz.fullname" . }}
+  Argo Rollouts is enabled. The Deployment supplies the pod template.
+  Updates pause for manual promotion. See docs/argo-rollouts.md before handover.
+{{- else }}
   kubectl -n {{ .Release.Namespace }} rollout status deployment/{{ include "buzz.fullname" . }}
+{{- end }}

deploy/charts/buzz/templates/_rollout.tpl

@@ -0,0 +1,38 @@
+{{/* Missing rollout values must remain compatible with --reuse-values. */}}
+{{- define "buzz.validateRollout" -}}
+{{- $rollout := .Values.rollout | default dict -}}
+{{- if $rollout.enabled -}}
+  {{- if .Values.autoscaling.enabled -}}
+    {{- fail "rollout.enabled requires autoscaling.enabled=false; this chart currently supports fixed-replica Rollouts only" -}}
+  {{- end -}}
+  {{- if or (not (hasKey $rollout "deploymentReplicas")) (eq $rollout.deploymentReplicas nil) (not (hasKey $rollout "replicas")) (eq $rollout.replicas nil) -}}
+    {{- fail "rollout.enabled requires explicit rollout.deploymentReplicas and rollout.replicas; start with replicaCount and 0 for an existing Deployment" -}}
+  {{- end -}}
+  {{- $deployment := int $rollout.deploymentReplicas -}}
+  {{- $replicas := int $rollout.replicas -}}
+  {{- $target := int .Values.replicaCount -}}
+  {{- if or (lt $deployment 0) (lt $replicas 0) (gt $deployment $target) (gt $replicas $target) (lt (add $deployment $replicas) $target) (gt (add $deployment $replicas) (add $target 1)) -}}
+    {{- fail "rollout replica counts must each be between 0 and replicaCount, and their sum must be replicaCount or replicaCount + 1" -}}
+  {{- end -}}
+  {{- if and (not .Values.redis.enabled) (not .Values.externalRedis.url) (not .Values.secrets.existingSecret) -}}
+    {{- fail "rollout.enabled requires Redis for overlapping relay pods; configure redis.enabled, externalRedis.url, or secrets.existingSecret with REDIS_URL" -}}
+  {{- end -}}
+  {{- if not $rollout.weights -}}
+    {{- fail "rollout.weights must contain increasing percentages ending at 100" -}}
+  {{- end -}}
+  {{- if le (int ($rollout.progressDeadlineSeconds | default 900)) (int ($rollout.minReadySeconds | default 60)) -}}
+    {{- fail "rollout.progressDeadlineSeconds must exceed rollout.minReadySeconds" -}}
+  {{- end -}}
+  {{- $previous := 0 -}}
+  {{- range $rollout.weights -}}
+    {{- $weight := int . -}}
+    {{- if or (le $weight $previous) (gt $weight 100) -}}
+      {{- fail "rollout.weights must contain increasing percentages ending at 100" -}}
+    {{- end -}}
+    {{- $previous = $weight -}}
+  {{- end -}}
+  {{- if ne $previous 100 -}}
+    {{- fail "rollout.weights must contain increasing percentages ending at 100" -}}
+  {{- end -}}
+{{- end -}}
+{{- end -}}

deploy/charts/buzz/templates/_validate.tpl

@@ -5,6 +5,8 @@ surface at template time regardless of which manifest helm renders first.
 
 {{- define "buzz.validate" -}}
 
+{{- include "buzz.validateRollout" . -}}
+
 {{/* relayUrl is required */}}
 {{- if not .Values.relayUrl -}}
   {{- fail "relayUrl is required: set --set relayUrl=wss://your.domain" -}}

deploy/charts/buzz/templates/deployment.yaml

@@ -1,4 +1,5 @@
 {{- include "buzz.validate" . -}}
+{{- $rollout := .Values.rollout | default dict -}}
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -6,7 +7,9 @@ metadata:
   labels:
     {{- include "buzz.labels" . | nindent 4 }}
 spec:
-  {{- if not .Values.autoscaling.enabled }}
+  {{- if $rollout.enabled }}
+  replicas: {{ $rollout.deploymentReplicas }}
+  {{- else if not .Values.autoscaling.enabled }}
   replicas: {{ .Values.replicaCount }}
   {{- end }}
   strategy:

deploy/charts/buzz/templates/rollout.yaml

@@ -0,0 +1,33 @@
+{{- include "buzz.validate" . -}}
+{{- $rollout := .Values.rollout | default dict -}}
+{{- if $rollout.enabled }}
+apiVersion: argoproj.io/v1alpha1
+kind: Rollout
+metadata:
+  name: {{ include "buzz.fullname" . }}
+  labels:
+    {{- include "buzz.labels" . | nindent 4 }}
+spec:
+  replicas: {{ $rollout.replicas }}
+  minReadySeconds: {{ $rollout.minReadySeconds | default 60 }}
+  progressDeadlineSeconds: {{ $rollout.progressDeadlineSeconds | default 900 }}
+  progressDeadlineAbort: false
+  selector:
+    matchLabels:
+      {{- include "buzz.relaySelectorLabels" . | nindent 6 }}
+  workloadRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: {{ include "buzz.fullname" . }}
+    scaleDown: never
+  strategy:
+    canary:
+      maxSurge: 1
+      maxUnavailable: 0
+      steps:
+        - pause: {}
+        {{- range $rollout.weights }}
+        - setWeight: {{ . }}
+        - pause: {}
+        {{- end }}
+{{- end }}

deploy/charts/buzz/tests/fixtures/rollout-values.yaml

@@ -0,0 +1,9 @@
+relayUrl: wss://buzz.example.com
+ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
+secrets:
+  existingSecret: buzz-secrets
+replicaCount: 2
+rollout:
+  enabled: true
+  deploymentReplicas: 0
+  replicas: 2

deploy/charts/buzz/tests/rollout_test.yaml

@@ -0,0 +1,173 @@
+suite: opt-in relay Rollout
+templates:
+  - templates/rollout.yaml
+  - templates/deployment.yaml
+  - templates/service.yaml
+  - templates/pdb.yaml
+  - templates/secret-chart.yaml
+set:
+  relayUrl: wss://buzz.example.com
+  ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
+  secrets.existingSecret: buzz-secrets
+  replicaCount: 2
+tests:
+  - it: tolerates missing rollout values from an older release
+    set:
+      rollout: null
+    asserts:
+      - hasDocuments:
+          count: 0
+        template: templates/rollout.yaml
+      - equal:
+          path: spec.replicas
+          value: 2
+        template: templates/deployment.yaml
+
+  - it: creates no Rollout by default
+    template: templates/rollout.yaml
+    asserts:
+      - hasDocuments:
+          count: 0
+
+  - it: keeps Deployment replicas and rolling update strategy by default
+    template: templates/deployment.yaml
+    asserts:
+      - equal:
+          path: spec.replicas
+          value: 2
+      - equal:
+          path: spec.strategy
+          value:
+            type: RollingUpdate
+            rollingUpdate:
+              maxSurge: 1
+              maxUnavailable: 0
+
+  - it: permits preparing a zero-pod Rollout without draining the Deployment
+    set:
+      rollout.enabled: true
+      rollout.deploymentReplicas: 2
+      rollout.replicas: 0
+    asserts:
+      - equal:
+          path: spec.replicas
+          value: 0
+        template: templates/rollout.yaml
+      - equal:
+          path: spec.replicas
+          value: 2
+        template: templates/deployment.yaml
+
+  - it: permits one extra pod during handover
+    set:
+      rollout.enabled: true
+      rollout.deploymentReplicas: 2
+      rollout.replicas: 1
+    asserts:
+      - equal:
+          path: spec.replicas
+          value: 1
+        template: templates/rollout.yaml
+      - equal:
+          path: spec.replicas
+          value: 2
+        template: templates/deployment.yaml
+
+  - it: retains the Deployment pod configuration at zero replicas
+    template: templates/deployment.yaml
+    set:
+      rollout.enabled: true
+      rollout.deploymentReplicas: 0
+      rollout.replicas: 2
+      image.repository: ghcr.io/example/buzz
+      image.tag: test-image
+    asserts:
+      - equal:
+          path: spec.replicas
+          value: 0
+      - equal:
+          path: spec.template.spec.containers[0].image
+          value: ghcr.io/example/buzz:test-image
+      - exists:
+          path: spec.template.spec.containers[0].readinessProbe
+
+  - it: references the same Deployment name with manual canary pauses
+    template: templates/rollout.yaml
+    set:
+      fullnameOverride: custom-relay
+      rollout.enabled: true
+      rollout.deploymentReplicas: 0
+      rollout.replicas: 2
+    asserts:
+      - isAPIVersion:
+          of: argoproj.io/v1alpha1
+      - isKind:
+          of: Rollout
+      - equal:
+          path: metadata.name
+          value: custom-relay
+      - equal:
+          path: spec.workloadRef
+          value:
+            apiVersion: apps/v1
+            kind: Deployment
+            name: custom-relay
+            scaleDown: never
+      - notExists:
+          path: spec.template
+      - equal:
+          path: spec.minReadySeconds
+          value: 60
+      - equal:
+          path: spec.progressDeadlineSeconds
+          value: 900
+      - equal:
+          path: spec.progressDeadlineAbort
+          value: false
+      - equal:
+          path: spec.strategy.canary
+          value:
+            maxSurge: 1
+            maxUnavailable: 0
+            steps:
+              - pause: {}
+              - setWeight: 50
+              - pause: {}
+              - setWeight: 100
+              - pause: {}
+      - equal:
+          path: spec.selector.matchLabels
+          value:
+            app.kubernetes.io/name: buzz
+            app.kubernetes.io/instance: RELEASE-NAME
+            app.kubernetes.io/component: relay
+
+  - it: keeps the Service selecting both sets of relay pods
+    template: templates/service.yaml
+    set:
+      rollout.enabled: true
+      rollout.deploymentReplicas: 1
+      rollout.replicas: 1
+    asserts:
+      - equal:
+          path: spec.selector
+          value:
+            app.kubernetes.io/name: buzz
+            app.kubernetes.io/instance: RELEASE-NAME
+            app.kubernetes.io/component: relay
+
+  - it: keeps the PodDisruptionBudget selecting both sets of relay pods
+    template: templates/pdb.yaml
+    set:
+      rollout.enabled: true
+      rollout.deploymentReplicas: 1
+      rollout.replicas: 2
+      podDisruptionBudget.enabled: true
+    asserts:
+      - equal:
+          path: spec.selector.matchLabels
+          value:
+            app.kubernetes.io/name: buzz
+            app.kubernetes.io/instance: RELEASE-NAME
+            app.kubernetes.io/component: relay
+

deploy/charts/buzz/tests/rollout_validation_test.yaml

@@ -0,0 +1,89 @@
+suite: Rollout validation
+templates:
+  - templates/rollout.yaml
+set:
+  relayUrl: wss://buzz.example.com
+  ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
+  secrets.existingSecret: buzz-secrets
+  replicaCount: 2
+tests:
+  - it: rejects enabling without explicit handover counts
+    template: templates/rollout.yaml
+    set:
+      rollout.enabled: true
+    asserts:
+      - failedTemplate:
+          errorPattern: requires explicit rollout.deploymentReplicas and rollout.replicas
+
+  - it: rejects simultaneous autoscaling and Rollout ownership
+    template: templates/rollout.yaml
+    set:
+      rollout.enabled: true
+      autoscaling.enabled: true
+    asserts:
+      - failedTemplate:
+          errorPattern: requires autoscaling.enabled=false
+
+  - it: rejects requested capacity below the fleet target
+    template: templates/rollout.yaml
+    set:
+      rollout.enabled: true
+      rollout.deploymentReplicas: 0
+      rollout.replicas: 1
+    asserts:
+      - failedTemplate:
+          errorPattern: their sum must be replicaCount or replicaCount \+ 1
+
+  - it: rejects doubling the fleet during handover
+    template: templates/rollout.yaml
+    set:
+      rollout.enabled: true
+      rollout.deploymentReplicas: 2
+      rollout.replicas: 2
+    asserts:
+      - failedTemplate:
+          errorPattern: their sum must be replicaCount or replicaCount \+ 1
+
+  - it: rejects fractional replica counts
+    template: templates/rollout.yaml
+    set:
+      rollout.enabled: true
+      rollout.deploymentReplicas: 1.5
+      rollout.replicas: 1
+    asserts:
+      - failedTemplate:
+          errorPattern: rollout.deploymentReplicas
+
+  - it: rejects decreasing weights
+    template: templates/rollout.yaml
+    set:
+      rollout.enabled: true
+      rollout.deploymentReplicas: 0
+      rollout.replicas: 2
+      rollout.weights: [50, 25, 100]
+    asserts:
+      - failedTemplate:
+          errorPattern: increasing percentages ending at 100
+
+  - it: rejects weights that never finish the update
+    template: templates/rollout.yaml
+    set:
+      rollout.enabled: true
+      rollout.deploymentReplicas: 0
+      rollout.replicas: 2
+      rollout.weights: [50]
+    asserts:
+      - failedTemplate:
+          errorPattern: increasing percentages ending at 100
+
+  - it: rejects a deadline shorter than the readiness window
+    template: templates/rollout.yaml
+    set:
+      rollout.enabled: true
+      rollout.deploymentReplicas: 0
+      rollout.replicas: 2
+      rollout.minReadySeconds: 120
+      rollout.progressDeadlineSeconds: 60
+    asserts:
+      - failedTemplate:
+          errorPattern: must exceed rollout.minReadySeconds

deploy/charts/buzz/values.schema.json

@@ -31,7 +31,23 @@
     "replicaCount": {
       "type": "integer",
       "minimum": 1,
-      "description": "Replica count for the relay Deployment. replicaCount > 1 requires Redis (for buzz-pubsub) — enforced by _validate.tpl. Git storage does NOT need ReadWriteMany: git state is object-store-backed and repo names live in Postgres, so ReadWriteOnce is fine per replica."
+      "description": "Replica count for the relay Deployment, or fleet target when rollout.enabled. replicaCount > 1 requires Redis (for buzz-pubsub) — enforced by _validate.tpl. Git storage does NOT need ReadWriteMany: git state is object-store-backed and repo names live in Postgres, so ReadWriteOnce is fine per replica."
+    },
+    "rollout": {
+      "type": ["object", "null"],
+      "additionalProperties": false,
+      "description": "Opt-in fixed-replica Argo Rollouts. Explicit counts control the handover; the existing Deployment remains the pod-template source.",
+      "properties": {
+        "enabled": { "type": "boolean" },
+        "deploymentReplicas": { "type": ["integer", "null"], "minimum": 0 },
+        "replicas": { "type": ["integer", "null"], "minimum": 0 },
+        "minReadySeconds": { "type": "integer", "minimum": 1 },
+        "progressDeadlineSeconds": { "type": "integer", "minimum": 1 },
+        "weights": {
+          "type": "array",
+          "items": { "type": "integer", "minimum": 1, "maximum": 100 }
+        }
+      }
     },
     "relayUrl": {
       "type": "string",

deploy/charts/buzz/values.yaml

@@ -35,8 +35,26 @@ image:
 # object-store-backed (each request hydrates an ephemeral repo from S3; writer
 # serialization is the object-store pointer CAS), and repo-name uniqueness lives
 # in Postgres. Each replica can use its own ReadWriteOnce volume (or none).
+# With rollout.enabled, this is the fleet target; rollout's two replica fields
+# explicitly divide that target between the Deployment and Rollout.
 replicaCount: 1
 
+# ── Argo Rollouts (opt-in, fixed replicas only) ────────────────────────────────
+# Requires a separately installed Argo Rollouts controller and CRDs.
+# The Deployment remains the pod-template source via workloadRef. Set BOTH
+# counts explicitly; enabling this flag alone cannot scale down the Deployment.
+# For a two-pod handover, see docs/argo-rollouts.md before changing these values.
+rollout:
+  enabled: false
+  deploymentReplicas: null
+  replicas: null
+  minReadySeconds: 60
+  progressDeadlineSeconds: 900
+  # Increasing percentages ending at 100. Each weight has an indefinite pause,
+  # with another pause before the first weight. Initial creation skips steps.
+  # [50, 100] is intended for two replicas; larger fleets need smaller steps.
+  weights: [50, 100]
+
 # ── Autoscaling ──────────────────────────────────────────────────────────────
 # Requires Metrics Server for CPU. Optional WebSocket scaling additionally
 # requires a custom-metrics adapter exposing its pod-level Prometheus gauge.