mirror of
https://github.com/block/buzz.git
synced 2026-10-28 21:11:48 +00:00
3ed623bb21
## Why A wedged relay boot pod holding a relation lock can park every other writer in the fleet behind it: DB load pins at pool capacity in `Lock:relation` waits while CPU stays flat, and nothing server-side releases the lock until the holder dies. We hit exactly this in production — ~1,400 sessions queued behind one crash-looping pod's boot transaction for ~20 minutes until kubelet killed the container. ## What Applies session-level Postgres timeouts to every **writer** connection inside the existing single `after_connect` hook in `buzz-db`, all env-tunable through the same `Config::from_env → DbConfig` path as the existing pool-size knobs: | Env var | GUC | Default | Effect | |---|---|---|---| | `BUZZ_DB_LOCK_TIMEOUT_MS` | `lock_timeout` | 5000 | statements waiting on any lock fail fast instead of parking behind a wedged holder | | `BUZZ_DB_IDLE_TXN_TIMEOUT_MS` | `idle_in_transaction_session_timeout` | 60000 | reaps wedged clients idling inside an open transaction while holding locks | | `BUZZ_DB_STATEMENT_TIMEOUT_MS` | `statement_timeout` | 0 (off) | opt-in runaway-statement cap; off by default because startup migrations/backfills legitimately run long statements | `0` disables a timeout (Postgres semantics) and deliberately passes through the env parsing — unlike the pool-size knobs where `0` falls back to the default. The reader pool is untouched: replica sessions never take contended locks and already fail acquire in 150 ms. Deployers tune these via plain env vars (`.env`, or `relay.extraEnv` in the Helm chart) — no code changes needed. ## Behavior change to note With the 5 s default `lock_timeout`, a boot-time migration or backfill that waits >5 s on a lock now errors (surfacing in logs / crash-looping the pod) instead of stalling silently. That is the intended visible-failure-over-fleet-stall tradeoff; deployers with slow contended migrations can set `BUZZ_DB_LOCK_TIMEOUT_MS=0`. ## Testing - `cargo test -p buzz-db -p buzz-relay` — buzz-db green; buzz-relay has 9 failures that also fail on clean `main` in this environment (api::admin/api::media/mesh_demo — unrelated, pre-existing). - New config test covers override / `0`-passthrough / invalid-fallback for all three env vars. - Extended the existing `writer_pool_safety_hook_is_single_and_composed` source-shape test so the timeouts can't drift out of the single `after_connect` hook (SQLx replaces hooks — a second hook would silently disarm the floor guard). - `cargo fmt --check` and `cargo clippy --all-targets` clean for the touched crates. Closest existing PR/issue: none found. --- **Update Aug 28, 17:06 EDT:** Rebased onto `main` at `a3730784fc` and addressed the latest correctness review. - Ported the timeout policy onto the refactored `buzz-db::runtime` pool constructor and kept the shared env overlay for relay, admin, deletion, and audit writers. - Migration/schema-destruction connections now disable `lock_timeout` and `statement_timeout` for their intentional long wait/DDL path. This supersedes the earlier “Behavior change to note”: contended boot migrations wait for the current migration owner rather than crash-looping after five seconds. - The audit worker now preserves and retries the same entry on PostgreSQL `55P03` lock timeouts, using exponential backoff capped at one second. Other database errors retain the existing terminal error behavior, and retries emit `buzz_audit_log_lock_retries_total`. - Added CI-backed PostgreSQL regressions for writer GUC installation/migration exemption, audit-pool lock timeouts, and worker recovery. The worker regression holds the real audit advisory lock past `lock_timeout`, observes a retry, releases the lock, and proves the original entry is appended exactly once. Current verification supersedes the earlier testing notes: workspace Rust clippy passed with warnings denied; all nine infrastructure-free backend unit-test lanes passed; all three focused PostgreSQL regressions passed against PostgreSQL 17; formatting, diff checks, file-size guards, and desktop frontend checks passed. The Linux Blox workstation could not run the unrelated Tauri native lane because `glib-2.0` is absent, so that platform check is left to PR CI. --- **Update Aug 31, 11:09 EDT:** Rebased onto current `main` at `c3132c3ee9` and reran the requested audit-lock contention scenario on Blox at head `896c3fe9ed`. - `git range-diff` reports both PR commits unchanged by the rebase; the branch remains two commits and the worktree is clean. - `cargo fmt --all -- --check` and clippy with warnings denied passed for `buzz-db`, `buzz-relay`, `buzz-admin`, and `buzz-deletion`. - All three focused PostgreSQL 17 regressions passed: writer session timeout/migration exemption, audit writer timeout bounds, and audit worker recovery of the original entry exactly once. - Live protocol verification used a head-built relay and CLI, native PostgreSQL 17/Redis, `BUZZ_DB_LOCK_TIMEOUT_MS=300`, and an eight-second hold on the community audit advisory lock. The real message was accepted and persisted once while the lock was held; its audit-row count remained zero during contention while retries accumulated. After release, exactly one `event_created` audit row appeared and remained exactly one after an additional two-second duplicate check. The run recorded nine lock-timeout retries, zero audit failures, and event ID `7f8c4ffae28e78555fcf2d56396d6e6c01b3712e5411288dc79e9a54af9d9444`. Generated with Codex --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com>
223 lines
20 KiB
Markdown
223 lines
20 KiB
Markdown
# Buzz Push Gateway deployment
|
||
|
||
`buzz-push-gateway` is the standalone public APNs last hop intended for `push.buzz.xyz`. Build it with `Dockerfile.push-gateway`; do not run it in the relay image or give relays APNs credentials.
|
||
|
||
## Network and health
|
||
|
||
- Public listener: `BUZZ_PUSH_BIND_ADDR` (default `0.0.0.0:8080`). Route `https://push.buzz.xyz` to this port.
|
||
- Private health listener: `BUZZ_PUSH_HEALTH_ADDR` (default `0.0.0.0:8081`). Probe `/_liveness` and `/_readiness`; do not expose this port publicly. The chart has no pod-ingress allowance for 8081; Kubernetes node/kubelet-origin probe traffic is exempt from NetworkPolicy. Add a narrowly selected monitoring source only if the target CNI requires pod-origin health scraping.
|
||
- Readiness fails when PostgreSQL authority is unavailable. Graceful shutdown stops accepting new requests before draining in-flight APNs calls.
|
||
|
||
## Required configuration
|
||
|
||
| Variable | Purpose |
|
||
|---|---|
|
||
| `DATABASE_URL` | PostgreSQL authority/admission store. Runtime credentials need DML on the six gateway tables, not DDL. |
|
||
| `BUZZ_PUSH_PUBLIC_DELIVERY_URL` | Exact externally signed URL, normally `https://push.buzz.xyz/v1/deliveries/apns`. |
|
||
| `BUZZ_PUSH_MAX_GRANT_LIFETIME_SECONDS` | Maximum delegation capability lifetime (`1..=31536000`). |
|
||
| `BUZZ_PUSH_MAX_INSTALLATION_LIFETIME_SECONDS` | Maximum encrypted-token installation lifetime (default 90 days, max one year). Clients must renew before expiry. |
|
||
| `BUZZ_PUSH_APP_ATTEST_ROOT_CERT_PATH` | Read-only mounted Apple App Attest root certificate PEM. |
|
||
| `BUZZ_PUSH_DOGFOOD_APP_ATTEST_APP_ID` | Exact server-owned Apple App Attest application identifier (`TEAMID.bundle-id`). |
|
||
| `BUZZ_PUSH_DOGFOOD_APNS_TOPIC` | Server-owned APNs topic. Never accepted from a client. |
|
||
| `BUZZ_PUSH_DOGFOOD_APNS_ENVIRONMENT` | `production` or `sandbox`, selected by deployment configuration. |
|
||
| `BUZZ_PUSH_DOGFOOD_APNS_CERT_PATH` | Read-only certificate/private-key PEM. |
|
||
| `BUZZ_PUSH_GRANT_KEYS` | Capability AEAD keyring, `id:base64-32-bytes[,predecessor...]`; current key first. |
|
||
| `BUZZ_PUSH_TOKEN_KEYS` | Independent token-custody AEAD keyring in the same format. Never reuse grant keys. |
|
||
|
||
The canonical `push.buzz.xyz` MVP serves the dogfood application identity
|
||
(`xyz.block.buzz.dogfood.mobile`). App Attest must cryptographically validate
|
||
the configured application ID before enrollment. Assertions and delivery use
|
||
the server-owned APNs topic, certificate-backed connection pool, and
|
||
environment. No client request or relay grant can supply or override an APNs
|
||
topic.
|
||
|
||
This MVP has exactly one compiled-in application profile,
|
||
`buzz-ios-dogfood`. The chart value
|
||
`profiles.dogfood.appAttestAppId` is rendered as
|
||
`BUZZ_PUSH_DOGFOOD_APP_ATTEST_APP_ID`; the gateway rejects startup when it is
|
||
missing or empty. The exact `TEAMID.bundle-id` is environment-owned,
|
||
non-secret deployment configuration. The chart's production values file leaves
|
||
it empty deliberately so a production renderer must supply it from the GitOps
|
||
environment rather than baking a Block team identifier into this repository.
|
||
Supporting another application identity requires an explicit code, schema,
|
||
chart, credential, and deployment change; this gateway does not currently
|
||
select among multiple application profiles.
|
||
|
||
Optional endpoint quota policy variables are `BUZZ_PUSH_ENDPOINT_QUOTA_WINDOW_SECONDS` (default `10`, max `86400`) and `BUZZ_PUSH_ENDPOINT_QUOTA_MAX_DELIVERIES` (default `10`, max `10000`). These are Buzz policy hypotheses, not Apple-published limits; tune under load while retaining a hard ceiling.
|
||
|
||
## Secret and key rotation rules
|
||
|
||
Mount the App Attest root read-only and startup will reject any byte mismatch. The sole accepted artifact is Apple’s **Apple App Attestation Root CA** from `https://www.apple.com/certificateauthority/Apple_App_Attestation_Root_CA.pem`: certificate SHA-256 fingerprint `1C:B9:82:3B:A2:8B:A6:AD:2D:33:A0:06:94:1D:E2:AE:4F:51:3E:F1:D4:E8:31:B9:F7:E0:FA:7B:62:42:C9:32`; exact PEM-file SHA-256 `c778d09ac341f7fd9f8f3b19e2b815af6aed4ad4490e1e92c05cb355212a5013`. Treat an Apple root rotation as a reviewed code/config rollout, not an unpinned mount replacement. Mount the APNs certificate identity and both AEAD keyrings from a secret manager; never place values in an image, manifest, log, or metrics label. Keep the current AEAD key first and retain decrypt-only predecessors until every capability/token encrypted under them has expired or been re-encrypted. Grant and token key ids and bytes must be distinct. Rotation is an operator rollout: add the new current key while retaining predecessors, deploy, wait through the retention window, then remove the old key.
|
||
|
||
The gateway stores APNs tokens encrypted in PostgreSQL. Database backups therefore contain ciphertext plus authority metadata and must receive the same access controls and retention treatment as the service secrets.
|
||
|
||
## PostgreSQL and replicas
|
||
|
||
The gateway's dedicated pool does not consume the relay-oriented `BUZZ_DB_LOCK_TIMEOUT_MS`, `BUZZ_DB_IDLE_TXN_TIMEOUT_MS`, or `BUZZ_DB_STATEMENT_TIMEOUT_MS` settings. Its session-timeout policy remains separate from the `buzz-db` writer policy and must be designed and rolled out independently.
|
||
|
||
All replicas must share one PostgreSQL database. Delivery authority, replay admission, and endpoint quota reservation are transactional there, so replica count does not multiply the abuse ceiling. The gateway owns a scoped migration history under `crates/buzz-push-gateway/migrations`; it creates only the six `push_gateway_*` authority tables plus SQLx's migration-history table and never runs relay migrations.
|
||
|
||
The Helm chart runs a single pre-install/pre-upgrade migration Job using `migration.existingSecret`; that secret contains a DDL-capable `DATABASE_URL`. The URL MUST name a dedicated gateway database, not the relay database: SQLx stores its `_sqlx_migrations` history in `public`, so sharing a database would collide with another application's migration history. `migration.runtimeDatabaseRole` names an existing LOGIN role (the default is `buzz_push_gateway_runtime`) used by runtime `DATABASE_URL`. After scoped migrations, the Job revokes database `CREATE` from that role and schema `CREATE` from both `PUBLIC` and the role, then grants only database `CONNECT`, schema `USAGE`, and `SELECT, INSERT, UPDATE, DELETE` on the six gateway tables. The migration role must own the database/schema objects or otherwise be allowed to issue those grants; it is never provided to runtime replicas. Readiness rejects an empty/partial schema, missing DML, or a runtime role that retains database/schema `CREATE`. Helm waits for the migration hook before updating replicas, so rolling deployments never race unconditional startup migration. Readiness must be removed from load-balancer service endpoints before terminating a pod.
|
||
|
||
The service reaps expired challenges and replay rows, idle quota rows, expired/revoked delegations, and retention-eligible installations (including their encrypted token ciphertext) at startup and every five minutes. Monitor reaper failures and table growth; retention does not depend on process restarts.
|
||
|
||
## Metrics and alerting
|
||
|
||
The gateway serves Prometheus metrics at `GET /metrics` on the **private health listener** (`BUZZ_PUSH_HEALTH_ADDR`, default `0.0.0.0:8081`) — the same port as the probes, never on the public `8080`. All series are sanitized and bounded-cardinality: label values are drawn only from closed sets (the five APNs outcome classes, the fixed admission results, the static error codes already returned to callers, and the readiness causes). No endpoint, device token, relay pubkey, request id, or any request-scoped identifier is ever used as a label.
|
||
|
||
| Metric | Type | Labels | Meaning |
|
||
|---|---|---|---|
|
||
| `push_gateway_apns_deliveries_total` | counter | `outcome` = `accepted` \| `invalid_endpoint` \| `retry` \| `configuration_fault` \| `permanent_request_fault` | Terminal APNs send outcomes. |
|
||
| `push_gateway_apns_delivery_seconds` | histogram | — | APNs send round-trip latency (seconds). |
|
||
| `push_gateway_admissions_total` | counter | `result` = `admitted` \| `rejected` \| `unavailable` | Outcome at the `authorize_delivery` replay/quota fence. |
|
||
| `push_gateway_delivery_errors_total` | counter | `class` (static) | Selected delivery-handler exit classes only (see note). |
|
||
| `push_gateway_reaper_failures_total` | counter | — | Retention reaper sweep failures. |
|
||
| `push_gateway_readiness_failures_total` | counter | `cause` = `not_accepting` \| `authority` | Readiness probe failures by cause. |
|
||
|
||
`push_gateway_delivery_errors_total` is intentionally **narrow**: it counts only selected exit classes of the `/v1/deliveries/apns` handler — `class` ∈ `invalid_grant` (grant rejected at the admission seam, before a permit is issued), `temporarily_unavailable` (authority unavailable at the admission seam), `profile_mismatch`, `token_custody` (endpoint-token open failure), `finish_failed` (detached disposition/join failure returned as 503). Request/auth/attestation/grant validation on the enrollment, delegation, rotation, and revocation handlers is **not** counted by this metric; it is a delivery-hot-path signal, not a total error rate across the API.
|
||
|
||
Scraping is **opt-in** and off by default, so the default chart render is unchanged and `8081` keeps no pod ingress. To enable it, set `podMonitor.enabled=true` (renders a prometheus-operator `PodMonitor` scraping the `health` port `/metrics`) and `networkPolicy.monitoring.enabled=true` with `networkPolicy.monitoring.namespaceSelector` / `podSelector` naming your scraper — this adds a single `8081` ingress rule scoped to that source, never a blanket allowance. Node/kubelet-origin probe traffic remains exempt from NetworkPolicy regardless.
|
||
|
||
Alerting rules ship as an opt-in prometheus-operator `PrometheusRule` (`prometheusRule.enabled=true`). Thresholds and operator actions:
|
||
|
||
| Alert | Fires when | Severity | Action |
|
||
|---|---|---|---|
|
||
| `PushGatewayConfigurationFault` | any `configuration_fault` outcomes for 10m | critical | The APNs certificate/topic/environment is unhealthy. Check `BUZZ_PUSH_DOGFOOD_APNS_*` configuration. No endpoints are being invalidated. |
|
||
| `PushGatewayAdmissionUnavailable` | any admission `unavailable` for 5m | critical | PostgreSQL authority store is unreachable. Check DB connectivity and the pod's `postgresEgressCidrs` NetworkPolicy. |
|
||
| `PushGatewayReadinessAuthorityFailing` | readiness `authority` failures for 5m | warning | Replicas are being pulled from the Service on DB check failure. Fix DB health before capacity drops below the PodDisruptionBudget. |
|
||
| `PushGatewayReaperFailing` | reaper failed ≥2 times within 30m (runs every 5m) | warning | Expired reservations aren't being swept, growing the bounded-until-expiry window. Check DB write availability. |
|
||
| `PushGatewayHighApnsRetryRate` | retryable fraction > `prometheusRule.apnsRetryRatioThreshold` (default `0.25`) over a 10m window, above `apnsRetryMinSamples` (default `20`) attempts, held true for 15m | warning | APNs is throttling or degraded (429/500/503). Deliveries are delayed, not lost. |
|
||
|
||
## Relay configuration
|
||
|
||
Relay push is an explicit deployment opt-in through `BUZZ_PUSH_ENABLED=true`;
|
||
the established strict boolean parser rejects unknown values and the default is
|
||
false. When enabled, an absent `BUZZ_PUSH_GATEWAY_DELIVERY_URL` selects the exact
|
||
canonical URL `https://push.buzz.xyz/v1/deliveries/apns`; operators can provide
|
||
another exact HTTPS `/v1/deliveries/apns` URL as an advanced override. An
|
||
explicitly empty URL while enabled is a startup error. Only an enabled relay
|
||
advertises its host-scoped NIP-PL descriptor, accepts leases, and starts the
|
||
matcher and delivery worker. Relays retain lease matching, authorization, durable
|
||
jobs/retries, and generation checks; they receive only opaque capabilities and
|
||
never APNs tokens or provider credentials.
|
||
|
||
An enabled relay exports the following bounded-cardinality series on its
|
||
existing Prometheus endpoint. None carries a community, account, relay key,
|
||
installation, event, or request identifier as a label.
|
||
|
||
| Metric | Type | Labels | Meaning |
|
||
|---|---|---|---|
|
||
| `buzz_push_enabled` | gauge | — | `1` only when the deployment opt-in is active. |
|
||
| `buzz_push_match_jobs_total` | counter | `result` = `matched` \| `unmatched` \| `error` \| `context_error` | Accepted message events evaluated by the matcher. |
|
||
| `buzz_push_match_queue_seconds` | histogram | — | Relay receipt to matcher evaluation latency. |
|
||
| `buzz_push_wakes_total` | counter | `result` = `enqueued` \| `duplicate` \| `inactive_lease` | Durable wake-enqueue outcomes. |
|
||
| `buzz_push_wake_enqueue_errors_total` | counter | — | Set-wise outbox transactions that failed. |
|
||
| `buzz_push_wake_queue_seconds` | histogram | — | First-attempt outbox enqueue-to-worker latency. |
|
||
| `buzz_push_gateway_requests_total` | counter | — | Relay requests that reached the gateway transport seam. |
|
||
| `buzz_push_gateway_request_seconds` | histogram | — | Relay-observed gateway request latency. |
|
||
| `buzz_push_deliveries_total` | counter | `outcome` (static closed set) | Accepted, retried, suppressed, exhausted, invalid, or failed relay delivery outcomes. |
|
||
|
||
## Relay integration status
|
||
|
||
The operational relay integration is complete: per-origin event matching with
|
||
read-authorization checks, durable enqueue, send-time revalidation, and NIP-98
|
||
delivery run only when `BUZZ_PUSH_ENABLED=true`. End-to-end use still requires
|
||
the client App Attest enrollment/delegation flow to place a gateway-issued opaque
|
||
capability—not a raw APNs token—into the encrypted relay lease.
|
||
|
||
## Internal dogfood evaluation and rollback
|
||
|
||
The MVP is ready to enable only when the canonical gateway's sole dogfood
|
||
profile is configured with its server-owned App Attest app ID, APNs topic,
|
||
production certificate identity, and production APNs environment, and only the
|
||
selected internal relay deployments set `BUZZ_PUSH_ENABLED=true`. Every iOS
|
||
artifact contains the native push bridge and Notification Service Extension,
|
||
but the client remains inactive until its current authenticated relay
|
||
advertises a fully valid NIP-11 `nip-pl` descriptor. There is no App Store
|
||
gateway profile in this MVP.
|
||
|
||
Physical-device validation must use an application whose App Attest identity
|
||
and APNs topic match the configured dogfood profile. The current gateway cannot
|
||
enroll `xyz.block.buzz.mobile` or another bundle identifier merely by changing
|
||
deployment values: adding another identity requires the explicit multi-profile
|
||
work described above.
|
||
|
||
Dogfood end-to-end release validation starts after this feature reaches `main`:
|
||
publish the next immutable `mobile-vX.Y.Z-rc.N` candidate from the exact current
|
||
`origin/main` commit, build that tag through the normal Block release pipeline,
|
||
and wait for the signed `xyz.block.buzz.dogfood.mobile` artifact to appear in
|
||
Mobile Releases/Comp Portal before installing it on a physical device. Verify
|
||
APNs delivery, fetched and signature-verified notification content, and
|
||
exact-message tap routing against the canonical gateway and a push-enabled
|
||
internal relay before widening the internal evaluation.
|
||
|
||
Before that first candidate, the private dogfood builder's manual signing and
|
||
export configuration must map separate distribution
|
||
profiles for both `xyz.block.buzz.dogfood.mobile` and
|
||
`xyz.block.buzz.dogfood.mobile.NotificationService`; an app-only profile does
|
||
not provision the extension. App Store rollout remains off through relay and
|
||
gateway deployment configuration until separately approved.
|
||
Before enabling rich message presentation, enable Apple's Communication
|
||
Notifications capability on the parent dogfood App ID and regenerate its app
|
||
provisioning profile. The extension profile does not need that capability.
|
||
Apply the same parent-App-ID prerequisite to the eventual App Store rollout;
|
||
updating Block Apple portal records is a separately authorized release step.
|
||
|
||
For each evaluation cohort, measure relay receipt-to-match, wake queue, relay-to-
|
||
gateway, and gateway-to-APNs latencies from the histograms above. Track the
|
||
ratio of accepted or replay-terminal relay outcomes to newly enqueued wakes,
|
||
gateway APNs accepted/retry/invalid/configuration outcomes, retry exhaustion,
|
||
and NSE resolution fallback. APNs acceptance cannot prove device presentation:
|
||
record a small manual physical-device sample with event-created, banner-visible,
|
||
and notification-tap timestamps, and verify that the visible title/body came
|
||
from fetched, signature-verified relay content and that the tap opened the exact
|
||
triggering message. Keep fallback-to-channel and placeholder/failure cases as
|
||
explicit counts in the manual sample until privacy-preserving client telemetry
|
||
is designed.
|
||
|
||
Rollback does not require deleting credentials or mutating existing leases.
|
||
Set `BUZZ_PUSH_ENABLED=false` on the enabled relays to stop advertisement, lease
|
||
acceptance, matching, workers, and new gateway traffic. If the gateway itself
|
||
is unhealthy, disable the gateway deployment only after relay delivery is off.
|
||
Existing leases and gateway authorities then expire naturally. Adding an App
|
||
Store application profile is outside this internal evaluation.
|
||
|
||
## Helm production inputs
|
||
|
||
The chart defaults to the `main` image tag because `.github/workflows/docker.yml` publishes it from the push-gateway lane. For a production rollout, open that workflow run's **Publish public push gateway image** job summary and copy its `sha256:...` digest. Verify the published subject and provenance before injecting it:
|
||
|
||
```bash
|
||
gh attestation verify \
|
||
oci://ghcr.io/block/buzz-push-gateway@sha256:<64-lowercase-hex> \
|
||
--owner block
|
||
```
|
||
|
||
Only after that command succeeds, set the exact digest as `image.digest`; the chart then renders `ghcr.io/block/buzz-push-gateway@sha256:...` and ignores the mutable tag. `values-production.yaml` is an intentionally invalid production-input contract: deployment CI must inject this verified `image.digest`, the provisioned dogfood Apple application identifier, an environment-owned Gateway parent reference, and the actual PostgreSQL network. Schema validation rejects the artifact when any remains empty; the render guard proves both rejection and a fully injected render.
|
||
|
||
Network policy keeps APNs HTTPS and PostgreSQL egress in separate CIDR lists. APNs currently requires broad TCP/443 reachability; `networkPolicy.postgresEgressCidrs` must be narrowed to the production database network, and the DNS namespace/pod selectors must match the cluster DNS deployment. The sample private CIDR is not a claim about the production topology.
|
||
|
||
Kubernetes does not restart pods when referenced Secret bytes change. AEAD or APNs certificate rotation therefore requires an explicit rolling restart after the secret manager update (for example, `kubectl rollout restart deployment/<release>-buzz-push-gateway`) and readiness verification before removing predecessor keys. Service-account token automount is disabled.
|
||
|
||
## Gateway chart release
|
||
|
||
The gateway chart has a collision-free release lane separate from the main
|
||
`buzz` chart. To publish version `X.Y.Z`, update both `version` and `appVersion`
|
||
in `deploy/charts/buzz-push-gateway/Chart.yaml`, validate the chart, and open a
|
||
same-repository PR whose branch is exactly `push-chart-release/X.Y.Z`:
|
||
|
||
```bash
|
||
deploy/charts/buzz-push-gateway/tests/render.sh
|
||
git switch -c push-chart-release/X.Y.Z
|
||
git add deploy/charts/buzz-push-gateway/Chart.yaml
|
||
git commit -m "release: push gateway chart X.Y.Z"
|
||
git push -u origin push-chart-release/X.Y.Z
|
||
```
|
||
|
||
When that PR merges, `.github/workflows/auto-tag-on-release-pr-merge.yml`
|
||
creates `push-chart-vX.Y.Z` and dispatches
|
||
`.github/workflows/push-gateway-helm-chart.yml` with that immutable tag and bare
|
||
version. The publisher verifies the checked-out commit is the tag target and the
|
||
chart version equals `X.Y.Z` before pushing
|
||
`oci://ghcr.io/block/buzz/charts/buzz-push-gateway`. A manually pushed
|
||
`push-chart-vX.Y.Z` tag is the documented rescue path and runs the same checks.
|