Files
Tom Brow 868e2f8f32 refactor(push): one app profile per gateway (#8206)
Buzz push currently selects an application profile, such as `dogfood`,
during client registration. We want each gateway to serve one Apple app
configured by the operator instead. This removes profile selection from
the client, relay, and gateway, while allowing internal and public
gateways to use their own Apple identities.

This PR implements that model in the Buzz source and upstream deployment
chart. Discovery, enrollment, grants, and leases no longer carry an
application-profile selector. Each gateway has one configured Apple
application and APNs transport.

The protocol changes are incompatible with the old registration format,
so relays advertise `buzz-push-v1` instead of `nip-pl`. Old clients do
not discover the new push capability. Updated clients use separate
Keychain records and register again. The [push
notes](https://github.com/block/buzz/blob/push-capability-v1/docs/buzz-push.md)
describe the experimental contract.

Landing this PR is the first step. Publish the matching gateway image
and upstream chart, then land the [shared chart
update](https://github.com/squareup/cash-helm-charts/pull/5740) and
[gateway deployment
update](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/1085).
Before cutover, stop old gateway instances and check legacy registration
state. Migrations refuse to proceed if old gateway installations or
active relay leases remain; they do not delete that state.
Physical-device verification remains necessary before production
activation.

### Validation
Gateway and focused Flutter push tests pass, as do the Swift package,
simulator build, and native notification/enrichment tests. A local relay
advertises `buzz-push-v1` without `nip-pl` or `app_profiles`. The
PostgreSQL suite passed 868 tests; its two migration-test failures were
corrected and pass a focused rerun together with the gateway cutover
guard. Chart rendering and release-contract checks pass.

Desktop JavaScript tests, desktop and web builds, native desktop
compilation, mobile analysis/tests, and desktop native tests pass. The
full Rust unit lane and prescribed `just test` command pass, including
workspace integration tests against task-isolated local services. The
production overlay now requires an explicitly supplied APNs topic, with
omission and rendered-value regression coverage. The corrected Swift
checksum assertion passes the complete package suite: 78 XCTest tests
and 42 Swift Testing tests. No physical-device enrollment or production
cutover is claimed.

Local merge validation: the embedded migration test and formatting pass
after reserving profile migration 0060 behind rollback migration 0059.
The local pre-push Rust lane failed ACP timing/steering tests
(`idle_resets_on_stdout_activity`,
`keepalive_resets_idle_past_deadline`, and the failed/missing-outcome
steering tests). ACP and its dependency manifests are unchanged by this
branch; all four pass in an isolated single-thread rerun. The documented
local-check exception skips only the Rust pre-push lane for publication;
required CI remains mandatory. Mobile and desktop pre-push lanes passed.

Combined migration validation: after rollback 0059 landed, the main
merge preserves profile migration 0060 and both embedded migration
assertions. The focused migration test and formatting pass. The next
local push again failed only the unchanged ACP idle/keepalive timing
tests (1053 passed, 2 failed, 3 ignored); other lanes passed.
Publication uses the same documented local Rust-lane exception; required
CI remains mandatory.

Follow-up review fix: removed the legacy profile column from the
copied-lease fixture. All four event-follow-up contract tests pass on
isolated PostgreSQL databases, covering desired and migrated schemas in
dual and app-only modes. Formatting passes. Publication retains the
prior passing local-lane evidence and uses the authorized pre-push
exception for the known unrelated ACP timing failures; required CI
remains mandatory.

---------

Signed-off-by: Tom Brow <tomb@block.xyz>
2026-10-09 15:09:53 -07:00

273 lines
13 KiB
Bash
Executable File

#!/usr/bin/env bash
set -euo pipefail
bash deploy/charts/buzz-push-gateway/tests/runtime-options.sh
bash deploy/charts/buzz-push-gateway/tests/grant-lifetime.sh
bash deploy/charts/buzz-push-gateway/tests/migration-annotations.sh
out=$(mktemp); production_out=$(mktemp); route_out=$(mktemp); datadog_out=$(mktemp)
trap 'rm -f "$out" "$production_out" "$route_out" "$datadog_out" "${monitoring_out:-}"' EXIT
gateway_origin_arg=(--set 'gatewayOrigin=https://push.example')
# Generic values work without an application origin.
helm lint deploy/charts/buzz-push-gateway >/dev/null
helm template push deploy/charts/buzz-push-gateway >"$out"
# Production values support a platform-owned ingress without rendering an
# HTTPRoute. The environment-owned inputs remain mandatory.
production_args=(
-f deploy/charts/buzz-push-gateway/values-production.yaml
--set 'image.digest=sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'
--set 'application.appAttestAppId=REALTEAM.com.example.production'
--set 'networkPolicy.postgresEgressCidrs[0]=10.42.0.0/16'
)
if helm template push deploy/charts/buzz-push-gateway "${production_args[@]}" >/dev/null 2>&1; then
echo 'expected production values without an APNs topic to fail' >&2
exit 1
fi
production_args+=(--set 'application.apnsTopic=com.example.production')
helm lint deploy/charts/buzz-push-gateway "${production_args[@]}" >/dev/null
helm template push deploy/charts/buzz-push-gateway "${production_args[@]}" >"$production_out"
# Gateway API remains an explicit supported ingress mode when an operator opts
# in and supplies the environment-owned parent.
helm template push deploy/charts/buzz-push-gateway \
"${gateway_origin_arg[@]}" \
--set httpRoute.enabled=true \
--set 'httpRoute.parentRefs[0].name=production-gateway' \
--set 'httpRoute.parentRefs[0].namespace=gateway-system' \
>"$route_out"
env -u GEM_HOME -u GEM_PATH -u RUBYLIB -u RUBYOPT ruby -ryaml -rset \
- "$out" "$production_out" "$route_out" <<'RUBY'
def assert!(condition, detail = "assertion failed")
raise detail unless condition
end
# The application serves plaintext HTTP behind ingress TLS termination. Check
# every ingress mode so service discovery never advertises application TLS.
ARGV.each do |path|
resources = YAML.load_stream(File.read(path)).compact
service = resources.find { |x| x["kind"] == "Service" }
ports = service.dig("spec", "ports")
assert!(ports == [{ "name" => "http", "port" => 8080, "targetPort" => "public" }], ports.inspect)
deployment = resources.find { |x| x["kind"] == "Deployment" }
container_ports = deployment.dig("spec", "template", "spec", "containers", 0, "ports")
assert!(container_ports.any? { |port| port["name"] == "public" && port["containerPort"] == 8080 })
end
xs = YAML.load_stream(File.read(ARGV[0])).compact
svc = xs.find { |x| x["kind"] == "Service" }
assert!(svc.dig("spec", "ports").map { |port| port["targetPort"] } == ["public"])
d = xs.find { |x| x["kind"] == "Deployment" }
j = xs.find { |x| x["kind"] == "Job" }
runtime = {
"app.kubernetes.io/name" => "buzz-push-gateway",
"app.kubernetes.io/instance" => "push",
"app.kubernetes.io/component" => "runtime",
}
migration = runtime.merge("app.kubernetes.io/component" => "migration")
assert!(svc.dig("spec", "selector") == runtime)
assert!(d.dig("spec", "selector", "matchLabels") == runtime)
assert!(d.dig("spec", "template", "metadata", "labels") == runtime)
assert!(d.dig("spec", "template", "metadata", "annotations").nil?)
assert!(j.dig("spec", "template", "metadata", "labels") == migration)
assert!(svc.dig("spec", "selector") != j.dig("spec", "template", "metadata", "labels"))
jenv = j.dig("spec", "template", "spec", "containers", 0, "env").to_h { |entry| [entry["name"], entry] }
assert!(jenv.dig("BUZZ_PUSH_RUNTIME_DATABASE_ROLE", "value") == "buzz_push_gateway_runtime")
assert!(jenv.fetch("DATABASE_URL").key?("valueFrom"))
assert!(j.dig("spec", "template", "spec", "containers", 0, "args") == ["--migrate-only"])
assert!(j.dig("metadata", "annotations") == {
"helm.sh/hook" => "pre-install,pre-upgrade",
"helm.sh/hook-weight" => "-5",
"helm.sh/hook-delete-policy" => "before-hook-creation,hook-succeeded",
})
env_names = d.dig("spec", "template", "spec", "containers", 0, "env")
.map { |entry| entry["name"] }.to_set
required = Set.new(%w[
DATABASE_URL BUZZ_PUSH_APNS_CERT_PATH
BUZZ_PUSH_APNS_TOPIC BUZZ_PUSH_APP_ATTEST_APP_ID
BUZZ_PUSH_GRANT_KEYS BUZZ_PUSH_TOKEN_KEYS BUZZ_PUSH_MAX_GRANT_LIFETIME_SECONDS
])
assert!(required.subset?(env_names))
assert!(!env_names.include?("BUZZ_PUSH_GATEWAY_ORIGIN"))
assert!(!env_names.any? { |name| name.include?("APP_STORE") })
apns_volume = d.dig("spec", "template", "spec", "volumes").find { |volume| volume["name"] == "apns" }
assert!(apns_volume.dig("secret", "defaultMode") == 0o400, apns_volume.inspect)
assert!(d.dig("spec", "replicas") >= 2)
assert!(!xs.any? { |x| x["kind"] == "HTTPRoute" })
# Observability is opt-in: default render exposes no scrape CRDs and 8081 stays
# free of pod ingress (only 8080 is reachable).
assert!(!xs.any? { |x| %w[PodMonitor PrometheusRule].include?(x["kind"]) })
nps = xs.select { |x| x["kind"] == "NetworkPolicy" }
np = nps.find { |x| x.dig("metadata", "name") == "push-buzz-push-gateway" }
migration_np = nps.find { |x| x.dig("metadata", "name") == "push-buzz-push-gateway-migration" }
assert!(np.dig("spec", "podSelector", "matchLabels") == runtime)
assert!(migration_np.dig("spec", "podSelector", "matchLabels") == migration)
assert!(migration_np.dig("metadata", "annotations") == {
"helm.sh/hook" => "pre-install,pre-upgrade",
"helm.sh/hook-weight" => "-10",
"helm.sh/hook-delete-policy" => "before-hook-creation",
})
assert!(migration_np.dig("metadata", "annotations", "helm.sh/hook-weight").to_i < j.dig("metadata", "annotations", "helm.sh/hook-weight").to_i)
assert!(migration_np.dig("spec", "ingress") == [])
assert!(migration_np.dig("spec", "policyTypes") == %w[Ingress Egress])
migration_ports = migration_np.dig("spec", "egress")
.flat_map { |rule| rule.fetch("ports", []) }.map { |port| port["port"] }.to_set
assert!(migration_ports == Set[53, 5432], migration_ports.inspect)
assert!(!migration_np.dig("spec", "egress").flat_map { |rule| rule.fetch("ports", []) }.any? { |port| port["port"] == 443 })
ingress_ports = np.dig("spec", "ingress")
.flat_map { |rule| rule.fetch("ports", []) }.map { |port| port["port"] }.to_set
assert!(ingress_ports == Set[8080], ingress_ports.inspect)
production = YAML.load_stream(File.read(ARGV[1])).compact
assert!(!production.any? { |x| x["kind"] == "HTTPRoute" })
production_deployment = production.find { |x| x["kind"] == "Deployment" }
production_image = production_deployment.dig("spec", "template", "spec", "containers", 0, "image")
assert!(production_image == "ghcr.io/block/buzz-push-gateway@sha256:#{"a" * 64}", production_image.inspect)
production_env = production_deployment.dig("spec", "template", "spec", "containers", 0, "env")
assert!(production_env.find { |env| env["name"] == "BUZZ_PUSH_APNS_TOPIC" }["value"] == "com.example.production")
route = YAML.load_stream(File.read(ARGV[2])).compact.find { |x| x["kind"] == "HTTPRoute" }
assert!(!route.dig("spec", "parentRefs").empty?)
assert!(route.dig("spec", "hostnames") == ["push.example"])
RUBY
# An origin is needed only to produce the optional routing hostname.
if helm template push deploy/charts/buzz-push-gateway \
--set httpRoute.enabled=true --set 'httpRoute.parentRefs[0].name=gateway' >/dev/null 2>&1; then
echo 'expected enabled HTTPRoute without gatewayOrigin to fail' >&2
exit 1
fi
for invalid_origin in 'http://push.example' 'https://push.example:8443' 'https://push.example/base'; do
if helm template push deploy/charts/buzz-push-gateway \
--set "gatewayOrigin=$invalid_origin" >/dev/null 2>&1; then
echo "expected malformed gatewayOrigin $invalid_origin to fail" >&2
exit 1
fi
done
# Legacy token-auth values must fail rather than silently selecting the default
# certificate Secret.
if helm template push deploy/charts/buzz-push-gateway \
"${gateway_origin_arg[@]}" \
--set apnsKey.secretName=legacy-apns-secret \
--set apnsKey.secretKey=legacy-provider.p8 >/dev/null 2>&1; then
echo 'expected legacy apnsKey values to fail schema validation' >&2
exit 1
fi
# Enabling a route without a Gateway attachment must fail schema validation.
if helm template push deploy/charts/buzz-push-gateway "${gateway_origin_arg[@]}" --set httpRoute.enabled=true >/dev/null 2>&1; then
echo 'expected httpRoute.enabled=true without parentRefs to fail' >&2
exit 1
fi
# The checked-in production contract is intentionally undeployable until CI or
# the release system supplies its environment-owned values.
if helm template push deploy/charts/buzz-push-gateway -f deploy/charts/buzz-push-gateway/values-production.yaml >/dev/null 2>&1; then
echo 'expected uninjected production values to fail' >&2
exit 1
fi
# Enabling observability renders the scrape CRDs and adds a scoped 8081 ingress
# keyed to the named monitoring source — never a blanket 8081 rule.
monitoring_out=$(mktemp)
helm template push deploy/charts/buzz-push-gateway \
"${gateway_origin_arg[@]}" \
--set podMonitor.enabled=true \
--set prometheusRule.enabled=true \
--set networkPolicy.monitoring.enabled=true \
--set 'networkPolicy.monitoring.namespaceSelector.kubernetes\.io/metadata\.name=monitoring' \
--set 'networkPolicy.monitoring.podSelector.app\.kubernetes\.io/name=prometheus' \
>"$monitoring_out"
env -u GEM_HOME -u GEM_PATH -u RUBYLIB -u RUBYOPT ruby -ryaml -rset \
- "$monitoring_out" <<'RUBY'
def assert!(condition, detail = "assertion failed")
raise detail unless condition
end
xs = YAML.load_stream(File.read(ARGV[0])).compact
pm = xs.find { |x| x["kind"] == "PodMonitor" }
endpoint = pm.dig("spec", "podMetricsEndpoints", 0)
assert!(endpoint["port"] == "health" && endpoint["path"] == "/metrics", endpoint.inspect)
assert!(!xs.find { |x| x["kind"] == "PrometheusRule" }.dig("spec", "groups").empty?)
np = xs.find do |x|
x["kind"] == "NetworkPolicy" && x.dig("metadata", "name") == "push-buzz-push-gateway"
end
monitoring = np.dig("spec", "ingress").select do |rule|
rule.fetch("ports", []).map { |port| port["port"] }.to_set == Set[8081]
end
assert!(monitoring.length == 1, "exactly one scoped 8081 ingress rule")
from = monitoring[0].fetch("from")[0]
# 8081 ingress must be scoped by both selectors, never empty/blanket.
assert!(!from.dig("namespaceSelector", "matchLabels").empty? && !from.dig("podSelector", "matchLabels").empty?, from.inspect)
RUBY
# Datadog discovers the same private endpoint from pod annotations and needs no
# prometheus-operator CRDs. Its agent ingress remains selector-scoped.
helm lint deploy/charts/buzz-push-gateway \
-f deploy/charts/buzz-push-gateway/tests/datadog-values.yaml \
"${gateway_origin_arg[@]}" >/dev/null
helm template push deploy/charts/buzz-push-gateway \
-f deploy/charts/buzz-push-gateway/tests/datadog-values.yaml \
"${gateway_origin_arg[@]}" \
>"$datadog_out"
env -u GEM_HOME -u GEM_PATH -u RUBYLIB -u RUBYOPT ruby -rjson -ryaml -rset \
- "$datadog_out" <<'RUBY'
def assert!(condition, detail = "assertion failed")
raise detail unless condition
end
xs = YAML.load_stream(File.read(ARGV[0])).compact
assert!(!xs.any? { |x| %w[PodMonitor PrometheusRule].include?(x["kind"]) })
deployment = xs.find { |x| x["kind"] == "Deployment" }
raw_check = deployment.dig(
"spec", "template", "metadata", "annotations",
"ad.datadoghq.com/gateway.checks",
)
check = JSON.parse(raw_check)
instance = check.dig("openmetrics", "instances", 0)
assert!(instance["openmetrics_endpoint"] == "http://%%host%%:8081/metrics", instance.inspect)
assert!(instance["metrics"] == ["push_gateway_.*"], instance.inspect)
np = xs.find do |x|
x["kind"] == "NetworkPolicy" && x.dig("metadata", "name") == "push-buzz-push-gateway"
end
monitoring = np.dig("spec", "ingress").select do |rule|
rule.fetch("ports", []).map { |port| port["port"] }.to_set == Set[8081]
end
assert!(monitoring.length == 1, "exactly one scoped Datadog 8081 ingress rule")
from = monitoring[0].fetch("from")[0]
assert!(!from.dig("namespaceSelector", "matchLabels").empty?, from.inspect)
assert!(!from.dig("podSelector", "matchLabels").empty?, from.inspect)
RUBY
# Negative: monitoring enabled with default empty selectors must fail (would
# otherwise render a blanket 8081 rule matching all namespaces/pods).
if helm template push deploy/charts/buzz-push-gateway \
"${gateway_origin_arg[@]}" \
--set podMonitor.enabled=true \
--set networkPolicy.monitoring.enabled=true >/dev/null 2>&1; then
echo 'expected monitoring.enabled with empty selectors to fail' >&2
exit 1
fi
# Negative: PodMonitor without ingress is an unreachable scraper and must fail.
# Scoped ingress without PodMonitor is valid for annotation-discovered agents.
if helm template push deploy/charts/buzz-push-gateway \
"${gateway_origin_arg[@]}" \
--set podMonitor.enabled=true \
--set 'networkPolicy.monitoring.namespaceSelector.kubernetes\.io/metadata\.name=monitoring' \
--set 'networkPolicy.monitoring.podSelector.app\.kubernetes\.io/name=prometheus' \
>/dev/null 2>&1; then
echo 'expected podMonitor.enabled without monitoring ingress to fail' >&2
exit 1
fi
# Negative: retry-ratio threshold is a fraction; a value > 1 must fail schema.
if helm template push deploy/charts/buzz-push-gateway \
"${gateway_origin_arg[@]}" \
--set prometheusRule.enabled=true \
--set prometheusRule.apnsRetryRatioThreshold=2 >/dev/null 2>&1; then
echo 'expected apnsRetryRatioThreshold=2 to fail' >&2
exit 1
fi