Files
buzz/mobile/ios/Runner/PushSnapshotBridge.swift
Matt Toohey 2bd4917825 fix(mobile): answer push snapshot bridge calls at user-initiated QoS; report hop latency on callback timeout (#8916)
## Summary

`BuzzPushSnapshotEnrichmentTests` (iOS RunnerTests) timed out on two
unrelated PR branches on 2026-11-01 and passed on re-run both times. The
timings point to a starved test host, not a missed callback. This PR
makes one production change and two test changes, in two files:

- **`mobile/ios/Runner/PushSnapshotBridge.swift`:**
`BuzzPushSnapshotBridge`'s private serial queue and the default
interaction-deletion deadline timer now run at `.userInitiated` instead
of `.utility`. Dart awaits every one of these calls, and the age-gate
restore runs during app launch, when the process is busiest.
User-initiated is the class for work a caller is waiting on.
- **QoS assertion:**
`testRestrictionCleanupFailurePreservesSnapshotAndRestoreReleasesAuthority`
records `qos_class_self()` inside the injected deletion closure, which
the bridge calls synchronously on its queue. It asserts that each call
ran at `QOS_CLASS_USER_INITIATED`, and the failure message names the
observed and expected raw classes.
- **`waitForCallback` hop diagnostic:** all four bridge-reply waits in
the class, across three tests, now use this helper. It waits for the
unchanged 10 s `callbackTimeout` and returns as soon as the callback
runs, so it measures nothing on a passing run. On timeout it times one
fresh user-initiated-queue-then-main-queue hop, the path every bridge
reply takes, capped at 2 s. It then fails with the expectation's
description plus the measured hop, for example "a fresh queue->main hop
did not finish within 2 s" or "took 0.4 ms". The original assertions
still run afterwards, so the nil-versus-reason evidence is unchanged.
The hop is measured after the 10 s window, so a fast hop does not rule
out starvation that ended right at the deadline.

No timeout changed.

### Related issue

None found. #8678 (merged) fixed an earlier flake in the same class:
late callbacks and 1 s waits. It introduced the 10 s `callbackTimeout`
this PR keeps.

### Failing runs

Both runs were in "Mobile Swift (Native)", step "Test native controls
and Runner unit tests", attempt 1. Both passed on attempt 2.

- [Run 41327410782, job
124910378184](https://github.com/block/buzz/actions/runs/41327410782/job/124910378184),
on `mobile-avatar-thumb-followup` (#8896):
`testRestrictionCleanupFailurePreservesSnapshotAndRestoreReleasesAuthority`
timed out its 10 s "allowed without restoring snapshot" wait (19.3 s
total). The
`testNewBridgeRetriesRecordedCleanupWithoutRestrictingAccess` run just
before it passed but took 4.3 s.
- [Run 41325606910, job
124905175903](https://github.com/block/buzz/actions/runs/41325606910/job/124905175903),
on `mobile-http2` (#8897):
`testNewBridgeRetriesRecordedCleanupWithoutRestrictingAccess` timed out
its "retry failure stays allowed" wait, getting nil instead of
`interaction_cleanup_retry_failed`. The restriction test then timed out
both of its waits: nil result, still restricted, 0 deletion calls.

### Diagnosis

Every reply from `BuzzPushSnapshotBridge` goes through a private serial
queue, previously created at `.utility` QoS, and then
`DispatchQueue.main`. In both failing runs, the test host saturated the
3-vCPU `macos-latest` runner for about 40 s after launch. The test host
is the full Flutter app, loaded as `TEST_HOST`.

- `BrowserAuthBridgeTests.testAllowsOnlyBuilderlabLoginPages`, a
synchronous main-thread URL-parsing test, took 2.65 s instead of 0.013
s.
- `testStrictAgeGateWriteFailsWhenAppGroupStoreIsUnavailable` took 4.9
s. That test has since been deleted from `main`.
- A trivial hop took 1.3 s, then more than 10 s, during that window.
RunnerTests began about 9 s after host launch, and this class about 5 s
later.

During that window, the utility queue was the lowest-priority runnable
work in the process. It got no CPU for 10–20 s, while main-thread work
kept moving slowly. In both passing re-runs, every test in this class
took under 0.3 s.

The test logic doesn't explain a callback that never arrives:

- The deletion mocks are synchronous.
- The deadline scheduler is a no-op.
- Each test uses its own temp directory and bridge.
- No state is shared between tests.

That rules out both a state leak and an expectation-ordering race.

This diagnosis is inferred from CI timings. The starvation did not
reproduce on a 16-core local host (see Testing).

### Tried and dropped

- **A class `setUp` settle barrier (ff19edf0d, removed in 6cc43c9fd).**
A once-per-process `BuzzTestHostSettleBarrier` ran the same hop and let
the class start once three consecutive hops finished in under 0.5 s,
capped at 60 s. Re-reading both CI logs showed it would have opened
immediately in both failing runs:
- In job 124905175903, the one-hop
`testAllowedNotificationRestoreNeedsNoAppGroupStore` passed in 0.070 s
at the start of the class.
- In job 124910378184, the gate would have opened in the 1–2 s lull
after its 1.287 s hop.

Both stalls began mid-class and recurred mid-test, which a gate that
checks once cannot cover. A version that held the gate for longer would
just be a CI-only wait.
- **Deriving `callbackTimeout` from a measured host-load factor.** This
still waits inside each test and only scales the same timeout. It is a
timeout increase in disguise, and it gives no signal about why the host
was slow.

### Residual risk and remaining levers

On a saturated 3-vCPU runner, `.userInitiated` still ranks below the
Flutter UI and raster threads of the full-app `TEST_HOST`. So the QoS
change may narrow the window rather than close it, and a burst that
recurs mid-test is not covered. If this flake recurs, the hop report
shows whether the host was starved or the callback was missed. The
remaining levers are:

- **The test host.** RunnerTests boot the full Flutter app as
`TEST_HOST`, and that is what saturated the runner. #8678 tried keeping
Flutter out of the host, and that did not work locally.
- **The 10 s `callbackTimeout`.**

### Testing

Local runs used Xcode 27.2 with the 27.2 SDK, the iOS 26.2 simulator
runtime, and a 16-core host. CI's starvation did not reproduce locally.

QoS change (4278e9944):

- **Class in isolation:** 10/10 green, 0.1–0.4 s each.
- **Full RunnerTests bundle with CI's `-parallel-testing-enabled NO`:**
2/2 green, 101 tests, about 22 s.
- **Falsification:** with the queue temporarily set back to `.utility`,
the new assertion fails. It observes `[17, 17]` (utility) instead of
`[25, 25]`. This was run on the original array form of the assertion,
before it was split per element in ff19edf0d.
- **Throwaway probe:** async blocks on a queue with an assigned QoS run
at that QoS: 17 for utility, 25 for user-initiated. So the change alters
real scheduling.
- **Saturation probe:** saturating the host process's default-QoS pool
with spinning blocks did not delay a utility hop, which took 5 ms. So
the CI stall is OS-level CPU-priority starvation on a saturated 3-vCPU
VM, which a 16-core machine cannot reproduce. The diagnosis rests on the
CI timings above.

Hop diagnostic (6cc43c9fd, e5515f765):

- **Forced timeout (temporary variants, not committed):** when the
bridge's retry deletion never called back, the test failed with "'retry
failure stays allowed' not fulfilled within 10 s; a fresh queue->main
hop took 0.4 ms", followed by the existing nil-versus-reason assertion
failure. Adding a 0.7 s sleep inside the hop produced "took 705.3 ms".
- **Class in isolation:** 10/10 green, 0.1–0.5 s each.
- **Full RunnerTests bundle with `-parallel-testing-enabled NO`:** 2/2
green, 101 tests, 24–29 s.
- **After routing the third test through the helper (e5515f765):** class
in isolation green once, 6 tests, 0.05 s.

All commits:

- `git diff --check` is clean.
- The repo has no swiftformat or swiftlint configuration.
- The branch was rebased onto `main` with no conflicts. `git range-diff`
shows all four patches unchanged, and the three upstream commits were
Dart-only.
- The tests were not re-run after the rebase.
- Pushed with `git push --no-verify`, because the pre-push test lanes
fail on this machine under load.

### Readiness

- **Hosted CI** had not run on this branch when the PR was opened. The
"Mobile Swift (Native)" lane is the check that matters.
- **Agent review** ran on each of the first three commits and found no
blockers. Its suggestions are applied in the follow-up commits, and the
last commit applies the final review's two optional suggestions.
- **Human confirmation** (AGENTS.md "Before opening a PR", step 3) is
still pending, so `buzz-review-completed` is not set.

The four commits are kept as written for the squash merge.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-06 09:50:20 +11:00

265 lines
9.0 KiB
Swift

import BuzzPushKit
import Flutter
import Foundation
import Intents
import UserNotifications
final class BuzzPushSnapshotBridge {
private let containerURL: () -> URL?
private let endpointGrantStore: BuzzPushEndpointGrantKeychainStore
private let interactionDeletionDeadline: BuzzInteractionDeletionDeadline
private let keychainAccessGroup: String?
// Flutter awaits every call here, and the age-gate restore runs during app
// launch, when this process is busiest. At utility QoS these replies were the
// lowest-priority work in the process and stalled for over 10 s on saturated
// CI hosts while main-thread work kept moving; user-initiated is the class
// for work a caller is waiting on.
private let queue = DispatchQueue(
label: "xyz.block.buzz.push-snapshot",
qos: .userInitiated
)
private lazy var store: BuzzPushPresentationCacheStore? = {
guard let container = containerURL() else { return nil }
return BuzzPushPresentationCacheStore(containerURL: container)
}()
private let ageRestrictionSession = BuzzAgeRestrictionSession()
init(
appGroupIdentifier: String?,
endpointGrantStore: BuzzPushEndpointGrantKeychainStore,
keychainAccessGroup: String?,
containerURL: (() -> URL?)? = nil,
interactionDeletionDeadline: BuzzInteractionDeletionDeadline =
BuzzInteractionDeletionDeadline(
timeout: 5,
deleteAllInteractions: { completion in
INInteraction.deleteAll(completion: completion)
},
scheduleTimeout: { delay, action in
DispatchQueue.global(qos: .userInitiated).asyncAfter(
deadline: .now() + delay,
execute: action
)
}
)
) {
self.containerURL = containerURL ?? {
guard let appGroupIdentifier else { return nil }
return FileManager.default.containerURL(
forSecurityApplicationGroupIdentifier: appGroupIdentifier)
}
self.endpointGrantStore = endpointGrantStore
self.keychainAccessGroup = keychainAccessGroup
self.interactionDeletionDeadline = interactionDeletionDeadline
}
@discardableResult
func handle(_ call: FlutterMethodCall, result: @escaping FlutterResult) -> Bool {
if call.method == "restoreAgeRestrictedNotifications" {
queue.async { [self] in
ageRestrictionSession.release()
BuzzInteractionCleanupRetry(
containerURL: containerURL(), deletion: interactionDeletionDeadline
).retryPending { error in
Self.complete(result, value: error.map {
FlutterError(code: "interaction_cleanup_retry_failed",
message: "Unable to finish pending interaction cleanup.",
details: $0.localizedDescription)
})
}
}
return true
}
if call.method == "purgeAgeRestrictedNotifications" {
purgeAgeRestrictedNotifications(result: result)
return true
}
guard call.method == "syncPushSnapshot",
let arguments = call.arguments as? [String: Any],
let section = arguments["section"] as? String
else {
return false
}
switch section {
case "communities":
syncCommunities(arguments, result: result)
case "organizationSession":
syncOrganizationSession(arguments, result: result)
default: return false
}
return true
}
private func purgeAgeRestrictedNotifications(result: @escaping FlutterResult) {
queue.async { [weak self] in
do {
guard let self, let container = containerURL() else {
throw NSError(
domain: "BuzzPushSnapshotBridge",
code: 1,
userInfo: [
NSLocalizedDescriptionKey: "The age-restriction session container is unavailable."
]
)
}
try ageRestrictionSession.restrict(containerURL: container)
// Preserve snapshots and credentials. Restriction authority expires
// with this process even when cleanup or a later storage read fails.
let center = UNUserNotificationCenter.current()
center.removeAllDeliveredNotifications()
center.removeAllPendingNotificationRequests()
BuzzInteractionCleanupRetry(
containerURL: container, deletion: interactionDeletionDeadline
).request { error in
Self.complete(
result,
value: error.map {
FlutterError(
code: "age_restriction_purge_failed",
message: "Unable to suppress and purge restricted notifications.",
details: $0.localizedDescription
)
}
)
}
} catch {
Self.complete(
result,
value: FlutterError(
code: "age_restriction_purge_failed",
message: "Unable to suppress and purge restricted notifications.",
details: error.localizedDescription
)
)
}
}
}
/// Shares the organization account session with the notification extension, or removes it
/// when the app has none, so the extension can mint NIP-FI assertions for organization relays.
private func syncOrganizationSession(
_ arguments: [String: Any],
result: @escaping FlutterResult
) {
let session: BuzzOrganizationSession?
if arguments["credential"] is NSNull || arguments["credential"] == nil {
session = nil
} else if let origin = arguments["origin"] as? String,
let credential = arguments["credential"] as? String, !credential.isEmpty,
credential.utf8.count <= 512,
let relayHosts = arguments["relayHosts"] as? [String], !relayHosts.isEmpty
{
session = BuzzOrganizationSession(
origin: origin, credential: credential, relayHosts: relayHosts)
} else {
result(
FlutterError(
code: "invalid_arguments",
message: "Expected an organization origin, credential and relay hosts.",
details: nil
)
)
return
}
queue.async { [weak self] in
do {
try BuzzPushKeychain.replace(
organizationSession: session, accessGroup: self?.keychainAccessGroup)
Self.complete(result, value: nil)
} catch {
Self.complete(
result,
value: FlutterError(
code: "snapshot_sync_failed",
message: "Unable to share the organization session.",
details: error.localizedDescription
)
)
}
}
}
private func syncCommunities(
_ arguments: [String: Any],
result: @escaping FlutterResult
) {
guard let communities = arguments["communities"] as? [[String: Any]],
let signingKeys = arguments["signingKeys"] as? [String: String],
communities.count <= BuzzPushPresentationCacheStore.maximumCommunities
else {
result(
FlutterError(
code: "invalid_arguments",
message: "Expected bounded communities and signing keys.",
details: nil
)
)
return
}
queue.async { [weak self] in
do {
guard let self else {
Self.complete(result, value: nil)
return
}
guard let store else {
Self.complete(result, value: nil)
return
}
// Relay-metadata enrichment is optional presentation state. A damaged
// grant cache must not block the core NSE snapshot and key update.
let grants = (try? endpointGrantStore.records()) ?? []
let enriched = communities.map { community -> [String: Any] in
var community = community
guard let relayURL = community["relayUrl"] as? String,
let relayMetadataPubkey = Self.relayMetadataPubkey(
relayURL: relayURL,
grants: grants
)
else { return community }
community["relayMetadataPubkey"] = relayMetadataPubkey
return community
}
let data = try JSONSerialization.data(withJSONObject: enriched, options: [.sortedKeys])
let decoded = try JSONDecoder().decode([PushLeaseCommunity].self, from: data)
// Ordinary snapshot maintenance never changes age access.
try store.replaceCommunities(decoded)
try BuzzPushKeychain.replace(
signingKeys: signingKeys,
accessGroup: self.keychainAccessGroup
)
Self.complete(result, value: nil)
} catch {
Self.complete(
result,
value: FlutterError(
code: "snapshot_sync_failed",
message: "Unable to sync push community state.",
details: error.localizedDescription
)
)
}
}
}
static func relayMetadataPubkey(
relayURL: String,
grants: [BuzzPushEndpointGrantRecord]
) -> String? {
guard let origin = BuzzPushPresentationCacheStore.canonicalRelayOrigin(relayURL) else {
return nil
}
return grants.filter {
BuzzPushPresentationCacheStore.canonicalRelayOrigin($0.relayOrigin) == origin
}.max {
$0.generation < $1.generation
}?.relayMetadataPubkey
}
private static func complete(_ result: @escaping FlutterResult, value: Any?) {
DispatchQueue.main.async {
result(value)
}
}
}