mirror of
https://github.com/block/buzz.git
synced 2026-11-06 17:31:19 +00:00
## Summary `BuzzPushSnapshotEnrichmentTests` (iOS RunnerTests) timed out on two unrelated PR branches on 2026-11-01 and passed on re-run both times. The timings point to a starved test host, not a missed callback. This PR makes one production change and two test changes, in two files: - **`mobile/ios/Runner/PushSnapshotBridge.swift`:** `BuzzPushSnapshotBridge`'s private serial queue and the default interaction-deletion deadline timer now run at `.userInitiated` instead of `.utility`. Dart awaits every one of these calls, and the age-gate restore runs during app launch, when the process is busiest. User-initiated is the class for work a caller is waiting on. - **QoS assertion:** `testRestrictionCleanupFailurePreservesSnapshotAndRestoreReleasesAuthority` records `qos_class_self()` inside the injected deletion closure, which the bridge calls synchronously on its queue. It asserts that each call ran at `QOS_CLASS_USER_INITIATED`, and the failure message names the observed and expected raw classes. - **`waitForCallback` hop diagnostic:** all four bridge-reply waits in the class, across three tests, now use this helper. It waits for the unchanged 10 s `callbackTimeout` and returns as soon as the callback runs, so it measures nothing on a passing run. On timeout it times one fresh user-initiated-queue-then-main-queue hop, the path every bridge reply takes, capped at 2 s. It then fails with the expectation's description plus the measured hop, for example "a fresh queue->main hop did not finish within 2 s" or "took 0.4 ms". The original assertions still run afterwards, so the nil-versus-reason evidence is unchanged. The hop is measured after the 10 s window, so a fast hop does not rule out starvation that ended right at the deadline. No timeout changed. ### Related issue None found. #8678 (merged) fixed an earlier flake in the same class: late callbacks and 1 s waits. It introduced the 10 s `callbackTimeout` this PR keeps. ### Failing runs Both runs were in "Mobile Swift (Native)", step "Test native controls and Runner unit tests", attempt 1. Both passed on attempt 2. - [Run 41327410782, job 124910378184](https://github.com/block/buzz/actions/runs/41327410782/job/124910378184), on `mobile-avatar-thumb-followup` (#8896): `testRestrictionCleanupFailurePreservesSnapshotAndRestoreReleasesAuthority` timed out its 10 s "allowed without restoring snapshot" wait (19.3 s total). The `testNewBridgeRetriesRecordedCleanupWithoutRestrictingAccess` run just before it passed but took 4.3 s. - [Run 41325606910, job 124905175903](https://github.com/block/buzz/actions/runs/41325606910/job/124905175903), on `mobile-http2` (#8897): `testNewBridgeRetriesRecordedCleanupWithoutRestrictingAccess` timed out its "retry failure stays allowed" wait, getting nil instead of `interaction_cleanup_retry_failed`. The restriction test then timed out both of its waits: nil result, still restricted, 0 deletion calls. ### Diagnosis Every reply from `BuzzPushSnapshotBridge` goes through a private serial queue, previously created at `.utility` QoS, and then `DispatchQueue.main`. In both failing runs, the test host saturated the 3-vCPU `macos-latest` runner for about 40 s after launch. The test host is the full Flutter app, loaded as `TEST_HOST`. - `BrowserAuthBridgeTests.testAllowsOnlyBuilderlabLoginPages`, a synchronous main-thread URL-parsing test, took 2.65 s instead of 0.013 s. - `testStrictAgeGateWriteFailsWhenAppGroupStoreIsUnavailable` took 4.9 s. That test has since been deleted from `main`. - A trivial hop took 1.3 s, then more than 10 s, during that window. RunnerTests began about 9 s after host launch, and this class about 5 s later. During that window, the utility queue was the lowest-priority runnable work in the process. It got no CPU for 10–20 s, while main-thread work kept moving slowly. In both passing re-runs, every test in this class took under 0.3 s. The test logic doesn't explain a callback that never arrives: - The deletion mocks are synchronous. - The deadline scheduler is a no-op. - Each test uses its own temp directory and bridge. - No state is shared between tests. That rules out both a state leak and an expectation-ordering race. This diagnosis is inferred from CI timings. The starvation did not reproduce on a 16-core local host (see Testing). ### Tried and dropped - **A class `setUp` settle barrier (ff19edf0d, removed in6cc43c9fd).** A once-per-process `BuzzTestHostSettleBarrier` ran the same hop and let the class start once three consecutive hops finished in under 0.5 s, capped at 60 s. Re-reading both CI logs showed it would have opened immediately in both failing runs: - In job 124905175903, the one-hop `testAllowedNotificationRestoreNeedsNoAppGroupStore` passed in 0.070 s at the start of the class. - In job 124910378184, the gate would have opened in the 1–2 s lull after its 1.287 s hop. Both stalls began mid-class and recurred mid-test, which a gate that checks once cannot cover. A version that held the gate for longer would just be a CI-only wait. - **Deriving `callbackTimeout` from a measured host-load factor.** This still waits inside each test and only scales the same timeout. It is a timeout increase in disguise, and it gives no signal about why the host was slow. ### Residual risk and remaining levers On a saturated 3-vCPU runner, `.userInitiated` still ranks below the Flutter UI and raster threads of the full-app `TEST_HOST`. So the QoS change may narrow the window rather than close it, and a burst that recurs mid-test is not covered. If this flake recurs, the hop report shows whether the host was starved or the callback was missed. The remaining levers are: - **The test host.** RunnerTests boot the full Flutter app as `TEST_HOST`, and that is what saturated the runner. #8678 tried keeping Flutter out of the host, and that did not work locally. - **The 10 s `callbackTimeout`.** ### Testing Local runs used Xcode 27.2 with the 27.2 SDK, the iOS 26.2 simulator runtime, and a 16-core host. CI's starvation did not reproduce locally. QoS change (4278e9944): - **Class in isolation:** 10/10 green, 0.1–0.4 s each. - **Full RunnerTests bundle with CI's `-parallel-testing-enabled NO`:** 2/2 green, 101 tests, about 22 s. - **Falsification:** with the queue temporarily set back to `.utility`, the new assertion fails. It observes `[17, 17]` (utility) instead of `[25, 25]`. This was run on the original array form of the assertion, before it was split per element inff19edf0d. - **Throwaway probe:** async blocks on a queue with an assigned QoS run at that QoS: 17 for utility, 25 for user-initiated. So the change alters real scheduling. - **Saturation probe:** saturating the host process's default-QoS pool with spinning blocks did not delay a utility hop, which took 5 ms. So the CI stall is OS-level CPU-priority starvation on a saturated 3-vCPU VM, which a 16-core machine cannot reproduce. The diagnosis rests on the CI timings above. Hop diagnostic (6cc43c9fd,e5515f765): - **Forced timeout (temporary variants, not committed):** when the bridge's retry deletion never called back, the test failed with "'retry failure stays allowed' not fulfilled within 10 s; a fresh queue->main hop took 0.4 ms", followed by the existing nil-versus-reason assertion failure. Adding a 0.7 s sleep inside the hop produced "took 705.3 ms". - **Class in isolation:** 10/10 green, 0.1–0.5 s each. - **Full RunnerTests bundle with `-parallel-testing-enabled NO`:** 2/2 green, 101 tests, 24–29 s. - **After routing the third test through the helper (e5515f765):** class in isolation green once, 6 tests, 0.05 s. All commits: - `git diff --check` is clean. - The repo has no swiftformat or swiftlint configuration. - The branch was rebased onto `main` with no conflicts. `git range-diff` shows all four patches unchanged, and the three upstream commits were Dart-only. - The tests were not re-run after the rebase. - Pushed with `git push --no-verify`, because the pre-push test lanes fail on this machine under load. ### Readiness - **Hosted CI** had not run on this branch when the PR was opened. The "Mobile Swift (Native)" lane is the check that matters. - **Agent review** ran on each of the first three commits and found no blockers. Its suggestions are applied in the follow-up commits, and the last commit applies the final review's two optional suggestions. - **Human confirmation** (AGENTS.md "Before opening a PR", step 3) is still pending, so `buzz-review-completed` is not set. The four commits are kept as written for the squash merge. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Matt Toohey <contact@matttoohey.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
265 lines
9.0 KiB
Swift
265 lines
9.0 KiB
Swift
import BuzzPushKit
|
|
import Flutter
|
|
import Foundation
|
|
import Intents
|
|
import UserNotifications
|
|
|
|
final class BuzzPushSnapshotBridge {
|
|
private let containerURL: () -> URL?
|
|
private let endpointGrantStore: BuzzPushEndpointGrantKeychainStore
|
|
private let interactionDeletionDeadline: BuzzInteractionDeletionDeadline
|
|
private let keychainAccessGroup: String?
|
|
// Flutter awaits every call here, and the age-gate restore runs during app
|
|
// launch, when this process is busiest. At utility QoS these replies were the
|
|
// lowest-priority work in the process and stalled for over 10 s on saturated
|
|
// CI hosts while main-thread work kept moving; user-initiated is the class
|
|
// for work a caller is waiting on.
|
|
private let queue = DispatchQueue(
|
|
label: "xyz.block.buzz.push-snapshot",
|
|
qos: .userInitiated
|
|
)
|
|
private lazy var store: BuzzPushPresentationCacheStore? = {
|
|
guard let container = containerURL() else { return nil }
|
|
return BuzzPushPresentationCacheStore(containerURL: container)
|
|
}()
|
|
private let ageRestrictionSession = BuzzAgeRestrictionSession()
|
|
|
|
init(
|
|
appGroupIdentifier: String?,
|
|
endpointGrantStore: BuzzPushEndpointGrantKeychainStore,
|
|
keychainAccessGroup: String?,
|
|
containerURL: (() -> URL?)? = nil,
|
|
interactionDeletionDeadline: BuzzInteractionDeletionDeadline =
|
|
BuzzInteractionDeletionDeadline(
|
|
timeout: 5,
|
|
deleteAllInteractions: { completion in
|
|
INInteraction.deleteAll(completion: completion)
|
|
},
|
|
scheduleTimeout: { delay, action in
|
|
DispatchQueue.global(qos: .userInitiated).asyncAfter(
|
|
deadline: .now() + delay,
|
|
execute: action
|
|
)
|
|
}
|
|
)
|
|
) {
|
|
self.containerURL = containerURL ?? {
|
|
guard let appGroupIdentifier else { return nil }
|
|
return FileManager.default.containerURL(
|
|
forSecurityApplicationGroupIdentifier: appGroupIdentifier)
|
|
}
|
|
self.endpointGrantStore = endpointGrantStore
|
|
self.keychainAccessGroup = keychainAccessGroup
|
|
self.interactionDeletionDeadline = interactionDeletionDeadline
|
|
}
|
|
|
|
@discardableResult
|
|
func handle(_ call: FlutterMethodCall, result: @escaping FlutterResult) -> Bool {
|
|
if call.method == "restoreAgeRestrictedNotifications" {
|
|
queue.async { [self] in
|
|
ageRestrictionSession.release()
|
|
BuzzInteractionCleanupRetry(
|
|
containerURL: containerURL(), deletion: interactionDeletionDeadline
|
|
).retryPending { error in
|
|
Self.complete(result, value: error.map {
|
|
FlutterError(code: "interaction_cleanup_retry_failed",
|
|
message: "Unable to finish pending interaction cleanup.",
|
|
details: $0.localizedDescription)
|
|
})
|
|
}
|
|
}
|
|
return true
|
|
}
|
|
if call.method == "purgeAgeRestrictedNotifications" {
|
|
purgeAgeRestrictedNotifications(result: result)
|
|
return true
|
|
}
|
|
guard call.method == "syncPushSnapshot",
|
|
let arguments = call.arguments as? [String: Any],
|
|
let section = arguments["section"] as? String
|
|
else {
|
|
return false
|
|
}
|
|
switch section {
|
|
case "communities":
|
|
syncCommunities(arguments, result: result)
|
|
case "organizationSession":
|
|
syncOrganizationSession(arguments, result: result)
|
|
default: return false
|
|
}
|
|
return true
|
|
}
|
|
|
|
private func purgeAgeRestrictedNotifications(result: @escaping FlutterResult) {
|
|
queue.async { [weak self] in
|
|
do {
|
|
guard let self, let container = containerURL() else {
|
|
throw NSError(
|
|
domain: "BuzzPushSnapshotBridge",
|
|
code: 1,
|
|
userInfo: [
|
|
NSLocalizedDescriptionKey: "The age-restriction session container is unavailable."
|
|
]
|
|
)
|
|
}
|
|
try ageRestrictionSession.restrict(containerURL: container)
|
|
// Preserve snapshots and credentials. Restriction authority expires
|
|
// with this process even when cleanup or a later storage read fails.
|
|
let center = UNUserNotificationCenter.current()
|
|
center.removeAllDeliveredNotifications()
|
|
center.removeAllPendingNotificationRequests()
|
|
BuzzInteractionCleanupRetry(
|
|
containerURL: container, deletion: interactionDeletionDeadline
|
|
).request { error in
|
|
Self.complete(
|
|
result,
|
|
value: error.map {
|
|
FlutterError(
|
|
code: "age_restriction_purge_failed",
|
|
message: "Unable to suppress and purge restricted notifications.",
|
|
details: $0.localizedDescription
|
|
)
|
|
}
|
|
)
|
|
}
|
|
} catch {
|
|
Self.complete(
|
|
result,
|
|
value: FlutterError(
|
|
code: "age_restriction_purge_failed",
|
|
message: "Unable to suppress and purge restricted notifications.",
|
|
details: error.localizedDescription
|
|
)
|
|
)
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Shares the organization account session with the notification extension, or removes it
|
|
/// when the app has none, so the extension can mint NIP-FI assertions for organization relays.
|
|
private func syncOrganizationSession(
|
|
_ arguments: [String: Any],
|
|
result: @escaping FlutterResult
|
|
) {
|
|
let session: BuzzOrganizationSession?
|
|
if arguments["credential"] is NSNull || arguments["credential"] == nil {
|
|
session = nil
|
|
} else if let origin = arguments["origin"] as? String,
|
|
let credential = arguments["credential"] as? String, !credential.isEmpty,
|
|
credential.utf8.count <= 512,
|
|
let relayHosts = arguments["relayHosts"] as? [String], !relayHosts.isEmpty
|
|
{
|
|
session = BuzzOrganizationSession(
|
|
origin: origin, credential: credential, relayHosts: relayHosts)
|
|
} else {
|
|
result(
|
|
FlutterError(
|
|
code: "invalid_arguments",
|
|
message: "Expected an organization origin, credential and relay hosts.",
|
|
details: nil
|
|
)
|
|
)
|
|
return
|
|
}
|
|
queue.async { [weak self] in
|
|
do {
|
|
try BuzzPushKeychain.replace(
|
|
organizationSession: session, accessGroup: self?.keychainAccessGroup)
|
|
Self.complete(result, value: nil)
|
|
} catch {
|
|
Self.complete(
|
|
result,
|
|
value: FlutterError(
|
|
code: "snapshot_sync_failed",
|
|
message: "Unable to share the organization session.",
|
|
details: error.localizedDescription
|
|
)
|
|
)
|
|
}
|
|
}
|
|
}
|
|
|
|
private func syncCommunities(
|
|
_ arguments: [String: Any],
|
|
result: @escaping FlutterResult
|
|
) {
|
|
guard let communities = arguments["communities"] as? [[String: Any]],
|
|
let signingKeys = arguments["signingKeys"] as? [String: String],
|
|
communities.count <= BuzzPushPresentationCacheStore.maximumCommunities
|
|
else {
|
|
result(
|
|
FlutterError(
|
|
code: "invalid_arguments",
|
|
message: "Expected bounded communities and signing keys.",
|
|
details: nil
|
|
)
|
|
)
|
|
return
|
|
}
|
|
queue.async { [weak self] in
|
|
do {
|
|
guard let self else {
|
|
Self.complete(result, value: nil)
|
|
return
|
|
}
|
|
guard let store else {
|
|
Self.complete(result, value: nil)
|
|
return
|
|
}
|
|
// Relay-metadata enrichment is optional presentation state. A damaged
|
|
// grant cache must not block the core NSE snapshot and key update.
|
|
let grants = (try? endpointGrantStore.records()) ?? []
|
|
let enriched = communities.map { community -> [String: Any] in
|
|
var community = community
|
|
guard let relayURL = community["relayUrl"] as? String,
|
|
let relayMetadataPubkey = Self.relayMetadataPubkey(
|
|
relayURL: relayURL,
|
|
grants: grants
|
|
)
|
|
else { return community }
|
|
community["relayMetadataPubkey"] = relayMetadataPubkey
|
|
return community
|
|
}
|
|
let data = try JSONSerialization.data(withJSONObject: enriched, options: [.sortedKeys])
|
|
let decoded = try JSONDecoder().decode([PushLeaseCommunity].self, from: data)
|
|
// Ordinary snapshot maintenance never changes age access.
|
|
try store.replaceCommunities(decoded)
|
|
try BuzzPushKeychain.replace(
|
|
signingKeys: signingKeys,
|
|
accessGroup: self.keychainAccessGroup
|
|
)
|
|
Self.complete(result, value: nil)
|
|
} catch {
|
|
Self.complete(
|
|
result,
|
|
value: FlutterError(
|
|
code: "snapshot_sync_failed",
|
|
message: "Unable to sync push community state.",
|
|
details: error.localizedDescription
|
|
)
|
|
)
|
|
}
|
|
}
|
|
}
|
|
|
|
static func relayMetadataPubkey(
|
|
relayURL: String,
|
|
grants: [BuzzPushEndpointGrantRecord]
|
|
) -> String? {
|
|
guard let origin = BuzzPushPresentationCacheStore.canonicalRelayOrigin(relayURL) else {
|
|
return nil
|
|
}
|
|
return grants.filter {
|
|
BuzzPushPresentationCacheStore.canonicalRelayOrigin($0.relayOrigin) == origin
|
|
}.max {
|
|
$0.generation < $1.generation
|
|
}?.relayMetadataPubkey
|
|
}
|
|
|
|
private static func complete(_ result: @escaping FlutterResult, value: Any?) {
|
|
DispatchQueue.main.async {
|
|
result(value)
|
|
}
|
|
}
|
|
}
|