From fae3f0346c44bbdf9629a7b5967965391c2bd22f Mon Sep 17 00:00:00 2001 From: Matt Toohey Date: Fri, 30 Oct 2026 11:58:40 +1100 Subject: [PATCH] test(mobile): keep late callbacks and fixed sleeps from failing tests (#8678) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Fixes three mobile flakes from the 21–28 Oct 2026 CI runs, in test code only: - In two iOS `RunnerTests` classes, late bridge callbacks no longer fail whichever test is running when they fire. - The waits on a busy test-host main thread now cap at 10 s. - Two Flutter avatar tests poll for the ready frame instead of sleeping a fixed 50 ms. This is one of seven PRs split out of the `flakey-tests` branch (siblings below). ### Related issue None found. ### Flaky tests fixed | Test | Failing runs | Cause | Change | |---|---|---|---| | `BuzzPushSnapshotEnrichmentTests` (RunnerTests) | 40362671989 (job 121712747802, main), 39681273328#1 (job 119504560518), 40658588384 (job 122715950541, PR #8640) | The 1 s waits ran out while the test host's Flutter app held the main thread, and the test returned and deleted its temp directory. The bridge callback's assertions then ran later and were blamed on whichever test was running (`ProfileSetupRecoveryTests`, `IdentityFilePluginTests`). | Callbacks only record the value and fulfill. Every assertion, including the `isRestricted` checks, runs after the wait, so a late callback writes into a dead local. The seven waits cap at 10 s (`callbackTimeout`). | | `ProfileSetupRecoveryTests` `testStartedEditorWithoutReadyRestoresProfileAndAllowsRetry` | 40585805357 (job 122477274956, PR #8568) | Engine release and editor dismissal must both happen within 2 s, on the main thread the host app keeps busy. | Both predicate waits cap at 10 s. | | Flutter `channel_detail_page_test.dart` "profile sheet shows the poster before autoplay and restores it on tap" and `settings_profile_header_test.dart` "shows the poster until the animated avatar is ready" | 40368315739 (job 121732711851, main) | After the image response is released, both tests slept a fixed 50 ms of real time before `pumpAndSettle`. A slower real decode left no frame scheduled, so the ready key was missing. | A new `WidgetHelpers.pumpUntilFound` pumps in 10 ms real-time steps until the ready key appears, capped at about 5 s. | Local evidence is from the original `flakey-tests` commit (53aaec97c), Xcode 27.1 on the iOS 26.2 simulator: - Enrichment, with `Self.complete` delayed 1.5 s and 4 s: 6 failures before, 107/107 after. With a 12 s delay, every failure lands in its owning test. A mutated error code is still caught. - Profile setup, with the engine release held 4 s: 2 failures before, 4/4 after. A leaked engine still fails. - Avatar tests: under load, 94/96 before and 96/96 after. With a 200 ms image delay, 0/5 before and 5/5 after. A widget that never signals ready still fails. Not included: - The planned `AppDelegate` / `Runner.xcscheme` change that keeps Flutter out of the RunnerTests host. It did not work locally: restored scene sessions and the `UIMainStoryboardFile` fallback, and a normal launch came up black afterwards. - `IdentityFilePluginTests` `testSyntheticDocumentsReadInPlaceAndRejectInvalidOrOversize` (40441255307#1). It is left unchanged and stays open. ### Verification This branch was pushed without local re-verification, by the user's choice: local test runs were overloading the machine and failing for that reason. **Hosted CI is the check.** It was pushed with `git push --no-verify`, because the pre-push hook runs the Flutter tests. - Every file is byte-identical to its final version on `flakey-tests` (67185df6a). - Checked for this PR: `git diff --check`, and a conflict-free merge with all six siblings onto `main`. - #8613's flake-check lane will repeat the two changed Dart test files. ### Readiness Test-only change. AGENTS.md's readiness steps (agent review, then human confirmation) are still outstanding, so `buzz-review-completed` is not set. ### Merge order Independent of the other six. It merges cleanly onto `main`, and with every sibling, in any order. ### Sibling PRs Split from `flakey-tests` (one PR per area): #8676 (PostgreSQL lease + lane), #8677 (relay E2E helper + Windows shell), #8679 (Tauri apt cache + Windows Rust cap), #8680 (desktop product races), #8681 (desktop E2E state gates), #8682 (desktop E2E composer + row actions). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Matt Toohey Co-authored-by: Claude Opus 5.5 Co-authored-by: Jitter <2b2e6415e748c35180847a37aadae06a11e30dddc76b784e7fd9354c4eb42e7a@buzz.block.builderlab.xyz> --- .../BuzzCommunicationNotificationTests.swift | 59 +++++++++++++------ .../ProfileSetupRecoveryTests.swift | 5 +- .../channels/channel_detail_page_test.dart | 13 ++-- .../profile/settings_profile_header_test.dart | 10 ++-- mobile/test/helpers/widget_helpers.dart | 13 ++++ scripts/flake-check.mjs | 6 +- scripts/flake-check.test.mjs | 17 ++++++ 7 files changed, 87 insertions(+), 36 deletions(-) diff --git a/mobile/ios/RunnerTests/BuzzCommunicationNotificationTests.swift b/mobile/ios/RunnerTests/BuzzCommunicationNotificationTests.swift index 28e5a34bbe..dc336bed41 100644 --- a/mobile/ios/RunnerTests/BuzzCommunicationNotificationTests.swift +++ b/mobile/ios/RunnerTests/BuzzCommunicationNotificationTests.swift @@ -242,7 +242,12 @@ final class BuzzCommunicationNotificationTests: XCTestCase { } } +/// Callbacks only record what they receive and assertions run after the wait, +/// so a callback that lands after its test returned cannot fail a later test. +/// The timeout is a cap; a passing test returns as soon as its callback runs. final class BuzzPushSnapshotEnrichmentTests: XCTestCase { + private let callbackTimeout: TimeInterval = 10 + func testAgeSignalRequestDoesNotDependOnNotificationStorage() { let delegate = MissingAppGroupDelegate() var completed = false @@ -268,28 +273,32 @@ final class BuzzPushSnapshotEnrichmentTests: XCTestCase { BuzzAgeSignalPayload.sharing(exclusiveUpperBound: 18, lowerBound: 13) } let minor = expectation(description: "native minor response") + var minorValue: Any? delegate.handleAgeSignalMethodCall( FlutterMethodCall(methodName: "requestAgeSignal", arguments: nil), viewController: UIViewController() ) { value in - XCTAssertEqual((value as? [String: Any])?["status"] as? String, "signal") - XCTAssertEqual((value as? [String: Any])?["ageUpper"] as? Int, 17) + minorValue = value minor.fulfill() } - await fulfillment(of: [minor], timeout: 1) + await fulfillment(of: [minor], timeout: callbackTimeout) + XCTAssertEqual((minorValue as? [String: Any])?["status"] as? String, "signal") + XCTAssertEqual((minorValue as? [String: Any])?["ageUpper"] as? Int, 17) delegate.requestPlatformAgeSignal = { _ in throw NSError(domain: "InjectedAgeFailure", code: 1) } let failed = expectation(description: "native error response") + var failedValue: Any? delegate.handleAgeSignalMethodCall( FlutterMethodCall(methodName: "requestAgeSignal", arguments: nil), viewController: UIViewController() ) { value in - XCTAssertEqual((value as? FlutterError)?.code, "age_signal_unavailable") + failedValue = value failed.fulfill() } - await fulfillment(of: [failed], timeout: 1) + await fulfillment(of: [failed], timeout: callbackTimeout) + XCTAssertEqual((failedValue as? FlutterError)?.code, "age_signal_unavailable") } func testAllowedNotificationRestoreNeedsNoAppGroupStore() { @@ -299,13 +308,15 @@ final class BuzzPushSnapshotEnrichmentTests: XCTestCase { keychainAccessGroup: nil ) let completed = expectation(description: "allowed without storage") + var received: Any? XCTAssertTrue(bridge.handle( FlutterMethodCall(methodName: "restoreAgeRestrictedNotifications", arguments: nil) ) { value in - XCTAssertNil(value) + received = value completed.fulfill() }) - wait(for: [completed], timeout: 1) + wait(for: [completed], timeout: callbackTimeout) + XCTAssertNil(received) } func testRestrictionCleanupFailurePreservesSnapshotAndRestoreReleasesAuthority() throws { @@ -335,25 +346,29 @@ final class BuzzPushSnapshotEnrichmentTests: XCTestCase { ) ) let restricted = expectation(description: "restriction despite cleanup failure") + var purgeResult: Any? XCTAssertTrue(bridge.handle( FlutterMethodCall(methodName: "purgeAgeRestrictedNotifications", arguments: nil) ) { value in - XCTAssertEqual((value as? FlutterError)?.code, "age_restriction_purge_failed") - XCTAssertTrue(BuzzAgeRestrictionSession.isRestricted(containerURL: directory)) + purgeResult = value restricted.fulfill() }) - wait(for: [restricted], timeout: 1) + wait(for: [restricted], timeout: callbackTimeout) + XCTAssertEqual((purgeResult as? FlutterError)?.code, "age_restriction_purge_failed") + XCTAssertTrue(BuzzAgeRestrictionSession.isRestricted(containerURL: directory)) XCTAssertEqual(try Data(contentsOf: snapshotURL), original) let restored = expectation(description: "allowed without restoring snapshot") + var restoreResult: Any? XCTAssertTrue(bridge.handle( FlutterMethodCall(methodName: "restoreAgeRestrictedNotifications", arguments: nil) ) { value in - XCTAssertNil(value) - XCTAssertFalse(BuzzAgeRestrictionSession.isRestricted(containerURL: directory)) + restoreResult = value restored.fulfill() }) - wait(for: [restored], timeout: 1) + wait(for: [restored], timeout: callbackTimeout) + XCTAssertNil(restoreResult) + XCTAssertFalse(BuzzAgeRestrictionSession.isRestricted(containerURL: directory)) XCTAssertEqual(deletionCalls, 2) XCTAssertEqual(try Data(contentsOf: snapshotURL), original) } @@ -365,9 +380,11 @@ final class BuzzPushSnapshotEnrichmentTests: XCTestCase { let failure = NSError(domain: "InjectedCleanupFailure", code: 1) let deletion = BuzzInteractionDeletionDeadline(timeout: 5, deleteAllInteractions: { $0(failure) }, scheduleTimeout: { _, _ in }) + var seedError: Error? BuzzInteractionCleanupRetry(containerURL: directory, deletion: deletion).request { - XCTAssertNotNil($0) + seedError = $0 } + XCTAssertNotNil(seedError) let bridge = BuzzPushSnapshotBridge( appGroupIdentifier: nil, endpointGrantStore: BuzzPushEndpointGrantKeychainStore(accessGroup: nil), @@ -375,14 +392,16 @@ final class BuzzPushSnapshotEnrichmentTests: XCTestCase { interactionDeletionDeadline: deletion ) let completed = expectation(description: "retry failure stays allowed") + var received: Any? XCTAssertTrue(bridge.handle( FlutterMethodCall(methodName: "restoreAgeRestrictedNotifications", arguments: nil) ) { value in - XCTAssertEqual((value as? FlutterError)?.code, "interaction_cleanup_retry_failed") - XCTAssertFalse(BuzzAgeRestrictionSession.isRestricted(containerURL: directory)) + received = value completed.fulfill() }) - wait(for: [completed], timeout: 1) + wait(for: [completed], timeout: callbackTimeout) + XCTAssertEqual((received as? FlutterError)?.code, "interaction_cleanup_retry_failed") + XCTAssertFalse(BuzzAgeRestrictionSession.isRestricted(containerURL: directory)) } func testStrictAgeGateWriteFailsWhenAppGroupStoreIsUnavailable() { @@ -392,6 +411,7 @@ final class BuzzPushSnapshotEnrichmentTests: XCTestCase { keychainAccessGroup: nil ) let completed = expectation(description: "strict write rejected") + var received: Any? XCTAssertTrue( bridge.handle( @@ -405,12 +425,13 @@ final class BuzzPushSnapshotEnrichmentTests: XCTestCase { ] ) ) { value in - XCTAssertEqual((value as? FlutterError)?.code, "snapshot_sync_unavailable") + received = value completed.fulfill() } ) - wait(for: [completed], timeout: 1) + wait(for: [completed], timeout: callbackTimeout) + XCTAssertEqual((received as? FlutterError)?.code, "snapshot_sync_unavailable") } func testMetadataAuthorityUsesNewestGrantForMatchingRelay() { diff --git a/mobile/ios/RunnerTests/ProfileSetupRecoveryTests.swift b/mobile/ios/RunnerTests/ProfileSetupRecoveryTests.swift index b7065ef274..0f9a48a121 100644 --- a/mobile/ios/RunnerTests/ProfileSetupRecoveryTests.swift +++ b/mobile/ios/RunnerTests/ProfileSetupRecoveryTests.swift @@ -163,8 +163,9 @@ final class ProfileSetupRecoveryTests: XCTestCase { XCTAssertFalse(avatar.isHidden) } // UIKit releases popped controllers at the end of its autorelease turn. + // The cap absorbs a busy main thread; a leaked engine never becomes nil. let released = XCTNSPredicateExpectation(predicate: NSPredicate { _, _ in engine == nil }, object: nil) - await fulfillment(of: [released], timeout: 2) + await fulfillment(of: [released], timeout: 10) } XCTAssertEqual(watchdogs.count, 2) skip.sendActions(for: .touchUpInside) @@ -178,7 +179,7 @@ final class ProfileSetupRecoveryTests: XCTestCase { XCTAssertTrue(notNow.isEnabled) notNow.sendActions(for: .touchUpInside) let dismissed = XCTNSPredicateExpectation(predicate: NSPredicate { _, _ in parent.presentedViewController == nil }, object: nil) - await fulfillment(of: [dismissed], timeout: 2) + await fulfillment(of: [dismissed], timeout: 10) XCTAssertEqual(messenger.completionAction, "skip") withExtendedLifetime(coordinator) {} nav.dismiss(animated: false) diff --git a/mobile/test/features/channels/channel_detail_page_test.dart b/mobile/test/features/channels/channel_detail_page_test.dart index 55f02451a2..0ae2f7fb11 100644 --- a/mobile/test/features/channels/channel_detail_page_test.dart +++ b/mobile/test/features/channels/channel_detail_page_test.dart @@ -90,6 +90,7 @@ import 'package:buzz/shared/widgets/skeleton.dart'; import 'package:shared_preferences/shared_preferences.dart'; import '../../helpers/thread_ids.dart'; +import '../../helpers/widget_helpers.dart'; part 'thread_reply_refresh_cases.dart'; part 'thread_title_capsule_cases.dart'; @@ -10119,17 +10120,13 @@ void main() { ); animationResponse.complete(http.Response.bytes(_transparentPng, 200)); - await tester.runAsync( - () => Future.delayed(const Duration(milliseconds: 50)), + final animationReady = find.byKey( + const ValueKey('progressive-animated-avatar-animation-ready'), ); + await WidgetHelpers.pumpUntilFound(tester, animationReady); await tester.pumpAndSettle(); - expect( - find.byKey( - const ValueKey('progressive-animated-avatar-animation-ready'), - ), - findsOneWidget, - ); + expect(animationReady, findsOneWidget); expect( find.byKey(const ValueKey('progressive-animated-avatar-poster')), findsNothing, diff --git a/mobile/test/features/profile/settings_profile_header_test.dart b/mobile/test/features/profile/settings_profile_header_test.dart index 79ec509e72..5ee803206c 100644 --- a/mobile/test/features/profile/settings_profile_header_test.dart +++ b/mobile/test/features/profile/settings_profile_header_test.dart @@ -128,15 +128,13 @@ void main() { containsAll([posterUrl, animationUrl]), ); animationResponse.complete(http.Response.bytes(_transparentPng, 200)); - await tester.runAsync( - () => Future.delayed(const Duration(milliseconds: 50)), + final animationReady = find.byKey( + const ValueKey('progressive-animated-avatar-animation-ready'), ); + await WidgetHelpers.pumpUntilFound(tester, animationReady); await tester.pumpAndSettle(); - expect( - find.byKey(const ValueKey('progressive-animated-avatar-animation-ready')), - findsOneWidget, - ); + expect(animationReady, findsOneWidget); expect( find.byKey(const ValueKey('progressive-animated-avatar-poster')), findsNothing, diff --git a/mobile/test/helpers/widget_helpers.dart b/mobile/test/helpers/widget_helpers.dart index 11b4b42141..1f15c54417 100644 --- a/mobile/test/helpers/widget_helpers.dart +++ b/mobile/test/helpers/widget_helpers.dart @@ -1,4 +1,5 @@ import 'package:flutter/material.dart'; +import 'package:flutter_test/flutter_test.dart'; import 'package:hooks_riverpod/hooks_riverpod.dart'; import 'package:hooks_riverpod/misc.dart'; import 'package:buzz/shared/theme/theme.dart'; @@ -24,4 +25,16 @@ class WidgetHelpers { ), ); } + + /// Pumps until [finder] matches, letting real async work such as an image + /// decode run between frames. Gives up after about 5 s so the caller's + /// `expect` still reports a real regression. + static Future pumpUntilFound(WidgetTester tester, Finder finder) async { + for (var i = 0; i < 500 && finder.evaluate().isEmpty; i++) { + await tester.runAsync( + () => Future.delayed(const Duration(milliseconds: 10)), + ); + await tester.pump(); + } + } } diff --git a/scripts/flake-check.mjs b/scripts/flake-check.mjs index 4c8693ea6c..13a0bbd53b 100644 --- a/scripts/flake-check.mjs +++ b/scripts/flake-check.mjs @@ -1115,6 +1115,9 @@ function repeatNode(files, iterations) { // Per-test statuses of a `dart test` JSON report (one event per line), keyed // ` › `. Hidden entries are the runner's own file-loading steps. +// A test declared through a helper (every `testWidgets` is) has the helper's +// `package:` URL as its `url`, so its file comes from its suite, else from +// `root_url`. export function parseDartReport(text, mobileRoot = `${root}mobile/`) { const started = new Map(); const suites = new Map(); @@ -1131,8 +1134,9 @@ export function parseDartReport(text, mobileRoot = `${root}mobile/`) { if (event.type !== "testDone" || event.hidden) continue; const test = started.get(event.testID); if (!test) continue; + const url = test.root_url ?? test.url; const source = suites.get(test.suiteID) ?? - (test.url?.startsWith("file:") ? fileURLToPath(test.url) : test.url ?? "unknown"); + (url?.startsWith("file:") ? fileURLToPath(url) : url ?? "unknown"); const file = source.replace(mobileRoot, ""); statuses.set( `${file} › ${test.name}`, diff --git a/scripts/flake-check.test.mjs b/scripts/flake-check.test.mjs index 5688a91478..304668d77a 100644 --- a/scripts/flake-check.test.mjs +++ b/scripts/flake-check.test.mjs @@ -953,6 +953,15 @@ test("reports give per-test statuses per iteration", () => { type: "testStart", test: { id: 4, name: "later", url: "file:///m/test/b_test.dart" }, }), + event({ + type: "testStart", + test: { + id: 5, + name: "widget renders", + url: "package:flutter_test/src/widget_tester.dart", + root_url: "file:///m/test/b_test.dart", + }, + }), "not json", event({ type: "testDone", @@ -975,6 +984,13 @@ test("reports give per-test statuses per iteration", () => { hidden: false, skipped: true, }), + event({ + type: "testDone", + testID: 5, + result: "success", + hidden: false, + skipped: false, + }), event({ type: "done", success: false }), ].join("\n"), "/m/", @@ -983,6 +999,7 @@ test("reports give per-test statuses per iteration", () => { ["test/a_test.dart › group renders", "passed"], ["test/a_test.dart › group fails", "failed"], ["test/b_test.dart › later", "skipped"], + ["test/b_test.dart › widget renders", "passed"], ]), ); // Playwright lists a --repeat-each journey once per repetition, in order.