jerry/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-10-27 04:30:52 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
edbfbf08db4bbcae8b557d077460ff3498e12fe1
buzz/.github/workflows/_ci-clients.yml
Wes edbfbf08db test(mobile): journey-first testing guidance and suite cleanup (#8397) 
## Summary

This PR does two things. It updates the mobile testing conventions to
favor a few journey tests over many small logic tests. It also applies
those conventions to the existing mobile suite, one commit per step.

### Testing conventions (`AGENTS.md`, `mobile/README.md`)

- Start with one widget journey per user-visible flow, driven through
the real widgets, notifiers and coordinators. Assert what the user sees
and what is durably saved.
- Cover retry, cancel and resume as steps inside the journey, not as
separate tests.
- Fake only external boundaries (network, relay, OS, browser, storage).
- Use focused, table-driven tests only for security and data-loss
invariants, protocol or crypto vectors, and pure functions with many
input combinations.
- When review finds a defect, extend an existing journey or table
instead of adding a test per finding.
- Delete a lower-level test only once a journey demonstrably catches the
same failures.
- Every test must fail when the production code it guards is broken.

These conventions keep the Review-Proven Rules' safety bar and change
only how coverage is organized.

### Suite inventory and plan

The suite has 234 Dart test files, about 110k lines and about 2,600
declared tests. Native tests are separate: BuzzPushKit, RunnerTests,
NativeControlTests and Android. Line-saving estimates are hypotheses,
not a deletion budget. Each deletion needs a replacement that rejects a
representative production mutation.

1. **CI and reliability.** Run the native tests CI skipped, make
time-dependent tests deterministic, and make tests that could not fail
falsifiable. *(done, see below)*
2. **Shared harness.** Add one fake relay (socket plus HTTP boundary,
with the real `RelaySessionNotifier`) in `test/helpers/`. Today 38 files
each write their own `RelaySessionNotifier` fake. *(done, see below)*
3. **Per-area journeys.** Convert page tests that fake the flow's own
notifiers (for example, `channel_detail_page_test` overrides about 60
providers). Fold near-duplicates into tables. Remove constant-pinning,
pixel-geometry and call-count tests once a journey covers the outcome.
Split `channel_detail_page_test.dart` (17.7k lines). Each area commit
reports before/after counts and the mutation it rejects.
4. **CI speed.** Shard by file instead of by test, since every shard
currently compiles the whole suite. *(done, see below)*

Keep as-is: relay protocol and NIP handling, crypto vectors, auth, push
security, persistence and journal invariants, recipient routing, and
real-socket transport and cancellation tests.

### Step 1 changes

- **Android CI** runs the 23 JVM unit tests, which were never run
before. It retries the APK build once, only on SDK or plugin download
failures (the cause of 4 recent `main` failures).
- **iOS CI** runs all RunnerTests (previously 2 of 99) and the
previously uncalled Me-agent accessibility script. The Swift Native lane
builds the simulator app once, in the background while the simulator
boots and runs the native-control scripts, instead of twice. The lane
stays non-required.
- **Stale RunnerTests.** Two reaction-details tests still expected the
filter list that #8280 replaced; they now cover the toggle list. #8393
moved navigation bar controls onto a shared glass surface and made
unknown Tabler symbols assert. The avatar and members-activity tests now
find controls on that surface, three tests use real symbol names, and a
240 pt title width pinned to the old control sizes is removed (its gap
assertions remain).
- **Known bug, not fixed here.** Since #8393 the DM title avatar image
is nil, because `makeItem` returns avatar data as a custom-view item. No
screen passes a title avatar yet. The test marks it `XCTExpectFailure`.
- **Typing retry** and **sections backoff** tests use fake time instead
of about 37 s of real sleeps and a `Stopwatch < 100ms` threshold.
- **Manage-channel sheet tests** previously asserted inside callbacks
that the sheet's `catch (_)` swallows. A wrong saved description made
the test loop until killed. The tests now record arguments, assert
outside the caught code, and cancel instead of re-presenting endlessly.
- **`video_viewer_transport_test` removed.** It exercised only
`package:http`. Removing the production `request.sink.close()` already
fails `video_viewer_test`.

### Step 2 changes

- **`test/helpers/fake_relay.dart`** fakes only the websocket (through
the session's socket factory) and the HTTP bridge (through its client).
The real `RelaySessionNotifier` still connects from `build()`, handles
OK and CLOSED, batches events, backs off and replays subscriptions.
- The fake serves the NIP-CW channel window. It can hold OK and live
echo independently, reject a chosen publish, and drop the connection.
Any request it cannot answer faithfully fails the test at teardown,
because the app often catches relay errors and falls back.
- **`channel_messages_journey_test`**: history through the channel
window, a live message, a send that is pending until OK and stays one
row after the echo, a rejected send rolled back, then drop and reconnect
with missed and later messages.
- **`channel_send_journey_test`**: type and send on `ChannelDetailPage`
with no feature-notifier overrides. The row and sending spinner show at
once, OK settles them, and a rejected send puts the text back in the
composer and shows a snackbar.
- Existing per-file relay fakes and `SidebarRelay` are unchanged. Step 3
migrates them area by area.

### Step 4 changes

- `scripts/mobile-test-shard.sh` splits the 235 test files across the 3
shards, balanced by line count. `flutter test --total-shards` split
tests, so each shard compiled and loaded every file. Locally at 4
workers, one shard took 63 s instead of 73 s and used 40% less CPU.
`channel_detail_page_test.dart` (17.7k lines) is now the long pole;
splitting it in step 3 shortens the slowest shard.

Out of scope here: requiring branches to be up to date before merge (a
repository setting), and the onboarding tests in #8375.

## Validation

- Mutation checks:
  - Typing retry bound: test fails.
  - Sections backoff reset: test fails.
- Wrong saved description: fails in about 11 s. The old tests hung
instead.
  - Removed sink close: `video_viewer_test` fails.
- Changed Dart files pass locally, and `flutter analyze` is clean.
- Android `testDebugUnitTest` passes locally (23 tests). The Me-agent
accessibility script passes on a local simulator.
- Step 2 mutation checks, each made a journey fail: accepted OK ignored,
rejected row not rolled back, optimistic row not added, draft not
restored, local message left pending after OK, channel window replaced
by the fallback, live subscriptions not replayed after reconnect.
- Both journeys run in about 1 s on fake time.
- All 99 RunnerTests pass on an iPhone 16 iOS 26.5 simulator, the CI
destination, after rebasing on #8398.
- The shard script partitions all 235 test files exactly once.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Co-authored-by: Carl <32a2e2c9d428ee08902cab75d956da2c1d235a22d4766b0dd4138bf6e2e5db1d@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 17:19:07 -06:00

266 lines
11 KiB
YAML
Raw Blame History

name: CI / Clients
on:
workflow_call:
inputs:
web:
required: true
type: boolean
mobile:
required: true
type: boolean
lane:
required: true
type: string
outputs:
web_result:
value: ${{ jobs.results.outputs.web_result }}
mobile_result:
value: ${{ jobs.results.outputs.mobile_result }}
env:
CARGO_TERM_COLOR: always
BUZZ_TEST_POSTGRES_PASSWORD: buzz_dev
PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/.cache/ms-playwright
jobs:
web:
name: Web
runs-on: ubuntu-latest
timeout-minutes: 15
if: inputs.lane == 'required' && (github.event_name == 'push' || inputs.web)
permissions:
contents: read
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 2
- uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
- name: Get pnpm store directory
id: pnpm-cache
run: echo "STORE_PATH=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
- name: Restore pnpm store cache
uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5
with:
path: ${{ steps.pnpm-cache.outputs.STORE_PATH }}
key: pnpm-${{ runner.os }}-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: pnpm-${{ runner.os }}-
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Web lint and format
run: just web-check
- name: Web build
run: just web-build
- name: Save pnpm store cache
if: github.event_name == 'push'
uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5
with:
path: ${{ steps.pnpm-cache.outputs.STORE_PATH }}
key: pnpm-${{ runner.os }}-${{ hashFiles('**/pnpm-lock.yaml') }}
# Mobile lanes run in parallel; the required check reads the combined
# matrix result. Test shards split files, so each compiles only its own.
mobile:
name: Mobile (${{ matrix.name }})
runs-on: ubuntu-latest
timeout-minutes: 30
if: inputs.lane == 'required' && (github.event_name == 'push' || inputs.mobile)
strategy:
fail-fast: false
matrix:
include:
- name: Checks
lane: checks
- name: Tests 1/3
lane: test
shard: 0
- name: Tests 2/3
lane: test
shard: 1
- name: Tests 3/3
lane: test
shard: 2
- name: Android
lane: android
permissions:
contents: read
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 2
- uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
- name: Compute Hermit cache key
id: hermit-bin-hash
run: |
hash="$(find ./bin ! -type d | sort | xargs openssl sha256 | openssl sha256 -r | cut -d' ' -f1)"
echo "hash=$hash" >> "$GITHUB_OUTPUT"
- name: Restore Hermit package cache
id: hermit-cache
uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5
with:
path: ~/.cache/hermit/pkg
key: ${{ runner.os }}-hermit-cache-${{ steps.hermit-bin-hash.outputs.hash }}
restore-keys: ${{ runner.os }}-hermit-cache-
- name: Prime Flutter SDK
run: flutter --version
- name: Save Hermit package cache
if: always() && steps.hermit-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5
continue-on-error: true
with:
path: ~/.cache/hermit/pkg
key: ${{ runner.os }}-hermit-cache-${{ steps.hermit-bin-hash.outputs.hash }}
- name: Restore pub cache
id: pub-cache
uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5
with:
path: ~/.pub-cache
key: pub-${{ runner.os }}-${{ hashFiles('mobile/pubspec.lock') }}
restore-keys: pub-${{ runner.os }}-
- name: Install dependencies
run: cd mobile && flutter pub get
- name: Save pub cache
if: always() && steps.pub-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5
continue-on-error: true
with:
path: ~/.pub-cache
key: pub-${{ runner.os }}-${{ hashFiles('mobile/pubspec.lock') }}
- name: Format check
if: matrix.lane == 'checks'
run: cd mobile && dart format --output=none --set-exit-if-changed .
- name: Analyze
if: matrix.lane == 'checks'
run: cd mobile && flutter analyze
- name: Test
if: matrix.lane == 'test'
run: /bin/bash scripts/mobile-test-shard.sh ${{ matrix.shard }} 3 --dart-define=BUZZ_PUSH_GATEWAY_URL=https://push.example
- name: Test configured authentication domains
if: matrix.lane == 'checks'
run: >-
cd mobile && flutter test
--dart-define=BUZZ_PUSH_GATEWAY_URL=https://push.example
--dart-define=BUZZ_AUTH_DOMAINS=buzz.xyz,builderlab.xyz,builderlabstaging.xyz
test/shared/auth/community_entry_requirements_test.dart
test/features/invites/invite_join_provider_test.dart
test/features/onboarding/onboarding_entry_routes_test.dart
test/shared/hosted/hosted_environment_test.dart
test/shared/hosted/self_hosted_entry_test.dart
test/features/onboarding/hosted_app_gate_environment_test.dart
test/features/onboarding/hosted_journeys_test.dart
test/features/onboarding/hosted_communities_test.dart
test/features/pairing/pairing_hosted_handoff_test.dart
- name: Test mobile without push configuration
if: matrix.lane == 'checks'
run: cd mobile && flutter test test/shared/push/push_unconfigured_build_test.dart
- name: Test mobile recipe argument forwarding
if: matrix.lane == 'checks'
run: /bin/bash scripts/test-mobile-gateway-recipes.sh
- name: Restore Gradle cache
if: matrix.lane == 'android'
uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5
with:
path: |
~/.gradle/caches
~/.gradle/wrapper
key: gradle-${{ runner.os }}-${{ hashFiles('mobile/android/**/*.gradle*', 'mobile/android/gradle/wrapper/gradle-wrapper.properties', 'mobile/pubspec.lock') }}
restore-keys: gradle-${{ runner.os }}-
# SDK component and plugin downloads intermittently fail on hosted
# runners; retry once only for those, so real build failures still fail.
- name: Build Android debug APK
if: matrix.lane == 'android'
run: |
set -o pipefail
log="$RUNNER_TEMP/android-build.log"
if ! just mobile-build-android 2>&1 | tee "$log"; then
grep -qE 'InstallFailedException|Could not (resolve|GET|download)' "$log"
just mobile-build-android
fi
- name: Test Android JVM units
if: matrix.lane == 'android'
run: cd mobile/android && ./gradlew :app:testDebugUnitTest --console=plain
- name: Save Gradle cache
if: matrix.lane == 'android' && github.event_name == 'push'
uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5
continue-on-error: true
with:
path: |
~/.gradle/caches
~/.gradle/wrapper
key: gradle-${{ runner.os }}-${{ hashFiles('mobile/android/**/*.gradle*', 'mobile/android/gradle/wrapper/gradle-wrapper.properties', 'mobile/pubspec.lock') }}
# The unsigned release build needs nothing from the native lane, so it runs
# beside it instead of after the simulator tests.
mobile-swift:
name: Mobile Swift (${{ matrix.name }})
runs-on: macos-latest
timeout-minutes: 45
if: inputs.lane == 'mobile-swift' && inputs.mobile
strategy:
fail-fast: false
matrix:
include:
- name: Native
lane: native
- name: Release
lane: release
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
- name: Install Flutter dependencies
run: cd mobile && flutter pub get
# `swift test` builds BuzzPushKit; the Release lane covers its release build.
- name: Test
if: matrix.lane == 'native'
run: swift test --package-path mobile/ios/BuzzPushKit
- name: Test native pagination geometry
if: matrix.lane == 'native'
run: /bin/bash scripts/test-mobile-pagination.sh
- name: Generate iOS build configuration
if: matrix.lane == 'native'
run: cd mobile && flutter build ios --simulator --debug --no-pub --config-only
# Build the complete simulator app and RunnerTests once, in the
# background, while the simulator boots and runs the native-control
# scripts. The build also guards Apple Silicon simulator linking (#7646).
- name: Test native controls and Runner unit tests
if: matrix.lane == 'native'
run: |
device=$(xcrun simctl create "Native tests" "iPhone 16")
trap 'xcrun simctl shutdown "$device"; xcrun simctl delete "$device"' EXIT
xcode=(xcodebuild -workspace mobile/ios/Runner.xcworkspace -scheme Runner
-configuration Debug -destination "platform=iOS Simulator,id=$device"
CODE_SIGNING_ALLOWED=NO)
build_log="$RUNNER_TEMP/runner-build.log"
"${xcode[@]}" build-for-testing > "$build_log" 2>&1 &
build=$!
xcrun simctl boot "$device"
xcrun simctl bootstatus "$device" -b
/bin/bash scripts/test-mobile-pagination-accessibility.sh "$device"
/bin/bash scripts/test-mobile-carousel.sh "$device"
/bin/bash scripts/test-mobile-member-picker.sh "$device"
/bin/bash scripts/test-mobile-me-agent-accessibility.sh "$device"
if ! wait "$build"; then
cat "$build_log"
exit 1
fi
tail -n 5 "$build_log"
"${xcode[@]}" -parallel-testing-enabled NO -only-testing:RunnerTests \
test-without-building
- name: Build complete unsigned iOS release
if: matrix.lane == 'release'
run: |
/bin/bash scripts/test-mobile-gateway-recipes.sh
cd mobile && flutter build ios --release --no-codesign --no-pub
results:
name: Results
if: ${{ always() }}
needs: [web, mobile, mobile-swift]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
outputs:
web_result: ${{ needs.web.result }}
mobile_result: ${{ needs.mobile.result }}
steps:
- run: echo "Captured client job results"
Reference in New Issue View Git Blame Copy Permalink