mirror of
https://github.com/block/buzz.git
synced 2026-11-02 15:31:27 +00:00
eafd77e9f6
## Summary
Two CI-workflow fixes from the 23–30 Oct flake audit. This is one of six
PRs split from the reviewed `flakey-tests` branch (siblings below). Each
PR applies on its own to `main`.
### Hermit tool downloads (5 runs, 1 on main)
- **Runs:** 40635610550 and 40941660945, in Detect Changed Paths
(`node`). The selection step printed nothing for 9–10 min, the job hit
its cap, and every gate failed as cancelled. Three more, 40373404267 on
main (PostgreSQL Tests and Desktop E2E Integration 1/3, `pgschema`),
40373003333 (Mobile Android, `cmake`) and 40373467575 (Desktop Build
macOS, `just`), got 504s during GitHub's release-asset outage on 10-26,
~17:54–18:00 UTC.
- **Cause:** jobs start with an empty Hermit cache, so a tool's first
call downloads it, with no deadline. Hermit v0.54.1 tries three times, a
second apart, then gives up.
- **Change:** a new `scripts/ci-hermit-retry.sh` runs a no-op tool
command with a 90 s deadline per attempt, three attempts and 15 s × n
backoff. The deadline is plain bash: job control gives each attempt its
own process group, which gets TERM and then KILL. macOS runners have no
`timeout(1)` and may run bash 3.2. A "Fetch Hermit tools" step calls it
for:
- `node` and `just` in Detect Changed Paths, where one stall cancels
every gate. One `sh -c` invocation covers both, so the worst case is 345
s, inside the 10-minute cap. Two invocations would take 690 s.
- `pgschema` in PostgreSQL Tests, Desktop E2E Integration and Backend
Integration (via `--help`, since it has no `--version`).
- `just` and `cmake` in the Android lane.
- `just` in Desktop Build (macOS).
The test, `scripts/ci-hermit-retry.test.mjs`, joins Detect Changed
Paths' `node --test` list, and the script joins the rust and mobile path
filters. One stalled attempt costs ~105 s, which fits every job's cap.
- A killed attempt doesn't break the retry. Hermit downloads to a temp
file and unpacks to a temp dir, and renames each into place only once
it's complete. Its lock is a kernel flock.
### sccache storage failures (2 runs)
- **Runs:** 40355289475 (Desktop E2E Relay postgres-tests, 10-26,
"Egress is over the account limit") and 40714034945 (relay-binaries,
10-28, "send http request").
- **Cause:** the sccache server checks its GHA cache storage at startup
and won't start if that fails. Cargo's first `sccache rustc -vV` spawns
it, so the build fails. `SCCACHE_IGNORE_SERVER_IO_ERROR` doesn't help
here: in v0.16.0 it is read only when a running server's compile
response fails.
- **Change:** a "Start sccache" step runs `sccache --start-server`. It
outputs `wrapper=sccache`, or an empty wrapper and a warning if the
server won't start. The three build steps take `RUSTC_WRAPPER` from it,
and cargo treats an empty `RUSTC_WRAPPER` as no wrapper.
- The fallback doesn't fail the job later, in sccache-action's post
step. At the pinned fc920bf, that step runs `sccache --show-stats` and
the JSON form, and fails the job if either exits non-zero. With no
server running, v0.16.0's `--show-stats` doesn't start one. It returns
zero stats from the client side and exits 0. So the step keeps its stats
annotation, and `disable_annotations` stays unset.
### Related issue
None found. **Overlaps #8213** ("ci: fix empty root-workspace Rust
caches and merge test-unit nextest runs"). #8213 already conflicts with
`main`, and this PR adds a conflict in `.github/workflows/ci.yml` (path
filters).
### Testing
Verified locally:
- With a fresh Hermit state dir, `node` was SIGKILLed mid-download, then
mid-unpack while holding the lock, then stopped with TERM by the
script's own 1 s deadline. Each retry produced a working v24.15.0, at
clean-unpack speed. The `sh -c` form fetched both tools, and a 1 s
deadline on `sh -c 'sleep 37 && ...'` stopped `sh` and its child,
leaving no orphan. Tested under macOS bash 3.2 and Debian bash 5.2.
- sccache v0.16.0 (aarch64-apple-darwin) ran against a GHA cache whose
`ACTIONS_RESULTS_URL` refused connections, in READ_ONLY and READ_WRITE
modes. `--start-server` exited 2 ("Server startup failed: cache storage
failed to read"). Both `--show-stats` forms then exited 0, with zero
stats and valid JSON. `--stop-server` exited 2, which shows no server
had started.
Only hosted CI can show:
- the script under the runners' real shells, and that healthy runs only
pay a few seconds;
- whether ~3 min of spaced attempts rides out the next outage. The 10-26
outage lasted ~6 min, longer than this covers;
- that the explicit `--start-server` behaves like cargo's implicit start
against the real GHA cache. The fallback branch needs a storage outage
to run in CI.
Not covered here: first fetches of other tools (rustup, uv, pnpm) still
have no deadline. A per-job Hermit package cache is the follow-up.
Pushed with `git push --no-verify`, because the pre-push suites fail
when the laptop is under load.
### Readiness
An agent review of the combined `flakey-tests` branch (b432453af) found
no blockers. `buzz-review-completed` is not added, because the human
test step (AGENTS.md "Before opening a PR", step 3) hasn't happened yet.
### Siblings
- #8740 test(relay): thread_window aux-page hold and `api::media`
Postgres tests
- #8741 test(mobile): RunnerTests title fade and alert wait
- #8742 fix(desktop): deep-linked message hold (product change)
- #8743 test(desktop): four specs racing scroll settles and focus
- #8744 test(desktop): five settle classes and remote-owned-mentions
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
286 lines
12 KiB
YAML
286 lines
12 KiB
YAML
name: CI / Clients
|
|
on:
|
|
workflow_call:
|
|
inputs:
|
|
web:
|
|
required: true
|
|
type: boolean
|
|
mobile:
|
|
required: true
|
|
type: boolean
|
|
lane:
|
|
required: true
|
|
type: string
|
|
outputs:
|
|
web_result:
|
|
value: ${{ jobs.results.outputs.web_result }}
|
|
mobile_result:
|
|
value: ${{ jobs.results.outputs.mobile_result }}
|
|
|
|
env:
|
|
CARGO_TERM_COLOR: always
|
|
BUZZ_TEST_POSTGRES_PASSWORD: buzz_dev
|
|
PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/.cache/ms-playwright
|
|
|
|
jobs:
|
|
web:
|
|
name: Web
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 15
|
|
if: inputs.lane == 'required' && (github.event_name == 'push' || inputs.web)
|
|
permissions:
|
|
contents: read
|
|
steps:
|
|
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
|
|
with:
|
|
fetch-depth: 2
|
|
- uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
|
|
- name: Get pnpm store directory
|
|
id: pnpm-cache
|
|
run: echo "STORE_PATH=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
|
|
- name: Restore pnpm store cache
|
|
uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5
|
|
with:
|
|
path: ${{ steps.pnpm-cache.outputs.STORE_PATH }}
|
|
key: pnpm-${{ runner.os }}-${{ hashFiles('**/pnpm-lock.yaml') }}
|
|
restore-keys: pnpm-${{ runner.os }}-
|
|
- name: Install dependencies
|
|
run: pnpm install --frozen-lockfile
|
|
- name: Web lint and format
|
|
run: just web-check
|
|
- name: Web build
|
|
run: just web-build
|
|
- name: Save pnpm store cache
|
|
if: github.event_name == 'push'
|
|
uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5
|
|
with:
|
|
path: ${{ steps.pnpm-cache.outputs.STORE_PATH }}
|
|
key: pnpm-${{ runner.os }}-${{ hashFiles('**/pnpm-lock.yaml') }}
|
|
|
|
# Mobile lanes run in parallel; the required check reads the combined
|
|
# matrix result. Test shards split files, so each compiles only its own.
|
|
mobile:
|
|
name: Mobile (${{ matrix.name }})
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 30
|
|
if: inputs.lane == 'required' && (github.event_name == 'push' || inputs.mobile)
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- name: Checks
|
|
lane: checks
|
|
- name: Tests 1/3
|
|
lane: test
|
|
shard: 0
|
|
- name: Tests 2/3
|
|
lane: test
|
|
shard: 1
|
|
- name: Tests 3/3
|
|
lane: test
|
|
shard: 2
|
|
- name: Android
|
|
lane: android
|
|
permissions:
|
|
contents: read
|
|
steps:
|
|
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
|
|
with:
|
|
fetch-depth: 2
|
|
- uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
|
|
- name: Compute Hermit cache key
|
|
id: hermit-bin-hash
|
|
run: |
|
|
hash="$(find ./bin ! -type d | sort | xargs openssl sha256 | openssl sha256 -r | cut -d' ' -f1)"
|
|
echo "hash=$hash" >> "$GITHUB_OUTPUT"
|
|
- name: Restore Hermit package cache
|
|
id: hermit-cache
|
|
uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5
|
|
with:
|
|
path: ~/.cache/hermit/pkg
|
|
key: ${{ runner.os }}-hermit-cache-${{ steps.hermit-bin-hash.outputs.hash }}
|
|
restore-keys: ${{ runner.os }}-hermit-cache-
|
|
- name: Prime Flutter SDK
|
|
run: flutter --version
|
|
- name: Save Hermit package cache
|
|
if: always() && steps.hermit-cache.outputs.cache-hit != 'true'
|
|
uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5
|
|
continue-on-error: true
|
|
with:
|
|
path: ~/.cache/hermit/pkg
|
|
key: ${{ runner.os }}-hermit-cache-${{ steps.hermit-bin-hash.outputs.hash }}
|
|
- name: Restore pub cache
|
|
id: pub-cache
|
|
uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5
|
|
with:
|
|
path: ~/.pub-cache
|
|
key: pub-${{ runner.os }}-${{ hashFiles('mobile/pubspec.lock') }}
|
|
restore-keys: pub-${{ runner.os }}-
|
|
- name: Install dependencies
|
|
run: cd mobile && flutter pub get
|
|
- name: Save pub cache
|
|
if: always() && steps.pub-cache.outputs.cache-hit != 'true'
|
|
uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5
|
|
continue-on-error: true
|
|
with:
|
|
path: ~/.pub-cache
|
|
key: pub-${{ runner.os }}-${{ hashFiles('mobile/pubspec.lock') }}
|
|
- name: Format check
|
|
if: matrix.lane == 'checks'
|
|
run: cd mobile && dart format --output=none --set-exit-if-changed .
|
|
- name: Analyze
|
|
if: matrix.lane == 'checks'
|
|
run: cd mobile && flutter analyze
|
|
- name: Test
|
|
if: matrix.lane == 'test'
|
|
run: /bin/bash scripts/mobile-test-shard.sh ${{ matrix.shard }} 3 --dart-define=BUZZ_PUSH_GATEWAY_URL=https://push.example
|
|
- name: Test native secp256k1 package
|
|
if: matrix.lane == 'checks'
|
|
run: cd mobile/packages/buzz_secp256k1 && dart pub get && dart test
|
|
- name: Test configured authentication domains
|
|
if: matrix.lane == 'checks'
|
|
run: >-
|
|
cd mobile && flutter test
|
|
--dart-define=BUZZ_PUSH_GATEWAY_URL=https://push.example
|
|
--dart-define=BUZZ_AUTH_DOMAINS=buzz.xyz,builderlab.xyz,builderlabstaging.xyz
|
|
test/shared/auth/community_entry_requirements_test.dart
|
|
test/features/invites/invite_join_provider_test.dart
|
|
test/features/onboarding/onboarding_entry_routes_test.dart
|
|
test/shared/hosted/hosted_environment_test.dart
|
|
test/shared/hosted/self_hosted_entry_test.dart
|
|
test/features/onboarding/hosted_app_gate_environment_test.dart
|
|
test/features/onboarding/hosted_journeys_test.dart
|
|
test/features/onboarding/hosted_communities_test.dart
|
|
test/features/pairing/pairing_hosted_handoff_test.dart
|
|
test/shared/agents/remote_agents_test.dart
|
|
test/features/agents
|
|
- name: Test mobile without push configuration
|
|
if: matrix.lane == 'checks'
|
|
run: cd mobile && flutter test test/shared/push/push_unconfigured_build_test.dart
|
|
- name: Test mobile recipe argument forwarding
|
|
if: matrix.lane == 'checks'
|
|
run: /bin/bash scripts/test-mobile-gateway-recipes.sh
|
|
# ubuntu-latest (26.04) defaults to JDK 25, which Gradle 8.14 does not
|
|
# support. Pin the JDK 17 the 24.04 image used.
|
|
- name: Set up JDK 17
|
|
if: matrix.lane == 'android'
|
|
uses: actions/setup-java@b843ebf9b913146b38da2eaa7aeddf76074ae1ad # v6.1.0
|
|
with:
|
|
distribution: temurin
|
|
java-version: "17"
|
|
- name: Restore Gradle cache
|
|
if: matrix.lane == 'android'
|
|
uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5
|
|
with:
|
|
path: |
|
|
~/.gradle/caches
|
|
~/.gradle/wrapper
|
|
key: gradle-${{ runner.os }}-${{ hashFiles('mobile/android/**/*.gradle*', 'mobile/android/gradle/wrapper/gradle-wrapper.properties', 'mobile/pubspec.lock') }}
|
|
restore-keys: gradle-${{ runner.os }}-
|
|
# The Hermit package cache above holds only Flutter, so the build's first
|
|
# just and cmake calls download them.
|
|
- name: Fetch Hermit tools
|
|
if: matrix.lane == 'android'
|
|
run: |
|
|
scripts/ci-hermit-retry.sh ./bin/just --version
|
|
scripts/ci-hermit-retry.sh ./bin/cmake --version
|
|
# SDK component and plugin downloads intermittently fail on hosted
|
|
# runners; retry once only for those, so real build failures still fail.
|
|
- name: Build Android debug APK
|
|
if: matrix.lane == 'android'
|
|
run: |
|
|
set -o pipefail
|
|
log="$RUNNER_TEMP/android-build.log"
|
|
if ! just mobile-build-android 2>&1 | tee "$log"; then
|
|
grep -qE 'InstallFailedException|Could not (resolve|GET|download)' "$log"
|
|
just mobile-build-android
|
|
fi
|
|
- name: Test Android JVM units
|
|
if: matrix.lane == 'android'
|
|
run: cd mobile/android && ./gradlew :app:testDebugUnitTest --console=plain
|
|
- name: Save Gradle cache
|
|
if: matrix.lane == 'android' && github.event_name == 'push'
|
|
uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5
|
|
continue-on-error: true
|
|
with:
|
|
path: |
|
|
~/.gradle/caches
|
|
~/.gradle/wrapper
|
|
key: gradle-${{ runner.os }}-${{ hashFiles('mobile/android/**/*.gradle*', 'mobile/android/gradle/wrapper/gradle-wrapper.properties', 'mobile/pubspec.lock') }}
|
|
|
|
# The unsigned release build needs nothing from the native lane, so it runs
|
|
# beside it instead of after the simulator tests.
|
|
mobile-swift:
|
|
name: Mobile Swift (${{ matrix.name }})
|
|
runs-on: macos-latest
|
|
timeout-minutes: 45
|
|
if: inputs.lane == 'mobile-swift' && inputs.mobile
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- name: Native
|
|
lane: native
|
|
- name: Release
|
|
lane: release
|
|
steps:
|
|
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
|
|
- uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
|
|
- name: Install Flutter dependencies
|
|
run: cd mobile && flutter pub get
|
|
# `swift test` builds BuzzPushKit; the Release lane covers its release build.
|
|
- name: Test
|
|
if: matrix.lane == 'native'
|
|
run: swift test --package-path mobile/ios/BuzzPushKit
|
|
- name: Test native pagination geometry
|
|
if: matrix.lane == 'native'
|
|
run: /bin/bash scripts/test-mobile-pagination.sh
|
|
- name: Generate iOS build configuration
|
|
if: matrix.lane == 'native'
|
|
run: cd mobile && flutter build ios --simulator --debug --no-pub --config-only
|
|
# Build the complete simulator app and RunnerTests once, in the
|
|
# background, while the simulator boots and runs the native-control
|
|
# scripts. The build also guards Apple Silicon simulator linking (#7646).
|
|
- name: Test native controls and Runner unit tests
|
|
if: matrix.lane == 'native'
|
|
run: |
|
|
device=$(xcrun simctl create "Native tests" "iPhone 16")
|
|
trap 'xcrun simctl shutdown "$device"; xcrun simctl delete "$device"' EXIT
|
|
xcode=(xcodebuild -workspace mobile/ios/Runner.xcworkspace -scheme Runner
|
|
-configuration Debug -destination "platform=iOS Simulator,id=$device"
|
|
CODE_SIGNING_ALLOWED=NO)
|
|
build_log="$RUNNER_TEMP/runner-build.log"
|
|
"${xcode[@]}" build-for-testing > "$build_log" 2>&1 &
|
|
build=$!
|
|
xcrun simctl boot "$device"
|
|
xcrun simctl bootstatus "$device" -b
|
|
/bin/bash scripts/test-mobile-pagination-accessibility.sh "$device"
|
|
/bin/bash scripts/test-mobile-carousel.sh "$device"
|
|
/bin/bash scripts/test-mobile-member-picker.sh "$device"
|
|
/bin/bash scripts/test-mobile-me-agent-accessibility.sh "$device"
|
|
if ! wait "$build"; then
|
|
cat "$build_log"
|
|
exit 1
|
|
fi
|
|
tail -n 5 "$build_log"
|
|
"${xcode[@]}" -parallel-testing-enabled NO -only-testing:RunnerTests \
|
|
test-without-building
|
|
- name: Build complete unsigned iOS release
|
|
if: matrix.lane == 'release'
|
|
run: |
|
|
/bin/bash scripts/test-mobile-gateway-recipes.sh
|
|
cd mobile && flutter build ios --release --no-codesign --no-pub
|
|
|
|
results:
|
|
name: Results
|
|
if: ${{ always() }}
|
|
needs: [web, mobile, mobile-swift]
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 5
|
|
permissions: {}
|
|
outputs:
|
|
web_result: ${{ needs.web.result }}
|
|
mobile_result: ${{ needs.mobile.result }}
|
|
steps:
|
|
- run: echo "Captured client job results"
|