From 52cb6818face2bcc1a2aeaceb44d183b6ff9f2e9 Mon Sep 17 00:00:00 2001 From: Tom Brow Date: Mon, 9 Nov 2026 07:21:21 -0800 Subject: [PATCH] perf(mobile): keep remote images on disk and show small ones as thumbnails (#9158) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary The mobile app now saves remote images on disk. After a relaunch, avatars, custom emoji, message attachments and agent artwork show immediately. The app does not download them again. Small images on a relay now load only the relay's thumbnail. The app never downloads the full-size upload to show a small image. ## Why Before this change, the app kept remote images only in memory. Each cold launch downloaded every image again. On a slow connection, attachments showed a "Loading image" placeholder for a long time. Also, when a relay had no thumbnail for an image, the app downloaded the full-size upload. That upload can be up to 20 MiB, for an avatar that is a few dozen points wide. ## What changes ### Images on disk - `MediaImageProvider` reads and writes a `MediaDiskCache`. Each account in each community has its own cache, at `/communities//media/`. The cache is built on `flutter_cache_manager`, with a JSON index. - On iOS, the root is the app group's `Library/Caches` (from a new `buzz/app_group` method channel). App extensions can then read the same cache. On other platforms, the root is the app cache directory. - Each cache holds a maximum of 250 MB. When it is full, the least recently used images go first. `MediaCacheIndex` measures each file when it writes it, because `flutter_cache_manager` limits only the number of files. Every disk read marks the file used, and every write schedules eviction, including a refresh that replaces bytes the package already holds in memory (by default the package does neither for those). - The cache does not save an image that is larger than 5 MiB. The app still shows that image from memory. Thus, one large image cannot push out all the others. - Relay media URLs contain a content hash, so their bytes never change. The cache keeps them until it evicts them. For other URLs, the cache shows the saved image, and then gets a new copy in the background after one week. A maximum of two of these refreshes run at the same time. The app decodes a refreshed image before it replaces the saved copy. - When you remove a community, the app deletes its cache. When a community gets a new signing key, the app deletes the media of the old account. At launch, the app also deletes caches of communities and accounts that are no longer present. - Startup does not wait for the cache. The cache directory resolves in the background. Image reads wait for it for a maximum of 3 seconds. ### Thumbnails for small images - A small relay image loads the relay's WebP thumbnail. If the relay has no WebP thumbnail, the image loads the JPEG thumbnail. If the relay has neither, the widget shows its placeholder (initials for an avatar, an empty tile for artwork). - Native avatars drawn by UIKit (the navigation bar, the community icon on the create button, and the New DM and Add Members pickers) use the same choice, through a new `smallMediaImageProviders`. They try the same thumbnails, fall back the same way, and show initials when neither thumbnail exists. Before, they downloaded the full-size upload and only decoded it small. - Larger layouts and the full-screen viewer load the full-size upload, as before. ### Agent artwork - Preset agent artwork (the tray, flight, video placeholder, avatar hero and the avatar editor's blurred backdrop) now loads through `MediaImage`, so it uses the disk cache too. - The create-agent and empty-bestie pages warm the same `MediaImageProvider` that their widgets read, through a new `mediaImageProviderFor(ref, url)`. - No code in `mobile/lib` uses `Image.network` or `NetworkImage` now. ## Demo A cold relaunch (the app is stopped, then started) into a synthetic "QA demo" Solo chat that holds a few generated images. Shown at half speed. Before: the attachment shows a "Loading image" placeholder while it downloads. https://github.com/user-attachments/assets/107f43a3-63d0-40c3-bd80-4ea9d7897a97 After: the attachment shows with the chat, from disk. https://github.com/user-attachments/assets/d3985bb2-838c-4efa-93cb-44cef51f677d ## Tests - `test/shared/relay/media_disk_cache_test.dart` follows one journey through the real provider and a real cache in a temporary directory. The first launch downloads and saves. An offline relaunch shows the full-size image, its thumbnail and an external image from disk, and the oversized image goes back to the network. Eight days later, only the external image refreshes. Removing the community deletes its directory. - A second test checks the order of eviction when the cache is full, that a file read from disk is evicted after files that were not, and that replacing a file's bytes after eviction last ran evicts again when the cache goes over budget. - `community_provider_test` checks that launch deletes orphan media directories, that a new signing key deletes only the old account's media, and that removing a community deletes its media. - `avatar_image_test` checks that a missing WebP thumbnail falls back to the JPEG thumbnail, and that an avatar with no thumbnail shows its initials. - `native_avatar_image_test` checks the same for native avatars: a WebP 404 paints the JPEG thumbnail, the full-size upload is never requested, a relay with no thumbnail paints initials and is not asked again, and any other failure stops at the initials. - `flutter analyze` and the full mobile test suite pass. - Automated app runs on an iPhone 17 Pro simulator (iOS 27): the channel list, a DM, Home, the Agents grid and an agent details sheet show their images. After a relaunch, the images came from disk and the app wrote no new files. In the "QA demo" chat, a new image from the app showed immediately. - Automated run of the native-thumbnail fix on an iPhone 17 Pro simulator (iOS 27): the native New DM picker shows photo avatars, and the cache gained only relay `.thumb.webp` entries for them, no full-size uploads. After a cold relaunch, the picker showed the same avatars within about 1.5 seconds, and the cache directory gained no files. The avatar editor backdrop was not exercised in the app. The test community has no remote agents, and the backdrop tests cover only emoji and picked-image selections. ## Before merge - [x] Automated simulator run of this head: avatars and agent artwork show their thumbnails, and a cold relaunch shows images from disk. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Tom Brow Co-authored-by: Tom Brow Co-authored-by: Claude Opus 5.5 Co-authored-by: studio --- mobile/ios/Podfile.lock | 7 + mobile/ios/Runner/AppDelegate.swift | 18 + .../agents/agent_avatar_backdrop.dart | 11 +- .../features/agents/agent_avatar_hero.dart | 7 +- .../agents/agent_character_artwork.dart | 8 +- .../features/agents/create_agent_page.dart | 5 +- .../features/channels/add_members_sheet.dart | 9 +- .../channels/channels_page/sheets.dart | 9 +- .../channels/empty_bestie_avatar.dart | 8 +- .../message_content/media_carousel.dart | 2 + mobile/lib/main.dart | 23 +- .../shared/community/community_provider.dart | 54 ++- mobile/lib/shared/relay/media_disk_cache.dart | 363 ++++++++++++++++ mobile/lib/shared/relay/media_image.dart | 271 +++++++++++- mobile/lib/shared/relay/relay.dart | 1 + .../shared/widgets/avatar_image_native.dart | 62 +-- .../widgets/ios_message_create_button.dart | 22 +- .../shared/widgets/ios_navigation_bar.dart | 10 +- mobile/pubspec.lock | 56 ++- mobile/pubspec.yaml | 3 + .../agents/create_agent_page_test.dart | 45 +- .../community/community_provider_test.dart | 55 ++- .../shared/relay/media_disk_cache_test.dart | 391 ++++++++++++++++++ .../shared/widgets/avatar_image_test.dart | 36 +- .../widgets/native_avatar_image_test.dart | 77 +++- 25 files changed, 1439 insertions(+), 114 deletions(-) create mode 100644 mobile/lib/shared/relay/media_disk_cache.dart create mode 100644 mobile/test/shared/relay/media_disk_cache_test.dart diff --git a/mobile/ios/Podfile.lock b/mobile/ios/Podfile.lock index c76613789a..55e8b9a988 100644 --- a/mobile/ios/Podfile.lock +++ b/mobile/ios/Podfile.lock @@ -98,6 +98,9 @@ PODS: - shared_preferences_foundation (0.0.1): - Flutter - FlutterMacOS + - sqflite_darwin (0.0.4): + - Flutter + - FlutterMacOS - url_launcher_ios (0.0.1): - Flutter - video_player_avfoundation (0.0.1): @@ -125,6 +128,7 @@ DEPENDENCIES: - record_ios (from `.symlinks/plugins/record_ios/ios`) - share_plus (from `.symlinks/plugins/share_plus/ios`) - shared_preferences_foundation (from `.symlinks/plugins/shared_preferences_foundation/darwin`) + - sqflite_darwin (from `.symlinks/plugins/sqflite_darwin/darwin`) - url_launcher_ios (from `.symlinks/plugins/url_launcher_ios/ios`) - video_player_avfoundation (from `.symlinks/plugins/video_player_avfoundation/darwin`) @@ -185,6 +189,8 @@ EXTERNAL SOURCES: :path: ".symlinks/plugins/share_plus/ios" shared_preferences_foundation: :path: ".symlinks/plugins/shared_preferences_foundation/darwin" + sqflite_darwin: + :path: ".symlinks/plugins/sqflite_darwin/darwin" url_launcher_ios: :path: ".symlinks/plugins/url_launcher_ios/ios" video_player_avfoundation: @@ -224,6 +230,7 @@ SPEC CHECKSUMS: record_ios: 980fd386a97a35987d0fce3dfda4b26a38c90f4c share_plus: 50da8cb520a8f0f65671c6c6a99b3617ed10a58a shared_preferences_foundation: 7036424c3d8ec98dfe75ff1667cb0cd531ec82bb + sqflite_darwin: 20b2a3a3b70e43edae938624ce550a3cbf66a3d0 url_launcher_ios: 7a95fa5b60cc718a708b8f2966718e93db0cef1b video_player_avfoundation: dd410b52df6d2466a42d28550e33e4146928280a diff --git a/mobile/ios/Runner/AppDelegate.swift b/mobile/ios/Runner/AppDelegate.swift index baa880fe56..e0d4fabb29 100644 --- a/mobile/ios/Runner/AppDelegate.swift +++ b/mobile/ios/Runner/AppDelegate.swift @@ -33,6 +33,7 @@ import os.log private var communityKeyboardChannel: FlutterMethodChannel? private var communityKeyboardObserver: NSObjectProtocol? private var hapticsChannel: FlutterMethodChannel? + private var appGroupChannel: FlutterMethodChannel? private var qrScannerChannel: FlutterMethodChannel? private var inlinePhotoPickerSupportChannel: FlutterMethodChannel? private var ageSignalChannel: FlutterMethodChannel? @@ -134,6 +135,23 @@ import os.log apnsRegistrationBuffer.attach { [weak self] update in self?.pushChannel?.invokeMethod(update.method, arguments: update.arguments) } + appGroupChannel = FlutterMethodChannel( + name: "buzz/app_group", + binaryMessenger: messenger + ) + appGroupChannel?.setMethodCallHandler { [weak self] call, result in + guard call.method == "cachesDirectory" else { + result(FlutterMethodNotImplemented) + return + } + // Caches the app's extensions can read too. The OS may purge it. + let caches = self?.appGroupIdentifier + .flatMap { + FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: $0) + }? + .appendingPathComponent("Library/Caches", isDirectory: true) + result(caches?.path) + } hapticsChannel = FlutterMethodChannel( name: "buzz/haptics", binaryMessenger: messenger diff --git a/mobile/lib/features/agents/agent_avatar_backdrop.dart b/mobile/lib/features/agents/agent_avatar_backdrop.dart index 8c2d6c20c6..535053f300 100644 --- a/mobile/lib/features/agents/agent_avatar_backdrop.dart +++ b/mobile/lib/features/agents/agent_avatar_backdrop.dart @@ -2,6 +2,8 @@ import 'dart:ui' as ui; import 'package:flutter/material.dart'; +import '../../shared/relay/relay.dart'; + import 'agent_avatar_picker.dart'; /// A soft artwork tint behind the avatar editor, matching desktop creation. @@ -82,11 +84,12 @@ class AgentAvatarBackdrop extends StatelessWidget { errorBuilder: (_, _, _) => const SizedBox.expand(), ) - : Image.network( - selection.url!, + : MediaImage( + url: selection.url!, fit: BoxFit.cover, - cacheWidth: 256, - excludeFromSemantics: true, + // About 256 physical pixels: the + // blur hides any more detail. + decodeWidth: 96, errorBuilder: (_, _, _) => const SizedBox.expand(), ), diff --git a/mobile/lib/features/agents/agent_avatar_hero.dart b/mobile/lib/features/agents/agent_avatar_hero.dart index 0b0ad55eb5..6454b2c6ff 100644 --- a/mobile/lib/features/agents/agent_avatar_hero.dart +++ b/mobile/lib/features/agents/agent_avatar_hero.dart @@ -5,6 +5,7 @@ import 'package:flutter_hooks/flutter_hooks.dart'; import '../../shared/animated_avatar.dart'; import '../../shared/widgets/avatar_shape.dart'; import 'agent_avatar_picker.dart'; +import 'agent_character_artwork.dart'; /// Keeps native playback mounted while a cached image carries the route motion. class AgentAvatarHero extends HookWidget { @@ -114,11 +115,7 @@ Widget _poster(AgentAvatarSelection selection, double offset) => ) : Transform.scale( scale: animatedAvatarArtworkScale, - child: Image.network( - selection.url!, - fit: BoxFit.contain, - gaplessPlayback: true, - ), + child: characterPoster(selection.url!), ), ), ); diff --git a/mobile/lib/features/agents/agent_character_artwork.dart b/mobile/lib/features/agents/agent_character_artwork.dart index 1a08a42df1..401757c8cb 100644 --- a/mobile/lib/features/agents/agent_character_artwork.dart +++ b/mobile/lib/features/agents/agent_character_artwork.dart @@ -1,5 +1,7 @@ import 'package:flutter/material.dart'; +import '../../shared/relay/media_image.dart'; + import 'agent_avatar_geometry.dart'; /// Compensates for the shared transparent padding in the Buzzy assets. @@ -22,9 +24,9 @@ class AgentCharacterAlignment extends StatelessWidget { /// One full-resolution cache entry shared by tray, flight, and video placeholder. /// Layout-sized decodes would otherwise reload at every animation frame. -Widget characterPoster(String url) => Image.network( - url, +Widget characterPoster(String url) => MediaImage( + url: url, fit: BoxFit.contain, - gaplessPlayback: true, + boundDecodeToLayout: false, errorBuilder: (_, _, _) => const Icon(Icons.smart_toy_outlined), ); diff --git a/mobile/lib/features/agents/create_agent_page.dart b/mobile/lib/features/agents/create_agent_page.dart index 01738f9fe2..c6e1fd9978 100644 --- a/mobile/lib/features/agents/create_agent_page.dart +++ b/mobile/lib/features/agents/create_agent_page.dart @@ -282,7 +282,10 @@ class CreateAgentPage extends HookConsumerWidget { FocusScope.of(context).unfocus(); if (avatar.value.url != null) { await precacheImage( - NetworkImage(avatar.value.url!), + mediaImageProviderFor( + ref, + avatar.value.url!, + ), context, onError: (_, _) {}, ).timeout( diff --git a/mobile/lib/features/channels/add_members_sheet.dart b/mobile/lib/features/channels/add_members_sheet.dart index b5362d9ec5..2a8a6a1656 100644 --- a/mobile/lib/features/channels/add_members_sheet.dart +++ b/mobile/lib/features/channels/add_members_sheet.dart @@ -215,6 +215,7 @@ class AddChannelMembersSheet extends HookConsumerWidget { if (nativePresentation) { final auth = ref.watch(mediaGetAuthServiceProvider); final client = ref.watch(mediaHttpClientProvider); + final diskCache = ref.watch(mediaDiskCacheProvider); final colors = Theme.of(context).colorScheme; String avatarKey(MentionCandidate user) => jsonEncode([ user.pubkey, @@ -248,8 +249,12 @@ class AddChannelMembersSheet extends HookConsumerWidget { isAgent: user.isAgent, background: colors.primaryContainer, foreground: colors.onPrimaryContainer, - networkImage: (url) => - MediaImageProvider(url: url, auth: auth, client: client), + networkImages: (url) => smallMediaImageProviders( + url: url, + auth: auth, + client: client, + diskCache: diskCache, + ), ); }, state: { diff --git a/mobile/lib/features/channels/channels_page/sheets.dart b/mobile/lib/features/channels/channels_page/sheets.dart index e1f511e28c..359987574e 100644 --- a/mobile/lib/features/channels/channels_page/sheets.dart +++ b/mobile/lib/features/channels/channels_page/sheets.dart @@ -401,6 +401,7 @@ class _NewDirectMessageSheet extends HookConsumerWidget { if (nativePresentation) { final auth = ref.watch(mediaGetAuthServiceProvider); final client = ref.watch(mediaHttpClientProvider); + final diskCache = ref.watch(mediaDiskCacheProvider); final colors = context.colors; String? avatarUrl(MentionCandidate user) => user.avatarUrl ?? userCache[user.pubkey]?.avatarUrl; @@ -437,8 +438,12 @@ class _NewDirectMessageSheet extends HookConsumerWidget { isAgent: user.isAgent, background: colors.primaryContainer, foreground: colors.onPrimaryContainer, - networkImage: (url) => - MediaImageProvider(url: url, auth: auth, client: client), + networkImages: (url) => smallMediaImageProviders( + url: url, + auth: auth, + client: client, + diskCache: diskCache, + ), ); }, state: { diff --git a/mobile/lib/features/channels/empty_bestie_avatar.dart b/mobile/lib/features/channels/empty_bestie_avatar.dart index 6a0f3f5e4a..0467b89574 100644 --- a/mobile/lib/features/channels/empty_bestie_avatar.dart +++ b/mobile/lib/features/channels/empty_bestie_avatar.dart @@ -2,8 +2,10 @@ import 'dart:async'; import 'package:flutter/material.dart'; import 'package:flutter_hooks/flutter_hooks.dart'; +import 'package:hooks_riverpod/hooks_riverpod.dart'; import '../../shared/agents/agent_avatar_catalog.dart'; +import '../../shared/relay/media_image.dart'; import '../../shared/widgets/agent_navigation_icon.dart'; import '../../shared/widgets/avatar_image.dart'; @@ -19,11 +21,11 @@ final emptyBestieCharacters = [ ]; /// A quiet invitation to create a first companion, cycling every five seconds. -class EmptyBestieAvatar extends HookWidget { +class EmptyBestieAvatar extends HookConsumerWidget { const EmptyBestieAvatar({super.key}); @override - Widget build(BuildContext context) { + Widget build(BuildContext context, WidgetRef ref) { final index = useState(0); final reducedMotion = MediaQuery.disableAnimationsOf(context); final lifecycle = useAppLifecycleState(); @@ -51,7 +53,7 @@ class EmptyBestieAvatar extends HookWidget { if (cancelled || !context.mounted) return; unawaited( precacheImage( - NetworkImage(nextPoster), + mediaImageProviderFor(ref, nextPoster), context, onError: (_, _) {}, // Keep the offline avatar fallback. ), diff --git a/mobile/lib/features/channels/message_content/media_carousel.dart b/mobile/lib/features/channels/message_content/media_carousel.dart index fb5dc1bda1..55b8e20f82 100644 --- a/mobile/lib/features/channels/message_content/media_carousel.dart +++ b/mobile/lib/features/channels/message_content/media_carousel.dart @@ -151,6 +151,7 @@ class _MessageImageCarousel extends HookConsumerWidget { final currentIndex = useState(0); final mediaAuth = ref.watch(mediaGetAuthServiceProvider); final mediaClient = ref.watch(mediaHttpClientProvider); + final mediaDiskCache = ref.watch(mediaDiskCacheProvider); return MessageGalleryFrame( count: items.length, @@ -189,6 +190,7 @@ class _MessageImageCarousel extends HookConsumerWidget { url: items[index].url, auth: mediaAuth, client: mediaClient, + diskCache: mediaDiskCache, ), ), ]; diff --git a/mobile/lib/main.dart b/mobile/lib/main.dart index db7c4cd744..5348edba84 100644 --- a/mobile/lib/main.dart +++ b/mobile/lib/main.dart @@ -1,4 +1,5 @@ import 'dart:async'; +import 'dart:io'; import 'package:flutter/material.dart'; import 'package:hooks_riverpod/hooks_riverpod.dart'; @@ -11,6 +12,7 @@ import 'features/agents/agents_page.dart'; import 'features/invites/invite_join_provider.dart'; import 'shared/agents/agents_surface.dart'; import 'shared/push/push_bridge.dart'; +import 'shared/relay/media_disk_cache.dart'; import 'shared/relay/relay_closed_policy.dart'; import 'shared/relay/relay_websocket.dart'; import 'shared/theme/theme_provider.dart'; @@ -29,7 +31,23 @@ Future runBuzzApp(Widget app) async { final prefs = await SharedPreferences.getInstance(); await userAgent; - runApp(buildRootProviderScope(prefs: prefs, child: app)); + runApp( + buildRootProviderScope( + prefs: prefs, + mediaCacheRoot: _mediaCacheRoot, + child: app, + ), + ); +} + +/// Without a caches directory images still load, just not from disk. +Future _mediaCacheRoot() async { + try { + return await resolveMediaCacheRoot(); + } catch (error) { + debugPrint('[MediaDiskCache] no caches directory: $error'); + return null; + } } /// The app's root scope, shared with tests so its production wiring is @@ -37,6 +55,7 @@ Future runBuzzApp(Widget app) async { ProviderScope buildRootProviderScope({ required SharedPreferences prefs, required Widget child, + Future Function()? mediaCacheRoot, }) => ProviderScope( // A relay statement deadline cannot resolve on retry — the same expensive // query would re-run and hit the same limit. Return null (no retry) for @@ -48,6 +67,8 @@ ProviderScope buildRootProviderScope({ (context, link) async => unawaited(showAgents(context, draft: link)), ), savedPrefsProvider.overrideWithValue(prefs), + if (mediaCacheRoot != null) + mediaCacheRootProvider.overrideWith((ref) => mediaCacheRoot()), inviteJoinRecoveryProvider.overrideWith( (ref) => (scope) => buildMobileInviteJoinRecovery(ref, scope), diff --git a/mobile/lib/shared/community/community_provider.dart b/mobile/lib/shared/community/community_provider.dart index 035ba67484..9d9cbfd5d7 100644 --- a/mobile/lib/shared/community/community_provider.dart +++ b/mobile/lib/shared/community/community_provider.dart @@ -16,6 +16,7 @@ import '../push/dev_push_lease.dart'; import '../push/push_bridge.dart'; import '../push/push_lease_revocation_outbox.dart'; import '../push/push_subscription.dart'; +import '../relay/media_disk_cache.dart'; import '../relay/signed_event_relay.dart'; import '../relay/relay_provider.dart' show RelayConfig, pubkeyFromNsec; import 'community.dart'; @@ -270,6 +271,7 @@ class CommunityListNotifier extends AsyncNotifier> { final storage = ref.read(communityStorageProvider); final communities = await storage.loadAll(); unawaited(_removeOrphanChannelListSnapshots(communities)); + unawaited(_removeOrphanMediaCaches(fallback: communities)); await syncCommunitySnapshot(ref, communities); return communities; } @@ -295,14 +297,17 @@ class CommunityListNotifier extends AsyncNotifier> { pubkey: community.pubkey, nsec: community.nsec, ); - if (_channelSnapshotPubkey(existing)?.toLowerCase() != - _channelSnapshotPubkey(updated)?.toLowerCase()) { - await _removeChannelListSnapshot(existing); - } + final identityChanged = + _channelSnapshotPubkey(existing)?.toLowerCase() != + _channelSnapshotPubkey(updated)?.toLowerCase(); + if (identityChanged) await _removeChannelListSnapshot(existing); await storage.save(updated); final updatedList = [...current]; updatedList[existingIndex] = updated; state = AsyncData(updatedList); + // The new account has its own image cache; drop the previous one's in + // the background. + if (identityChanged) unawaited(_removeOrphanMediaCaches()); await _removeOrphanChannelListSnapshots(updatedList); await syncCommunitySnapshot(ref, updatedList); return existing.id; @@ -359,6 +364,9 @@ class CommunityListNotifier extends AsyncNotifier> { ref.invalidate(authProvider); } } + // Deleting up to a full cache can take a while; never hold the removal + // on it. The launch sweep retries a failure. + unawaited(_removeOrphanMediaCaches()); if (revocationJournaled) { unawaited( ref.read(communityPushLeaseRevocationTriggerProvider)().catchError(( @@ -419,6 +427,44 @@ class CommunityListNotifier extends AsyncNotifier> { ]); } + /// Deletes the image caches of communities no longer in the list, and of + /// accounts a community no longer signs in as. A failure, or a + /// write that lands after the delete, is swept on the next launch. + /// + /// Sweeps run one at a time, each keeping what the community list holds + /// when it starts, so an older sweep can never delete a newer account's + /// cache. [fallback] stands in for the list while it is still loading. + Future _removeOrphanMediaCaches({List? fallback}) => + _mediaSweep = _mediaSweep.then((_) => _sweepMediaCaches(fallback)); + + Future _mediaSweep = Future.value(); + + Future _sweepMediaCaches(List? fallback) async { + try { + final root = await ref.read(mediaCacheRootProvider.future); + if (root == null) return; + List? current() => + state.isLoading ? fallback ?? state.value : state.value; + var communities = current(); + if (communities == null) return; + await MediaDiskCache.removeOrphans(root, () { + // A list briefly unavailable mid-sweep keeps the last one seen. + communities = current() ?? communities; + return { + for (final community in communities!) + community.id: _channelSnapshotPubkey(community)?.toLowerCase(), + }; + }); + } catch (error, stackTrace) { + developer.log( + 'Media cache deletion failed; the next launch will retry', + name: 'buzz.community', + error: error, + stackTrace: stackTrace, + ); + } + } + Future switchCommunity(String id) { return ref.read(communityTransitionProvider).runExclusive(() async { final storage = ref.read(communityStorageProvider); diff --git a/mobile/lib/shared/relay/media_disk_cache.dart b/mobile/lib/shared/relay/media_disk_cache.dart new file mode 100644 index 0000000000..ad8df2d51b --- /dev/null +++ b/mobile/lib/shared/relay/media_disk_cache.dart @@ -0,0 +1,363 @@ +import 'dart:async'; +import 'dart:io'; + +import 'package:file/file.dart' as pf; +import 'package:file/local.dart'; +import 'package:flutter/foundation.dart'; +import 'package:flutter/services.dart'; +import 'package:flutter_cache_manager/flutter_cache_manager.dart'; +import 'package:hooks_riverpod/hooks_riverpod.dart'; +import 'package:path/path.dart' as p; +import 'package:path_provider/path_provider.dart'; + +import '../community/community_provider.dart'; +import 'relay_provider.dart' show pubkeyFromNsec; + +/// Image bytes saved on disk, and whether they are due a background refresh. +typedef MediaDiskCacheHit = ({Uint8List bytes, bool isStale}); + +/// Persistent store for one community's remote image bytes: custom emoji, +/// avatars, attachments and anything else [MediaImageProvider] loads. +/// +/// Built on `flutter_cache_manager`. It keeps the bytes each URL served, so a +/// relay thumbnail and its original are separate entries. Each community and +/// account has its own directory under the caches root (see +/// [mediaCacheRootProvider]), deleted with the community or account. Least recently used files are evicted once +/// the directory holds more than [maxBytes]. +/// +/// Relay media URLs are content hashes and never go stale. Other URLs can +/// change in place, so they are served from disk but refetched in the +/// background once older than [externalRefreshAge]. +class MediaDiskCache { + /// Opens the store in [directory], creating it on first write, or a store + /// that saves nothing when [directory] is null. Reads and writes wait for + /// [directory] to resolve, so a load never skips the disk while it does. + /// [now] is the clock staleness is judged by. + /// + /// Tests can shrink the budgets and the delay after a write or first read + /// before eviction runs (10 s in `flutter_cache_manager`). + MediaDiskCache.open( + FutureOr directory, { + @visibleForTesting DateTime Function() now = DateTime.now, + @visibleForTesting int byteBudget = maxBytes, + @visibleForTesting int fileBudget = maxFiles, + @visibleForTesting Duration? evictionDelay, + }) : _now = now { + _manager = Future.value(directory) + // Images wait for the disk, so never wait on it for long. + .timeout(_openTimeout, onTimeout: () => null) + .then((directory) { + if (directory == null || isClosed) return null; + return _openManager( + directory, + byteBudget: byteBudget, + fileBudget: fileBudget, + evictionDelay: evictionDelay, + ); + }) + .catchError((Object error) { + debugPrint('[MediaDiskCache] open failed: $error'); + return null; + }); + } + + static CacheManager _openManager( + Directory directory, { + required int byteBudget, + required int fileBudget, + required Duration? evictionDelay, + }) { + final manager = CacheManager( + Config( + directory.path, + // Age never evicts: the byte budget below does. + stalePeriod: _neverStale, + maxNrOfCacheObjects: fileBudget, + repo: MediaCacheIndex( + File(p.join(directory.path, 'index.json')), + directory, + byteBudget, + ), + fileSystem: _DirectoryFileSystem(directory), + ), + ); + if (evictionDelay != null) { + manager.store.cleanupRunMinInterval = evictionDelay; + } + return manager; + } + + static const _openTimeout = Duration(seconds: 3); + + /// Most bytes one community's cache keeps on disk. + static const maxBytes = 250 * 1024 * 1024; + + /// Most files one community's cache keeps, so many tiny files (emoji, + /// avatars) cannot cost far more disk and index work than their bytes. + static const maxFiles = 10000; + + /// Larger images still show, from memory, but are never written to disk, + /// so one image cannot evict the rest of the cache. + static const maxFileBytes = 5 * 1024 * 1024; + + /// Age at which a non-relay image is refetched in the background. + static const externalRefreshAge = Duration(days: 7); + + static const _neverStale = Duration(days: 365 * 100); + + late final Future _manager; + final DateTime Function() _now; + final _closed = Completer(); + + /// Completes once the cache is disposed, when its community or account is + /// retired. + Future get closed => _closed.future; + + /// Whether the cache was disposed. + bool get isClosed => _closed.isCompleted; + + /// The directory holding everything one community persists under [root]. + static Directory communityDirectory(Directory root, String communityId) => + Directory( + p.join(root.path, 'communities', Uri.encodeComponent(communityId)), + ); + + /// Where [communityId]'s image cache for the account [signer] lives under + /// [root]. Each account has its own, so one account's images never serve + /// another's. + static Directory directoryFor( + Directory root, + String communityId, + String signer, + ) => Directory( + p.join(communityDirectory(root, communityId).path, 'media', signer), + ); + + /// Keeps only each retained community's cache for its current signer + /// (community id to signer pubkey) and deletes every other directory under + /// [root]: removed communities and replaced accounts, including writes + /// that landed after their removal. A failed delete is retried on the + /// next call. + /// + /// [retained] is read again before each delete, so a community or account + /// added while the sweep runs is kept. + static Future removeOrphans( + Directory root, + Map Function() retained, + ) async { + Set keep() => { + for (final MapEntry(key: id, value: signer) in retained().entries) + if (signer != null) directoryFor(root, id, signer).path, + }; + final communities = Directory(p.join(root.path, 'communities')); + await _deleteExcept(communities, () { + return {for (final path in keep()) p.dirname(p.dirname(path))}; + }); + if (!await communities.exists()) return; + await for (final community in communities.list(followLinks: false)) { + if (community is! Directory) continue; + await _deleteExcept(Directory(p.join(community.path, 'media')), keep); + } + } + + /// Deletes the entries of [parent] whose paths are not in [keep]. + static Future _deleteExcept( + Directory parent, + Set Function() keep, + ) async { + if (!await parent.exists()) return; + await for (final entry in parent.list(followLinks: false)) { + if (keep().contains(entry.path)) continue; + try { + await entry.delete(recursive: true); + } on FileSystemException catch (error) { + debugPrint('[MediaDiskCache] orphan cleanup failed: $error'); + } + } + } + + /// Removes [url]'s saved bytes, for example when they no longer decode. + /// Failures are logged; a later read finds the same bad bytes. + Future remove(String url) async { + try { + await (await _manager)?.removeFile(url); + } on Object catch (error) { + _logFailure(error); + } + } + + /// [url]'s saved bytes, or null when they are not on disk. + Future read(String url) async { + final FileInfo? info; + final Uint8List bytes; + final manager = await _manager; + if (manager == null) return null; + try { + // Past the first lookup, the package serves a URL from memory without + // marking it used, so a file read often would be evicted first. Going + // to the index marks it used, and schedules eviction. + info = await manager.getFileFromCache(url, ignoreMemCache: true); + if (info == null) return null; + bytes = await info.file.readAsBytes(); + } on FileSystemException { + // The OS may purge cache files under the index. Drop the entry and + // load from the network. + unawaited(remove(url)); + return null; + } + if (bytes.isEmpty) return null; + return (bytes: bytes, isStale: info.validTill.isBefore(_now())); + } + + /// Saves [bytes] for [url]. [immutable] bytes (relay media) are never + /// refetched; others are refetched after [externalRefreshAge]. Bytes over + /// [maxFileBytes] are not saved, and drop any older copy so it is not + /// shown in their place. Failures are logged, never thrown: a missing + /// entry only costs a later download. + Future write( + String url, + Uint8List bytes, { + required bool immutable, + }) async { + // A download that finishes after its account or community was retired + // must not repopulate the cache. + if (isClosed || bytes.isEmpty) return; + final manager = await _manager; + if (manager == null || isClosed) return; + try { + if (bytes.length > maxFileBytes) { + await manager.removeFile(url); + return; + } + await manager.putFile( + url, + bytes, + maxAge: immutable ? _neverStale : externalRefreshAge, + ); + // Replacing a URL the package already holds in memory schedules no + // eviction, so a refresh that grows the cache would leave it over + // budget. An index lookup schedules it. + await manager.getFileFromCache(url, ignoreMemCache: true); + } on Object catch (error) { + _logFailure(error); + } + } + + /// Stops writes, then flushes the index. + Future dispose() { + if (_closed.isCompleted) return _disposed; + _closed.complete(); + return _disposed = _manager + .then((manager) => manager?.dispose()) + .catchError((Object error) { + debugPrint('[MediaDiskCache] close failed: $error'); + }); + } + + Future _disposed = Future.value(); + + static void _logFailure(Object error) => + debugPrint('[MediaDiskCache] write failed: $error'); +} + +/// `flutter_cache_manager` caps a cache only by file count. This index keeps +/// its JSON format but evicts by bytes: once the files total more than +/// [_maxBytes], the least recently used are over capacity. +@visibleForTesting +class MediaCacheIndex extends JsonCacheInfoRepository { + MediaCacheIndex(super.file, this._directory, this._maxBytes) + : super.withFile(); + + final Directory _directory; + final int _maxBytes; + + /// Least recently used files past [capacity] files or past the byte + /// budget, whichever keeps fewer. + @override + Future> getObjectsOverCapacity(int capacity) async { + final newestFirst = await getAllObjects() + ..sort((a, b) => b.touched!.compareTo(a.touched!)); + var total = 0; + var count = 0; + return [ + for (final object in newestFirst) + if ((total += object.length ?? 0) > _maxBytes || ++count > capacity) + object, + ]; + } + + /// `putFile` records no length, and replacing a file's bytes keeps the old + /// one, so measure the file on every write. + @override + Future updateOrInsert(CacheObject cacheObject) async { + final file = File(p.join(_directory.path, cacheObject.relativePath)); + final length = await file.exists() ? await file.length() : 0; + return super.updateOrInsert(cacheObject.copyWith(length: length)); + } +} + +/// Keeps cache files in one directory of our choosing rather than the +/// package's default under the temporary directory. +class _DirectoryFileSystem implements FileSystem { + _DirectoryFileSystem(Directory directory) + : _directory = const LocalFileSystem().directory(directory.path); + + final pf.Directory _directory; + + @override + Future createFile(String name) async { + await _directory.create(recursive: true); + return _directory.childFile(name); + } +} + +const _appGroupChannel = MethodChannel('buzz/app_group'); + +/// The directory community caches live under: the shared app group's caches +/// on iOS, so app extensions can read them, and the app's cache directory +/// elsewhere. Null when the iOS app group is unavailable. +Future resolveMediaCacheRoot() async { + if (!Platform.isIOS) return getApplicationCacheDirectory(); + final path = await _appGroupChannel.invokeMethod('cachesDirectory'); + if (path == null) { + debugPrint('[MediaDiskCache] no app group container; disk cache is off'); + return null; + } + return Directory(path); +} + +/// The caches root, overridden at startup with [resolveMediaCacheRoot]. Null +/// (isolated tests, or no app group) turns the disk cache off. Startup does +/// not wait for it: images that load first skip the disk. +final mediaCacheRootProvider = FutureProvider((ref) async => null); + +/// The active community's image cache, or null when there is no caches root +/// or no community. Switching community or account disposes the previous one. +final mediaDiskCacheProvider = Provider((ref) { + final (communityId, signer) = ref.watch( + activeCommunityProvider.select((value) { + final community = value.value; + if (community == null) return (null, null); + final signer = pubkeyFromNsec(community.nsec?.trim()) ?? community.pubkey; + return (community.id, signer); + }), + ); + if (communityId == null || signer == null) return null; + // The cache exists at once and opens its directory when the root + // resolves, so images shown during startup still read the disk. + final cache = MediaDiskCache.open( + ref + .watch(mediaCacheRootProvider.future) + .then( + (root) => root == null + ? null + : MediaDiskCache.directoryFor( + root, + communityId, + signer.toLowerCase(), + ), + ), + ); + ref.onDispose(() => unawaited(cache.dispose())); + return cache; +}); diff --git a/mobile/lib/shared/relay/media_image.dart b/mobile/lib/shared/relay/media_image.dart index 7608cef60b..7a2305e73a 100644 --- a/mobile/lib/shared/relay/media_image.dart +++ b/mobile/lib/shared/relay/media_image.dart @@ -1,4 +1,5 @@ import 'dart:async'; +import 'dart:collection'; import 'dart:ui' as ui; import 'package:flutter/foundation.dart'; @@ -10,6 +11,7 @@ import 'package:http/http.dart' as http; import '../hosted/hosted_relay_auth.dart'; import '../utils/bounded_fetch.dart'; import 'media_auth.dart'; +import 'media_disk_cache.dart'; /// Shared, keep-alive HTTP client for media fetches. One client per container /// so TLS connections are reused across images instead of re-handshaking per @@ -20,6 +22,50 @@ final mediaHttpClientProvider = Provider((ref) { return client; }); +/// The provider [MediaImage] shows [url] with at full size (no decode +/// bound), for warming it with `precacheImage`. +MediaImageProvider mediaImageProviderFor(WidgetRef ref, String url) => + MediaImageProvider( + url: url, + auth: ref.read(mediaGetAuthServiceProvider), + client: ref.read(mediaHttpClientProvider), + diskCache: ref.read(mediaDiskCacheProvider), + ); + +/// The providers to try, best first, for [url] drawn at most +/// [MediaImage.thumbnailMaxPhysicalSize] physical pixels outside a +/// [MediaImage], such as a native bar's avatar bitmap. This is the source +/// [MediaImage.preferThumbnail] picks: a relay image gets only its generated +/// thumbnails, minus those the relay already returned 404 for, so it can get +/// none; any other URL gets itself. Try the next provider only after a 404. +List smallMediaImageProviders({ + required String url, + required MediaGetAuthService auth, + required http.Client client, + MediaDiskCache? diskCache, +}) { + final thumbnails = relayThumbnailUrls(url, auth); + return [ + if (thumbnails.isEmpty) + MediaImageProvider( + url: url, + auth: auth, + client: client, + diskCache: diskCache, + ) + else + for (final thumbnail in thumbnails) + if (!MediaImage._missingThumbnails.contains(thumbnail)) + MediaImageProvider( + url: thumbnail, + auth: auth, + client: client, + diskCache: diskCache, + isThumbnailFallback: true, + ), + ]; +} + /// An [ImageProvider] for relay-hosted (and arbitrary remote) media that fixes /// the fetch-stampede failure modes of `Image.network` + per-build auth /// headers: @@ -46,8 +92,8 @@ class MediaImageProvider extends ImageProvider { final ValueChanged? onBytesLoaded; /// Set when [url] is a thumbnail [MediaImage] chose in place of an - /// original: its 404 then records a missing thumbnail, so later builds load - /// the original, rather than a failure cooldown. Any other 404, including a + /// original: its 404 then records a missing thumbnail, so later builds try + /// the next thumbnail, rather than a failure cooldown. Any other 404, including a /// profile URL that is itself a thumbnail, keeps the cooldown. Part of /// equality, the shared download's key and the cooldown's key, so a /// fallback and a direct load of the same URL each keep their own failure @@ -57,6 +103,10 @@ class MediaImageProvider extends ImageProvider { /// Excluded from equality: transport, not identity. final http.Client client; + /// Where loaded bytes persist across launches, or null to keep them in + /// memory only. Excluded from equality: storage, not identity. + final MediaDiskCache? diskCache; + /// Longest wait for response headers or the next body chunk. A slow link /// that keeps delivering bytes finishes; one that stops is aborted without /// closing the shared client. @@ -70,6 +120,7 @@ class MediaImageProvider extends ImageProvider { required this.url, required this.auth, required this.client, + this.diskCache, this.onBytesLoaded, this.isThumbnailFallback = false, this.scale = 1.0, @@ -90,6 +141,19 @@ class MediaImageProvider extends ImageProvider { static final Map<(String, MediaGetAuthService, bool), _SharedFetch> _inFlight = {}; + /// Background refreshes of stale disk bytes running or queued, per cache + /// and URL. + static final Set<(MediaDiskCache, String)> _refreshing = {}; + static final _refreshQueue = Queue Function()>(); + static var _runningRefreshes = 0; + + /// Most background refreshes at once; the rest wait in a queue. + static const _maxRefreshes = 2; + + /// Most refreshes waiting. A stale image dropped past this keeps its + /// saved bytes and refreshes the next time it loads from disk. + static const _maxQueuedRefreshes = 64; + /// Injectable clock for cooldown tests. @visibleForTesting static DateTime Function() debugNow = DateTime.now; @@ -114,14 +178,28 @@ class MediaImageProvider extends ImageProvider { ImageDecoderCallback decode, ) { final abandoned = Completer(); + _SharedFetch? fetched; // The image cache listens to a pending load, so this fires mid-download // only once a consumer evicts the entry and stops listening (see // [abandonPendingImage]), as [MediaImage] does when it stops showing an // image before its first frame. Abort the download then. return _MediaImageStreamCompleter( - codec: _loadAsync(key, decode, abandoned.future), + codec: _loadAsync( + key, + decode, + abandoned.future, + onFetch: (fetch) => fetched = fetch, + ), scale: key.scale, debugLabel: url, + // Save only bytes this load decoded a frame from (a codec can open + // from bytes whose frames then fail to decode), and drop saved bytes + // that no longer decode. Then drop the fetch: the cached completer + // must not keep the encoded bytes alive. + onSettled: (decoded) { + fetched?.settle(decoded: decoded); + fetched = null; + }, )..addOnLastListenerRemovedCallback(() { if (!abandoned.isCompleted) abandoned.complete(); }); @@ -130,8 +208,9 @@ class MediaImageProvider extends ImageProvider { Future _loadAsync( MediaImageProvider key, ImageDecoderCallback decode, - Future abandoned, - ) async { + Future abandoned, { + required void Function(_SharedFetch) onFetch, + }) async { var wasAbandoned = false; try { // Each decode size is its own cache entry (see [ResizeImage]), so one @@ -139,12 +218,13 @@ class MediaImageProvider extends ImageProvider { // fetching the same bytes once per size. final fetchKey = (url, auth, isThumbnailFallback); final fetch = _inFlight[fetchKey] ??= _SharedFetch( - _fetchBytes, + _loadBytes, onFinished: (fetch) { if (_inFlight[fetchKey] == fetch) _inFlight.remove(fetchKey); }, ); fetch.join(); + onFetch(fetch); var left = false; void leave() { if (left) return; @@ -183,6 +263,96 @@ class MediaImageProvider extends ImageProvider { } } + /// [url]'s bytes from [diskCache] when saved there, else downloaded (and + /// saved once they decode). Stale saved bytes are shown, and once they + /// decode a background download replaces them; saved bytes that fail to + /// decode are removed instead. + Future _loadBytes(_SharedFetch fetch) async { + final cache = diskCache; + if (cache == null) return _fetchBytes(fetch.aborted); + final hit = await cache.read(url); + if (hit != null) { + // Refresh only after the decode, so a refresh can never land before + // the removal of bytes that failed it. + if (hit.isStale) fetch.onDecoded = () => _scheduleRefresh(cache); + fetch.onFailed = () => unawaited(cache.remove(url)); + return hit.bytes; + } + final bytes = await _fetchBytes(fetch.aborted); + final immutable = _isContentAddressed; + fetch.onDecoded = () => + unawaited(cache.write(url, bytes, immutable: immutable)); + return bytes; + } + + void _scheduleRefresh(MediaDiskCache cache) { + final key = (cache, url); + if (_refreshQueue.length >= _maxQueuedRefreshes || !_refreshing.add(key)) { + return; + } + _refreshQueue.add(() async { + try { + await _refresh(cache); + } finally { + _refreshing.remove(key); + } + }); + _pumpRefreshes(); + } + + static void _pumpRefreshes() { + while (_runningRefreshes < _maxRefreshes && _refreshQueue.isNotEmpty) { + final refresh = _refreshQueue.removeFirst(); + _runningRefreshes++; + unawaited( + refresh().whenComplete(() { + _runningRefreshes--; + _pumpRefreshes(); + }), + ); + } + } + + Future _refresh(MediaDiskCache cache) async { + try { + // Nobody is waiting on it, so it stops only when its cache retires. + if (cache.isClosed) return; + final bytes = await _fetchBytes(cache.closed); + // Nothing displays these bytes yet, so check they decode before they + // replace the saved copy. A tiny target keeps the check from + // allocating a full-resolution frame. + try { + final codec = await ui.instantiateImageCodec( + bytes, + targetWidth: 1, + targetHeight: 1, + allowUpscaling: false, + ); + try { + (await codec.getNextFrame()).image.dispose(); + } finally { + codec.dispose(); + } + } on Object { + // A body that never decodes is a failed fetch: back off. + _cooldownUntil[_cooldownKey] = debugNow().add(_defaultCooldown); + rethrow; + } + await cache.write(url, bytes, immutable: _isContentAddressed); + } on Object catch (error) { + // The stale bytes stay; the next load after the cooldown retries. + debugPrint('[MediaImageProvider] refresh of $url failed: $error'); + } + } + + /// A relay blob or its thumbnail: named by content hash, so its bytes + /// never change. + bool get _isContentAddressed { + if (!auth.isRelayMediaUrl(url)) return false; + final uri = Uri.parse(url); + return !uri.hasQuery && _relayContentPath.hasMatch(uri.path); + } + /// Downloads [url]. Completing [abort] means every load waiting for this /// download was abandoned, so the URL is not put on cooldown. Future _fetchBytes(Future abort) async { @@ -218,7 +388,7 @@ class MediaImageProvider extends ImageProvider { // Older or externally populated media may have no generated thumbnail. // Record that here rather than in a widget's error builder: the widget // that asked may be gone by the time a slow 404 arrives, and a - // cooldown would then keep every later build off the original. + // cooldown would then keep every later build off the next thumbnail. MediaImage._rememberMissingThumbnail(url); throw NetworkImageLoadException(statusCode: 404, uri: uri); } @@ -301,11 +471,47 @@ class _MediaImageStreamCompleter extends MultiFrameImageStreamCompleter { _MediaImageStreamCompleter({ required super.codec, required super.scale, + required this.onSettled, super.debugLabel, }); + /// Called once: with true when the first frame has decoded, or with false + /// when the load fails before it. A codec can be created from bytes whose + /// frames then fail to decode. + void Function(bool decoded)? onSettled; + + void _settle({required bool decoded}) { + final settled = onSettled; + onSettled = null; + settled?.call(decoded); + } + final _listeners = []; + @override + void setImage(ImageInfo image) { + _settle(decoded: true); + super.setImage(image); + } + + @override + void reportError({ + DiagnosticsNode? context, + required Object exception, + StackTrace? stack, + InformationCollector? informationCollector, + bool silent = false, + }) { + _settle(decoded: false); + super.reportError( + context: context, + exception: exception, + stack: stack, + informationCollector: informationCollector, + silent: silent, + ); + } + @override void addListener(ImageStreamListener listener) { _listeners.add(listener); @@ -325,10 +531,10 @@ class _MediaImageStreamCompleter extends MultiFrameImageStreamCompleter { /// download. class _SharedFetch { _SharedFetch( - Future Function(Future abort) fetch, { + Future Function(_SharedFetch fetch) load, { required this.onFinished, }) { - bytes = fetch(_abort.future).whenComplete(() { + bytes = load(this).whenComplete(() { _done = true; onFinished(this); }); @@ -342,6 +548,24 @@ class _SharedFetch { var _waiting = 0; var _done = false; + /// Completes when every load waiting for this download was abandoned. + Future get aborted => _abort.future; + + /// Run by the first load that decodes a frame: saves downloaded bytes, + /// or refreshes stale saved ones. + VoidCallback? onDecoded; + + /// Run when a load fails before decoding a frame: removes saved bytes. + VoidCallback? onFailed; + + /// Runs [onDecoded] or [onFailed] once, by whether a load decoded a frame. + void settle({required bool decoded}) { + final pending = decoded ? onDecoded : onFailed; + onDecoded = null; + onFailed = null; + pending?.call(); + } + void join() => _waiting++; /// Called once per joined load, when it is abandoned or has its result. @@ -366,6 +590,10 @@ class MediaImageCooldownException implements Exception { 'MediaImageCooldownException: $url is cooling down until $until'; } +final _relayContentPath = RegExp( + r'^/media/[0-9a-f]{64}(?:\.thumb\.(?:jpg|webp)|\.[a-z0-9]{1,8})?$', +); + final _relayBlobPath = RegExp(r'^/media/([0-9a-f]{64})(?:\.[a-z0-9]{1,8})?$'); /// The relay's generated thumbnails for a relay-hosted image, best first, or @@ -401,7 +629,9 @@ List relayThumbnailUrls(String url, MediaGetAuthService auth) { /// /// With [preferThumbnail], a relay image laid out small enough loads the /// relay's generated thumbnail instead of the original upload, falling back -/// to the JPEG thumbnail and then the original when the relay has none. +/// to the JPEG thumbnail when the relay has no WebP one. It never falls back +/// to the original: with no thumbnail it shows [errorBuilder], so a small +/// image cannot download and decode a multi-megabyte upload. class MediaImage extends ConsumerWidget { final String url; final BoxFit? fit; @@ -449,13 +679,14 @@ class MediaImage extends ConsumerWidget { ), assert( !preferThumbnail || errorBuilder != null, - 'a thumbnail failure other than 404 needs an error widget', + 'a thumbnail failure needs an error widget', ); @override Widget build(BuildContext context, WidgetRef ref) { final auth = ref.watch(mediaGetAuthServiceProvider); final client = ref.watch(mediaHttpClientProvider); + final diskCache = ref.watch(mediaDiskCacheProvider); MediaImageProvider providerFor( String source, { bool isThumbnailFallback = false, @@ -463,6 +694,7 @@ class MediaImage extends ConsumerWidget { url: source, auth: auth, client: client, + diskCache: diskCache, onBytesLoaded: onBytesLoaded, isThumbnailFallback: isThumbnailFallback, ); @@ -487,13 +719,24 @@ class MediaImage extends ConsumerWidget { final rest = candidates .skipWhile(_missingThumbnails.contains) .toList(); - if (rest.isEmpty) return _image(providerFor(url), cacheWidth); + if (rest.isEmpty) { + if (thumbnails.isEmpty) return _image(providerFor(url), cacheWidth); + return errorBuilder!( + context, + NetworkImageLoadException( + statusCode: 404, + uri: Uri.parse(thumbnails.last), + ), + null, + ); + } return _image( providerFor(rest.first, isThumbnailFallback: true), cacheWidth, errorBuilder: (context, error, stackTrace) { if (error is! NetworkImageLoadException || - error.statusCode != 404) { + error.statusCode != 404 || + rest.length == 1) { return errorBuilder!(context, error, stackTrace); } // The fetch already recorded the missing thumbnail, so later @@ -510,7 +753,7 @@ class MediaImage extends ConsumerWidget { /// Thumbnails the relay returned 404 for, oldest first. Bounded so a /// community full of pre-thumbnail media cannot grow it without limit; an - /// evicted entry costs one more 404 before the original loads again. + /// evicted entry costs one more 404 before the next source loads again. static final Set _missingThumbnails = {}; static const _maxMissingThumbnails = 256; diff --git a/mobile/lib/shared/relay/relay.dart b/mobile/lib/shared/relay/relay.dart index c2490968b8..360e0e2a2b 100644 --- a/mobile/lib/shared/relay/relay.dart +++ b/mobile/lib/shared/relay/relay.dart @@ -1,6 +1,7 @@ export 'app_lifecycle_provider.dart'; export 'identity_scoped_prefs.dart'; export 'media_auth.dart'; +export 'media_disk_cache.dart'; export 'media_image.dart'; export 'media_upload.dart'; export 'nostr_filters.dart'; diff --git a/mobile/lib/shared/widgets/avatar_image_native.dart b/mobile/lib/shared/widgets/avatar_image_native.dart index 2b18b41f46..f7feaed3c5 100644 --- a/mobile/lib/shared/widgets/avatar_image_native.dart +++ b/mobile/lib/shared/widgets/avatar_image_native.dart @@ -8,7 +8,7 @@ Future nativeAvatarImage({ bool circular = false, required Color background, required Color foreground, - required ImageProvider Function(String) networkImage, + required List Function(String url) networkImages, }) async { const dimension = 72.0; final animatedAvatar = parseAnimatedAvatarUrl(url); @@ -69,26 +69,16 @@ Future nativeAvatarImage({ } rendered = true; case _NetworkAvatarSource(:final url): - final stream = ResizeImage( - networkImage(url), - width: dimension.toInt(), - height: dimension.toInt(), - policy: ResizeImagePolicy.fit, - ).resolve(ImageConfiguration.empty); - final completer = Completer(); - final listener = ImageStreamListener( - (image, _) { - if (!completer.isCompleted) completer.complete(image.clone()); - }, - onError: (Object error, StackTrace? stack) { - if (!completer.isCompleted) completer.completeError(error, stack); - }, - ); - stream.addListener(listener); - try { - final image = await completer.future.timeout( - const Duration(seconds: 10), - ); + final providers = networkImages(url); + for (var i = 0; i < providers.length; i++) { + final ImageInfo image; + try { + image = await _loadNativeAvatar(providers[i], dimension.toInt()); + } on NetworkImageLoadException catch (error) { + // A missing relay thumbnail moves on to the next one. + if (error.statusCode == 404 && i < providers.length - 1) continue; + rethrow; + } paintImage( canvas: canvas, rect: imageBounds, @@ -97,8 +87,7 @@ Future nativeAvatarImage({ ); image.dispose(); rendered = true; - } finally { - stream.removeListener(listener); + break; } case null: break; @@ -119,6 +108,33 @@ Future nativeAvatarImage({ } } +Future _loadNativeAvatar( + ImageProvider provider, + int dimension, +) async { + final stream = ResizeImage( + provider, + width: dimension, + height: dimension, + policy: ResizeImagePolicy.fit, + ).resolve(ImageConfiguration.empty); + final completer = Completer(); + final listener = ImageStreamListener( + (image, _) { + if (!completer.isCompleted) completer.complete(image.clone()); + }, + onError: (Object error, StackTrace? stack) { + if (!completer.isCompleted) completer.completeError(error, stack); + }, + ); + stream.addListener(listener); + try { + return await completer.future.timeout(const Duration(seconds: 10)); + } finally { + stream.removeListener(listener); + } +} + void _paintNativeAvatarText( Canvas canvas, String text, diff --git a/mobile/lib/shared/widgets/ios_message_create_button.dart b/mobile/lib/shared/widgets/ios_message_create_button.dart index 619569a8c7..7ed01526cf 100644 --- a/mobile/lib/shared/widgets/ios_message_create_button.dart +++ b/mobile/lib/shared/widgets/ios_message_create_button.dart @@ -5,12 +5,14 @@ import 'package:flutter/foundation.dart'; import 'package:flutter/gestures.dart'; import 'package:flutter/services.dart'; import 'package:flutter_hooks/flutter_hooks.dart'; +import 'package:hooks_riverpod/hooks_riverpod.dart'; +import '../relay/relay.dart'; import '../theme/theme.dart'; import 'avatar_image.dart'; /// The four desktop creation destinations, presented by UIKit on iOS. -class IosMessageCreateButton extends HookWidget { +class IosMessageCreateButton extends HookConsumerWidget { const IosMessageCreateButton({ super.key, required this.actions, @@ -66,7 +68,7 @@ class IosMessageCreateButton extends HookWidget { final List<({String label, String symbol, VoidCallback? onPressed})> actions; @override - Widget build(BuildContext context) { + Widget build(BuildContext context, WidgetRef ref) { final route = ModalRoute.of(context); final coveringProgress = useAnimation( route?.secondaryAnimation ?? kAlwaysDismissedAnimation, @@ -76,6 +78,14 @@ class IosMessageCreateButton extends HookWidget { final brightness = context.theme.brightness; final foreground = context.colors.onSurface.toARGB32(); + // Only an avatar trigger fetches media, so only it depends on the relay. + final media = avatarInitial == null + ? null + : ( + auth: ref.watch(mediaGetAuthServiceProvider), + client: ref.watch(mediaHttpClientProvider), + diskCache: ref.watch(mediaDiskCacheProvider), + ); final avatarFuture = useMemoized( () => avatarInitial == null ? null @@ -85,10 +95,16 @@ class IosMessageCreateButton extends HookWidget { circular: true, background: context.colors.primaryContainer, foreground: context.colors.onPrimaryContainer, - networkImage: NetworkImage.new, + networkImages: (url) => smallMediaImageProviders( + url: url, + auth: media!.auth, + client: media.client, + diskCache: media.diskCache, + ), ), [ avatarUrl, + media, avatarInitial, context.colors.primaryContainer, context.colors.onPrimaryContainer, diff --git a/mobile/lib/shared/widgets/ios_navigation_bar.dart b/mobile/lib/shared/widgets/ios_navigation_bar.dart index 86223a2a10..3d19c6ea7a 100644 --- a/mobile/lib/shared/widgets/ios_navigation_bar.dart +++ b/mobile/lib/shared/widgets/ios_navigation_bar.dart @@ -183,6 +183,7 @@ class IosNavigationBar extends HookConsumerWidget { }; final auth = ref.watch(mediaGetAuthServiceProvider); final client = ref.watch(mediaHttpClientProvider); + final diskCache = ref.watch(mediaDiskCacheProvider); final colors = Theme.of(context).colorScheme; String imageKey(IosNavigationAction action) => jsonEncode([ action.avatarIdentity, @@ -211,8 +212,12 @@ class IosNavigationBar extends HookConsumerWidget { ? Colors.transparent : colors.primaryContainer, foreground: colors.onPrimaryContainer, - networkImage: (url) => - MediaImageProvider(url: url, auth: auth, client: client), + networkImages: (url) => smallMediaImageProviders( + url: url, + auth: auth, + client: client, + diskCache: diskCache, + ), ).then( (bytes) { if (!active || !context.mounted || bytes == null) return; @@ -236,6 +241,7 @@ class IosNavigationBar extends HookConsumerWidget { avatarKey, auth, client, + diskCache, colors.primaryContainer, colors.onPrimaryContainer, ], diff --git a/mobile/pubspec.lock b/mobile/pubspec.lock index 7d3ac999f5..a3a747f4cd 100644 --- a/mobile/pubspec.lock +++ b/mobile/pubspec.lock @@ -393,7 +393,7 @@ packages: source: hosted version: "0.1.2" file: - dependency: transitive + dependency: "direct main" description: name: file sha256: a3b4f84adafef897088c160faf7dfffb7696046cb13ae90b508c2cbc95d3b8d4 @@ -477,6 +477,14 @@ packages: description: flutter source: sdk version: "0.0.0" + flutter_cache_manager: + dependency: "direct main" + description: + name: flutter_cache_manager + sha256: a42024cf8bbb7e1f4d2875301660f62543601a4c3b20119ea9a683e4d191d0f4 + url: "https://pub.dev" + source: hosted + version: "3.5.0" flutter_hooks: dependency: "direct main" description: @@ -1040,7 +1048,7 @@ packages: source: hosted version: "4.1.0" path: - dependency: transitive + dependency: "direct main" description: name: path sha256: "75cca69d1490965be98c73ceaea117e8a04dd21217b37b292c9ddbec0d955bc5" @@ -1428,6 +1436,46 @@ packages: url: "https://pub.dev" source: hosted version: "1.10.2" + sqflite: + dependency: transitive + description: + name: sqflite + sha256: "564cfed0746fe53140c23b70b308e045c3b31f17778f2f326ccb7d804ea0250a" + url: "https://pub.dev" + source: hosted + version: "2.4.2+1" + sqflite_android: + dependency: transitive + description: + name: sqflite_android + sha256: "881e28efdcc9950fd8e9bb42713dcf1103e62a2e7168f23c9338d82db13dec40" + url: "https://pub.dev" + source: hosted + version: "2.4.2+3" + sqflite_common: + dependency: transitive + description: + name: sqflite_common + sha256: "1581ffbf7a0e333b380d6a30737d78516b826cb35beb7fb0bf8a3ea0c678b465" + url: "https://pub.dev" + source: hosted + version: "2.5.8" + sqflite_darwin: + dependency: transitive + description: + name: sqflite_darwin + sha256: "279832e5cde3fe99e8571879498c9211f3ca6391b0d818df4e17d9fff5c6ccb3" + url: "https://pub.dev" + source: hosted + version: "2.4.2" + sqflite_platform_interface: + dependency: transitive + description: + name: sqflite_platform_interface + sha256: "8dd4515c7bdcae0a785b0062859336de775e8c65db81ae33dd5445f35be61920" + url: "https://pub.dev" + source: hosted + version: "2.4.0" stack_trace: dependency: transitive description: @@ -1600,10 +1648,10 @@ packages: dependency: "direct main" description: name: uuid - sha256: "1fef9e8e11e2991bb773070d4656b7bd5d850967a2456cfc83cf47925ba79489" + sha256: "9b129329f58692f6e6578329498a8fe9fbe98f090beb764ffbb8ee2eadd01dcd" url: "https://pub.dev" source: hosted - version: "4.5.3" + version: "4.6.0" vector_graphics: dependency: transitive description: diff --git a/mobile/pubspec.yaml b/mobile/pubspec.yaml index 1dc431d675..02e4763216 100644 --- a/mobile/pubspec.yaml +++ b/mobile/pubspec.yaml @@ -52,6 +52,9 @@ dependencies: share_plus: ^13.3.0 local_auth: ^3.0.2 unorm_dart: ^0.3.3 + flutter_cache_manager: ^3.5.0 + file: ^7.0.1 + path: ^1.9.1 dev_dependencies: leak_tracker: ^11.0.2 diff --git a/mobile/test/features/agents/create_agent_page_test.dart b/mobile/test/features/agents/create_agent_page_test.dart index 6b1b41117a..277de48237 100644 --- a/mobile/test/features/agents/create_agent_page_test.dart +++ b/mobile/test/features/agents/create_agent_page_test.dart @@ -1,5 +1,4 @@ -import 'dart:io'; - +import 'package:buzz/shared/relay/media_image.dart'; import 'package:buzz/shared/widgets/frosted_app_bar.dart'; import 'package:buzz/shared/widgets/preset_agent_avatar.dart'; import 'package:buzz/shared/agents/agent_avatar_catalog.dart'; @@ -13,6 +12,7 @@ import 'package:buzz/features/agents/agents_page.dart'; import 'package:buzz/features/agents/create_agent_page.dart'; import 'package:flutter/material.dart'; import 'package:flutter_test/flutter_test.dart'; +import 'package:http/http.dart' as http; import 'package:nostr/nostr.dart' as nostr; import 'agents_harness.dart'; @@ -26,7 +26,12 @@ void main() { final h = await AgentsHarness.create(); final controller = h.controller; final added = nostr.Keys('0' * 63 + '2').public; - await h.open(tester, directory: [existing]); + final posters = _PosterClient(); + await h.open( + tester, + directory: [existing], + overrides: [mediaHttpClientProvider.overrideWithValue(posters)], + ); final list = tester.element(find.byType(AgentsPage)); if (!supportsRemoteAgents(h.community)) { expect(find.byTooltip('Add agent'), findsNothing); @@ -121,17 +126,15 @@ void main() { ) .onPressed!; // A stalled CDN must not prevent entry to the editor. - final stalledClient = _StalledPosterClient(); - debugNetworkImageHttpClientProvider = () => stalledClient; - addTearDown(() => debugNetworkImageHttpClientProvider = null); - await NetworkImage(originalAvatar).evict(); + posters.stalled = true; + MediaImageProvider.debugResetCooldowns(); openAvatar(); openAvatar(); await tester.pump(); await tester.pump(const Duration(milliseconds: 301)); await tester.pumpAndSettle(); - expect(stalledClient.requests, greaterThan(0)); - debugNetworkImageHttpClientProvider = null; + expect(posters.stalledRequests, greaterThan(0)); + posters.release(); expect(find.byType(AgentAvatarEditorPage), findsOneWidget); expect(find.text('Instructions'), findsNothing); await tester.tap(find.byKey(ValueKey('agent-avatar-$targetIndex'))); @@ -311,14 +314,24 @@ void main() { }); } -class _StalledPosterClient implements HttpClient { - int requests = 0; - @override - Future getUrl(Uri url) { - requests++; - return Completer().future; +/// Serves no posters; while [stalled], requests hang until [release]. +class _PosterClient extends http.BaseClient { + bool stalled = false; + final _pending = >[]; + int get stalledRequests => _pending.length; + + void release() { + stalled = false; + for (final pending in _pending) { + pending.complete(http.StreamedResponse(const Stream.empty(), 404)); + } } @override - dynamic noSuchMethod(Invocation invocation) => null; + Future send(http.BaseRequest request) { + if (!stalled) { + return Future.value(http.StreamedResponse(const Stream.empty(), 404)); + } + return (_pending..add(Completer())).last.future; + } } diff --git a/mobile/test/shared/community/community_provider_test.dart b/mobile/test/shared/community/community_provider_test.dart index 4afe19dcd7..f8c2ac0bfe 100644 --- a/mobile/test/shared/community/community_provider_test.dart +++ b/mobile/test/shared/community/community_provider_test.dart @@ -1,5 +1,6 @@ import 'dart:async'; import 'dart:convert'; +import 'dart:io'; import 'package:flutter_test/flutter_test.dart'; import 'package:hooks_riverpod/hooks_riverpod.dart'; @@ -8,6 +9,7 @@ import 'package:buzz/shared/channels/channel_list_snapshot.dart'; import 'package:buzz/shared/theme/theme_provider.dart'; import 'package:shared_preferences/shared_preferences.dart'; import 'package:shared_preferences_platform_interface/shared_preferences_platform_interface.dart'; +import 'package:buzz/shared/relay/media_disk_cache.dart'; import 'package:buzz/shared/relay/relay_provider.dart' show RelayConfig; import 'package:buzz/shared/community/community.dart'; import 'package:buzz/shared/community/community_provider.dart'; @@ -516,13 +518,30 @@ void main() { ]); }); - test('startup and credential replacement discard orphan snapshots', () async { + test('startup and credential replacement discard orphan snapshots and ' + 'image caches', () async { + final mediaRoot = await Directory.systemTemp.createTemp('media_root'); + addTearDown(() => mediaRoot.delete(recursive: true)); + Future seedMedia(Directory directory) async { + await File('${directory.path}/image').create(recursive: true); + return directory; + } + + // Media sweeps run in the background. + Future eventuallyExists(Directory directory) async { + for (var i = 0; i < 50 && await directory.exists(); i++) { + await Future.delayed(const Duration(milliseconds: 10)); + } + return directory.exists(); + } + SharedPreferences.setMockInitialValues({}); final prefs = await SharedPreferences.getInstance(); final cache = ChannelListSnapshotStorage(prefs); const snapshot = ChannelListSnapshot(channels: [], hiddenDmIds: {}); final oldKeys = nostr.Keys.generate(); final newKeys = nostr.Keys.generate(); + final midKeys = nostr.Keys.generate(); final community = Community.create( name: 'Stored', relayUrl: 'wss://cached.example.com:443', @@ -534,21 +553,45 @@ void main() { await cache.write(oldToken, snapshot); await cache.write(restoredToken, snapshot); await prefs.setString('unrelated', 'keep'); + final oldMedia = await seedMedia( + MediaDiskCache.directoryFor(mediaRoot, community.id, oldKeys.public), + ); + final orphanMedia = await seedMedia( + MediaDiskCache.communityDirectory(mediaRoot, 'restored'), + ); container = createContainer( - overrides: [savedPrefsProvider.overrideWithValue(prefs)], + overrides: [ + savedPrefsProvider.overrideWithValue(prefs), + mediaCacheRootProvider.overrideWith((ref) async => mediaRoot), + ], ); await container.read(communityListProvider.future); expect(cache.read(oldKeys.public, community.relayUrl), isNotNull); expect(cache.read('restored', 'https://orphan.example'), isNull); expect(await cache.write(restoredToken, snapshot), isFalse); + expect(await eventuallyExists(orphanMedia), isFalse); + expect(await oldMedia.exists(), isTrue); final notifier = container.read(communityListProvider.notifier); await notifier.addCommunity(community); expect(cache.read(oldKeys.public, community.relayUrl), isNotNull); expect(await cache.write(oldToken, snapshot), isTrue); + expect(await oldMedia.exists(), isTrue); + final midMedia = await seedMedia( + MediaDiskCache.directoryFor(mediaRoot, community.id, midKeys.public), + ); + final newMedia = await seedMedia( + MediaDiskCache.directoryFor(mediaRoot, community.id, newKeys.public), + ); + // Two quick replacements: the first one's sweep must not delete the + // newest account's cache. + await notifier.addCommunity(community.copyWith(nsec: midKeys.nsec)); await notifier.addCommunity(community.copyWith(nsec: newKeys.nsec)); + expect(await eventuallyExists(midMedia), isFalse); expect(cache.read(oldKeys.public, community.relayUrl), isNull); expect(await cache.write(oldToken, snapshot), isFalse); + expect(await eventuallyExists(oldMedia), isFalse); + expect(await newMedia.exists(), isTrue); final newToken = cache.capture(newKeys.public, community.relayUrl); await cache.write(newToken, snapshot); await communityStorage.saveActiveId(community.id); @@ -556,6 +599,14 @@ void main() { expect(await communityStorage.loadAll(), isEmpty); expect(prefs.getKeys(), {'unrelated'}); expect(await cache.write(newToken, snapshot), isFalse); + expect(await eventuallyExists(newMedia), isFalse); + expect( + await MediaDiskCache.communityDirectory( + mediaRoot, + community.id, + ).exists(), + isFalse, + ); // A backup restored without secure community records is swept at startup. container.dispose(); diff --git a/mobile/test/shared/relay/media_disk_cache_test.dart b/mobile/test/shared/relay/media_disk_cache_test.dart new file mode 100644 index 0000000000..9ae3b01b25 --- /dev/null +++ b/mobile/test/shared/relay/media_disk_cache_test.dart @@ -0,0 +1,391 @@ +import 'dart:async'; +import 'dart:convert'; +import 'dart:io'; +import 'dart:typed_data'; + +import 'package:buzz/shared/relay/media_auth.dart'; +import 'package:buzz/shared/relay/media_disk_cache.dart'; +import 'package:buzz/shared/relay/media_image.dart'; +import 'package:flutter/painting.dart'; +import 'package:flutter_test/flutter_test.dart'; +import 'package:http/http.dart' as http; +import 'package:http/testing.dart' as http_testing; +import 'package:path/path.dart' as p; + +// Minimal valid 1x1 transparent PNG. +final _pngBytes = base64Decode( + 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYAAA' + 'AAYAAjCB0C8AAAAASUVORK5CYII=', +); + +const _relayBase = 'https://relay.example.com'; +const _hash = + 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'; +const _relayUrl = '$_relayBase/media/$_hash.png'; +const _thumbnailUrl = '$_relayBase/media/$_hash.thumb.jpg'; +const _hugeUrl = + '$_relayBase/media/' + 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.png'; +const _corruptUrl = + '$_relayBase/media/' + 'cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc.png'; +const _truncatedUrl = + '$_relayBase/media/' + 'dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd.png'; +const _externalUrls = [ + 'https://images.example.org/a.png', + 'https://images.example.org/b.png', + 'https://images.example.org/c.png', +]; + +void main() { + TestWidgetsFlutterBinding.ensureInitialized(); + + late Directory root; + final auth = MediaGetAuthService(baseUrl: _relayBase, nsec: null); + + setUp(() async { + root = await Directory.systemTemp.createTemp('media_disk_cache_test'); + MediaImageProvider.debugResetCooldowns(); + PaintingBinding.instance.imageCache.clear(); + PaintingBinding.instance.imageCache.clearLiveImages(); + }); + + tearDown(() => root.delete(recursive: true)); + + test('images load from disk after a relaunch, and only stale external ' + 'ones are refetched', () async { + final fetched = []; + var online = true; + // While set, external images answer once it completes, with an image + // too large to save. + Completer? replaceExternal; + // While set, external images answer with a body that never decodes. + var malformedExternal = false; + // Trailing bytes after IEND still decode, so this is a 5 MiB+ image. + final hugeBytes = [..._pngBytes, ...Uint8List(MediaDiskCache.maxFileBytes)]; + final client = http_testing.MockClient((request) async { + final url = request.url.toString(); + fetched.add(url); + if (!online) throw const SocketException('offline'); + if (url == _hugeUrl) return http.Response.bytes(hugeBytes, 200); + if (malformedExternal && _externalUrls.contains(url)) { + return http.Response('not an image', 200); + } + if (replaceExternal != null && _externalUrls.contains(url)) { + await replaceExternal.future; + return http.Response.bytes(hugeBytes, 200); + } + if (request.url.toString() == _corruptUrl) { + return http.Response('not an image', 200); + } + // A valid header, so a codec opens, but the frame fails to decode. + if (request.url.toString() == _truncatedUrl) { + return http.Response.bytes(_pngBytes.sublist(0, 41), 200); + } + return http.Response.bytes(_pngBytes, 200); + }); + final directory = MediaDiskCache.directoryFor( + root, + 'community-1', + 'signer-1', + ); + var now = DateTime.now(); + + Future launch( + Future Function(MediaDiskCache cache) body, + ) async { + // The caches root resolves after the first images start loading. + final cache = MediaDiskCache.open( + Future.delayed( + const Duration(milliseconds: 20), + () => directory, + ), + now: () => now, + ); + try { + await body(cache); + } finally { + await cache.dispose(); + // A relaunch starts with nothing decoded in memory. + PaintingBinding.instance.imageCache.clear(); + PaintingBinding.instance.imageCache.clearLiveImages(); + } + } + + Future show(MediaDiskCache cache, String url) => _waitError( + MediaImageProvider( + url: url, + auth: auth, + client: client, + diskCache: cache, + ).resolve(ImageConfiguration.empty), + ); + + const saved = [_relayUrl, _thumbnailUrl, ..._externalUrls]; + const urls = [...saved, _hugeUrl]; + + // First launch downloads everything and saves what fits. + await launch((cache) async { + for (final url in urls) { + expect(await show(cache, url), isNull, reason: url); + } + expect(await show(cache, _corruptUrl), isNotNull); + expect(await show(cache, _truncatedUrl), isNotNull); + await _settle(); + }); + expect(fetched, [...urls, _corruptUrl, _truncatedUrl]); + + // Offline relaunch: the original and its thumbnail are separate entries, + // both shown from disk. The oversized image and the bodies that failed + // to decode were never saved. + fetched.clear(); + online = false; + await launch((cache) async { + for (final url in saved) { + expect(await show(cache, url), isNull, reason: url); + } + expect(await show(cache, _hugeUrl), isNotNull); + expect(await show(cache, _corruptUrl), isNotNull); + expect(await show(cache, _truncatedUrl), isNotNull); + }); + expect(fetched, [_hugeUrl, _corruptUrl, _truncatedUrl]); + + // A week on, external images show from disk at once and refresh in the + // background, two at a time. Relay media never goes stale. + fetched.clear(); + online = true; + now = now.add(const Duration(days: 8)); + final replacement = replaceExternal = Completer(); + await launch((cache) async { + for (final url in saved) { + expect(await show(cache, url), isNull, reason: url); + } + await _settle(); + expect(fetched, _externalUrls.take(2)); + // The third waits for a slot. The replacements are too large to save, + // so the old copies go rather than showing in their place. + replacement.complete(); + await _until(() async { + for (final url in _externalUrls) { + if (await cache.read(url) != null) return false; + } + return true; + }); + expect(fetched, _externalUrls); + }); + replaceExternal = null; + + fetched.clear(); + online = false; + await launch((cache) async { + for (final url in _externalUrls) { + expect(await show(cache, url), isNotNull, reason: url); + } + }); + expect(fetched, _externalUrls); + + // A saved copy that no longer decodes is dropped and fetched again. + fetched.clear(); + online = true; + await launch((cache) async { + await cache.write( + _relayUrl, + Uint8List.fromList(utf8.encode('not an image')), + immutable: true, + ); + expect(await show(cache, _relayUrl), isNotNull); + await _settle(); + expect(await show(cache, _relayUrl), isNull); + await cache.write(_externalUrls.first, _pngBytes, immutable: false); + }); + expect(fetched, [_relayUrl]); + + // A refresh whose body never decodes backs off rather than refetching + // on every load. + fetched.clear(); + malformedExternal = true; + // Forget the failures of the offline launch. + MediaImageProvider.debugResetCooldowns(); + now = now.add(const Duration(days: 8)); + await launch((cache) async { + for (var i = 0; i < 2; i++) { + expect(await show(cache, _externalUrls.first), isNull); + await _settle(); + PaintingBinding.instance.imageCache.clear(); + } + }); + expect(fetched, [_externalUrls.first]); + + // A stale saved copy that fails to decode is dropped without starting a + // refresh, which could otherwise save a new copy just before the drop. + fetched.clear(); + malformedExternal = false; + MediaImageProvider.debugResetCooldowns(); + await launch((cache) async { + await cache.write( + _externalUrls[1], + Uint8List.fromList(utf8.encode('not an image')), + immutable: false, + ); + }); + now = now.add(const Duration(days: 8)); + await launch((cache) async { + expect(await show(cache, _externalUrls[1]), isNotNull); + await _settle(); + expect(fetched, isEmpty); + expect(await show(cache, _externalUrls[1]), isNull); + }); + expect(fetched, [_externalUrls[1]]); + + // Refreshes queued when the community or account retires never run. + fetched.clear(); + malformedExternal = false; + MediaImageProvider.debugResetCooldowns(); + await launch((cache) async { + for (final url in _externalUrls) { + await cache.write(url, _pngBytes, immutable: false); + } + }); + now = now.add(const Duration(days: 8)); + final held = replaceExternal = Completer(); + await launch((cache) async { + for (final url in _externalUrls) { + expect(await show(cache, url), isNull, reason: url); + } + await _settle(); + }); + held.complete(); + await _settle(); + replaceExternal = null; + expect(fetched, _externalUrls.take(2)); + + // A replaced account's directory goes, including writes that land after + // its delete; removing the community deletes its directory. + expect(await directory.exists(), isTrue); + await MediaDiskCache.removeOrphans(root, () => {'community-1': 'signer-2'}); + expect(await directory.exists(), isFalse); + await directory.create(recursive: true); + await MediaDiskCache.removeOrphans(root, () => {'community-1': 'signer-1'}); + expect(await directory.exists(), isTrue); + await MediaDiskCache.removeOrphans(root, () => {'community-2': 'signer-1'}); + expect( + await MediaDiskCache.communityDirectory(root, 'community-1').exists(), + isFalse, + ); + }); + + test( + 'evicts least recently used files past the byte or file budget', + () async { + // (byte budget, file budget, evicted): three 4-byte files make 12 bytes. + const cases = [ + (10, 100, ['old']), + (100, 1, ['old', 'middle']), + ]; + for (final (bytes, files, evicted) in cases) { + final directory = MediaDiskCache.directoryFor( + root, + '$bytes-$files', + 's', + ); + final cache = MediaDiskCache.open( + directory, + byteBudget: bytes, + fileBudget: files, + // Like the app's 10 s: long enough for the writes to finish. + evictionDelay: const Duration(milliseconds: 200), + ); + for (final name in ['old', 'middle', 'newest']) { + await cache.write(name, Uint8List(4), immutable: true); + // Distinct last-used times. + await Future.delayed(const Duration(milliseconds: 5)); + } + // Saving a new URL schedules eviction, as in the app. + await Future.delayed(const Duration(milliseconds: 400)); + for (final name in ['old', 'middle', 'newest']) { + expect( + await cache.read(name), + evicted.contains(name) ? isNull : isNotNull, + reason: '$bytes bytes, $files files: $name', + ); + } + // Evicted files are gone from disk too; the index (and a save of it + // in progress) remains. + final names = directory + .listSync() + .map((entry) => p.basename(entry.path)) + .where((name) => !name.startsWith('index.json')); + expect(names, hasLength(3 - evicted.length), reason: '$bytes $files'); + await cache.dispose(); + } + + Future open(String name) async => MediaDiskCache.open( + MediaDiskCache.directoryFor(root, name, 's'), + byteBudget: 10, + evictionDelay: const Duration(milliseconds: 200), + ); + const evictionRan = Duration(milliseconds: 400); + + // Reading a file from disk marks it used, even when the cache already + // holds it in memory. + final reads = await open('reads'); + await reads.write('old', Uint8List(4), immutable: true); + await Future.delayed(const Duration(milliseconds: 5)); + await reads.write('middle', Uint8List(4), immutable: true); + await Future.delayed(const Duration(milliseconds: 5)); + expect(await reads.read('old'), isNotNull); + await reads.write('newest', Uint8List(4), immutable: true); + await Future.delayed(evictionRan); + expect(await reads.read('middle'), isNull); + expect(await reads.read('old'), isNotNull); + await reads.dispose(); + + // Replacing a file's bytes after eviction last ran, as a background + // refresh does, evicts again once the cache is over budget. + final refresh = await open('refresh'); + await refresh.write('other', Uint8List(4), immutable: true); + await Future.delayed(const Duration(milliseconds: 5)); + await refresh.write('refreshed', Uint8List(4), immutable: false); + // Eviction ran with the cache in budget. A read would schedule it again. + await Future.delayed(evictionRan); + await refresh.write('refreshed', Uint8List(8), immutable: false); + await Future.delayed(evictionRan); + expect(await refresh.read('other'), isNull); + expect((await refresh.read('refreshed'))?.bytes, hasLength(8)); + await refresh.dispose(); + }, + ); +} + +/// Waits for background work that a fixed delay cannot bound on a slow +/// machine, such as several refreshes running one after another. +Future _until(Future Function() done) async { + final deadline = DateTime.now().add(const Duration(seconds: 10)); + while (!await done()) { + if (DateTime.now().isAfter(deadline)) fail('timed out'); + await Future.delayed(const Duration(milliseconds: 20)); + } +} + +/// Lets fire-and-forget disk writes and refreshes finish. +Future _settle() => + Future.delayed(const Duration(milliseconds: 200)); + +Future _waitError(ImageStream stream) { + final done = Completer(); + late final ImageStreamListener listener; + listener = ImageStreamListener( + (image, sync) { + image.dispose(); + stream.removeListener(listener); + if (!done.isCompleted) done.complete(null); + }, + onError: (error, stack) { + stream.removeListener(listener); + if (!done.isCompleted) done.complete(error); + }, + ); + stream.addListener(listener); + return done.future; +} diff --git a/mobile/test/shared/widgets/avatar_image_test.dart b/mobile/test/shared/widgets/avatar_image_test.dart index 2c885c532e..543cc3c482 100644 --- a/mobile/test/shared/widgets/avatar_image_test.dart +++ b/mobile/test/shared/widgets/avatar_image_test.dart @@ -112,7 +112,7 @@ void main() { expect(shown, ['$relay/media/$hash.thumb.webp', original]); }); - testWidgets('missing thumbnails fall back to the JPEG, then the original', ( + testWidgets('missing thumbnails fall back to the JPEG, never the original', ( tester, ) async { const relay = 'https://relay.example'; @@ -161,16 +161,17 @@ void main() { String thumb(String url, String ext) => url.replaceFirst('.jpg', '.thumb.$ext'); - // A 404 tries the next source; any other failure shows the fallback. + // A 404 tries the next thumbnail; any other failure, or running out of + // thumbnails, shows the fallback. expect( requested, containsAll([ thumb(missing, 'webp'), thumb(missing, 'jpg'), - missing, thumb(broken, 'webp'), ]), ); + expect(find.text('M'), findsOneWidget); expect(requested, isNot(contains(thumb(broken, 'jpg')))); expect(requested, isNot(contains(broken))); expect(find.text('B'), findsOneWidget); @@ -293,14 +294,8 @@ void main() { } } - String shownUrl(WidgetTester tester) { - final image = tester.widget(find.byType(Image)); - final provider = (image.image as ResizeImage).imageProvider; - return (provider as MediaImageProvider).url; - } - testWidgets( - 'an avatar that left before its thumbnail 404 later shows the original', + 'an avatar that left before its thumbnail 404 later shows its fallback', (tester) async { final original = '$relay/media/${'12' * 32}.png'; final thumbnail = '$relay/media/${'12' * 32}.thumb.jpg'; @@ -325,8 +320,8 @@ void main() { slow404.complete(http.Response('', 404)); await show(tester, const SizedBox()); - // Back at another size, it loads and renders the original rather - // than the initials. + // Back at another size, it shows the initials without requesting the + // thumbnail again. await show( tester, AvatarImage( @@ -335,9 +330,8 @@ void main() { fallback: const Text('M'), ), ); - expect(requested.where((url) => url == original), hasLength(1)); - expect(shownUrl(tester), original); - expect(find.text('M'), findsNothing); + expect(requested, [thumbnail.replaceFirst('.jpg', '.webp'), thumbnail]); + expect(find.text('M'), findsOneWidget); expect(MediaImageProvider.debugCooldownUrls, isEmpty); // A profile URL that is itself thumbnail-shaped, on another host or @@ -404,7 +398,7 @@ void main() { // The direct URL is cooling down, and the fallback recorded the // missing thumbnail, so the original avatar shown again at another - // size loads the original instead of showing initials. + // size shows initials. expect(MediaImageProvider.debugCooldownUrls, [thumbnail]); await show( tester, @@ -414,13 +408,12 @@ void main() { fallback: const Text('F'), ), ); - expect(shownUrl(tester), original); - expect(find.text('F'), findsNothing); + expect(find.text('F'), findsOneWidget); }, ); testWidgets( - 'a fallback after a direct thumbnail 404 still loads the original', + 'a fallback after a direct thumbnail 404 still records it missing', (tester) async { final original = '$relay/media/${'9b' * 32}.png'; final thumbnail = '$relay/media/${'9b' * 32}.thumb.jpg'; @@ -444,7 +437,7 @@ void main() { // During that cooldown, an avatar for the original falls back to the // same thumbnail. Its own request records the thumbnail as missing, - // so the avatar ends up on the original rather than initials. + // so the avatar shows initials. await show( tester, AvatarImage( @@ -454,8 +447,7 @@ void main() { ), ); expect(requested.where((url) => url == thumbnail), hasLength(2)); - expect(requested, contains(original)); - expect(find.text('F'), findsNothing); + expect(find.text('F'), findsOneWidget); }, ); }); diff --git a/mobile/test/shared/widgets/native_avatar_image_test.dart b/mobile/test/shared/widgets/native_avatar_image_test.dart index efd7e88fc9..84d4ab2e8f 100644 --- a/mobile/test/shared/widgets/native_avatar_image_test.dart +++ b/mobile/test/shared/widgets/native_avatar_image_test.dart @@ -2,9 +2,13 @@ import 'dart:ui' as ui; import 'dart:convert'; import 'package:buzz/shared/emoji/emoji_avatar.dart'; +import 'package:buzz/shared/relay/media_auth.dart'; +import 'package:buzz/shared/relay/media_image.dart'; import 'package:buzz/shared/widgets/avatar_image.dart'; import 'package:flutter/material.dart'; import 'package:flutter_test/flutter_test.dart'; +import 'package:http/http.dart' as http; +import 'package:http/testing.dart' as http_testing; void main() { testWidgets('bounds remote avatar decoding before painting the native PNG', ( @@ -26,7 +30,7 @@ void main() { initial: 'A', background: Colors.white, foreground: Colors.black, - networkImage: (_) => provider, + networkImages: (_) => [provider], ); expect(png, isNotNull); expect(provider.decodedWidth, lessThanOrEqualTo(72)); @@ -35,6 +39,73 @@ void main() { }); }); + testWidgets('native relay avatars load only thumbnails, like MediaImage', ( + tester, + ) async { + await tester.runAsync(() async { + MediaImage.debugResetMissingThumbnails(); + final recorder = ui.PictureRecorder(); + Canvas(recorder).drawColor(Colors.blue, BlendMode.src); + final picture = recorder.endRecording(); + final image = await picture.toImage(8, 8); + final blue = (await image.toByteData( + format: ui.ImageByteFormat.png, + ))!.buffer.asUint8List(); + image.dispose(); + picture.dispose(); + + const relay = 'https://relay.example'; + final jpegOnly = '$relay/media/${'a1' * 32}.png'; + final missing = '$relay/media/${'b2' * 32}.png'; + final broken = '$relay/media/${'c3' * 32}.png'; + String thumb(String url, String ext) => + url.replaceFirst('.png', '.thumb.$ext'); + final requested = []; + final auth = MediaGetAuthService(baseUrl: relay, nsec: null); + final client = http_testing.MockClient((request) async { + final url = request.url.toString(); + requested.add(url); + if (url == thumb(jpegOnly, 'jpg')) { + return http.Response.bytes(blue, 200); + } + if (url == thumb(broken, 'webp')) return http.Response('', 500); + return http.Response('', 404); + }); + Future paintsImage(String url) async { + final png = await nativeAvatarImage( + url: url, + initial: 'A', + background: Colors.black, + foreground: Colors.white, + networkImages: (url) => + smallMediaImageProviders(url: url, auth: auth, client: client), + ); + final codec = await ui.instantiateImageCodec(png!); + final frame = await codec.getNextFrame(); + final pixels = (await frame.image.toByteData())!; + frame.image.dispose(); + codec.dispose(); + // Left of center, clear of the initial: blue artwork or black. + return pixels.getUint8((36 * 72 + 10) * 4 + 2) > 200; + } + + // A 404 moves to the JPEG thumbnail; the original is never fetched. + expect(await paintsImage(jpegOnly), isTrue); + expect(requested, [thumb(jpegOnly, 'webp'), thumb(jpegOnly, 'jpg')]); + // With no thumbnail at all, the initials show. + requested.clear(); + expect(await paintsImage(missing), isFalse); + expect(requested, [thumb(missing, 'webp'), thumb(missing, 'jpg')]); + // Known-missing thumbnails are skipped on the next rasterization. + requested.clear(); + expect(await paintsImage(missing), isFalse); + expect(requested, isEmpty); + // Any other failure stops at the initials. + expect(await paintsImage(broken), isFalse); + expect(requested, [thumb(broken, 'webp')]); + }); + }); + for (final size in [const Size(1440, 720), const Size(720, 1440)]) { testWidgets('bounds inline raster decode at $size before native painting', ( tester, @@ -66,7 +137,7 @@ void main() { initial: 'A', background: Colors.white, foreground: Colors.black, - networkImage: (_) => throw StateError('Unexpected network image'), + networkImages: (_) => throw StateError('Unexpected network image'), ); expect(png, isNotNull); // The first image is the codec frame, before the final 72px canvas. @@ -92,7 +163,7 @@ void main() { initial: 'K', background: Colors.black, foreground: Colors.white, - networkImage: (_) => throw StateError('Unexpected network image'), + networkImages: (_) => throw StateError('Unexpected network image'), ); expect(png, isNotNull); final codec = await ui.instantiateImageCodec(png!);