From ef57fec4d26cee1939a329acd7c8c2b079b56576 Mon Sep 17 00:00:00 2001 From: klopez4212 Date: Wed, 11 Nov 2026 21:22:25 +0000 Subject: [PATCH] Fix gallery video selection, thumbnails, and swipe dismissal (#9356) ## Summary Photos currently filters out videos, video messages without poster metadata stay grey until playback, and the native gallery rejects downward dismissal on video pages. - Include photos and videos in the native inline/popover pickers, system picker, and recent-media grid. Mixed selections retain their order and videos use the video upload path. Remove the redundant Video menu row on iOS and Android, whose Photos picker accepts both media types; reduce their menu height to four rows. Native pickers request the current asset representation to avoid compatibility transcoding and export up to three files concurrently, preserving order and cleaning up cancelled/failed batches. - Show local stills for queued videos. For older messages without posters, serialize thumbnail generation for the four rendered stack cards, bound/cancel downloads, remove temporary videos, and save stills in the existing account/community cache. Existing poster metadata is used directly. The native viewer uses the poster while loading instead of a play-button snapshot. - Allow downward dismissal on native video pages while preserving horizontal paging/scrubbing, zoomed-photo panning, and the accessible Close button. Fire selection-click haptics as a swipe begins rather than after the page settles, with no duplicate on completion or cancellation. Page-indicator changes retain immediate feedback. Replace visible video-loading text with the existing white arc-spinner treatment, implemented natively with reduced-motion and lifecycle handling. Completed swipes release the previous video player even after UIKit removes it from the active children. Legacy videos need one initial download (up to the existing 100 MB limit) to generate their cached preview. This does not start playback. ### Related issue Follow-up to merged #9259 and #9314. Searched open and closed gallery/video issues and PRs; no duplicate fix found. ### Testing - Full `just mobile-check` and `just mobile-test` passed at `12e6ed6ae3`: 4,688 Flutter tests (71 skipped), 4 unconfigured-push tests, and 27 crypto-package tests. File-size checks passed. Regressions cover mixed image/video upload routing, queued still previews, the rendered-card poster window, authenticated/cancelled downloads, caching/cleanup, and native poster loading. - Native gallery simulator regression fixture passed: photo/video dismissal guards, real MP4 decoding, paging, selected-only playback loading, and teardown. - Swift type-check of the native picker and its dependencies passed, with existing deprecation/capture warnings. - `just ci` passed its repository-wide lint/type checks. The unrelated Rust test rebuild was stopped after consuming about 13 GB to preserve remaining disk space; only that run's generated caches were removed. Release build at `fb09f9cb8` installed in place and launched on the physical iPhone Air as **Buzz Photo Stack**, with `BUZZ_NATIVE_PHOTO_ZOOM=true`. Existing app updated in place with the same bundle and Keychain identity; no uninstall or reset. Owner tested the physical-device preview and approved readiness for review. No before/after capture was produced. The simulator evidence above is an automated native fixture, not a full Buzz device journey. Native preparation follow-up: `scripts/test-mobile-picker-preparation.sh` passed on an iPhone Air simulator, covering bounded overlap, out-of-order completion, cancellation, and failure cleanup. The updated physical Air preview launched successfully with unchanged preferences metadata. Cloud-only assets may still require preparation. Loader/haptics follow-up native fixture passed: real video readiness hides the loader, error/retry and detach stop/restart it correctly, swipe start triggers one haptic before selection changes, and completion/cancellation/duplicate-start callbacks add none. Updated release installed and launched on the physical Air; owner approved the final preview for review. Review: focused code review and native regression checks completed; physical Air preview tested by the owner, who approved readiness. P2 follow-up: video presentation uses an already decoded poster or the source snapshot immediately, then forwards a pending poster asynchronously. Late poster updates are limited to the selected native page and do not restart playback. Added pending-provider presentation coverage and native late/stale-poster checks. This follow-up has not been installed on the physical preview yet. --------- Signed-off-by: kenny lopez Co-authored-by: Terp <5856f6900e8e0a4e0ea1f4f8d2ee33e4753d2119bc122f94ecafc660b46c0511@buzz.block.builderlab.xyz> --- .../NativePhotoZoomTests.swift | 77 ++++++++++- .../PickerFilePreparationTests.swift | 65 ++++++++++ mobile/ios/Runner/InlinePhotoPicker.swift | 57 ++++++++- .../ios/Runner/NativeAttachmentPopover.swift | 13 +- .../NativeAttachmentPopoverCoordinator.swift | 2 +- .../Runner/NativeAttachmentPopoverMenu.swift | 9 -- .../Runner/NativePhotoZoomCoordinator.swift | 82 ++++++++++-- mobile/ios/RunnerTests/RunnerTests.swift | 2 +- mobile/lib/features/channels/compose_bar.dart | 2 + .../channels/compose_bar/attachments.dart | 40 ++++-- .../compose_bar/ios_photo_picker.dart | 6 +- .../compose_bar/photo_gallery_picker.dart | 17 ++- .../channels/compose_bar/video_thumbnail.dart | 28 ++++ .../features/channels/message_content.dart | 1 + .../channels/message_content/media_stack.dart | 36 +++++- .../message_content/video_preview.dart | 36 +++++- .../features/channels/message_gallery.dart | 2 +- .../channels/native_photo_zoom_preview.dart | 45 +++++-- .../features/channels/native_video_files.dart | 6 + .../lib/features/channels/photo_library.dart | 25 +++- .../lib/features/channels/video_poster.dart | 60 +++++++++ mobile/lib/shared/relay/media_upload.dart | 9 ++ .../relay/media_upload/platform_bindings.dart | 3 +- .../features/channels/compose_bar_test.dart | 53 ++------ .../compose_bar_test/mixed_gallery_tests.dart | 83 ++++++++++++ .../channels/message_content_test.dart | 1 + .../channels/message_stack_cases.dart | 31 ++++- .../native_photo_zoom_preview_test.dart | 55 +++++++- .../features/channels/video_poster_test.dart | 120 ++++++++++++++++++ scripts/test-mobile-picker-preparation.sh | 18 +++ 30 files changed, 854 insertions(+), 130 deletions(-) create mode 100644 mobile/ios/NativeControlTests/PickerFilePreparationTests.swift create mode 100644 mobile/lib/features/channels/compose_bar/video_thumbnail.dart create mode 100644 mobile/lib/features/channels/video_poster.dart create mode 100644 mobile/test/features/channels/compose_bar_test/mixed_gallery_tests.dart create mode 100644 mobile/test/features/channels/video_poster_test.dart create mode 100644 scripts/test-mobile-picker-preparation.sh diff --git a/mobile/ios/NativeControlTests/NativePhotoZoomTests.swift b/mobile/ios/NativeControlTests/NativePhotoZoomTests.swift index f1d78ecaa7..074399ccdf 100644 --- a/mobile/ios/NativeControlTests/NativePhotoZoomTests.swift +++ b/mobile/ios/NativeControlTests/NativePhotoZoomTests.swift @@ -21,6 +21,14 @@ struct NativePhotoZoomTests { viewer.view.layoutIfNeeded() precondition(!requests.isEmpty && requests.allSatisfy { $0.1 <= 2 }, "Initial requests must stay adjacent") let pager = viewer.children.compactMap { $0 as? UIPageViewController }.first! + let location = CGPoint(x: 195, y: 400) + precondition(viewer.shouldBeginDismissal(location: location, velocity: CGVector(dx: 0, dy: 300)), "Resting photos dismiss downward") + precondition(!viewer.shouldBeginDismissal(location: location, velocity: CGVector(dx: 300, dy: 10)), "Horizontal swipes must page") + let scroll = descendants(pager.viewControllers!.first!.view).compactMap { $0 as? UIScrollView }.first! + scroll.zoomScale = 2 + precondition(!viewer.shouldBeginDismissal(location: location, velocity: CGVector(dx: 0, dy: 300)), "Zoomed photos must pan instead of dismiss") + scroll.zoomScale = 1 + let indicator = descendants(viewer.view).compactMap { $0 as? UIPageControl }.first! let stale = requests requests.removeAll() @@ -64,6 +72,7 @@ struct NativePhotoZoomTests { precondition(closedPhoto.image === closedImage, "Closed viewers must ignore continuing frames") print("PASS: native photo window, stale responses, eviction and teardown") videoJourney(data: data) + loaderJourney() } } @@ -73,6 +82,8 @@ struct NativePhotoZoomTests { let viewer = NativePhotoZoomViewController(image: UIImage(data: data)!, labels: ["Clip", "Photo", "Second clip"], initialIndex: 0, canReply: true, canMore: true, canDelete: false, videoIndices: [0, 2]) + let feedback = GalleryFeedbackSpy() + viewer.selectionFeedback = feedback var requested: [Int] = [] var lateVideo: FlutterResult? viewer.request = { method, index, _, completion in @@ -87,30 +98,88 @@ struct NativePhotoZoomTests { viewer.viewDidAppear(false) let pager = viewer.children.compactMap { $0 as? UIPageViewController }.first! let indicator = descendants(viewer.view).compactMap { $0 as? UIPageControl }.first! + precondition(viewer.shouldBeginDismissal(location: CGPoint(x: 195, y: 400), velocity: CGVector(dx: 0, dy: 300)), "Videos must dismiss downward too") + precondition(!viewer.shouldBeginDismissal(location: CGPoint(x: 195, y: 400), velocity: CGVector(dx: 300, dy: 0)), "Video horizontal scrubbing must not dismiss") let first = pager.viewControllers!.first! first.loadViewIfNeeded() + let posterView = descendants(first.view).compactMap { $0 as? UIImageView }.first! + let oldPoster = posterView.image + viewer.updateFrame(data, at: 0) + precondition(posterView.image !== oldPoster, "A late poster updates the selected video") + let selectedPoster = posterView.image + viewer.updateFrame(data, at: 2) + precondition(posterView.image === selectedPoster, "A stale poster cannot replace the selected video") let playerController = first.children.compactMap { $0 as? AVPlayerViewController }.first! let player = playerController.player! let item = player.currentItem! + let firstSpinner = descendants(first.view).compactMap { $0 as? NativeGalleryLoadingIndicator }.first! let deadline = Date().addingTimeInterval(5) - while item.status == .unknown && Date() < deadline { + while (item.status == .unknown || !firstSpinner.isHidden) && Date() < deadline { RunLoop.current.run(until: Date().addingTimeInterval(0.01)) } precondition(item.status == .readyToPlay, "Real MP4 fixture must become playable") + precondition(firstSpinner.isHidden, "Playable video removes the loader") precondition(requested == [0], "Only the selected video may download") + precondition(feedback.clicks == 0, "Opening must not trigger selection haptics") + viewer.pageViewController(pager, willTransitionTo: [first]) + precondition(feedback.clicks == 1, "Swipe feedback fires before the page changes") + viewer.pageViewController(pager, willTransitionTo: [first]) + precondition(feedback.clicks == 1, "Repeated start callbacks do not duplicate swipe feedback") + viewer.pageViewController(pager, didFinishAnimating: true, previousViewControllers: [first], transitionCompleted: false) + precondition(feedback.clicks == 1, "Cancellation does not add a second haptic") indicator.currentPage = 2; sendValueChanged(indicator) precondition(player.currentItem == nil && playerController.player == nil, "Paging releases the old player before Dart deletes its file") precondition(requested == [0, 2], "Paging requests the selected video") + precondition(feedback.clicks == 2, "Changing to a video triggers one selection click") + let loadingPage = pager.viewControllers!.first! + loadingPage.loadViewIfNeeded() + let spinner = descendants(loadingPage.view).compactMap { $0 as? NativeGalleryLoadingIndicator }.first! + precondition(!spinner.isHidden && spinner.accessibilityLabel == "Loading video", "Show the shared arc treatment while loading") + precondition(!descendants(loadingPage.view).compactMap { $0 as? UILabel }.contains { $0.text == "Loading video…" }, "No visible loading copy") + lateVideo?(nil) + precondition(spinner.isHidden, "Failure stops the loader") + let retry = descendants(loadingPage.view).compactMap { $0 as? UIButton }.first { $0.currentTitle == "Retry video" }! + precondition(!retry.isHidden, "Failure retains retry") + for target in retry.allTargets { + for action in retry.actions(forTarget: target, forControlEvent: .touchUpInside) ?? [] { + _ = (target as AnyObject).perform(NSSelectorFromString(action), with: retry) + } + } + precondition(!spinner.isHidden && requested == [0, 2, 2], "Retry restarts the loader") indicator.currentPage = 0; sendValueChanged(indicator) lateVideo?(path) let current = pager.viewControllers!.first! current.loadViewIfNeeded() let active = current.children.compactMap { $0 as? AVPlayerViewController }.first! precondition(active.player != nil, "Returning to a video restores its player") + precondition(spinner.isHidden, "Leaving a loading video stops its loader") + let next = viewer.pageViewController(pager, viewControllerAfter: current)! + viewer.pageViewController(pager, willTransitionTo: [next]) + precondition(feedback.clicks == 4, "Next swipe fires before the new photo is selected") + pager.setViewControllers([next], direction: .forward, animated: false) + viewer.pageViewController(pager, didFinishAnimating: true, previousViewControllers: [current], transitionCompleted: true) + precondition(feedback.clicks == 4, "Completed swipe does not add a delayed haptic") + viewer.pageViewController(pager, didFinishAnimating: true, previousViewControllers: [current], transitionCompleted: true) + precondition(feedback.clicks == 4, "Repeated callbacks for the same page do not duplicate haptics") viewer.viewDidDisappear(false) precondition(active.player == nil, "Closing releases the player before returning to Flutter") print("PASS: native video decoding, selected-only loading, paging and teardown") } + @MainActor static func loaderJourney() { + let window = UIWindow(frame: CGRect(x: 0, y: 0, width: 100, height: 100)) + let root = UIViewController(); window.rootViewController = root + let loader = NativeGalleryLoadingIndicator(frame: CGRect(x: 28, y: 28, width: 44, height: 44)) + root.view.addSubview(loader); window.isHidden = false + loader.setLoading(true); loader.layoutIfNeeded() + if !UIAccessibility.isReduceMotionEnabled { + precondition(loader.layer.animation(forKey: "loading")?.duration == 0.5, "Match the shared spinner cadence") + } + loader.setLoading(false) + precondition(loader.layer.animation(forKey: "loading") == nil && loader.isHidden) + loader.setLoading(true); loader.removeFromSuperview() + precondition(loader.layer.animation(forKey: "loading") == nil, "Detached loaders must stop animating") + window.isHidden = true + } @MainActor static func sendValueChanged(_ control: UIControl) { // A command-line UIKit fixture has no UIApplication action dispatcher. for target in control.allTargets { @@ -123,3 +192,9 @@ struct NativePhotoZoomTests { [view] + view.subviews.flatMap { descendants($0) } } } + +@MainActor +private final class GalleryFeedbackSpy: UISelectionFeedbackGenerator { + var clicks = 0 + override func selectionChanged() { clicks += 1 } +} diff --git a/mobile/ios/NativeControlTests/PickerFilePreparationTests.swift b/mobile/ios/NativeControlTests/PickerFilePreparationTests.swift new file mode 100644 index 0000000000..d499f808cc --- /dev/null +++ b/mobile/ios/NativeControlTests/PickerFilePreparationTests.swift @@ -0,0 +1,65 @@ +import Foundation + +@MainActor +private final class ExportGate { + var pending: [Int: CheckedContinuation] = [:] + var started: [Int] = [] + let directory = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString) + + init() throws { try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true) } + + func export(_ index: Int) async throws -> String { + started.append(index) + return try await withCheckedThrowingContinuation { pending[index] = $0 } + } + + func waitFor(_ count: Int) async { + while started.count < count { await Task.yield() } + } + + func finish(_ index: Int) throws { + let file = directory.appendingPathComponent("\(index).mov") + try Data([UInt8(index)]).write(to: file) + pending.removeValue(forKey: index)!.resume(returning: file.path) + } +} + +@main +struct PickerFilePreparationTests { + @MainActor static func main() async throws { + let gate = try ExportGate() + defer { try? FileManager.default.removeItem(at: gate.directory) } + let task = Task { try await PickerFilePreparation.prepare(count: 5, export: gate.export) } + await gate.waitFor(3) + precondition(gate.pending.count == 3, "Exports overlap but stay bounded") + try gate.finish(2) + await gate.waitFor(4) + precondition(gate.pending.count == 3) + try gate.finish(0) + await gate.waitFor(5) + for index in [4, 3, 1] { try gate.finish(index) } + let paths = try await task.value + precondition(paths.map { URL(fileURLWithPath: $0).lastPathComponent } == (0..<5).map { "\($0).mov" }, "Completion order must not reorder the selection") + + for cancel in [false, true] { + let gate = try ExportGate() + defer { try? FileManager.default.removeItem(at: gate.directory) } + let task = Task { try await PickerFilePreparation.prepare(count: cancel ? 5 : 3, export: gate.export) } + await gate.waitFor(3) + if cancel { task.cancel() } + else { gate.pending.removeValue(forKey: 0)!.resume(throwing: CocoaError(.fileReadUnknown)) } + if cancel { try gate.finish(0) } + // Providers may return files even after cancellation or another export fails. + try gate.finish(1) + try gate.finish(2) + do { + _ = try await task.value + preconditionFailure("Failed/cancelled batches must not become attachments") + } catch {} + precondition(gate.started.count == 3, "Stop scheduling after failure or cancellation") + let remaining = try FileManager.default.contentsOfDirectory(atPath: gate.directory.path) + precondition(remaining.isEmpty, "Remove every completed temporary file") + } + print("Picker export batching: order, concurrency, failure and cancellation passed") + } +} diff --git a/mobile/ios/Runner/InlinePhotoPicker.swift b/mobile/ios/Runner/InlinePhotoPicker.swift index 4126734779..fb3001f34d 100644 --- a/mobile/ios/Runner/InlinePhotoPicker.swift +++ b/mobile/ios/Runner/InlinePhotoPicker.swift @@ -90,10 +90,10 @@ final class InlinePhotoPickerPlatformView: NSObject, FlutterPlatformView { @available(iOS 17.0, *) private func installPicker() { var configuration = PHPickerConfiguration(photoLibrary: .shared()) - configuration.filter = .images + configuration.filter = .any(of: [.images, .videos]) configuration.selectionLimit = 0 configuration.selection = .continuousAndOrdered - configuration.preferredAssetRepresentationMode = .compatible + configuration.preferredAssetRepresentationMode = .current configuration.disabledCapabilities = [ .search, .stagingArea, @@ -129,7 +129,7 @@ final class InlinePhotoPickerPlatformView: NSObject, FlutterPlatformView { guard let typeIdentifier = provider.registeredTypeIdentifiers.first(where: { guard let type = UTType($0) else { return false } - return type.conforms(to: .image) + return type.conforms(to: .image) || type.conforms(to: .movie) }) else { throw InlinePhotoPickerError.unsupportedImage @@ -200,9 +200,8 @@ extension InlinePhotoPickerPlatformView: PHPickerViewControllerDelegate { guard let self else { return } var paths: [String] = [] do { - for result in results { - try Task.checkCancellation() - paths.append(try await self.exportPickerResult(result)) + paths = try await PickerFilePreparation.prepare(count: results.count) { + try await self.exportPickerResult(results[$0]) } try Task.checkCancellation() await MainActor.run { @@ -235,3 +234,49 @@ private enum InlinePhotoPickerError: Error { case missingFile case unsupportedImage } + +// Keep Photos exports bounded without serializing an entire multi-selection. +// Drain cancelled exports before cleaning up: providers can finish after cancel. +enum PickerFilePreparation { + @MainActor + static func prepare( + count: Int, + export: @escaping @MainActor (Int) async throws -> String + ) async throws -> [String] { + var paths = Array(repeating: nil, count: count) + var failure: Error? + await withTaskGroup(of: (Int, Result).self) { group in + func enqueue(_ index: Int) { + group.addTask { + do { + try Task.checkCancellation() + return (index, .success(try await export(index))) + } catch { + return (index, .failure(error)) + } + } + } + var next = min(3, count) + for index in 0.. Void)? @@ -271,7 +272,7 @@ private final class NativePhotoZoomViewController: UIViewController, UIPageViewC closeCenterConstraint?.constant = navigationSizingBar.sizeThatFits(view.bounds.size).height / 2 } func shouldBeginDismissal(location: CGPoint, velocity: CGVector) -> Bool { - !paging && !finishing && !videoIndices.contains(index) && pager.view.frame.contains(location) + !paging && !finishing && pager.view.frame.contains(location) && velocity.dy > abs(velocity.dx) * 1.5 && ((pager.viewControllers?.first as? NativePhotoPage)?.isAtRestingZoom ?? false) } @@ -359,8 +360,9 @@ private final class NativePhotoZoomViewController: UIViewController, UIPageViewC // Flutter decodes authenticated originals and forwards continuing frames. // Ignore old pages and closed presentations before allocating a UIImage. func updateFrame(_ data: Data, at value: Int) { - guard !finishing, !videoIndices.contains(value), value == index, let image = UIImage(data: data) else { return } - images[value] = image + guard !finishing, value == index, let image = UIImage(data: data) else { return } + if videoIndices.contains(value) { previews[value] = image } + else { images[value] = image } for page in pager.children.compactMap({ $0 as? NativePhotoPage }) where page.index == value { page.setImage(image) } @@ -397,6 +399,7 @@ private final class NativePhotoZoomViewController: UIViewController, UIPageViewC } override func viewDidAppear(_ animated: Bool) { super.viewDidAppear(animated); appeared = true + selectionFeedback.prepare() (pager.viewControllers?.first as? NativePhotoPage)?.playVideo() } override func viewWillDisappear(_ animated: Bool) { @@ -410,12 +413,18 @@ private final class NativePhotoZoomViewController: UIViewController, UIPageViewC guard let page = viewController as? NativePhotoPage, page.index + 1 < labels.count else { return nil }; return self.page(page.index + 1) } func pageViewController(_ pageViewController: UIPageViewController, willTransitionTo pendingViewControllers: [UIViewController]) { + // Respond when the swipe starts, not after UIKit finishes settling the page. + if !paging { selectionFeedback.selectionChanged() } paging = true; indicator.isEnabled = false for page in pager.children.compactMap({ $0 as? NativePhotoPage }) { page.pauseVideo() } } func pageViewController(_ pageViewController: UIPageViewController, didFinishAnimating finished: Bool, previousViewControllers: [UIViewController], transitionCompleted completed: Bool) { paging = false; indicator.isEnabled = true - if completed, let page = pager.viewControllers?.first as? NativePhotoPage { selected(page.index) } + selectionFeedback.prepare() + if completed, let page = pager.viewControllers?.first as? NativePhotoPage { + for previous in previousViewControllers.compactMap({ $0 as? NativePhotoPage }) where previous !== page { previous.stopVideo() } + selected(page.index) + } else if appeared { (pager.viewControllers?.first as? NativePhotoPage)?.playVideo() } } private func updateMore() { @@ -425,6 +434,7 @@ private final class NativePhotoZoomViewController: UIViewController, UIPageViewC more.accessibilityElementsHidden = video } private func selected(_ value: Int) { + guard value != index else { return } stopVideos() index = value; indicator.currentPage = value updateMore() @@ -436,6 +446,7 @@ private final class NativePhotoZoomViewController: UIViewController, UIPageViewC guard !paging, !finishing else { return } let target = indicator.currentPage guard target != index else { return } + selectionFeedback.selectionChanged() // Scrubbing must not disable its own control or wait for a page animation. let direction: UIPageViewController.NavigationDirection = target >= index ? .forward : .reverse selected(target) @@ -505,9 +516,10 @@ private final class NativePhotoPage: UIViewController, UIScrollViewDelegate { private let imageView = UIImageView() private let scroll = UIScrollView() private let status = UILabel() + private let loading = NativeGalleryLoadingIndicator() private var previousSize = CGSize.zero var image: UIImage? { imageView.image } - var isAtRestingZoom: Bool { !isVideo && scroll.zoomScale <= scroll.minimumZoomScale + 0.01 } + var isAtRestingZoom: Bool { isVideo || scroll.zoomScale <= scroll.minimumZoomScale + 0.01 } func photoFrame(in target: UIView) -> CGRect? { if isVideo, video.player != nil, !video.videoBounds.isEmpty { return video.view.convert(video.videoBounds, to: target) } guard let image, image.size.width > 0, image.size.height > 0 else { return nil } @@ -519,7 +531,7 @@ private final class NativePhotoPage: UIViewController, UIScrollViewDelegate { init(index: Int, label: String, image: UIImage?, isVideo: Bool = false) { self.index = index; self.isVideo = isVideo; super.init(nibName: nil, bundle: nil) imageView.image = image; imageView.accessibilityLabel = label; imageView.isAccessibilityElement = !isVideo - status.text = isVideo ? "Loading video…" : image == nil ? "Loading photo…" : nil + status.text = !isVideo && image == nil ? "Loading photo…" : nil } required init?(coder: NSCoder) { nil } override func viewDidLoad() { @@ -542,6 +554,7 @@ private final class NativePhotoPage: UIViewController, UIScrollViewDelegate { retry.titleLabel?.adjustsFontForContentSizeCategory = true retry.addTarget(self, action: #selector(retryVideo), for: .touchUpInside) retry.isHidden = true; view.addSubview(retry) + view.addSubview(loading); loading.setLoading(true) } let tap = UITapGestureRecognizer(target: self, action: #selector(toggleZoom)); tap.numberOfTapsRequired = 2; if !isVideo { scroll.addGestureRecognizer(tap) } } @@ -549,12 +562,13 @@ private final class NativePhotoPage: UIViewController, UIScrollViewDelegate { imageView.image = image if !isVideo { status.text = image == nil ? "Image unavailable" : nil } } - func showVideoLoading() { guard isVideo else { return }; status.text = "Loading video…"; retry.isHidden = true } + func showVideoLoading() { guard isVideo else { return }; loadViewIfNeeded(); status.text = nil; loading.setLoading(true); retry.isHidden = true } func setVideo(_ path: String?, autoplay: Bool) { guard isVideo else { return } loadViewIfNeeded() stopVideo() guard let path, FileManager.default.fileExists(atPath: path) else { videoFailed(); return } + showVideoLoading() let item = AVPlayerItem(url: URL(fileURLWithPath: path)) let player = AVPlayer(playerItem: item) video.player = player @@ -562,7 +576,7 @@ private final class NativePhotoPage: UIViewController, UIScrollViewDelegate { DispatchQueue.main.async { guard let self, let item, self.video.player?.currentItem === item else { return } if item.status == .failed { self.videoFailed() } - else if item.status == .readyToPlay { self.status.text = nil; self.video.view.isHidden = false; self.scroll.isHidden = true } + else if item.status == .readyToPlay { self.loading.setLoading(false); self.status.text = nil; self.video.view.isHidden = false; self.scroll.isHidden = true } } } if autoplay { player.play() } @@ -572,6 +586,7 @@ private final class NativePhotoPage: UIViewController, UIScrollViewDelegate { func stopVideo() { guard isVideo else { return } observation = nil + loading.setLoading(false) video.player?.pause(); video.player?.replaceCurrentItem(with: nil); video.player = nil if isViewLoaded { video.view.isHidden = true; scroll.isHidden = false } } @@ -584,6 +599,7 @@ private final class NativePhotoPage: UIViewController, UIScrollViewDelegate { super.viewDidLayoutSubviews(); scroll.frame = view.bounds; status.frame = view.bounds.insetBy(dx: 24, dy: 100) if isVideo { video.view.frame = view.bounds + loading.frame = CGRect(x: view.bounds.midX - 22, y: view.bounds.midY - 22, width: 44, height: 44) retry.frame = CGRect(x: 24, y: view.bounds.midY + 32, width: max(0, view.bounds.width - 48), height: 48) } guard previousSize != scroll.bounds.size else { return }; previousSize = scroll.bounds.size @@ -592,3 +608,53 @@ private final class NativePhotoPage: UIViewController, UIScrollViewDelegate { func viewForZooming(in scrollView: UIScrollView) -> UIView? { imageView } @objc private func toggleZoom() { scroll.setZoomScale(scroll.zoomScale > 1 ? 1 : 2.5, animated: !UIAccessibility.isReduceMotionEnabled) } } + +// Native counterpart of BuzzLoadingIndicator: 44pt white quarter arc, 4pt +// stroke, 10% track and a linear 500ms turn, as used by the Flutter video viewer. +private final class NativeGalleryLoadingIndicator: UIView { + private let track = CAShapeLayer() + private let arc = CAShapeLayer() + private var loading = false + + override init(frame: CGRect) { + super.init(frame: frame) + isUserInteractionEnabled = false + isAccessibilityElement = true + accessibilityLabel = "Loading video" + accessibilityTraits = .updatesFrequently + for shape in [track, arc] { + shape.fillColor = UIColor.clear.cgColor + shape.lineWidth = 4 + layer.addSublayer(shape) + } + track.strokeColor = UIColor.white.withAlphaComponent(0.1).cgColor + arc.strokeColor = UIColor.white.cgColor + NotificationCenter.default.addObserver(self, selector: #selector(updateAnimation), + name: UIAccessibility.reduceMotionStatusDidChangeNotification, object: nil) + isHidden = true + } + required init?(coder: NSCoder) { nil } + deinit { NotificationCenter.default.removeObserver(self) } + func setLoading(_ value: Bool) { + loading = value; isHidden = !value; updateAnimation() + } + override func didMoveToWindow() { super.didMoveToWindow(); updateAnimation() } + override func layoutSubviews() { + super.layoutSubviews() + let center = CGPoint(x: bounds.midX, y: bounds.midY) + let radius = max(0, (min(bounds.width, bounds.height) - 4) / 2) + track.path = UIBezierPath(arcCenter: center, radius: radius, startAngle: 0, endAngle: 2 * .pi, clockwise: true).cgPath + arc.path = UIBezierPath(arcCenter: center, radius: radius, startAngle: -.pi / 2, endAngle: 0, clockwise: true).cgPath + } + @objc private func updateAnimation() { + guard loading, window != nil, !UIAccessibility.isReduceMotionEnabled else { + layer.removeAnimation(forKey: "loading"); return + } + guard layer.animation(forKey: "loading") == nil else { return } + let rotation = CABasicAnimation(keyPath: "transform.rotation.z") + rotation.fromValue = 0; rotation.toValue = 2 * Double.pi + rotation.duration = 0.5; rotation.repeatCount = .infinity + rotation.timingFunction = CAMediaTimingFunction(name: .linear) + layer.add(rotation, forKey: "loading") + } +} diff --git a/mobile/ios/RunnerTests/RunnerTests.swift b/mobile/ios/RunnerTests/RunnerTests.swift index 5f5dbdfb9c..71159a730d 100644 --- a/mobile/ios/RunnerTests/RunnerTests.swift +++ b/mobile/ios/RunnerTests/RunnerTests.swift @@ -1343,7 +1343,7 @@ class RunnerTests: XCTestCase { let size = NativeAttachmentMenuLayout.size(compatibleWith: traits) XCTAssertEqual(size.width, 216) - XCTAssertEqual(size.height, 324) // Five actions, including voice notes. + XCTAssertEqual(size.height, 264) // Photos includes videos; four actions. XCTAssertEqual(NativeAttachmentMenuLayout.contentPadding, 16) XCTAssertEqual( NativeAttachmentMenuLayout.itemHeight(compatibleWith: traits), diff --git a/mobile/lib/features/channels/compose_bar.dart b/mobile/lib/features/channels/compose_bar.dart index a73ab1882a..549990a5bb 100644 --- a/mobile/lib/features/channels/compose_bar.dart +++ b/mobile/lib/features/channels/compose_bar.dart @@ -17,6 +17,7 @@ import 'package:flutter/scheduler.dart'; import 'package:flutter_hooks/flutter_hooks.dart'; import 'package:hooks_riverpod/hooks_riverpod.dart'; import 'package:image_picker/image_picker.dart'; + import 'package:buzz/shared/theme/buzz_icons.dart'; import 'package:nostr/nostr.dart' as nostr; @@ -67,6 +68,7 @@ part 'compose_bar/suggestions.dart'; part 'compose_bar/mention_chooser.dart'; part 'compose_bar/formatting_toolbar.dart'; part 'compose_bar/attachments.dart'; +part 'compose_bar/video_thumbnail.dart'; part 'compose_bar/upload_progress_pill.dart'; part 'compose_bar/photo_gallery_picker.dart'; part 'compose_bar/ios_photo_picker.dart'; diff --git a/mobile/lib/features/channels/compose_bar/attachments.dart b/mobile/lib/features/channels/compose_bar/attachments.dart index 33962f27c6..1ad81a0bcc 100644 --- a/mobile/lib/features/channels/compose_bar/attachments.dart +++ b/mobile/lib/features/channels/compose_bar/attachments.dart @@ -44,6 +44,20 @@ bool _queueComposerAttachment( return true; } +final _pickedVideoExtension = RegExp( + r'\.(mp4|mov|m4v|webm|3gp|avi|mkv)$', + caseSensitive: false, +); + +bool _isPickedVideo(XFile file) { + final mime = file.mimeType; + if (mime != null && mime != 'application/octet-stream') { + return mime.startsWith('video/'); + } + return _pickedVideoExtension.hasMatch(file.name) || + _pickedVideoExtension.hasMatch(file.path); +} + bool _queueComposerImages( List images, _ComposerVoiceNote voiceNote, @@ -63,7 +77,9 @@ bool _queueComposerImages( for (final image in images) _PendingAttachment( file: image, - kind: _PendingAttachmentKind.image, + kind: _isPickedVideo(image) + ? _PendingAttachmentKind.video + : _PendingAttachmentKind.image, deleteAfterUse: deleteAfterUse, ), ]; @@ -72,6 +88,10 @@ bool _queueComposerImages( enum _AttachmentSurface { closed, menu, camera, photos } +bool get _hasCombinedMediaPicker => + defaultTargetPlatform == TargetPlatform.iOS || + defaultTargetPlatform == TargetPlatform.android; + const _attachmentMenuWidth = 216.0; const _attachmentMenuPadding = Grid.xs; const _attachmentMenuItemHeight = 52.0; @@ -104,10 +124,11 @@ class _AttachmentMenuLayout { textPainter.height + (Grid.xxs * 2), ); textPainter.dispose(); + final itemCount = _hasCombinedMediaPicker ? 4 : 5; final contentHeight = (_attachmentMenuPadding * 2) + - (itemHeight * 5) + - (_attachmentMenuItemSpacing * 4); + (itemHeight * itemCount) + + (_attachmentMenuItemSpacing * (itemCount - 1)); return _AttachmentMenuLayout( itemHeight: itemHeight, @@ -545,7 +566,7 @@ class _AttachmentMenu extends StatelessWidget { final items = <(IconData, String, VoidCallback)>[ (BuzzIcons.camera, 'Camera', onCamera), (BuzzIcons.images, 'Photos', onPhotos), - (BuzzIcons.video, 'Video', onVideo), + if (!_hasCombinedMediaPicker) (BuzzIcons.video, 'Video', onVideo), (BuzzIcons.mic, 'Voice note', onVoiceNote), (BuzzIcons.file, 'Files', onFiles), ]; @@ -692,16 +713,7 @@ class _AttachmentStrip extends StatelessWidget { ClipRRect( borderRadius: BorderRadius.circular(Radii.md), child: attachment.kind == _PendingAttachmentKind.video - ? ColoredBox( - color: Colors.black, - child: Center( - child: Icon( - BuzzIcons.video, - color: Colors.white, - size: 24, - ), - ), - ) + ? _PendingVideoThumbnail(file: attachment.file) : attachment.kind == _PendingAttachmentKind.image && attachment.file.path.isNotEmpty ? Image.file( diff --git a/mobile/lib/features/channels/compose_bar/ios_photo_picker.dart b/mobile/lib/features/channels/compose_bar/ios_photo_picker.dart index d4552fb7d4..29e55a7164 100644 --- a/mobile/lib/features/channels/compose_bar/ios_photo_picker.dart +++ b/mobile/lib/features/channels/compose_bar/ios_photo_picker.dart @@ -183,14 +183,14 @@ class _IOSInlinePhotoPicker extends HookWidget { child: BuzzLoadingIndicator( size: 20, color: Colors.white, - semanticLabel: 'Preparing selected photos', + semanticLabel: 'Preparing selected items', ), ) : Text( selectedCount.value == 0 ? 'All Photos' : 'Add ${selectedCount.value} ' - '${selectedCount.value == 1 ? 'photo' : 'photos'}', + '${selectedCount.value == 1 ? 'item' : 'items'}', ), ), ), @@ -202,7 +202,7 @@ class _IOSInlinePhotoPicker extends HookWidget { child: BuzzLoadingIndicator( size: 44, color: Colors.white, - semanticLabel: 'Preparing selected photos', + semanticLabel: 'Preparing selected items', ), ), ), diff --git a/mobile/lib/features/channels/compose_bar/photo_gallery_picker.dart b/mobile/lib/features/channels/compose_bar/photo_gallery_picker.dart index eaccb00b66..778ace7b50 100644 --- a/mobile/lib/features/channels/compose_bar/photo_gallery_picker.dart +++ b/mobile/lib/features/channels/compose_bar/photo_gallery_picker.dart @@ -101,11 +101,13 @@ class _RecentPhotoGalleryPicker extends HookConsumerWidget { }); final selectedCount = selection.value.length; + final hasVideos = selection.value.any((photo) => photo.isVideo); + final selectionNoun = hasVideos ? 'item' : 'photo'; final actionLabel = selectedCount == 0 ? 'All photos' : selectedCount == 1 - ? 'Add 1 photo' - : 'Add $selectedCount photos'; + ? 'Add 1 $selectionNoun' + : 'Add $selectedCount ${selectionNoun}s'; Widget buildGalleryBody() { if (recentSnapshot.connectionState != ConnectionState.done) { @@ -293,12 +295,13 @@ class _RecentPhotoTile extends StatelessWidget { @override Widget build(BuildContext context) { final isSelected = selectionIndex >= 0; + final mediaName = photo.isVideo ? 'Video' : 'Photo'; return Semantics( button: true, selected: isSelected, label: isSelected - ? 'Photo ${selectionIndex + 1} selected' - : 'Select photo', + ? '$mediaName ${selectionIndex + 1} selected' + : 'Select ${mediaName.toLowerCase()}', child: GestureDetector( key: ValueKey('recent-photo-${photo.id}'), behavior: HitTestBehavior.opaque, @@ -314,6 +317,12 @@ class _RecentPhotoTile extends StatelessWidget { gaplessPlayback: true, ), ), + if (photo.isVideo) + const Positioned( + left: Grid.half, + bottom: Grid.half, + child: Icon(BuzzIcons.video, color: Colors.white, size: 20), + ), AnimatedContainer( duration: reducedMotion ? Duration.zero diff --git a/mobile/lib/features/channels/compose_bar/video_thumbnail.dart b/mobile/lib/features/channels/compose_bar/video_thumbnail.dart new file mode 100644 index 0000000000..c026ec65cc --- /dev/null +++ b/mobile/lib/features/channels/compose_bar/video_thumbnail.dart @@ -0,0 +1,28 @@ +part of '../compose_bar.dart'; + +class _PendingVideoThumbnail extends HookConsumerWidget { + const _PendingVideoThumbnail({required this.file}); + final XFile file; + + @override + Widget build(BuildContext context, WidgetRef ref) { + final service = ref.watch(mediaUploadServiceProvider); + final poster = useFuture( + useMemoized(() => service.prepareVideoPoster(file.path), [ + file.path, + service, + ]), + ); + return Stack( + fit: StackFit.expand, + children: [ + const ColoredBox(color: Colors.black), + if (poster.data case final bytes? when bytes.isNotEmpty) + Image.memory(bytes, fit: BoxFit.cover, excludeFromSemantics: true), + const Center( + child: Icon(BuzzIcons.video, color: Colors.white, size: 24), + ), + ], + ); + } +} diff --git a/mobile/lib/features/channels/message_content.dart b/mobile/lib/features/channels/message_content.dart index b86a201ac7..457c576fa6 100644 --- a/mobile/lib/features/channels/message_content.dart +++ b/mobile/lib/features/channels/message_content.dart @@ -42,6 +42,7 @@ import 'local_send_bubble_surface.dart'; import 'media_viewer_page.dart'; import 'native_photo_zoom_preview.dart'; import 'native_video_files.dart'; +import 'video_poster.dart'; import 'message_content/link_normalizer.dart'; import 'message_content/markdown_escapes.dart'; import 'message_content/preview_text.dart'; diff --git a/mobile/lib/features/channels/message_content/media_stack.dart b/mobile/lib/features/channels/message_content/media_stack.dart index 17a5807ac3..b3138632b3 100644 --- a/mobile/lib/features/channels/message_content/media_stack.dart +++ b/mobile/lib/features/channels/message_content/media_stack.dart @@ -42,6 +42,21 @@ class _MessageImageStack extends HookConsumerWidget { final reducedMotion = MediaQuery.disableAnimationsOf(context); final auth = ref.watch(mediaGetAuthServiceProvider); final index = value.round() % items.length; + // Match the rendered cards, including the next card during a swipe. Do not + // download every posterless video in a large attachment group. + final posterIndices = { + for (var slot = 0; slot <= 3; slot++) + (value.floor() + slot) % items.length, + }; + final generatedPosters = [ + for (var i = 0; i < items.length; i++) + if (posterIndices.contains(i) && + items[i].isVideo && + items[i].posterUrl == null) + ref.watch(videoPosterProvider(items[i].url)).asData?.value + else + null, + ]; final hasVideos = items.any((item) => item.isVideo); final itemName = hasVideos ? 'Attachment' : 'Photo'; @@ -76,15 +91,17 @@ class _MessageImageStack extends HookConsumerWidget { (cardWidth * MediaQuery.devicePixelRatioOf(context)) .ceilToDouble(); final providers = [ - for (final item in items) - if (item.isVideo && item.posterUrl == null) - null + for (var i = 0; i < items.length; i++) + if (items[i].isVideo && items[i].posterUrl == null) + generatedPosters[i] == null + ? null + : MemoryImage(generatedPosters[i]!) else ResizeImage.resizeIfNeeded( physicalDecodeWidth.toInt(), null, MediaImageProvider( - url: item.isVideo ? item.posterUrl! : item.url, + url: items[i].isVideo ? items[i].posterUrl! : items[i].url, auth: auth, client: ref.watch(mediaHttpClientProvider), ), @@ -251,7 +268,16 @@ class _MessageImageStack extends HookConsumerWidget { Widget card(int i) { final item = items[i]; Widget photo = item.isVideo && item.posterUrl == null - ? ColoredBox(color: context.colors.surfaceContainerHigh) + ? generatedPosters[i] == null + ? const _MediaPreviewFallback( + icon: BuzzIcons.video, + label: 'Video preview unavailable', + ) + : Image.memory( + generatedPosters[i]!, + fit: BoxFit.cover, + excludeFromSemantics: true, + ) : MediaImage( url: item.isVideo ? item.posterUrl! : item.url, decodeWidth: decodeWidth, diff --git a/mobile/lib/features/channels/message_content/video_preview.dart b/mobile/lib/features/channels/message_content/video_preview.dart index 7a0727d725..d797a03fe1 100644 --- a/mobile/lib/features/channels/message_content/video_preview.dart +++ b/mobile/lib/features/channels/message_content/video_preview.dart @@ -1,7 +1,7 @@ part of '../message_content.dart'; /// A video bubble. It shows the NIP-71 poster when the event has one; -/// otherwise a static placeholder, so scrolling never starts a video decoder. +/// older attachments generate a still without creating a playback controller. class _MessageVideoPreview extends HookConsumerWidget { final String url; final ImetaEntry? imeta; @@ -19,6 +19,9 @@ class _MessageVideoPreview extends HookConsumerWidget { final opening = useRef(false); final aspectRatio = messageVideoAspectRatio(imeta?.aspectRatio); final posterUrl = imeta?.posterUrl; + final generatedPoster = posterUrl == null + ? ref.watch(videoPosterProvider(url)).asData?.value + : null; return Padding( padding: const EdgeInsets.only(top: Grid.half), @@ -37,12 +40,27 @@ class _MessageVideoPreview extends HookConsumerWidget { final fallback = await NativePhotoZoomPreview.open( context: context, sourceKeys: [sourceKey], + previewProviders: [ + if (generatedPoster != null) + MemoryImage(generatedPoster) + else if (posterUrl != null) + MediaImageProvider( + url: posterUrl, + auth: auth, + client: client, + ) + else + null, + ], providers: [ - MediaImageProvider( - url: posterUrl ?? url, - auth: auth, - client: client, - ), + if (generatedPoster != null) + MemoryImage(generatedPoster) + else + MediaImageProvider( + url: posterUrl ?? url, + auth: auth, + client: client, + ), ], urls: [url], labels: [imeta?.alt ?? 'Video'], @@ -90,6 +108,12 @@ class _MessageVideoPreview extends HookConsumerWidget { label: 'Video preview unavailable', ), ) + else if (generatedPoster != null) + Image.memory( + generatedPoster, + fit: BoxFit.cover, + excludeFromSemantics: true, + ) else const _MediaPreviewFallback( icon: BuzzIcons.video, diff --git a/mobile/lib/features/channels/message_gallery.dart b/mobile/lib/features/channels/message_gallery.dart index 862278309f..bf3a335c66 100644 --- a/mobile/lib/features/channels/message_gallery.dart +++ b/mobile/lib/features/channels/message_gallery.dart @@ -17,7 +17,7 @@ class MessageGalleryItem { /// Whether this item opens video playback rather than an image decoder. final bool isVideo; - /// Optional video poster; absent posters use a static video placeholder. + /// Optional video poster; absent posters are generated lazily by the stack. final String? posterUrl; const MessageGalleryItem({ diff --git a/mobile/lib/features/channels/native_photo_zoom_preview.dart b/mobile/lib/features/channels/native_photo_zoom_preview.dart index 3f3c286c8b..33b455419d 100644 --- a/mobile/lib/features/channels/native_photo_zoom_preview.dart +++ b/mobile/lib/features/channels/native_photo_zoom_preview.dart @@ -30,7 +30,7 @@ class NativePhotoZoomPreview { /// JPEG, PNG and HEIC/HEIF photos use the native gallery. Continuing frames /// are forwarded for animated PNGs; other formats keep the Flutter renderer. /// [mimeTypes], when provided, contains one declared type per [urls] entry. - /// [videoIndices] use rendered source snapshots and [loadVideo] local files; + /// [videoIndices] use posters (or source snapshots) and [loadVideo] local files; /// their URLs never enter the image decoder. static Future open({ required BuildContext context, @@ -94,13 +94,21 @@ class NativePhotoZoomPreview { }); }); window = loads; - final bytes = videoIndices.contains(initialIndex) - ? await _snapshot(sourceKeys[initialIndex]) + final initialPoster = previewProvider ?? previewProviders?[initialIndex]; + final isVideo = videoIndices.contains(initialIndex); + final posterLoad = isVideo && initialPoster != null + ? loads.load(initialIndex, initialPoster, preview: true) + : null; + // Only wait for PNG encoding when the poster was already decoded. + // Network-backed posters must never delay opening playable video. + final cachedPoster = + loads.loads[(initialIndex, true)]?.receivedFrame == true; + final bytes = isVideo + ? (cachedPoster ? await posterLoad : null) ?? + await _snapshot(sourceKeys[initialIndex]) : await loads.load( initialIndex, - previewProvider ?? - previewProviders?[initialIndex] ?? - providers[initialIndex], + initialPoster ?? providers[initialIndex], preview: true, ); if (bytes == null) return true; @@ -123,10 +131,13 @@ class NativePhotoZoomPreview { } return loadVideo?.call(index); case 'preview': - if (videoIndices.contains(index)) { - return _snapshot(sourceKeys[index]); - } final provider = previewProviders?[index]; + if (videoIndices.contains(index)) { + return (provider == null + ? null + : await loads.load(index, provider, preview: true)) ?? + await _snapshot(sourceKeys[index]); + } return provider == null ? null : loads.load(index, provider, preview: true); @@ -150,7 +161,7 @@ class NativePhotoZoomPreview { } return null; }); - final result = await channel.invokeMethod('present', { + final presentation = channel.invokeMethod('present', { ...geometry, 'bytes': bytes, 'session': session, @@ -162,6 +173,20 @@ class NativePhotoZoomPreview { 'canDelete': actions?.canDelete ?? false, 'reduceMotion': reduceMotion, }); + if (posterLoad != null && !cachedPoster) { + unawaited( + posterLoad + .then((poster) async { + if (poster != null && + !loads.disposed && + loads.index == initialIndex) { + await loads.onFrame(initialIndex, poster); + } + }) + .catchError((Object _) {}), + ); + } + final result = await presentation; if (result == 'reply' && context.mounted) onReply?.call(); return result == null || result == 'unavailable'; } catch (_) { diff --git a/mobile/lib/features/channels/native_video_files.dart b/mobile/lib/features/channels/native_video_files.dart index 977e9915c5..ce8f666f9f 100644 --- a/mobile/lib/features/channels/native_video_files.dart +++ b/mobile/lib/features/channels/native_video_files.dart @@ -83,6 +83,7 @@ class _VideoFileDownload { Directory? directory; bool disposed = false; bool finished = false; + void Function()? _cancelTimeouts; Future _fetch() async { Timer? stall; @@ -96,6 +97,10 @@ class _VideoFileDownload { } final deadline = Timer(owner.maxDuration, abort); + _cancelTimeouts = () { + stall?.cancel(); + deadline.cancel(); + }; StreamIterator>? body; RandomAccessFile? output; var success = false; @@ -168,6 +173,7 @@ class _VideoFileDownload { Future dispose() async { disposed = true; + _cancelTimeouts?.call(); if (!stop.isCompleted) stop.complete(); await result; await _delete(); diff --git a/mobile/lib/features/channels/photo_library.dart b/mobile/lib/features/channels/photo_library.dart index 8a01de01e0..c79b14cc2c 100644 --- a/mobile/lib/features/channels/photo_library.dart +++ b/mobile/lib/features/channels/photo_library.dart @@ -6,12 +6,12 @@ import 'package:photo_manager/photo_manager.dart'; const _recentPhotoCount = 30; const _photoPermissionRequest = PermissionRequestOption( androidPermission: AndroidPermission( - type: RequestType.image, + type: RequestType.common, mediaLocation: false, ), ); -/// A recent photo exposed to Buzz's compact in-composer gallery. +/// A recent photo or video exposed to Buzz's compact in-composer gallery. @immutable class RecentPhoto { /// The platform photo-library identifier. @@ -20,13 +20,20 @@ class RecentPhoto { /// Thumbnail bytes sized for the compact picker grid. final Uint8List thumbnailBytes; + /// Whether this library asset is a video. + final bool isVideo; + /// Creates a recent photo value. - const RecentPhoto({required this.id, required this.thumbnailBytes}); + const RecentPhoto({ + required this.id, + required this.thumbnailBytes, + this.isVideo = false, + }); } -/// Reads recent photos and resolves selected library assets to uploadable files. +/// Reads recent photos/videos and resolves selected assets to uploadable files. abstract interface class PhotoLibrary { - /// Requests access when needed and returns the newest visible photos. + /// Requests access when needed and returns the newest visible photos/videos. Future> loadRecentPhotos(); /// Resolves selected photos in the supplied selection order. @@ -64,7 +71,7 @@ class DevicePhotoLibrary implements PhotoLibrary { final assets = await PhotoManager.getAssetListPaged( page: 0, pageCount: _recentPhotoCount, - type: RequestType.image, + type: RequestType.common, filterOption: FilterOptionGroup( imageOption: const FilterOption(needTitle: true), orders: const [ @@ -85,7 +92,11 @@ class DevicePhotoLibrary implements PhotoLibrary { quality: 84, ); if (bytes == null || bytes.isEmpty) return null; - return RecentPhoto(id: asset.id, thumbnailBytes: bytes); + return RecentPhoto( + id: asset.id, + thumbnailBytes: bytes, + isVideo: asset.type == AssetType.video, + ); } @override diff --git a/mobile/lib/features/channels/video_poster.dart b/mobile/lib/features/channels/video_poster.dart new file mode 100644 index 0000000000..adb3494b3f --- /dev/null +++ b/mobile/lib/features/channels/video_poster.dart @@ -0,0 +1,60 @@ +import 'dart:async'; +import 'dart:typed_data'; + +import 'package:hooks_riverpod/hooks_riverpod.dart'; + +import '../../shared/relay/relay.dart'; +import 'native_video_files.dart'; + +// Serialize legacy-poster work so a timeline never launches many video downloads +// or decoders at once. This queue belongs to the current provider container. +final _posterQueueProvider = Provider((ref) => _PosterQueue()); + +class _PosterQueue { + Future tail = Future.value(); + Future run(Future Function() work) { + final result = tail.then((_) => work()); + tail = result.then((_) {}, onError: (Object _, StackTrace _) {}); + return result; + } +} + +/// Generates a still for older video messages that have no poster metadata. +/// Downloads are bounded/cancellable and temporary video files are removed. +/// Stills use the account/community image cache; no playback controller is created. +final videoPosterProvider = FutureProvider.autoDispose + .family((ref, url) async { + final auth = ref.watch(mediaGetAuthServiceProvider); + final client = ref.watch(mediaHttpClientProvider); + final service = ref.watch(mediaUploadServiceProvider); + final queue = ref.watch(_posterQueueProvider); + final cache = ref.watch(mediaDiskCacheProvider); + final cacheKey = 'video-poster:v1:$url'; + final files = NativeVideoFiles(auth: auth, client: client, index: 0); + var disposed = false; + ref.onDispose(() { + disposed = true; + unawaited(files.dispose()); + }); + final cached = await cache?.read(cacheKey); + if (disposed) return null; + if (cached != null && !cached.isStale) return cached.bytes; + return queue.run(() async { + if (disposed) return null; + try { + final path = await files.load(0, url); + if (path == null || disposed) return null; + final bytes = await service.prepareVideoPoster(path); + if (!disposed && bytes != null && bytes.isNotEmpty) { + await cache?.write( + cacheKey, + bytes, + immutable: auth.isRelayMediaUrl(url), + ); + } + return bytes; + } finally { + await files.dispose(); + } + }); + }); diff --git a/mobile/lib/shared/relay/media_upload.dart b/mobile/lib/shared/relay/media_upload.dart index ac6c612512..c814c899d5 100644 --- a/mobile/lib/shared/relay/media_upload.dart +++ b/mobile/lib/shared/relay/media_upload.dart @@ -333,6 +333,15 @@ class MediaUploadService { return XFile.fromData(bytes, name: 'Pasted image'); } + /// Extracts a local poster without starting playback or uploading the video. + Future prepareVideoPoster(String path) async { + try { + return await _generateVideoPoster(path); + } catch (_) { + return null; + } + } + /// Opens the system gallery video picker. Future pickGalleryVideo() => _pickGalleryVideo(); diff --git a/mobile/lib/shared/relay/media_upload/platform_bindings.dart b/mobile/lib/shared/relay/media_upload/platform_bindings.dart index a192c9b5cd..47e1675510 100644 --- a/mobile/lib/shared/relay/media_upload/platform_bindings.dart +++ b/mobile/lib/shared/relay/media_upload/platform_bindings.dart @@ -24,7 +24,8 @@ final mediaUploadServiceProvider = Provider((ref) { source: ImageSource.gallery, requestFullMetadata: false, ), - pickGalleryImages: () => picker.pickMultiImage(requestFullMetadata: false), + pickGalleryImages: () => + picker.pickMultipleMedia(requestFullMetadata: false), pickGalleryVideo: () => picker.pickVideo(source: ImageSource.gallery), pickAttachmentFile: file_selector.openFile, ); diff --git a/mobile/test/features/channels/compose_bar_test.dart b/mobile/test/features/channels/compose_bar_test.dart index bc973f8abf..071ed02d6d 100644 --- a/mobile/test/features/channels/compose_bar_test.dart +++ b/mobile/test/features/channels/compose_bar_test.dart @@ -43,6 +43,7 @@ import 'package:buzz/shared/widgets/mobile_tab_footer_backdrop.dart'; import 'package:shared_preferences/shared_preferences.dart'; part 'compose_bar_test/exact_mention_tests.dart'; +part 'compose_bar_test/mixed_gallery_tests.dart'; part 'compose_bar_test/solo_draft_tests.dart'; part 'compose_bar_test/durable_mention_tests.dart'; part 'compose_bar_test/agent_prefill_tests.dart'; @@ -414,6 +415,9 @@ class _FakeVideoUploadService extends MediaUploadService { @override Future pickGalleryVideo() async => video; + @override + Future> pickGalleryImages() async => [video]; + @override Future uploadVideo( XFile pickedVideo, { @@ -735,6 +739,7 @@ class _IterationCountingChannels extends ListBase { } void main() { + _mixedGalleryTests(); soloDraftTests(); exactMentionTests(); durableMentionTests(); @@ -3122,15 +3127,10 @@ void main() { final menu = find.byKey(const ValueKey('attachment-menu')); final surface = find.byKey(const ValueKey('attachment-surface-popover')); final rows = [ - for (final label in [ - 'camera', - 'photos', - 'video', - 'voice note', - 'files', - ]) + for (final label in ['camera', 'photos', 'voice note', 'files']) find.byKey(ValueKey('attachment-menu-item-$label')), ]; + expect(find.text('Video'), findsNothing); final menuRect = tester.getRect(menu); final material = tester.widget(surface); @@ -3142,41 +3142,23 @@ void main() { material.shadowColor, appPopoverShadowColor(tester.element(surface)), ); - expect(menuRect.size, const Size(216, 324)); + expect(menuRect.size, const Size(216, 264)); for (final row in rows) { expect(tester.getSize(row).height, 52); expect(tester.getRect(row).left - menuRect.left, Grid.xs); expect(menuRect.right - tester.getRect(row).right, Grid.xs); } - for (final label in [ - 'Camera', - 'Photos', - 'Video', - 'Voice note', - 'Files', - ]) { + for (final label in ['Camera', 'Photos', 'Voice note', 'Files']) { final text = tester.widget(find.text(label)); expect(text.style?.fontSize, 20); expect(text.style?.fontFamily, 'Inter'); } final icons = [ - for (final label in [ - 'camera', - 'photos', - 'video', - 'voice note', - 'files', - ]) + for (final label in ['camera', 'photos', 'voice note', 'files']) find.byKey(ValueKey('attachment-menu-icon-$label')), ]; final labels = [ - for (final label in [ - 'camera', - 'photos', - 'video', - 'voice note', - 'files', - ]) + for (final label in ['camera', 'photos', 'voice note', 'files']) find.byKey(ValueKey('attachment-menu-label-$label')), ]; for (final icon in icons) { @@ -3271,13 +3253,7 @@ void main() { final menu = find.byKey(const ValueKey('attachment-menu')); final rows = [ - for (final label in [ - 'camera', - 'photos', - 'video', - 'voice note', - 'files', - ]) + for (final label in ['camera', 'photos', 'voice note', 'files']) find.byKey(ValueKey('attachment-menu-item-$label')), ]; final scrollView = tester.widget( @@ -5962,7 +5938,7 @@ void main() { expect(find.byTooltip('Remove attachment'), findsOneWidget); }); - testWidgets('taps Video in chooser sheet and uploads video', ( + testWidgets('selects a video through Photos and uploads video', ( tester, ) async { final pickedVideo = XFile.fromData( @@ -5987,8 +5963,7 @@ void main() { ), ); - await _openAttachmentMenu(tester); - await tester.tap(find.text('Video')); + await _openSystemPhotoPicker(tester); await tester.pumpAndSettle(); expect(find.byIcon(BuzzIcons.video), findsOneWidget); diff --git a/mobile/test/features/channels/compose_bar_test/mixed_gallery_tests.dart b/mobile/test/features/channels/compose_bar_test/mixed_gallery_tests.dart new file mode 100644 index 0000000000..689f3399c7 --- /dev/null +++ b/mobile/test/features/channels/compose_bar_test/mixed_gallery_tests.dart @@ -0,0 +1,83 @@ +part of '../compose_bar_test.dart'; + +class _MixedGalleryService extends _FakeVideoUploadService { + _MixedGalleryService(super.video); + final uploaded = []; + @override + Future prepareVideoPoster(String path) async => _gifBytes; + @override + Future> pickGalleryImages() async => [ + XFile.fromData(_pngBytes, name: 'photo.png'), + video, + ]; + @override + Future uploadImage( + XFile image, { + ValueChanged? onProgress, + UploadCancellationToken? cancellationToken, + }) async { + uploaded.add('image'); + return const BlobDescriptor( + url: 'https://relay.example/photo.png', + sha256: 'photo', + size: 10, + type: 'image/png', + uploaded: 1, + ); + } + + @override + Future uploadVideo( + XFile video, { + ValueChanged? onProgress, + UploadCancellationToken? cancellationToken, + }) async { + uploaded.add('video'); + return super.uploadVideo( + video, + onProgress: onProgress, + cancellationToken: cancellationToken, + ); + } +} + +void _mixedGalleryTests() { + for (final filename in ['camera.MOV', 'clip.mp4', 'clip.m4v']) { + testWidgets('Photos queues $filename as a video alongside photos', ( + tester, + ) async { + // XFile.fromData ignores its name on IO platforms; picker results are paths. + final video = XFile('/fixtures/$filename'); + final service = _MixedGalleryService(video); + String? sent; + await tester.pumpWidget( + _buildComposeBar( + uploadService: service, + onSend: + (content, mentions, {mediaTags = const >[]}) async { + sent = content; + }, + ), + ); + await _openSystemPhotoPicker(tester); + await tester.pumpAndSettle(); + expect(find.byTooltip('Remove attachment'), findsNWidgets(2)); + expect(service.uploaded, isEmpty); + expect( + find.byWidgetPredicate( + (widget) => + widget is Image && + widget.image is MemoryImage && + identical((widget.image as MemoryImage).bytes, _gifBytes), + ), + findsOneWidget, + ); + await _expandComposer(tester); + await tester.tap(find.byIcon(BuzzIcons.arrowUp)); + await tester.pumpAndSettle(); + expect(service.uploaded, ['image', 'video']); + expect(service.uploadedVideo, same(video)); + expect(sent, contains('![video]')); + }); + } +} diff --git a/mobile/test/features/channels/message_content_test.dart b/mobile/test/features/channels/message_content_test.dart index 67d6626a10..7d1fa0d1b3 100644 --- a/mobile/test/features/channels/message_content_test.dart +++ b/mobile/test/features/channels/message_content_test.dart @@ -1,4 +1,5 @@ import 'package:buzz/shared/widgets/buzz_agent_icon.dart'; +import 'package:buzz/features/channels/video_poster.dart'; import 'package:buzz/features/channels/local_send_bubble_surface.dart'; import 'dart:async'; import 'dart:convert'; diff --git a/mobile/test/features/channels/message_stack_cases.dart b/mobile/test/features/channels/message_stack_cases.dart index 102fd04da1..8af2156c72 100644 --- a/mobile/test/features/channels/message_stack_cases.dart +++ b/mobile/test/features/channels/message_stack_cases.dart @@ -1,6 +1,34 @@ part of 'message_content_test.dart'; void stackJourneyTests() { + testWidgets('large video stacks only fetch posters for rendered cards', ( + tester, + ) async { + final requested = []; + await tester.pumpWidget( + _testable( + MessageContent( + bubble: true, + content: List.generate( + 20, + (i) => '![Video $i](https://example.com/clip-$i.mp4)', + ).join('\n'), + ), + disableAnimations: true, + overrides: [ + videoPosterProvider.overrideWith((ref, url) async { + requested.add(url); + return null; + }), + ], + ), + ); + await tester.pumpAndSettle(); + expect(requested.toSet(), { + for (var i = 0; i < 4; i++) 'https://example.com/clip-$i.mp4', + }); + }); + testWidgets('mixed stacks keep video order and return from the fallback player', ( tester, ) async { @@ -39,7 +67,8 @@ void stackJourneyTests() { await tester.pumpAndSettle(); expect(requested, contains(photo)); expect(requested, contains(poster)); - expect(requested, isNot(contains(clip))); + // Only the legacy video without poster metadata needs a background fetch. + expect(requested, contains(clip)); expect(requested, isNot(contains('https://example.com/second.mov'))); final stack = find.byKey(const ValueKey('message-media-stack')); final spoken = find.byKey(const ValueKey('message-media-stack-semantics')); diff --git a/mobile/test/features/channels/native_photo_zoom_preview_test.dart b/mobile/test/features/channels/native_photo_zoom_preview_test.dart index 74830373c1..9b57339763 100644 --- a/mobile/test/features/channels/native_photo_zoom_preview_test.dart +++ b/mobile/test/features/channels/native_photo_zoom_preview_test.dart @@ -2,6 +2,7 @@ import 'dart:convert'; import 'dart:async'; import 'dart:io'; import 'dart:ui' as ui; + import 'package:flutter/services.dart'; import 'package:buzz/features/channels/native_photo_zoom_preview.dart'; @@ -483,9 +484,11 @@ void main() { debugDefaultTargetPlatformOverride = null; }, skip: !NativePhotoZoomPreview.enabled); - testWidgets( - 'video opens from its rendered source and only loads the selected video', - (tester) async { + for (final posterMode in ['none', 'cached', 'pending']) { + final usePoster = posterMode != 'none'; + testWidgets('video uses $posterMode poster without delaying presentation', ( + tester, + ) async { debugDefaultTargetPlatformOverride = TargetPlatform.iOS; final key = GlobalKey(); late BuildContext context; @@ -509,6 +512,8 @@ void main() { ), ); final image = _PendingPreview(); + final pendingPoster = _PendingPreview(); + late Completer posterUpdated; final videos = []; const codec = StandardMethodCodec(); Future nativeCall(String method, int session, int index) async { @@ -527,9 +532,39 @@ void main() { call, ) async { if (call.method == 'isSupported') return true; + if (call.method == 'imageFrame') { + posterUpdated.complete(); + return null; + } final args = call.arguments as Map; + if (posterMode == 'pending') { + expect( + pendingPoster.frame.isCompleted, + isFalse, + reason: 'Presentation must not wait for the remote poster', + ); + final decoder = await ui.instantiateImageCodec(provider.bytes); + pendingPoster.frame.complete( + ImageInfo(image: (await decoder.getNextFrame()).image), + ); + decoder.dispose(); + await posterUpdated.future.timeout(const Duration(seconds: 5)); + } expect(args['videoIndices'], [0, 2]); expect(args['bytes'], isA()); + if (posterMode == 'cached') { + final codec = await ui.instantiateImageCodec( + args['bytes'] as Uint8List, + ); + final frame = await codec.getNextFrame(); + expect( + frame.image.width, + 2, + reason: 'Use the poster, not the 100-pixel source snapshot', + ); + frame.image.dispose(); + codec.dispose(); + } final session = args['session'] as int; expect(await nativeCall('image', session, 0), isNull); expect(await nativeCall('video', session, 1), isNull); @@ -542,11 +577,20 @@ void main() { return 'closed'; }); await tester.runAsync(() async { + posterUpdated = Completer(); + if (posterMode == 'cached') await precacheImage(provider, context); expect( await NativePhotoZoomPreview.open( context: context, sourceKeys: [key, key, key], providers: [image, provider, image], + previewProviders: usePoster + ? [ + posterMode == 'pending' ? pendingPoster : provider, + provider, + provider, + ] + : null, urls: [ 'https://example.org/clip.mp4', 'https://example.org/photo.jpg', @@ -571,9 +615,8 @@ void main() { reason: 'A video URL must never enter the image decoder', ); debugDefaultTargetPlatformOverride = null; - }, - skip: !NativePhotoZoomPreview.enabled, - ); + }, skip: !NativePhotoZoomPreview.enabled); + } testWidgets( 'unsupported platform preserves existing gallery without decoding', diff --git a/mobile/test/features/channels/video_poster_test.dart b/mobile/test/features/channels/video_poster_test.dart new file mode 100644 index 0000000000..a94fcd3265 --- /dev/null +++ b/mobile/test/features/channels/video_poster_test.dart @@ -0,0 +1,120 @@ +import 'dart:async'; +import 'dart:io'; +import 'dart:typed_data'; + +import 'package:buzz/features/channels/video_poster.dart'; +import 'package:buzz/shared/relay/relay.dart'; +import 'package:flutter_test/flutter_test.dart'; +import 'package:hooks_riverpod/hooks_riverpod.dart'; +import 'package:http/http.dart' as http; +import 'package:http/testing.dart'; +import 'package:nostr/nostr.dart' as nostr; +import 'package:path_provider_platform_interface/path_provider_platform_interface.dart'; +import 'package:plugin_platform_interface/plugin_platform_interface.dart'; + +class _TemporaryPath extends Fake + with MockPlatformInterfaceMixin + implements PathProviderPlatform { + _TemporaryPath(this.path); + final String path; + @override + Future getTemporaryPath() async => path; +} + +class _PosterCache extends MediaDiskCache { + _PosterCache() : super.open(null); + final entries = {}; + @override + Future read(String url) async { + final bytes = entries[url]; + return bytes == null ? null : (bytes: bytes, isStale: false); + } + + @override + Future write( + String url, + Uint8List bytes, { + required bool immutable, + }) async { + entries[url] = bytes; + } +} + +void main() { + TestWidgetsFlutterBinding.ensureInitialized(); + test( + 'legacy posters serialize downloads, authenticate, cache and clean files', + () async { + final root = await Directory.systemTemp.createTemp('poster-test-'); + final previous = PathProviderPlatform.instance; + PathProviderPlatform.instance = _TemporaryPath(root.path); + final release = Completer(); + final extracting = Completer(); + final requests = []; + final paths = []; + final client = MockClient((request) async { + requests.add(request); + return http.Response.bytes([1, 2, 3], 200); + }); + final service = MediaUploadService( + baseUrl: 'https://relay.example', + nsec: null, + pickGalleryImage: () async => null, + pickGalleryVideo: () async => null, + generateVideoPoster: (path) async { + paths.add(path); + expect(await File(path).readAsBytes(), [1, 2, 3]); + if (paths.length == 1) { + extracting.complete(); + await release.future; + } + return Uint8List.fromList([4, 5, 6]); + }, + ); + final cache = _PosterCache(); + final container = ProviderContainer( + overrides: [ + mediaDiskCacheProvider.overrideWithValue(cache), + mediaGetAuthServiceProvider.overrideWithValue( + MediaGetAuthService( + baseUrl: 'https://relay.example', + nsec: nostr.Keys.generate().nsec, + ), + ), + mediaHttpClientProvider.overrideWithValue(client), + mediaUploadServiceProvider.overrideWithValue(service), + ], + ); + addTearDown(() async { + container.dispose(); + service.dispose(); + client.close(); + PathProviderPlatform.instance = previous; + await root.delete(recursive: true); + }); + final first = videoPosterProvider('https://relay.example/media/one.mp4'); + final second = videoPosterProvider('https://elsewhere.example/two.mp4'); + final sub1 = container.listen(first, (_, _) {}); + final sub2 = container.listen(second, (_, _) {}); + final cancelled = container.listen( + videoPosterProvider('https://relay.example/media/cancelled.mp4'), + (_, _) {}, + ); + await extracting.future; + cancelled.close(); + await container.pump(); + expect(requests, hasLength(1)); + release.complete(); + expect(await container.read(first.future), [4, 5, 6]); + expect(await container.read(second.future), [4, 5, 6]); + expect(requests.first.headers['authorization'], startsWith('Nostr ')); + expect(requests.last.headers['authorization'], isNull); + expect(await root.list().toList(), isEmpty); + sub1.close(); + sub2.close(); + await container.pump(); + expect(await container.read(first.future), [4, 5, 6]); + expect(requests, hasLength(2)); + }, + ); +} diff --git a/scripts/test-mobile-picker-preparation.sh b/scripts/test-mobile-picker-preparation.sh new file mode 100644 index 0000000000..1dc2a5ebaf --- /dev/null +++ b/scripts/test-mobile-picker-preparation.sh @@ -0,0 +1,18 @@ +#!/bin/bash +# Exercise the bounded Photos export pipeline on a booted iOS simulator. +set -euo pipefail +repo_root=$(cd "$(dirname "$0")/.." && pwd) +device=${1:?Pass a booted iOS simulator UDID} +flutter_root=$(sed -n 's/^FLUTTER_ROOT=//p' "$repo_root/mobile/ios/Flutter/Generated.xcconfig") +frameworks="$flutter_root/bin/cache/artifacts/engine/ios/Flutter.xcframework/ios-arm64_x86_64-simulator" +fixture=$(mktemp -d) +trap 'rm -r "$fixture"' EXIT +xcrun --sdk iphonesimulator swiftc \ + -sdk "$(xcrun --sdk iphonesimulator --show-sdk-path)" \ + -target "$(uname -m)-apple-ios18.0-simulator" \ + -F "$frameworks" -framework Flutter \ + -Xlinker -rpath -Xlinker "$frameworks" \ + "$repo_root/mobile/ios/Runner/InlinePhotoPicker.swift" \ + "$repo_root/mobile/ios/NativeControlTests/PickerFilePreparationTests.swift" \ + -o "$fixture/picker-preparation-tests" +xcrun simctl spawn "$device" "$fixture/picker-preparation-tests"