Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[firebase_storage] Impossible to upload multiple files simultaneously? #1935

Closed
fysoul17 opened this issue Feb 5, 2020 · 5 comments
Closed
Labels
impact: customer A bug with low impact (e.g. affecting only a few customers or has a workaround). (P3) plugin: storage Stale Issue with no recent activity type: bug Something isn't working

Comments

@fysoul17
Copy link

fysoul17 commented Feb 5, 2020

Describe the bug
I was trying to upload 10 image files with under 20KB of file size, and it took around 15~17 seconds every time.

To Reproduce

So I ran a test if it performs upload tasks concurrently as below:

uploadAllImages(List<File> imagesToUpload) {
   final FirebaseStorage _storage = FirebaseStorage(storageBucket: "gs://xxxxxx.appspot.com/");
    for (int index = 0; index < imagesToUpload.length; index++) {
      uploadFile(_storage, imagesToUpload[index], index);
    }
}

uploadFile(FirebaseStorage storage, File file, int index) async {
    DateTime start = DateTime.now();

    int imageNo = index ~/ 2;
    String fileName = index % 2 == 0 ? "$imageNo.jpg" : "${imageNo}_thumbnail.jpg";
    // Temp path for testing.
    String path = "temp/$fileName";

    StorageUploadTask task = storage.ref().child(path).putFile(file);

    print("Task added: $path (size: ${file.lengthSync()} bytes)");

    StorageTaskSnapshot snap = await task.onComplete;

    print("$fileName completed");

    await snap.ref.getDownloadURL();

    print("$fileName getDownloadURL done");

    DateTime end = DateTime.now();
    print("$fileName took ${end.difference(start).inMilliseconds} ms");
}

The result shows it performed sequentially like:

I/flutter (27240): Task added: temp/0.jpg (size: 21312 bytes)
I/flutter (27240): Task added: temp/0_thumbnail.jpg (size: 3687 bytes)
I/flutter (27240): Task added: temp/1.jpg (size: 21312 bytes)
I/flutter (27240): Task added: temp/1_thumbnail.jpg (size: 3687 bytes)
I/flutter (27240): Task added: temp/2.jpg (size: 21312 bytes)
I/flutter (27240): Task added: temp/2_thumbnail.jpg (size: 3687 bytes)
I/flutter (27240): Task added: temp/3.jpg (size: 21312 bytes)
I/flutter (27240): Task added: temp/3_thumbnail.jpg (size: 3687 bytes)
I/flutter (27240): Task added: temp/4.jpg (size: 21321 bytes)
I/flutter (27240): Task added: temp/4_thumbnail.jpg (size: 3694 bytes)
I/flutter (27240): 0.jpg completed
I/flutter (27240): 0_thumbnail.jpg completed
I/flutter (27240): 0.jpg getDownloadURL done
I/flutter (27240): 0.jpg took 3288 ms
I/flutter (27240): 0_thumbnail.jpg getDownloadURL done
I/flutter (27240): 0_thumbnail.jpg took 3669 ms
I/flutter (27240): 1.jpg completed
I/flutter (27240): 1_thumbnail.jpg completed
I/flutter (27240): 1.jpg getDownloadURL done
I/flutter (27240): 1.jpg took 6732 ms
I/flutter (27240): 1_thumbnail.jpg getDownloadURL done
I/flutter (27240): 1_thumbnail.jpg took 7070 ms
I/flutter (27240): 2.jpg completed
I/flutter (27240): 2_thumbnail.jpg completed
I/flutter (27240): 2.jpg getDownloadURL done
I/flutter (27240): 2.jpg took 10014 ms
I/flutter (27240): 2_thumbnail.jpg getDownloadURL done
I/flutter (27240): 2_thumbnail.jpg took 10166 ms
I/flutter (27240): 3.jpg completed
I/flutter (27240): 3_thumbnail.jpg completed
I/flutter (27240): 3.jpg getDownloadURL done
I/flutter (27240): 3.jpg took 13186 ms
I/flutter (27240): 3_thumbnail.jpg getDownloadURL done
I/flutter (27240): 3_thumbnail.jpg took 13459 ms
I/flutter (27240): 4.jpg completed
I/flutter (27240): 4.jpg getDownloadURL done
I/flutter (27240): 4.jpg took 16009 ms
I/flutter (27240): 4_thumbnail.jpg completed
I/flutter (27240): 4_thumbnail.jpg getDownloadURL done
I/flutter (27240): 4_thumbnail.jpg took 16964 ms

So is the StorageUploadTask intended to be performed one-by-one when one task is completed? or is it a bug/coding mistake?

@fysoul17 fysoul17 added the type: bug Something isn't working label Feb 5, 2020
@iapicca
Copy link

iapicca commented Mar 6, 2020

Hi @fysoul17
can you please provide your flutter doctor -v ,
your flutter run --verbose
and your pubspec.yaml
Thank you

@iapicca iapicca added the blocked: customer-response Waiting for customer response, e.g. more information was requested. label Mar 6, 2020
@fysoul17
Copy link
Author

fysoul17 commented Mar 6, 2020

No problem @iapicca !

flutter doctor -v
[√] Flutter (Channel stable, v1.12.13+hotfix.8, on Microsoft Windows [Version 10.0.18362.657], locale ko-KR)
• Flutter version 1.12.13+hotfix.8 at D:\Flutter
• Framework revision 0b8abb4724 (3 weeks ago), 2020-02-11 11:44:36 -0800
• Engine revision e1e6ced81d
• Dart version 2.7.0

[√] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
• Android SDK at C:\Users\Terry\AppData\Local\Android\sdk
• Android NDK location not configured (optional; useful for native profiling support)
• Platform android-29, build-tools 28.0.3
• Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
• Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b03)
• All Android licenses accepted.

[√] Android Studio (version 3.5)
• Android Studio at C:\Program Files\Android\Android Studio
• Flutter plugin version 42.1.1
• Dart plugin version 191.8593
• Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b03)

[√] VS Code (version 1.42.1)
• VS Code at C:\Users\Terry\AppData\Local\Programs\Microsoft VS Code
• Flutter extension version 3.8.1

[√] Connected device (1 available)
• AOSP on IA Emulator • emulator-5554 • android-x86 • Android 9 (API 28) (emulator)

• No issues found!

pubspec.yaml
name: firestore_concurrent_upload_test
description: A new Flutter project.

version: 1.0.0+1

environment:
sdk: ">=2.1.0 <3.0.0"

dependencies:
flutter:
sdk: flutter

cupertino_icons: ^0.1.2

firebase_core: ^0.4.4+2
firebase_storage: ^3.1.3
uuid: ^2.0.4

dev_dependencies:
flutter_test:
sdk: flutter

flutter:
uses-material-design: true

flutter run --verbose
[+1803 ms] I/flutter (12875): Task added: temp/0.jpg (size: 77190 bytes)
[ ] I/flutter (12875): Task added: temp/1.jpg (size: 73446 bytes)
[ ] I/flutter (12875): Task added: temp/2.jpg (size: 47357 bytes)
[ ] I/flutter (12875): Task added: temp/3.jpg (size: 49342 bytes)
[ +24 ms] I/flutter (12875): 0.jpg completed
[ +317 ms] I/flutter (12875): 0.jpg getDownloadURL done
[ +1 ms] I/flutter (12875): 0.jpg took 3367 ms
[ +11 ms] I/flutter (12875): 1.jpg completed
[ +573 ms] I/flutter (12875): 1.jpg getDownloadURL done
[ +1 ms] I/flutter (12875): 1.jpg took 4338 ms
[+1033 ms] I/flutter (12875): 2.jpg completed
[ +311 ms] I/flutter (12875): 2.jpg getDownloadURL done
[ +1 ms] I/flutter (12875): 2.jpg took 6066 ms
[ +731 ms] I/flutter (12875): 3.jpg completed
[ +300 ms] I/flutter (12875): 3.jpg getDownloadURL done
[ +1 ms] I/flutter (12875): 3.jpg took 7103 ms

I just used the example project of official firestorage project by replacing firebase to my own one and below code for upload preparation.

I also found that the example code also seems to upload sequentially. Just press upload button quickly a few times then watch it. It does not seem to uploading concurrently.

List<File> files = List<File>();

  for (int index = 1; index <= 4; index++) {
       ByteData bytes = await rootBundle.load('assets/image$index.jpg');
       List<int> imageData = bytes.buffer.asUint8List();
       final Directory systemTempDir = Directory.systemTemp;
       final File tempPath = File('${systemTempDir.path}');
       String fileName = "${Uuid().v1()}.jpg";
       final File orgFile = await File("${tempPath.path}/$fileName").create()
          ..writeAsBytesSync(imageData);
        files.add(orgFile);
   }

// This code is at first comment. (just remove codes regarding thumbnail)
uploadAllImages(files);   

@iapicca iapicca removed the blocked: customer-response Waiting for customer response, e.g. more information was requested. label Mar 8, 2020
@Ehesp Ehesp added plugin: storage impact: customer A bug with low impact (e.g. affecting only a few customers or has a workaround). (P3) labels Apr 21, 2020
@greghesp
Copy link
Contributor

@fysoul17
I've tested this on the latest roadmap version

I did a similar test myself, and logged everything out below. As you can see, the log order and timestamps suggest that these uploads are happening simultaneously.

Please keep an eye out for a release, and let us know if you're still experiencing the issue

I/flutter (12904): Index: 0 Start: 2020-09-22 10:16:17.224210
I/flutter (12904): Index: 0 Task: Added
I/flutter (12904): Index: 1 Start: 2020-09-22 10:16:17.226510
I/flutter (12904): Index: 1 Task: Added
I/flutter (12904): Index: 2 Start: 2020-09-22 10:16:17.226954
I/flutter (12904): Index: 2 Task: Added
I/flutter (12904): Index: 3 Start: 2020-09-22 10:16:17.227499
I/flutter (12904): Index: 3 Task: Added
I/flutter (12904): Index: 4 Start: 2020-09-22 10:16:17.230897
I/flutter (12904): Index: 4 Task: Added
I/flutter (12904): Index: 5 Start: 2020-09-22 10:16:17.231517
I/flutter (12904): Index: 5 Task: Added
I/flutter (12904): Index: 6 Start: 2020-09-22 10:16:17.232380
I/flutter (12904): Index: 6 Task: Added
I/flutter (12904): Index: 7 Start: 2020-09-22 10:16:17.233319
I/flutter (12904): Index: 7 Task: Added
I/flutter (12904): Index: 8 Start: 2020-09-22 10:16:17.234375
I/flutter (12904): Index: 8 Task: Added
I/flutter (12904): Index: 9 Start: 2020-09-22 10:16:17.235088
I/flutter (12904): Index: 9 Task: Added
I/flutter (12904): Index: 0 Task: Upload Complete
I/flutter (12904): Index: 2 Task: Upload Complete
I/flutter (12904): Index: 0 Task: Got Download URL
I/flutter (12904): Index: 0 End: 2020-09-22 10:16:18.400255
I/flutter (12904): Index: 0 Task: Completed in 1176 ms
I/flutter (12904): Index: 2 Task: Got Download URL
I/flutter (12904): Index: 2 End: 2020-09-22 10:16:18.412605
I/flutter (12904): Index: 2 Task: Completed in 1185 ms
I/flutter (12904): Index: 1 Task: Upload Complete
I/flutter (12904): Index: 3 Task: Upload Complete
I/flutter (12904): Index: 1 Task: Got Download URL
I/flutter (12904): Index: 1 End: 2020-09-22 10:16:19.337491
I/flutter (12904): Index: 1 Task: Completed in 2110 ms
I/flutter (12904): Index: 3 Task: Got Download URL
I/flutter (12904): Index: 3 End: 2020-09-22 10:16:19.362573
I/flutter (12904): Index: 3 Task: Completed in 2135 ms
I/flutter (12904): Index: 5 Task: Upload Complete
I/flutter (12904): Index: 4 Task: Upload Complete
I/flutter (12904): Index: 5 Task: Got Download URL
I/flutter (12904): Index: 5 End: 2020-09-22 10:16:20.278068
I/flutter (12904): Index: 5 Task: Completed in 3046 ms
I/flutter (12904): Index: 4 Task: Got Download URL
I/flutter (12904): Index: 4 End: 2020-09-22 10:16:20.298661
I/flutter (12904): Index: 4 Task: Completed in 3067 ms
I/flutter (12904): Index: 7 Task: Upload Complete
I/flutter (12904): Index: 8 Task: Upload Complete
I/flutter (12904): Index: 7 Task: Got Download URL
I/flutter (12904): Index: 7 End: 2020-09-22 10:16:21.160583
I/flutter (12904): Index: 7 Task: Completed in 3927 ms
I/flutter (12904): Index: 8 Task: Got Download URL
I/flutter (12904): Index: 8 End: 2020-09-22 10:16:21.241619
I/flutter (12904): Index: 8 Task: Completed in 4007 ms
I/flutter (12904): Index: 9 Task: Upload Complete
I/flutter (12904): Index: 9 Task: Got Download URL
I/flutter (12904): Index: 9 End: 2020-09-22 10:16:22.188572
I/flutter (12904): Index: 9 Task: Completed in 4953 ms
I/flutter (12904): Index: 6 Task: Upload Complete
I/flutter (12904): Index: 6 Task: Got Download URL
I/flutter (12904): Index: 6 End: 2020-09-22 10:16:22.408898
I/flutter (12904): Index: 6 Task: Completed in 5176 ms

@fysoul17
Copy link
Author

fysoul17 commented Sep 22, 2020

@greghesp

Thanks for the information!

Just out of curiosity, are files all the same size?? or is the file on index 6 has the largest size?
If all file sizes are the same, it doesn't seem working simultaneously, it is just happening sequentially by random order. (as you can see each file takes about 900ms to be uploaded. It has to be around 1000~2000ms to be completed for uploading all files if it happened simultaneously - or first and last uploads shouldn't have like 5000ms gap)

I will test it with the latest version anyway! Thanks again for the comment.

@Salakar Salakar added the Stale Issue with no recent activity label Apr 1, 2021
@Salakar
Copy link
Member

Salakar commented Apr 16, 2021

Given the age of this issue and that there's been no further issue reported after this was tested on the new version of Storage I'm going to go ahead and close this issue. Thanks

@Salakar Salakar closed this as completed Apr 16, 2021
@firebase firebase locked and limited conversation to collaborators May 17, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
impact: customer A bug with low impact (e.g. affecting only a few customers or has a workaround). (P3) plugin: storage Stale Issue with no recent activity type: bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants