Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FileBinary[Interface] to support large files without loading them in memory unnecessarily #705

Merged
merged 1 commit into from
Feb 12, 2016

Conversation

Seldaek
Copy link
Contributor

@Seldaek Seldaek commented Feb 4, 2016

This works in combination with php-imagine/Imagine#479 to allow me to generate thumbnails for very large images (>1GB) with low memory usage (tested with 50MB) vs needing to increase the memory usage to a few gigs.

It should be BC as far as I can tell.

@makasim
Copy link
Collaborator

makasim commented Feb 5, 2016

I understand the problem, but solution looks a bit hacky. I have to think of a better approach. If nothing comes in my mind I merge this one.

@Seldaek Maybe you already have some ideas?

@Seldaek
Copy link
Contributor Author

Seldaek commented Feb 5, 2016

I don't really see how it is hacky.. sometimes we have files, sometimes we just have data blobs, and converting it all to data blobs is extremely inefficient if a path to a file does the job. The only other way would be to always convert to file and write data blobs in a temp dir, but that's also less efficient and tends to cause new types of issues with permissions and whatnot. The current patch takes care of both of these in the best possible way as far as I can tell.

@lsmith77 lsmith77 added the State: Reviewing This item is being reviewed to determine if it should be accepted. label Feb 8, 2016
@lsmith77
Copy link
Contributor

@makasim can you elaborate about your concerns?

@makasim
Copy link
Collaborator

makasim commented Feb 12, 2016

First of all I am a big fan of things like this but since we have a BC promise we have to develop a good solution and here's what do not like in current approach:

  • It adds if else logic to every place where we have to create image from binary object
  • It works only for filesystem loader.
  • What about other loaders?
  • Rename FileBinary to BinarySteram

@Seldaek
Copy link
Contributor Author

Seldaek commented Feb 12, 2016

  • The if else logic I agree isn't great but I don't see a way around it, it's not even a repeating one that we could easily abstract.
  • I don't see how it can apply to other loaders.. if your files aren't stored locally then you will need to read them into memory somehow before processing them, or we could migrate them to a temp dir and create a FileBinary with that.
  • The name issue if there is a consensus I don't mind renaming it, it's a detail. I don't really see this as a stream though because as mentioned above it can't be guaranteed to work with any php stream. For example if you have a flysystem or s3 stream wrapper and pass that as a file to Imagine, I have no idea what will happen.

makasim added a commit that referenced this pull request Feb 12, 2016
Add FileBinary[Interface] to support large files without loading them in memory unnecessarily
@makasim makasim merged commit 708785a into liip:master Feb 12, 2016
@lsmith77 lsmith77 removed the State: Reviewing This item is being reviewed to determine if it should be accepted. label Feb 12, 2016
@makasim
Copy link
Collaborator

makasim commented Feb 12, 2016

@Seldaek
Copy link
Contributor Author

Seldaek commented Feb 12, 2016

Ok then ;) Thanks

@Seldaek Seldaek deleted the large-file-support branch February 12, 2016 10:42
@makasim
Copy link
Collaborator

makasim commented Feb 12, 2016

Make sense to me. Thank you

@MAXakaWIZARD
Copy link

@Seldaek Last stable release of https://github.com/avalanche123/Imagine was on 19 Sep 2015. As I got it, we need new stable release of imagine to be tagged or install it as dev-master.
UPD: I managed to plug it in via "imagine/imagine": "dev-master#eb3c8d2 as 0.6.x-dev"

@MAXakaWIZARD
Copy link

However, even after plugging optimized version, I am still not able to resize 5Mb image and getting:
Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 71663617 bytes) in /home/proj/vendor/imagine/imagine/lib/Imagine/Gd/Image.php on line 617

@Seldaek
Copy link
Contributor Author

Seldaek commented Mar 26, 2016

Yeah? Maybe something else broke it then, or maybe that doesn't work with GD. I used imagick.

But yes there needs a new tag either way. I'm running on my forks for now as I don't really trust people here since there is no regression test in place I am afraid it'll break in the future. Writing tests would be good but no time.

@MAXakaWIZARD
Copy link

@Seldaek I've used GD, maybe it is the culprit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants