Enhancement: add tilecache before use of smartcrop to avoid over-computation

[coffeebite]

Using resize to make an image bigger than its original size is very slow. Is this expected?
E.g. I have a 1px x 260px image that I am resizing to 464px x 260px. It takes 30 seconds to do that.

A less dramatic change, e.g. changing a 283x500 image to 540x283 still takes about 350ms.

Is that to be expected? If so, is there a way to make things faster?

[lovell]

This doesn't sound right. Are you able to share the code you're using and a sample image?

[coffeebite]

Thanks.

Image: https://padlet-assets.s3.amazonaws.com/wallpapers/violet.png

Here's my code

const downsize = { width: 512, height: 512 }
// In the ideal world, this changing to buffer shouldn't be required and we should be able to chain the next resize.
// But it looks like the crop results is non integer crop x and y which the second resize doesn't like.
// Buffering, presumably, rounds things before we resize them again.
const buffer = await img.resize(downsize.width, downsize.height).crop(sharp.strategy.attention).toBuffer()
img = sharp(buffer).resize(transforms.width, transforms.height)
img.toBuffer()

[lovell]

Thank you. This sample code does not appear to be standalone (e.g. img is initially undefined) and doesn't use the same dimensions as the original report.

Using your sample image (which is in JPEG rather than PNG format as the extension might suggest) as Buffer-based input with the originally reported problem dimensions and the following code:

const fs = require('fs');
const sharp = require('sharp');

const img = fs.readFileSync('violet.jpg');

console.time('violet-upsample');
sharp(img)
  .resize(464, 260)
  .toBuffer()
  .then(() => console.timeEnd('violet-upsample'));

I see the following locally (Linux, i3-4170):

violet-upsample: 6.681ms

which, at less than 7 milliseconds, is the correct order of magnitude of expected performance.

[coffeebite]

I'm sorry for providing such a shitty code sample.

So it seems like using attention based cropping is what slows things down. 23ms vs 37s. On my machine:

console.time('violet-upsample');
sharp(img)
  .resize(464, 260)
  .toBuffer()
  .then(() => console.timeEnd('violet-upsample'));

// Takes 23ms

and

console.time('violet-upsample');
sharp(img)
  .resize(464, 260)
  .crop(sharp.strategy.attention)
  .toBuffer()
  .then(() => console.timeEnd('violet-upsample'));

// Takes 37s

Any good way to handle it?

[lovell]

What's happening here is that, with target dimensions of 464x260, a 1x256 image has to be enlarged to 464x118784 in order to look for the best area.

This is quite a big image. libvips doesn't store all of it in memory, instead re-computing regions as required. What we're seeing are the effects of over-computation of the enlargement operation (I see >60 billion CPU cycles on this step alone when profiling your code example locally).

We should be able to alleviate some of this in sharp by adding a tile cache just before the smartcrop operation, which trades increased memory usage for performance. I'll take a look.

[lovell]

I see a ~18% improvement with commit 1fec132 on the suit v0.19.0 work-in-progress branch.

Any further performance improvements can only come from libvips itself.

At 1 pixel wide, this example input is somewhat contrived. Perhaps you could make the use of a crop strategy conditional in your code based on input dimensions.

Thanks for reporting this.

[coffeebite]

@lovell Thanks so much.

The example is certainly a special case. I did try using a different strategy: If I have to change a 1x256 image to, say, 500x500, I first crop it to 1x1 using attention strategy and then enalarge that image up to 500x500. However, I ran into #1030

[lovell]

sharp v0.19.0 now available.