Stateful cache mode for sequential access

[s1ddok]

🚀 The feature

Our usecase is training volumetric videos, so we may have >100 synchronized video streams and we need to sequentially slice each of them frame-by-frame, so slice 100 frames, train a volumetric frame, then slice next frame in every other video.

We'd love to have a dataloader that can efficiently store >100 video decoders in memory while only storing minimal data cache in order to efficiently retrieve next frame (opposite to storing the whole video in RAM).

As NERF/3DGS method are evolving we believe it will be a major usecase.

Motivation, pitch

No response

[NicolasHug]

Thanks for sharing your use case @s1ddok

We'd love to have a dataloader that can efficiently store >100 video decoders in memory while only storing minimal data cache in order to efficiently retrieve next frame (opposite to storing the whole video in RAM).

Have you tried the naive approach of simply creating 100 VideoDecoder objects, i.e. one decoder for each video? Each of the VideoDecoder should only store a reasonable amount of metadata in RAM, not the entire video.

If the memory footprint of each decoder is still too high, then maybe there are opportunities to save some space when using the appoximate mode (#427, CC @scotts)?

[s1ddok]

@NicolasHug API doesn't seem to be optimized for sequential access, rather a random access patterns. I think that inherently should mean excessive seeks unless you are maintaining an internal state and reusing cached data for close frame indices.

do you think torchcodec can benefit from designing a separate API optimized for accessing frames one-by-one?

[NicolasHug]

It is true that we have optimized routines for random access, but that is not at the expense of sequential access!
There are no unnecessary seeks when accessing frames sequentially.

My understanding of your use-case is that you want to do something like this:

all_videos = [...]
all_decoders = [VideoDecoder(video, ...) for video in all_videos]

for i in range(num_frames_to_decode):
    volumetric_frame = [decoder[i] for decoder in all_decoders]  # or decoder.get_frame_at(i)

I encourage you to try, let us know if there are gaps that could be filled in terms of perf or memory usage.