zstd decompression using "--patch-from" cost memory too high (when the dict is a big file)

[rongekuta]

Is your feature request related to a problem? Please describe.

zstd -d -M128MB --patch-from=d:/code-shared/apks/wzry_old.apk d:/code-shared/apks/wzry.diff -o d:/code-shared/apks/wzry_new_d.apk

when i use zstd (--patch-from) to decompress a patch file using a big dict, i find "-M#" option is not valid, the big dict is load to memory all, and cost too many memory. i search code for reason, and find:
https://github.com/facebook/zstd/blob/v1.5.2/programs/fileio.c#L2019

these code cause "-M" memory limit is not work under "--patch-from" mode

Describe the solution you'd like
decompression under a big dict with "--patch-from" is also be limited memory

Describe alternatives you've considered
the big dict also be cache from file system, not load to memory totally

Additional context

[JiapengLi]

a workaround is to use swap to bypass this limitation.

This is what i do the test it under a docker container.:

Host enable a 4G swap area:

dd if=/dev/zero of=/swapfile bs=1M count=4096
mkswap /swapfile 
swapon /swapfile
swapon --show
free

docker compose file to creat a container

version: "3"
services:
  zstd:
    restart: always
    image: some-image-contain-zstd
    container_name: zstd
    init: true
    privileged: true
    network_mode: host
    ulimits:
      nofile:
        soft: 64000
        hard: 64000
    deploy:
      resources:
        limits:
          memory: 512M
        reservations:
          memory: 128M
    memswap_limit: 6G

to test

docker exec -it zstd bash
> zstd -vv --memory=some_valueM --patch-from file1 file1-2.diff -o file2

It is very recommended to use zstd patch-from under docker if your file is very large, this can avoid zstd drain your memory and make bad things happen.