413 Payload Too Large when recursive remove a large directory

[pvanderlinden]

When calling rm(path, recursive=True), the request fails without an actual error response:

_call non-retriable exception: 
Traceback (most recent call last):
  File "/home/paul/anaconda3/envs/antenna/lib/python3.7/site-packages/gcsfs/core.py", line 487, in _call
    validate_response(r, path)
  File "/home/paul/anaconda3/envs/antenna/lib/python3.7/site-packages/gcsfs/core.py", line 132, in validate_response
    raise HttpError({"code": r.status_code})
gcsfs.utils.HttpError

When digging deeper:
<Response [413]>

This happens basically on large directories.

[martindurant]

rm includes a batchsize= parameter, default 20. Can you find a smaller value that works?
Perhaps the code logic is wrong, and you can find out by debugging a little how many files to delete were actually included in the call that failed.

I would appreciate if, while digging, you also helped improve the code to dig out the actual error from the general HttpError case. Note that I have deleted large numbers of files in a single call before.

[pvanderlinden]

I just checked, and the batchsize parameter is only available in the latest version. Unfortunately the latest version is unusable due to #279
Closing this issue though as it has been resolved in the latest version (I think however that 20 is quiet low, this code has ran before with a large amount of files, but I assume it has crossed the threshold now).

[martindurant]

Let's get that issue fixed, then. Did you try #280?

I think however that 20 is quite low

Since we can now send many requests concurrently, there is less of a need to pack them into large batches. Indeed, this very issue shows the problem. I don't know what an optimal batch-size would be...

[pvanderlinden]

I didn't know the new version automatically did many calls at once, that's a great improvement (as well as asyncio support!). I will try out the PR, didn't see that last week.