Speed up glob patterns with a prefix

[JacobHayes]

Resolves the issue described in #223 (comment) by applying @martindurant's first suggestion.

I was experimenting with a path like gs://bucket/partial_folder_name**.csv.gz and these changes returned sub-second versus me canceling after 18 minutes.

The changes are a bit more indirect/wide than I'd like, but I think that's mostly from the async fs and the overridden find not delegating to ls like in the simple spec.

Any suggestions on how to add a test or benchmark around this to ensure it isn't lost? Perhaps I can use VCR or something to assert that the prefix is passed in the API request? I think the VCR recording should be updated accordingly now. The VCR process/docs have gotten much better, so thanks! 😃

[martindurant]

I haven't had time to look in detail yet. Is this essentially the same change as fsspec/s3fs#360 for s3fs? Note that there are some follow-on PRs there, to deal with caching the listings and making sure maxdepth is respected in find.

[martindurant]

On second reading, is it just about providing "partial_folder_name" to find in your example?

[JacobHayes]

On second reading, is it just about providing "partial_folder_name" to find in your example?

Yes, exactly. I have folders like:

• gs://bucket/folder/
• gs://bucket/folder_backfill/
• gs://bucket/some_really_large_folder

and a pattern like gs://bucket/folder**.csv.gz. some_really_large_folder wouldn't match the glob, but the old code wouldn't pass along the prefix to the API call, so all of the objects in some_really_large_folder would also be retrieved and filtered in python.

[martindurant]

OK, got it.
So in this case, you would expect "bucket/folder" and "bucket/folder_backfill" to be in the directory cache, but not "bucket" or "bucket/some_really_large_folder".

[martindurant]

Thank you, this is going in - and congratulations for making vcrpy work for you.