Fix appending of random directory within find() method

[gutzbenj]

At #506 we've seen that when using the HTTPFileSystem's find method, it appears that the method would randomly add one of the found directories to the returned files listing. This is due to conflicting variable naming here:
https://github.com/intake/filesystem_spec/blob/a5bb95155b450ec2443872a45a4a58ce90748297/fsspec/spec.py#L430-L437

To fix this I renamed resulting path from self.walk() to path_. See here:
https://github.com/earthobservations/filesystem_spec/blob/e3956827181393b30a57ab6d11ede52931a4dec8/fsspec/spec.py#L430-L437

[martindurant]

Thanks for finding this!

I expect this didn't show up with existing tests of find because, unlike the HTTP case, we can filter out directories easily, right? It would be nice to have a test for HTTP that would have shown the failure, but is now fixed after this change.

[martindurant]

Fixes #506

[amotl]

Maybe just phrase it like this?

Suggested change

	for path_, dirs, files in self.walk(path, maxdepth, detail=True, **kwargs):
	for _, dirs, files in self.walk(path, maxdepth, detail=True, **kwargs):

[gutzbenj]

Dear Martin,

I added two tests for the FTP and HTTP filesystem querying the Debian repo at [1]. As you can see from the test

• the url which was queried is included in HTTPs result while for FTP this one is missing
• for the case of HTTP this one has found the additional filter hrefs at this directory
• the FTP method returns the relative paths

I propose that the FTP method is aligned with the one from HTTP by adding the base url to the found files thus returning the full filepath. Would you agree on this point?

[1] https://ftp.fau.de/debian-cd/current/amd64/log/success/

[martindurant]

I propose that the FTP method is aligned with the one from HTTP by adding the base url to the found files thus returning the full filepath. Would you agree on this point?

I don't think I do. In the case of FTP, the host/port are arguments to the class, so each set of arguments is an independent file-system with a set of paths within it (which may depend on the specific user info given) and a single persistent connection. Conversely, HTTP sees the internet as a single filesystem, and makes new connections for every access. Although it's true that you can phrase a unique FTP path like "ftp://user:pass@host:port/path", this is for convenience only. Obviously for a case like this, where you have FTP-on-HTTP, the distinction is blurred.

for the case of HTTP this one has found the additional filter hrefs at this directory

I suppose this is OK, though one wonders what these things are, and whether they are likely to stay the same in the future.

[martindurant]

Ah, those hrefs are new sorted views of the same directory. There is no way to know that they are not new content.

Ready to merge now?

[martindurant]

Fixes fsspec/s3fs#393

[gutzbenj]

Yes, ready.