Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

put(recursive=True) failes. #358

Closed
roy-ht opened this issue Sep 6, 2020 · 2 comments · Fixed by #360
Closed

put(recursive=True) failes. #358

roy-ht opened this issue Sep 6, 2020 · 2 comments · Fixed by #360

Comments

@roy-ht
Copy link

roy-ht commented Sep 6, 2020

What happened:
Below code causes an error:

Traceback (most recent call last):
  File "temp.py", line 6, in <module>
    fs.put('tempdir', 's3://our_bucket/key/', recursive=True)
  File "/home/jovyan/temp/venv/lib/python3.7/site-packages/fsspec/asyn.py", line 227, in put
    sync(self.loop, self._put, lpaths, rpaths, **kwargs)
  File "/home/jovyan/temp/venv/lib/python3.7/site-packages/fsspec/asyn.py", line 51, in sync
    raise exc.with_traceback(tb)
  File "/home/jovyan/temp/venv/lib/python3.7/site-packages/fsspec/asyn.py", line 35, in f
    result[0] = await future
  File "/home/jovyan/temp/venv/lib/python3.7/site-packages/fsspec/asyn.py", line 214, in _put
    for lpath, rpath in zip(lpaths, rpaths)
  File "/home/jovyan/temp/venv/lib/python3.7/site-packages/s3fs/core.py", line 601, in _put_file
    with open(lpath, 'rb') as f0:
IsADirectoryError: [Errno 21] Is a directory: '/home/jovyan/temp/tempdir'

And sf3s v0.4 does not.

What you expected to happen:

if recursive=True, all directory contents should be copied to remote.

Minimal Complete Verifiable Example:

$ mkdir tempdir
$ touch tempdir/a.txt
import s3fs
fs = s3fs.S3FileSystem()
fs.put('tempdir', 's3://our_bucket/key/', recursive=True)

Anything else we need to know?:

https://github.com/intake/filesystem_spec/blob/66f8bec62c510161b6bd0b93eb0ff143ad34c72f/fsspec/spec.py#L670

put_file treats directory path, but awaitable "_put_file" doesn't.

Environment:

  • Dask version:
  • Python version: 3.7.5
  • Operating System: Linux
  • Install method (conda, pip, source): pip
@martindurant
Copy link
Member

Would you please like to test again with the latest versions?

@roy-ht
Copy link
Author

roy-ht commented Sep 10, 2020

v0.5.1 looks good to me, Thanks!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants