-
-
Notifications
You must be signed in to change notification settings - Fork 381
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix #684 Abort S3 MultipartUpload if exception is raised #752
Conversation
# https://github.com/RaRe-Technologies/smart_open/issues/684 | ||
# If exception is raised multipart writer should be terminated to abort file uploading to S3 | ||
# If not incomplete file will be saved in bucket | ||
if type(inner) == MultipartWriter and ex: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi 👋 can this be generalized such that it doesn't just handle S3, but also the other adapters like Azure?
Azure exception handling ref #783
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
e.g. by refactoring tweak_close
to become tweak_exit
, that patches .__exit__()
instead of .close()
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
opened #786
What do I need to do to install this? I'm running into situation where larger transfers fail with this error:
|
|
Closing, thank you |
Motivation
AWS Supports multipart upload of a file. I big file split by chunks uploaded one by one and on S3 concatenated again.
In our case, if an exception is raised while processing one of the parts we have to abort uploading to avoid corrupted file creation.
Tests
test_write_gz_with_error