Handle exceptions during writes to Azure #783
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ddelange
changed the title
ddelange
changed the title
This was referenced Oct 2, 2023
ddelange
commented
Oct 5, 2023
| raise ValueError | ||
| except ValueError: | ||
| # FakeBlobClient.commit_block_list was not called | ||
| self.assertGreater(len(blob_client._staged_contents), 0) |
There was a problem hiding this comment.
should add a similar test including compression, after #786 merges
…open into patch-2 * 'develop' of https://github.com/RaRe-Technologies/smart_open: Propagate __exit__ call to underlying filestream (piskvorky#786) Retry finalizing multipart s3 upload (piskvorky#785) Fix `KeyError: 'ContentRange'` when received full content from S3 (piskvorky#789) Add support for SSH connection via aliases from `~/.ssh/config` (piskvorky#790) Make calls to smart_open.open() for GCS 1000x faster by avoiding unnecessary GCS API call (piskvorky#788) Add zstandard compression feature (piskvorky#801) Support moto 4 & 5 (piskvorky#802) Secure the connection using SSL when connecting to the FTPS server (piskvorky#793) upgrade dev status classifier to stable (piskvorky#798) Fix formatting of python code (piskvorky#795)
ddelange
commented
Feb 21, 2024
| raise ValueError | ||
| except ValueError: | ||
| # FakeBlobClient.commit_block_list was not called | ||
| self.assertGreater(len(blob_client._staged_contents), 0) |
There was a problem hiding this comment.
this one still fails, because this test case covers the TextIOWrapper case which calls close on the azure binary io before our FileLikeProxy had the chance to call __exit__ on the azure binary io (which circumvents close when __exit__ is called during exception handling ref #786)
|
@mpenkov one more PR green :) added some documentation and extra tests |
…open into patch-2 * 'develop' of https://github.com/RaRe-Technologies/smart_open: fix test, for real this time update integration test Add advanced usage sections to README.rst (piskvorky#741) Add logic for handling large files in MultipartWriter uploads to s3 (piskvorky#796) Fix __str__ method in SinglepartWriter (piskvorky#791)
|
I've also opened an analogous gcs issue googleapis/python-storage#1228 |
|
Thank you for your work! |
|
Thanks for the reviews! 🙏 |
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Title
Avoid lingering incomplete uploads on Azure storage blob in case of upload interrupt/failure.
Analogous to the
s3.Writer.__exit__implementation.Motivation
Housekeeping, expected behaviour, cost savings.
If you're adding a new feature, then consider opening a ticket and discussing it with the maintainers before you actually do the hard work.
Tests
If you're fixing a bug, consider test-driven development:
If you're implementing a new feature, include unit tests for it.
Make sure all existing unit tests pass.
You can run them locally using:
If there are any failures, please fix them before creating the PR (or mark it as WIP, see below).
Work in progress
If you're still working on your PR, include "WIP" in the title.
We'll skip reviewing it for the time being.
Once you're ready to review, remove the "WIP" from the title, and ping one of the maintainers (e.g. mpenkov).
Checklist
Before you create the PR, please make sure you have:
Workflow
Please avoid rebasing and force-pushing to the branch of the PR once a review is in progress.
Rebasing can make your commits look a bit cleaner, but it also makes life more difficult from the reviewer, because they are no longer able to distinguish between code that has already been reviewed, and unreviewed code.