Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update support for deleting files from non-AWS S3 endpoints #1163

Closed
wants to merge 22 commits into from
Closed

Update support for deleting files from non-AWS S3 endpoints #1163

wants to merge 22 commits into from

Conversation

lbeckman314
Copy link
Contributor

Link to JIRA ticket if there is one: N/A

New Features

  • Adds support for deleting files from non-AWS S3-compatible endpoints (e.g. MinIO, Wasabi, Ceph).

Breaking Changes

  • None

Bug Fixes

  • Fixes the "The AWS Access Key Id you provided does not exist" error when attempting to delete a file from a non-AWS bucket.

Improvements

  • Allows for Gen3 administrators to have greater flexibility in choosing their preferred data storage, while keeping the AWS S3 endpoint as the "default" storage option.

Dependency updates

  • None

Deployment changes

  • None

BinamB and others added 20 commits October 16, 2023 14:26
@BinamB
fix non-aws s3 bucket stuff
b86a8bc
@BinamB
add imports
91ea49f
@BinamB
Merge branch 'master' into feat/multi_part_bucket_param
fc7b446
@BinamB
Merge branch 'master' into feat/multi_part_bucket_param
4ba4233
@BinamB
Resolve some reviews
4c65be6
@Avantol13
Merge branch 'master' into feat/multi_part_bucket_param
b9c6efb
@lbeckman314
Update support for multipart uploads to non-AWS S3 endpoints
4db98a9
@lbeckman314
Add initial endpoint_url parameter to boto3 client
bb0883a
@lbeckman314
Add initial multi-client support to Boto Manager
6703a91
@lbeckman314
Remove debugging statements
3839a4d
@lbeckman314
Revert changes to indexd data blueprint
fbcb64c
@lbeckman314
Revert changes to multipart_upload.py
36e86cd
@lbeckman314
Revert change to s3_client call in Boto Manager
83032c4
@lbeckman314
Update BotoManager to use specified credentials for each bucket
915edd2
@lbeckman314
Resolve unit tests
7893d17
@lbeckman314
Use correct s3_client to delete files
f64baff
@lbeckman314
Add unit tests for creating S3 clients and deleting files
51f83fd
@lbeckman314
Update Boto Manager
d7f91d5
@lbeckman314
Add minor fix to resolve S3 Mock test
acfe08f
@lbeckman314
Merge remote-tracking branch 'upstream/master' into feat/s3-delete-files
41b20eb
@lbeckman314 lbeckman314 changed the title Feat/s3 delete files Update support for deleting files from non-AWS S3 endpoints Jul 19, 2024
BinamB
BinamB previously approved these changes Jul 31, 2024
View reviewed changes
@lbeckman314 lbeckman314 dismissed BinamB’s stale review August 7, 2024 19:29

The merge-base changed after approval.

@BinamB
Copy link
Contributor

BinamB commented Aug 7, 2024

Merged with #1171

@BinamB BinamB closed this Aug 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BinamB BinamB BinamB left review comments

Assignees
No one assigned
Labels
@requires-fence
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@lbeckman314 @BinamB @Avantol13