🐛 Destination GCS: Multi-part upload permissions are not validated by the connection check step

[alafanechere]

While troubleshooting this user's problem I realized that the connection check could be improved to avoid false-positive connection checks.

Current Behavior

On check GCS destination performs a putObject operation to check it has permission to write to the destination bucket:

s3Client.putObject(destinationConfig.getBucketName(), "test", "check-content");

On write the S3StreamTransferManagerHelper is performing a multipart upload.
On GCS storage.multipartUploads.create permission is required to write an object to the bucket, the check only requires storage.objects.create permission.
If the service account has storage.objects.create permission but not storage.multipartUploads.create the check will pass but the sync will fail.

Expected Behavior

check should fail if the service account does not have storage.multipartUploads.create permissions.

Logs

Logs_Mayur_Mahajan_20211220.txt

[alafanechere]

@sherifnada I'm marking this as critical as it's a popular destination.

[sherifnada]

Great catch @alafanechere , thanks for the great error report.

[etsybaev]

Hi @tuliren.
Can I ask you please to share a user with such permissions for testing? I guess Airbyte side only have an access to do that.
By the way, since we use a check method from the destination-gcs connector, I think this issue will be also reproducible in the plain destination-gcs connector.
Thanks.

[sherifnada]

full logs:
logs-3-2 (1).txt

[cgardens]

I don't think storage.multipartuploads.create is enough. I was still encountering permissions issues when attempting to use a service account with Storage Object Admin. According to GCP (here) that role includes this permission. Adding Storage Admin fixed it. I don't know if this is a good solution as that is a pretty powerful and broad permission. It does include storage.multipartUploads.*, which is what I am assuming fixed it (though I did not verify that it was that specific piece of the permission and not something else that was blocking)

[cgardens]

I tried to add information about this to the readme so that if someone at least follows the golden path they should not run into it. #9255

[tuliren]

Hi @tuliren. Can I ask you please to share a user with such permissions for testing? I guess Airbyte side only have an access to do that. By the way, since we use a check method from the destination-gcs connector, I think this issue will be also reproducible in the plain destination-gcs connector. Thanks.

@etsybaev, I have created a new service account: gcs-destination-connector-test@dataline-integration-testing.iam.gserviceaccount.com, and a new role: GCS Destination User which has the following permissions:

storage.multipartUploads.abort
storage.multipartUploads.create
storage.objects.create
storage.objects.delete
storage.objects.get
storage.objects.list

The new service account is only associated with this new role under the dataline-integration-testing project.

A new HMAC key is generated for this service account. I have updated the destination gcs creds entry in LastPass to use this new key.

GCS integration test passes locally with the new key. You should be able to test your change with the test command.

[tuliren]

A second service (gcs-destination-failure-test@dataline-integration-testing.iam.gserviceaccount.com) is created associated with a role (GCS Destination User Without Multipart Permission) without the multipart permission.

Its key is uploaded to LastPass under the entry destination gcs creds (no multipart permission). You can use that config to test the check method.