clarify staging setup guide for bq destination #9255
Conversation
github-actions
bot
added
the
area/documentation
| * See [this](https://cloud.google.com/storage/docs/authentication/hmackeys) on how to generate an access key. | ||
| * We recommend creating an Airbyte-specific user or service account. This user or account will require read and write permissions to objects in the bucket. | ||
| * See [this](https://cloud.google.com/storage/docs/authentication/managing-hmackeys) on how to generate an access key. For more information on hmac keys please reference the [GCP docs](https://cloud.google.com/storage/docs/authentication/hmackeys) | ||
| * We recommend creating an Airbyte-specific user or service account. This user or account will require the following permissions for the bucket: `Storage Object Admin`. You can set those by going to the permissions tab in the GCS bucket and adding the appropriate the email address of the service account or user and adding the aforementioned permissions. |
There was a problem hiding this comment.
is there a reason we didn't originally specify the the exact permissions needed? the GCP permission model has so many permissions here that seem vaguely related. it took me a few tries to get the right one. maybe there's another combination of Legacy Storage Reader, etc, that works as well? If so maybe we should document it as well.
| * See [this](https://cloud.google.com/storage/docs/authentication/hmackeys) on how to generate an access key. | ||
| * We recommend creating an Airbyte-specific user or service account. This user or account will require read and write permissions to objects in the bucket. | ||
| * See [this](https://cloud.google.com/storage/docs/authentication/managing-hmackeys) on how to generate an access key. For more information on hmac keys please reference the [GCP docs](https://cloud.google.com/storage/docs/authentication/hmackeys) | ||
| * We recommend creating an Airbyte-specific user or service account. This user or account will require the following permissions for the bucket: `Storage Object Admin` and `Storage Admin`. You can set those by going to the permissions tab in the GCS bucket and adding the appropriate the email address of the service account or user and adding the aforementioned permissions. |
There was a problem hiding this comment.
There was a problem hiding this comment.
Hi ✋ what was the error you saw after adding the storage.multipartUploads.create permission but it still didnt work? I'm going to guess it was something about not seeing a bucket, and the actual permission you needed was storage.buckets.list. Storage Admin probably wouldnt be the best thing to suggest people give to this SA.
I've always felt that the default BigQuery/Storage Roles GCP creates for users dont include some basic permissions I think they should. And then people end up granting an Admin role because of this. I usually create a custom role "Storage Viewer" with this:
And use it in combination with one of the Storage Object _ roles
There was a problem hiding this comment.
Are you okay with me publishing this as is or is there a more limited permission that you know works?
I am fine with merging this PR as is. I don't know a more limited permission that works.
Update: Actually I second Noah and it would be helpful to post the error message after adding the multipart upload permission. I guess many users won't be comfortable with giving out admin permission just for a staging bucket. It's totally fine if you don't have bandwidth to do that. In that case, we can merge this PR as is, but we should create follow-up ticket to look into this problem.
|
A smaller set of required permissions are: I will update this PR accordingly. |
karinakuz
added
connectors/destinations-warehouse
and removed
connectors/destinations-api
labels
What
How