Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

📖 Add service account doc to gcs connector readme #9347

Merged
merged 2 commits into from
Jan 6, 2022
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions airbyte-integrations/connectors/destination-gcs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,35 @@ As a community contributor, you can follow these steps to run integration tests.
- Access the `destination gcs creds` secrets on Last Pass, and put it in `sample_secrets/config.json`.
- Rename the directory from `sample_secrets` to `secrets`.

### GCP Service Account for Testing
Two service accounts have been created in our GCP for testing this destination. Both of them have access to Cloud Storage through HMAC keys. The keys are persisted together with the connector integration test credentials in LastPass.

- Account: `gcs-destination-connector-test@dataline-integration-testing.iam.gserviceaccount.com`
- This account has the required permission to pass the integration test. Note that the uploader needs `storage.multipartUploads` permissions, which may not be intuitive.
- Role: `GCS Destination User`
- Permissions:
```
storage.multipartUploads.abort
storage.multipartUploads.create
storage.objects.create
storage.objects.delete
storage.objects.get
storage.objects.list
```
- LastPass entry: `destination gcs creds`

- Account: `gcs-destination-failure-test@dataline-integration-testing.iam.gserviceaccount.com`
- This account does not have the `storage.multipartUploads` permissions, and will fail the integration test. The purpose of this account is to test that the `check` command can correctly detect the lack of these permissions and return an error message.
- Role: `GCS Destination User Without Multipart Permission`
- Permissions:
```
storage.objects.create
storage.objects.delete
storage.objects.get
storage.objects.list
```
- LastPass entry: `destination gcs creds (no multipart permission)`

## Add New Output Format
- Add a new enum in `S3Format`.
- Modify `spec.json` to specify the configuration of this new format.
Expand Down