Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run the new DAT(s) against all destinations and find destinations that are not compliant #9444

Closed
edgao opened this issue Jan 12, 2022 · 3 comments · Fixed by #11579, #11965, #11971, #12073 or #12143
Closed

Run the new DAT(s) against all destinations and find destinations that are not compliant #9444

edgao opened this issue Jan 12, 2022 · 3 comments · Fixed by #11579, #11965, #11971, #12073 or #12143
Assignees
@DoNotPanicUA

Comments

@edgao
Copy link
Contributor

edgao commented Jan 12, 2022 •
edited
Loading

Depends on #9443

After the tests are written, run them against all our destinations.

For destinations which are failing because they don't support the optionally-supported types (large numbers and unions), add them in a comment on this issue.

For destinations which don't support one or more of the mandatory types, create an issue to fix them. Add that issue to the "Noncompliant destinations" section of the main epic (https://github.com/airbytehq/airbyte-internal-issues/issues/371).
This was referenced Jan 12, 2022
Closed
Closed
Closed
@alexandr-shegeda alexandr-shegeda moved this to Ready for implementation in GL Roadmap Jan 18, 2022
@oustynova oustynova moved this from Ready for implementation (prioritized) to Implementation in progress in GL Roadmap Mar 4, 2022
@DoNotPanicUA
Copy link
Contributor

DoNotPanicUA commented Mar 10, 2022 •
edited
Loading

Destination name Base test Object test Array test Object + Array test
MySql 🩹 🩹 🩹 🩹
MsSql 🩹 🩹 🩹 🩹
Oracle ✔️ ✔️ ✔️ ✔️
Postgres 🩹 🩹 🩹 🩹
Clickhouse 🩹 🩹 🩹 🩹
MongoDb ✔️ ✔️ ✔️ ✔️
Azure Blob Storage 🩹 ✔️ 🩹 🩹
BigQuery ✔️ ✔️ 🩹 🩹
BigQuery Denormalized ✔️ ✔️ ❓ ❓
Elasticsearch ✔️ ✔️ ✔️ ✔️
GCS ✔️ ✔️ ✔️ ✔️
Kafka ✔️ ✔️ ✔️ ✔️
Keen ✔️ ✔️ ✔️ ✔️
Kinesis ✔️ ✔️ ✔️ ✔️
Local-json ✔️ ✔️ ✔️ ✔️
MariaDb ✔️ ✔️ ✔️ ✔️
MeiliSearch ✔️ ✔️ ✔️ ✔️
Mqtt ✔️ ✔️ ✔️ ✔️
Pulsar ✔️ ✔️ ✔️ ✔️
Pubsub ✔️ ✔️ ✔️ ✔️
Redis ✔️ ✔️ ✔️ ✔️
RedShift ✔️ ✔️ 🩹 🩹
Rockset ✔️ ✔️ ✔️ ✔️
S3 ✔️ ✔️ ✔️ ✔️
Scylla ✔️ ✔️ ✔️ ✔️
Snowflake 🩹 🩹 🩹 🩹

✅ - the destination is fully compatible but the test is not enabled on the Master.
✔️ - the destination is fully compatible with the test. The test is enabled.
❌ - destination is not compatible with the test.
❓ - unable to run this test without significant rework. In the common case, we need to develop proper result data fetch.
🩹 - fixed.

@DoNotPanicUA DoNotPanicUA moved this from Implementation in progress to In review (Airbyte) in GL Roadmap Mar 30, 2022
@DoNotPanicUA
Copy link
Contributor

Hi @edgao ,
I've executed our tests for all Java destinations and now we have a full picture.
Please note that I did some improvements that I mentioned in the #9443 issue. I will create a PR for the common part and will prepare separated PRs to enable tests for destinations where it works fine.
What is the next step? Shall we create tickets for non-compatible destinations and fix them?

@edgao
Copy link
Contributor Author

edgao commented Mar 30, 2022

nice! Yep, next step would be to make tickets and fix the destinations that don't work. Can you add them to the Noncompliant destinations section on the epic? https://github.com/airbytehq/airbyte-internal-issues/issues/371

This was linked to pull requests Apr 11, 2022
Destination acceptence test : improve comparison methods #11579
Merged
Oracle & MongoDb destinations : Enable DAT tests #11965
Merged
ElasticSearch, GCS, Keen, Kafka, Kinesis destinations : Enable DAT tests #11971
Merged
Local_json, Maridb, Meilisearch, Mongodb, Mqtt destinations : Enable DAT tests #12073
Merged
@alexandr-shegeda alexandr-shegeda moved this from In review (Airbyte) to Done in GL Roadmap Apr 18, 2022
@DoNotPanicUA DoNotPanicUA linked a pull request Apr 20, 2022 that will close this issue
Pubsub, Pulsar, Redis, Redshift, Rocket destinations : Enable DAT tests #12143
Merged
@sashaNeshcheret sashaNeshcheret linked a pull request Apr 25, 2022 that will close this issue
Mssql destination: enable DAT tests, use nvarchar and datetime2 by default #12305
Merged
38 tasks
This was linked to pull requests May 4, 2022
Destination Postgres : Enable DAT and fix the data fetch. #12543
Merged
Destination Clickhouse : Enable and fix DAT tests #12566
Merged
@sashaNeshcheret sashaNeshcheret linked a pull request May 12, 2022 that will close this issue
AzureBlobStorage destination: enable DAT tests #12764
Merged
@DoNotPanicUA DoNotPanicUA linked a pull request May 16, 2022 that will close this issue
Scylla destinations : Enable DAT tests #12860
Merged
@DoNotPanicUA DoNotPanicUA mentioned this issue May 24, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DoNotPanicUA DoNotPanicUA

Labels
None yet
Projects
No open projects
GL Roadmap
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Destination acceptence test : improve comparison methods airbytehq/airbyte
Oracle & MongoDb destinations : Enable DAT tests airbytehq/airbyte
ElasticSearch, GCS, Keen, Kafka, Kinesis destinations : Enable DAT tests airbytehq/airbyte
Local_json, Maridb, Meilisearch, Mongodb, Mqtt destinations : Enable DAT tests airbytehq/airbyte
Pubsub, Pulsar, Redis, Redshift, Rocket destinations : Enable DAT tests airbytehq/airbyte
Mssql destination: enable DAT tests, use nvarchar and datetime2 by default airbytehq/airbyte
Destination Postgres : Enable DAT and fix the data fetch. airbytehq/airbyte
Destination Clickhouse : Enable and fix DAT tests airbytehq/airbyte
AzureBlobStorage destination: enable DAT tests airbytehq/airbyte
Scylla destinations : Enable DAT tests airbytehq/airbyte
3 participants
@bleonard @edgao @DoNotPanicUA