Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Snowflake destination : Enable DAT tests #12912

Merged
merged 3 commits into from
May 17, 2022
Merged

Snowflake destination : Enable DAT tests #12912

merged 3 commits into from
May 17, 2022

Conversation

DoNotPanicUA
Copy link
Contributor

@DoNotPanicUA DoNotPanicUA commented May 17, 2022 •
edited
Loading

What

The Snowflake destination fails the DAT tests. It means that we can't guarantee full data type compliance.

How

Enable DAT tests and fix all errors.

Recommended reading order

  1. SnowflakeDestinationIntegrationTest.java
  2. SnowflakeTestDataComparator.java
  3. SnowflakeTestSourceOperations.java

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented May 17, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2338731709
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2338731709
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_config/transform.py                         159     31    81%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 543    352    35%
-------------------------------------------------------------------------------------
TOTAL                                                              1333    559    58%

@github-actions github-actions bot added the area/connectors Connector related issues label May 17, 2022
@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented May 17, 2022 •
edited
Loading

/test connector=connectors/destination-mssql

🕑 connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/2339270230
❌ connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/2339270230
🐛 https://gradle.com/s/ayi7pwagtckz6

looks like we have a new bug for the Python team.
our tests are fine.

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented May 17, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-postgres

🕑 connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2339271257
✅ connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2339271257
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_config/transform.py                         159     31    81%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 543    352    35%
-------------------------------------------------------------------------------------
TOTAL                                                              1333    559    58%

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented May 17, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2339273396
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2339273396
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_config/transform.py                         159     31    81%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 543    352    35%
-------------------------------------------------------------------------------------
TOTAL                                                              1333    559    58%

@DoNotPanicUA DoNotPanicUA self-assigned this May 17, 2022
@DoNotPanicUA DoNotPanicUA marked this pull request as ready for review May 17, 2022 14:28
sashaNeshcheret
sashaNeshcheret approved these changes May 17, 2022
View reviewed changes
.../io/airbyte/integrations/destination/snowflake/SnowflakeInsertDestinationAcceptanceTest.java
Comment on lines -121 to -126
// Temporarily disabling the behavior of the ExtendedNameTransformer, see (issue #1785) so we don't
// use quoted names
// if (!tableName.startsWith("\"")) {
// // Currently, Normalization always quote tables identifiers
// tableName = "\"" + tableName + "\"";
// }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to delete the comment?

Copy link
Contributor Author

@DoNotPanicUA DoNotPanicUA May 17, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like an ancient artifact. Probably waiting for archaeologists.
Let's do this!
P.S. Actually it's not relevant because we handle it inside the method resolveIdentifier.

@DoNotPanicUA DoNotPanicUA merged commit 28ccd06 into master May 17, 2022
@DoNotPanicUA DoNotPanicUA deleted the aleonets/DAT-snowflake branch May 17, 2022 15:01
@octavia-squidington-iii octavia-squidington-iii mentioned this pull request May 20, 2022
Merged
suhomud pushed a commit that referenced this pull request May 23, 2022
@DoNotPanicUA @suhomud
Snowflake destination : Enable DAT tests (#12912)
e92ab7c 
* enable new DAT for Snowflake and fix timezone transformation

* Make Snowflake in line with new DAT tests.
+ move method of putting text values into node to util a class
+ format

* Rollback to DestinationAcceptanceTest as a parent class.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sashaNeshcheret sashaNeshcheret sashaNeshcheret approved these changes

@alexandr-shegeda alexandr-shegeda Awaiting requested review from alexandr-shegeda

Assignees

@DoNotPanicUA DoNotPanicUA

Labels
area/connectors Connector related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@DoNotPanicUA @sashaNeshcheret