Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Destination Postgres : Enable DAT and fix the data fetch. #12543

Merged
merged 3 commits into from
May 4, 2022
Merged

Destination Postgres : Enable DAT and fix the data fetch. #12543

merged 3 commits into from
May 4, 2022

Conversation

DoNotPanicUA
Copy link
Contributor

@DoNotPanicUA DoNotPanicUA commented May 3, 2022 •
edited
Loading

What

The Postgres destination fails the DAT tests. It means that we can't guarantee full data type compliance.

How

Enable DAT tests and fix all errors.

Recommended reading order

  1. JdbcDestinationAcceptanceTest.java
  2. PostgresDestinationAcceptanceTest.java

@DoNotPanicUA
Enable DAT for Postgres and fix the data fetch.
27ae0e9 
Move JDBC abstract part for tests to the JdbcDestinationAcceptanceTest.java
@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented May 3, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2265560705
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2265560705
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     31    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           34      7    79%
normalization/transform_catalog/dbt_macro.py                                                                                       22      7    68%
normalization/transform_catalog/catalog_processor.py                                                                              147     80    46%
normalization/transform_catalog/transform.py                                                                                       61     38    38%
normalization/transform_catalog/stream_processor.py                                                                               534    345    35%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1447    550    62%

@github-actions github-actions bot added the area/connectors Connector related issues label May 3, 2022
@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA DoNotPanicUA self-assigned this May 3, 2022
@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented May 4, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-postgres

🕑 connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2269162952
✅ connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/2269162952
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     31    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           34      7    79%
normalization/transform_catalog/dbt_macro.py                                                                                       22      7    68%
normalization/transform_catalog/catalog_processor.py                                                                              147     80    46%
normalization/transform_catalog/transform.py                                                                                       61     38    38%
normalization/transform_catalog/stream_processor.py                                                                               534    345    35%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1447    550    62%

@DoNotPanicUA DoNotPanicUA marked this pull request as ready for review May 4, 2022 11:03
sashaNeshcheret
sashaNeshcheret approved these changes May 4, 2022
View reviewed changes
Copy link
Contributor

@sashaNeshcheret sashaNeshcheret left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@DoNotPanicUA DoNotPanicUA merged commit ade0056 into master May 4, 2022
@DoNotPanicUA DoNotPanicUA deleted the aleonets/DAT-fix-postgres branch May 4, 2022 11:27
@octavia-squidington-iii octavia-squidington-iii mentioned this pull request May 4, 2022
Merged
@DoNotPanicUA DoNotPanicUA linked an issue May 4, 2022 that may be closed by this pull request
Run the new DAT(s) against all destinations and find destinations that are not compliant #9444
Closed
suhomud pushed a commit that referenced this pull request May 23, 2022
@DoNotPanicUA @suhomud
Destination Postgres : Enable DAT and fix the data fetch. (#12543)
781be94 
* Enable DAT for Postgres and fix the data fetch.
Move JDBC abstract part for tests to the JdbcDestinationAcceptanceTest.java

* Remove unnecessary deserialization + add jsonb to json transformation.

* Remove unnecessary deserialization from ssh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sashaNeshcheret sashaNeshcheret sashaNeshcheret approved these changes

@alexandr-shegeda alexandr-shegeda Awaiting requested review from alexandr-shegeda

Assignees

@DoNotPanicUA DoNotPanicUA

Labels
area/connectors Connector related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Run the new DAT(s) against all destinations and find destinations that are not compliant
2 participants
@DoNotPanicUA @sashaNeshcheret