Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🎉 Postgres source: retrieve only the tables in the publication under cdc mode #14447

Merged
merged 14 commits into from
Jul 10, 2022
Merged

🎉 Postgres source: retrieve only the tables in the publication under cdc mode #14447

merged 14 commits into from
Jul 10, 2022

Conversation

tuliren
Copy link
Contributor

@tuliren tuliren commented Jul 6, 2022

What

@github-actions github-actions bot added the area/connectors Connector related issues label Jul 6, 2022
@tuliren tuliren changed the title Liren/check publication in discover 🎉 Source postgres: retrieve only the tables in the publication under cdc mode Jul 6, 2022
tuliren
tuliren commented Jul 6, 2022
View reviewed changes
...rs/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSource.java
}

return stream;
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These methods related to CDC catalog manipulation are moved to PostgresCdcCatalogHelper.

tuliren
tuliren commented Jul 6, 2022
View reviewed changes
...rs/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSource.java
&& config.get("replication_method").hasNonNull("publication");
LOGGER.info("using CDC: {}", isCdc);
return isCdc;
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is moved to PostgresUtils.

@tuliren tuliren marked this pull request as ready for review July 6, 2022 13:33
@tuliren
Copy link
Contributor Author

tuliren commented Jul 6, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2623148411
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2623148411
No Python unittests run

Build Passed

Test summary info:

All Passed

@tuliren tuliren requested a review from subodh1810 July 6, 2022 13:33
@tuliren tuliren changed the title 🎉 Source postgres: retrieve only the tables in the publication under cdc mode 🎉 Postgres source: retrieve only the tables in the publication under cdc mode Jul 6, 2022
@tuliren tuliren requested a review from edgao July 6, 2022 14:21
@tuliren tuliren requested a review from a team as a code owner July 9, 2022 17:45
@tuliren
Copy link
Contributor Author

tuliren commented Jul 9, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2641892558
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2641892558
No Python unittests run

Build Passed

Test summary info:

All Passed

@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Jul 10, 2022
@tuliren
Copy link
Contributor Author

tuliren commented Jul 10, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/source-postgres

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/2643256971


Connector Did it publish? Were definitions generated?
connectors/source-postgres ✅ ✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@tuliren
Copy link
Contributor Author

tuliren commented Jul 10, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/source-postgres-strict-encrypt

🕑 Publishing the following connectors:
connectors/source-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/2643257700


Connector Did it publish? Were definitions generated?
connectors/source-postgres-strict-encrypt ✅ ❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@tuliren tuliren merged commit 078f5fc into master Jul 10, 2022
@tuliren tuliren deleted the liren/check-publication-in-discover branch July 10, 2022 04:54
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets July 10, 2022 04:55 Inactive
@subodh1810 subodh1810 mentioned this pull request Jul 12, 2022
Merged
@octavia-squidington-iii octavia-squidington-iii mentioned this pull request Jul 13, 2022
Merged
@tuliren tuliren mentioned this pull request Sep 14, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@subodh1810 subodh1810 Awaiting requested review from subodh1810

@edgao edgao Awaiting requested review from edgao

Assignees
No one assigned
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/postgres connectors/source/postgres-strict-encrypt connectors/source/relational-db
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Postgres source: add a check for publications in discover method for CDC
2 participants
@tuliren @octavia-squidington-iii