Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Postgres source: add a check for publications in discover method for CDC #13207

Closed
tuliren opened this issue May 25, 2022 · 1 comment · Fixed by #14447
Closed

Postgres source: add a check for publications in discover method for CDC #13207

tuliren opened this issue May 25, 2022 · 1 comment · Fixed by #14447

Comments

@tuliren
Copy link
Contributor

tuliren commented May 25, 2022

Tell us about the problem you're trying to solve

  • In Investigate the effect of huge WAL logs on Postgres CDC source sync #5837 we concluded that under the CDC mode, all tables and only the tables to be synced should be in the publication.
  • We should add a check in the discover method for this. The check can only be added to the discover method, not the check method, because the check method does not know the catalog.

Describe the solution you’d like

  • Query the pg_publication and pg_publication_tables to see which tables are included in the publication.
  • Limit the streams in the catalog to only those exist in the publication
    • Should we enforce this? Or just provide a warning message?

Describe the alternative you’ve considered or used

N/A

Additional context

N/A

@matt-ripple
Copy link

@tuliren It's no longer possible to sync views with a connection that uses CDC replication. This is causing us issues. The only way around this would be to create two sources for each of our postgres databases (We have 10+).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants