-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Source-postgres] Set default cursor value for cdc mode #27442
Conversation
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
Affected Connector ReportNOTE
|
Connector | Version | Changelog | Publish |
---|---|---|---|
source-alloydb |
2.0.28 |
✅ | ✅ |
source-alloydb-strict-encrypt |
2.0.28 |
🔵 (ignored) |
🔵 (ignored) |
source-postgres-strict-encrypt |
2.0.33 |
🔵 (ignored) |
🔵 (ignored) |
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Destinations (0)
Connector | Version | Changelog | Publish |
---|
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Other Modules (0)
Actionable Items
(click to expand)
Category | Status | Actionable Item |
---|---|---|
Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
Publish | ⚠ not in seed |
The connector is not in the cloud or oss registry, so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that you have added a metadata.yaml file and the expected registries are enabled. |
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-alloydb-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check. | ✅ |
Connector version increment check. | ❌ |
QA checks | ✅ |
Build connector tar | ✅ |
Build source-alloydb-strict-encrypt docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-alloydb-strict-encrypt test
Coverage report for source-postgres
|
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-postgres-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check. | ✅ |
Connector version increment check. | ❌ |
QA checks | ✅ |
Build connector tar | ✅ |
Build source-postgres-strict-encrypt docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-postgres-strict-encrypt test
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-mssql/metadata.yaml | ✅ |
Connector version semver check. | ✅ |
Connector version increment check. | ❌ |
QA checks | ✅ |
Build connector tar | ✅ |
Build source-mssql docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-mssql test
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-alloydb/metadata.yaml | ✅ |
Connector version semver check. | ✅ |
Connector version increment check. | ❌ |
QA checks | ✅ |
Build connector tar | ✅ |
Build source-alloydb docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-alloydb test
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-alloydb-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check. | ✅ |
Connector version increment check. | ❌ |
QA checks | ✅ |
Build connector tar | ✅ |
Build source-alloydb-strict-encrypt docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-alloydb-strict-encrypt test
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-mssql/metadata.yaml | ✅ |
Connector version semver check. | ❌ |
Connector version increment check. | ❌ |
QA checks | ✅ |
Build connector tar | ✅ |
Build source-mssql docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-mssql test
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check. | ✅ |
Connector version increment check. | ❌ |
QA checks | ✅ |
Build connector tar | ✅ |
Build source-mysql-strict-encrypt docker image for platform linux/x86_64 | ✅ |
Unit tests | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-mysql-strict-encrypt test
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml | ✅ |
Connector version semver check. | ✅ |
Connector version increment check. | ❌ |
QA checks | ✅ |
Build connector tar | ✅ |
Build source-postgres docker image for platform linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-postgres test
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So just to confirm, _ab_cdc_lsn
field and associated value is already present in the data that we serialize, and that is why we don't need to add this field as we reading data?
@prateekmukhedkar yes, we are currently emitting the lsn value in the data we serialize so we don't need to add a new column. |
running legacy normalization tests in #27670 (comment). Will approve once those look good. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
tests are green on #27670.
thanks for putting this together so quickly!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
THis looks fine. I just want you to carry out few tests
- Create a postgres source cdc connector on an old version with a destination that supports normalization (snowflake), run a sync and then change image version of the connector to use changes from this PR and then run a sync and make sure everything works.
- Do everything as mentioned in step 1 but this time refresh the schema as well.
- Create a connector on this new version and then roll back to the old version and see what happens (both with and without refresh schema)
48b836d
to
052a5ae
Compare
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build source-postgres docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-postgres:integrationTest | ✅ |
Acceptance tests | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-postgres test
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build source-mysql docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-mysql:integrationTest | ❌ |
Acceptance tests | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-mysql test
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-postgres-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build source-postgres-strict-encrypt docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-postgres-strict-encrypt:integrationTest | ✅ |
Acceptance tests | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-postgres-strict-encrypt test
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build source-mysql-strict-encrypt docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-mysql-strict-encrypt:integrationTest | ❌ |
Acceptance tests | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-mysql-strict-encrypt test
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-alloydb-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build source-alloydb-strict-encrypt docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-alloydb-strict-encrypt:integrationTest | ✅ |
Acceptance tests | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-alloydb-strict-encrypt test
b661b1f
to
54f4a11
Compare
/legacy-test connector=connectors/source-postgres
Build PassedTest summary info:
|
/legacy-test connector=connectors/source-postgres-strict-encrypt
Build PassedTest summary info:
|
/legacy-test connector=connectors/source-alloydb
Build PassedTest summary info:
|
/test connector=connectors/source-alloydb-strict-encrypt
|
/legacy-test connector=connectors/source-alloydb-strict-encrypt
Build PassedTest summary info:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Didn't review much on the tests side, leaving that to the sources team.
From ops side looks good, seeing breaking changes checks correctly indicated as skipped in the test logs for the affected version:
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:624: Backward compatibility tests are disabled for version 2.1.1.
Should make sure acceptance tests are passing for the connectors to be published - it looks like they are.
Build failure for destination-snowflake is also failing on master and this PR does not touch it, so should be ok to release.
.map(PostgresCatalogHelper::setIncrementalToSourceDefined) | ||
.map(PostgresCatalogHelper::setDefaultCursorFieldForCdc) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍🏼 only for cdc, we are setting the default cursor.
/approve-and-merge reason="All CI steps already passed. And manual tests done to verify. Author: Duy Nguyen, copilot=Ed Gao" |
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build source-mysql-strict-encrypt docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:source-mysql-strict-encrypt:integrationTest | ❌ |
Acceptance tests | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-mysql-strict-encrypt test
) * use LSN as default cursor for postgres CDC * Fixed static constant * Set lsn default cursor value for postgres sync * Bumped metadata and dockerfile versions * Disable acceptance backwards compatibility discovery test as this is a breaking change --------- Co-authored-by: Conor <cpdeethree@users.noreply.github.com>
What
26807
To prepare for Destination v2, Cdc-enabled DB sources need to have a pre-defined cursor for normalization to operate properly.
How
During the discover phase,

source_defined_cursor
is set totrue
, and_ab_cdc_lsn
is selected as the predefined cursor. NOTE: This value could change if we believe it's not unique.AirbyteStateMessage:
Recommended reading order
PostgresCatalogHelper.java
PostgresSource.java
🚨 User Impact 🚨
New CDC syncs that have refreshed their source schema will see this cursor field be chosen. Customers who reset their data will continue to see old behavior until their source goes through the
discover()
phase again.