cdc: fix debezium shutdown interruption bug #24166

subodh1810 · 2023-03-16T21:15:04Z

For issue https://github.com/airbytehq/oncall/issues/1596
See this comment for detailed explanation

github-actions · 2023-03-16T21:18:19Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (8)

Connector	Version	Changelog	Publish
`source-alloydb`	`2.0.13`	✅	✅
`source-alloydb-strict-encrypt`	`2.0.13`	🔵 (ignored)	🔵 (ignored)
`source-mssql`	`1.0.8`	✅	✅
`source-mssql-strict-encrypt`	`1.0.8`	🔵 (ignored)	🔵 (ignored)
`source-mysql`	`2.0.11`	✅	✅
`source-mysql-strict-encrypt`	`2.0.11`	🔵 (ignored)	🔵 (ignored)
`source-postgres`	`2.0.13`	✅	✅
`source-postgres-strict-encrypt`	`2.0.13`	🔵 (ignored)	🔵 (ignored)

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

sergio-ropero · 2023-03-22T09:34:59Z

Why don't you just trigger the shutdown in a different thread and let Airbyte main thread to finish the consumption of the queue as usual?
I just did a quick read on the changes and it's super confusing without any details on the PR or comments in the code itself. Do we really need this type of classes: DebeziumShutdownProcedure<ChangeEvent<String, String>>?

...ebezium/src/main/java/io/airbyte/integrations/debezium/internals/DebeziumRecordIterator.java

.../src/test/java/io/airbyte/integrations/debezium/internals/DebeziumShutdownProcedureTest.java

sergio-ropero

Nice work!! hard to tackle with the spaghetti code we have... 🤦

rodireich · 2023-03-27T19:24:40Z

...zium/src/main/java/io/airbyte/integrations/debezium/internals/DebeziumPropertiesManager.java

@@ -55,7 +55,8 @@ protected Properties getDebeziumProperties() {
    props.setProperty("max.queue.size", "8192");

    props.setProperty("errors.max.retries", "10");
-    props.setProperty("errors.retry.delay.initial.ms", "1000");
+    props.setProperty("errors.retry.delay.initial.ms", "300");


Why changing these delays timeouts?

Cause in our Airbyte code we only wait for 60 seconds for Debezium to return a record. So Debezium needs to hurry and restart in case of an error. If it waits for 10 seconds, there is a possibility by the time it restarts, we would ask it to shutdown anyway

rodireich · 2023-03-27T19:28:59Z

...ebezium/src/main/java/io/airbyte/integrations/debezium/internals/DebeziumRecordIterator.java

+    while (!debeziumShutdownProcedure.getRecordsRemainingAfterShutdown().isEmpty()) {
+      final ChangeEvent<String, String> event;
+      try {
+        event = debeziumShutdownProcedure.getRecordsRemainingAfterShutdown().poll(10, TimeUnit.SECONDS);


Wouldn't keep reading from the regular queue make space for any outstanding ChangeEvents after shutdown?
If not we should discard these.

WE cant read form the regular queue cause the reading happens inside the main thread and the main thread pauses once we request shutdown and thus the reading stops. The records can not be discarded.
#24166 (comment)

rodireich · 2023-03-27T19:30:55Z

...ebezium/src/main/java/io/airbyte/integrations/debezium/internals/DebeziumRecordIterator.java

+        continue;
+      }
+      hasSnapshotFinished = hasSnapshotFinished(Jsons.deserialize(event.value()));
+      return event;


Are events received after shutdown still going to be inside the bounds of our target LSN?
If later than we should discard it, no?

We can not/should not discard it cause it's still relevant data. Debezium updates its offset uptill the point it has fetched records so we need to process these records. See this

rodireich · 2023-03-27T20:42:36Z

...zium/src/main/java/io/airbyte/integrations/debezium/internals/DebeziumShutdownProcedure.java

+        try {
+          T event = sourceQueue.poll(10, TimeUnit.SECONDS);
+          if (event != null) {
+            targetQueue.put(event);


Suggestion: use offer to avoid blocking / error in case target queue is at capacity

The targetQueue is not bound by size so it should not be at capacity.

subodh1810 · 2023-03-28T10:01:46Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4541777327
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4541777327
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
=================== 68 passed, 5 skipped in 81.70s (0:01:21) ===================

subodh1810 · 2023-03-28T10:16:22Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/4541918364
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/4541918364
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
================= 22 passed, 6 skipped, 30 warnings in 18.18s ==================

subodh1810 · 2023-03-28T10:16:31Z

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/4541919472
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/4541919472
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
================= 22 passed, 6 skipped, 30 warnings in 17.27s ==================

subodh1810 · 2023-03-28T10:57:10Z

/publish connector=connectors/source-alloydb

🕑 Publishing the following connectors:
connectors/source-alloydb
https://github.com/airbytehq/airbyte/actions/runs/4542255199

Connector	Did it publish?	Were definitions generated?
connectors/source-alloydb	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

subodh1810 · 2023-03-28T10:57:17Z

/publish connector=connectors/source-alloydb-strict-encrypt

🕑 Publishing the following connectors:
connectors/source-alloydb-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/4542256236

Connector	Did it publish?	Were definitions generated?
connectors/source-alloydb-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

subodh1810 · 2023-03-28T10:57:24Z

/publish connector=connectors/source-postgres-strict-encrypt

🕑 Publishing the following connectors:
connectors/source-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/4542257119

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

subodh1810 · 2023-03-28T10:57:30Z

/publish connector=connectors/source-postgres

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/4542257803

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

subodh1810 · 2023-03-28T10:58:05Z

/publish connector=connectors/source-mysql-strict-encrypt

🕑 Publishing the following connectors:
connectors/source-mysql-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/4542261852

Connector	Did it publish?	Were definitions generated?
connectors/source-mysql-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

subodh1810 · 2023-03-28T10:58:11Z

/publish connector=connectors/source-mysql

🕑 Publishing the following connectors:
connectors/source-mysql
https://github.com/airbytehq/airbyte/actions/runs/4542262664

Connector	Did it publish?	Were definitions generated?
connectors/source-mysql	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

subodh1810 · 2023-03-28T10:58:27Z

/publish connector=connectors/source-mssql

🕑 Publishing the following connectors:
connectors/source-mssql
https://github.com/airbytehq/airbyte/actions/runs/4542264576

Connector	Did it publish?	Were definitions generated?

THe publish was successful but action hit rate limit as a result didn't update the comment

Run peter-evans/create-or-update-comment@v1
Error: API rate limit exceeded for installation ID 6913263.

subodh1810 · 2023-03-28T10:58:35Z

/publish connector=connectors/source-mssql-strict-encrypt run-tests=false

🕑 Publishing the following connectors:
connectors/source-mssql-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/4542265582

Connector	Did it publish?	Were definitions generated?
connectors/source-mssql-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

ifun248 · 2023-05-11T13:29:42Z

@subodh1810 Hi Subodh1810, I am testing source MySQL 2.0.17, and find an interesting problem maybe is related to this bug. refs #25988.

cdc: fix debezium shutdown interruption bug

e48492e

subodh1810 self-assigned this Mar 16, 2023

subodh1810 added 6 commits March 17, 2023 03:02

update retry properties

5d3d54a

Merge branch 'master' into fix-cdc-debezium-shutdown-issue

f631bb2

add test for queue transfer

5e51dbc

refactor to simplify

67940ac

Merge branch 'master' into fix-cdc-debezium-shutdown-issue

679e77f

rename parameter

68b9168

subodh1810 requested review from akashkulk, sergio-ropero and rodireich March 22, 2023 09:10

subodh1810 marked this pull request as ready for review March 22, 2023 09:10

sergio-ropero suggested changes Mar 22, 2023

View reviewed changes

subodh1810 added 3 commits March 23, 2023 12:32

Merge branch 'master' into fix-cdc-debezium-shutdown-issue

597259a

fix test

3b81afb

add comments

b624776

subodh1810 requested a review from sergio-ropero March 23, 2023 07:48

add check for engine already closed

b119d84

sergio-ropero approved these changes Mar 23, 2023

View reviewed changes

Merge branch 'master' into fix-cdc-debezium-shutdown-issue

522e318

rodireich reviewed Mar 27, 2023

View reviewed changes

subodh1810 added 2 commits March 28, 2023 14:25

Merge branch 'master' into fix-cdc-debezium-shutdown-issue

5db443a

disable checkpointing

7f7603e

subodh1810 requested a review from a team as a code owner March 28, 2023 10:01

octavia-squidington-iii added area/connectors Connector related issues connectors/source/postgres labels Mar 28, 2023

bump version

1c08168

octavia-squidington-iii and others added 2 commits March 28, 2023 11:06

auto-bump connector version

7d5038a

manual bump

ca9070c

subodh1810 merged commit 29a7e2e into master Mar 28, 2023

subodh1810 deleted the fix-cdc-debezium-shutdown-issue branch March 28, 2023 14:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cdc: fix debezium shutdown interruption bug #24166

cdc: fix debezium shutdown interruption bug #24166

subodh1810 commented Mar 16, 2023 •

edited

Loading

github-actions bot commented Mar 16, 2023 •

edited

Loading

✅ Sources (8)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

sergio-ropero commented Mar 22, 2023

sergio-ropero left a comment •

edited

Loading

rodireich Mar 27, 2023

subodh1810 Mar 28, 2023

rodireich Mar 27, 2023

subodh1810 Mar 28, 2023

rodireich Mar 27, 2023

subodh1810 Mar 28, 2023

rodireich Mar 27, 2023

subodh1810 Mar 28, 2023

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

ifun248 commented May 11, 2023

cdc: fix debezium shutdown interruption bug #24166

cdc: fix debezium shutdown interruption bug #24166

Conversation

subodh1810 commented Mar 16, 2023 • edited Loading

github-actions bot commented Mar 16, 2023 • edited Loading

Affected Connector Report

✅ Sources (8)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

sergio-ropero commented Mar 22, 2023

sergio-ropero left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

subodh1810 commented Mar 28, 2023 • edited by github-actions bot Loading

Build Passed

subodh1810 commented Mar 28, 2023 • edited by github-actions bot Loading

Build Passed

subodh1810 commented Mar 28, 2023 • edited by github-actions bot Loading

Build Passed

subodh1810 commented Mar 28, 2023 • edited by github-actions bot Loading

subodh1810 commented Mar 28, 2023 • edited by github-actions bot Loading

subodh1810 commented Mar 28, 2023 • edited by github-actions bot Loading

subodh1810 commented Mar 28, 2023 • edited by github-actions bot Loading

subodh1810 commented Mar 28, 2023 • edited by github-actions bot Loading

subodh1810 commented Mar 28, 2023 • edited by github-actions bot Loading

subodh1810 commented Mar 28, 2023 • edited Loading

subodh1810 commented Mar 28, 2023 • edited by github-actions bot Loading

ifun248 commented May 11, 2023

subodh1810 commented Mar 16, 2023 •

edited

Loading

github-actions bot commented Mar 16, 2023 •

edited

Loading

sergio-ropero left a comment •

edited

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Mar 28, 2023 •

edited

Loading

subodh1810 commented Mar 28, 2023 •

edited by github-actions bot

Loading