Force legacy if feature flag not set #14694

benmoriceau · 2022-07-13T22:08:24Z

What

for the legacy mode for if the feature flag is not set to true.
Issue : https://github.com/airbytehq/oncall/issues/339

benmoriceau · 2022-07-13T22:47:43Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2666786353

lmossman · 2022-07-13T22:49:06Z

...rs/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSource.java

+    if (!featureFlags.useStreamCapableState()) {
+      return List.of(new AirbyteStateMessage()
+          .withType(AirbyteStateType.LEGACY)
+          .withData(Jsons.emptyObject()));
+    }


@benmoriceau it looks like the non-overriden method does .withData(Jsons.jsonNode(new DbState())), but here you are using .withData(Jsons.emptyObject()):

airbyte/airbyte-integrations/connectors/source-relational-db/src/main/java/io/airbyte/integrations/source/relationaldb/AbstractDbSource.java

Line 551 in 7bff12a

return List.of(new AirbyteStateMessage().withType(AirbyteStateType.LEGACY).withData(Jsons.jsonNode(new DbState())));

Should these be the same?

I think so. I have updated it

…ce-legacy-postgres

benmoriceau · 2022-07-13T23:23:21Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2666925534

lmossman · 2022-07-13T23:59:24Z

...ce-postgres/src/test/java/io/airbyte/integrations/source/postgres/CdcPostgresSourceTest.java

-    assertTrue(streamsInStateAfterFirstSyncCompletion.contains(new StreamDescriptor().withName(MODELS_STREAM_NAME).withNamespace(MODELS_SCHEMA)));*/
-    assertNotNull(stateMessageEmittedAfterFirstSyncCompletion.getData());
+    assertTrue(streamsInStateAfterFirstSyncCompletion.contains(new StreamDescriptor().withName(MODELS_STREAM_NAME).withNamespace(MODELS_SCHEMA)));
+    assertNotNull(stateMessageEmittedAfterFirstSyncCompletion.getData());*/


included this line in the commented out code, because we are doing the same assert on line 411

lmossman · 2022-07-13T23:59:54Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2667056870
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2667056870
🐛 https://gradle.com/s/zdsjgvzy5odly

Build Failed

Test summary info:

Could not find result summary

benmoriceau · 2022-07-14T01:59:56Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2667460329
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2667460329
🐛 https://gradle.com/s/hbbjxo6gykeje

Build Failed

Test summary info:

Could not find result summary

subodh1810 · 2022-07-14T07:49:47Z

I have created a separate commit #14703 to disable the test newTableSnapshotTest cause its testing addition of a new table in the catalog and then seeing how the connector behaves in CDC mode. But this addition without resetting the state can only happen in GLOBAL state mode and not in LEGACY so as a safety measure the connector doesn't identify new table in the catalog when the state is LEGACY. The addition of new tables in catalog without causing the state to reset can only happen when the connector is in GLOBAL state mode and not in LEGACY state mode. (edited)

benmoriceau · 2022-07-14T14:32:55Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2671048908
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2671048908
No Python unittests run

Build Passed

Test summary info:

All Passed

benmoriceau · 2022-07-14T15:49:32Z

/publish connector=connectors/source-postgres

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/2671520470

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

benmoriceau · 2022-07-14T18:01:51Z

/publish connector=connectors/source-postgres-strict-encrypt

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Force legacy if feature flag not set

0923865

github-actions bot added the area/connectors Connector related issues label Jul 13, 2022

octavia-squidington-iii added the connectors/source/postgres label Jul 13, 2022

benmoriceau requested review from gosusnp and lmossman July 13, 2022 22:17

benmoriceau added 2 commits July 13, 2022 15:42

Fix acceptance test

0536780

bump version

9eb695b

github-actions bot added the area/documentation Improvements or additions to documentation label Jul 13, 2022

octavia-squidington-iii added the connectors/source/postgres-strict-encrypt label Jul 13, 2022

lmossman reviewed Jul 13, 2022

View reviewed changes

PR comments

979b93c

benmoriceau requested a review from lmossman July 13, 2022 23:05

Revert Integration test

4975382

benmoriceau force-pushed the bmoric/force-legacy-postgres branch from 4cdb0b5 to 4975382 Compare July 13, 2022 23:07

benmoriceau added 2 commits July 13, 2022 16:22

Update some integration test

c9bb9ff

Merge branch 'master' of github.com:airbytehq/airbyte into bmoric/for…

ff7257a

…ce-legacy-postgres

lmossman added 2 commits July 13, 2022 16:57

asset legacy type and data is correct in source tests

c1daa0e

Merge branch 'master' into bmoric/force-legacy-postgres

ff86c2f

lmossman reviewed Jul 13, 2022

View reviewed changes

benmoriceau added 2 commits July 13, 2022 18:59

better diff

5024a99

Merge branch 'master' into bmoric/force-legacy-postgres

84b7c8b

subodh1810 mentioned this pull request Jul 14, 2022

disable test cause its not valid for LEGACY state #14703

Merged

disable test cause its not valid for LEGACY state (#14703)

9d99993

gosusnp approved these changes Jul 14, 2022

View reviewed changes

benmoriceau marked this pull request as ready for review July 14, 2022 15:50

benmoriceau requested a review from a team as a code owner July 14, 2022 15:50

benmoriceau requested a review from lmossman July 14, 2022 15:50

grishick approved these changes Jul 14, 2022

View reviewed changes

auto-bump connector version

6685e43

octavia-squidington-iii temporarily deployed to more-secrets July 14, 2022 17:13 Inactive

benmoriceau merged commit 939afb7 into master Jul 14, 2022

benmoriceau deleted the bmoric/force-legacy-postgres branch July 14, 2022 17:53

octavia-squidington-iii mentioned this pull request Jul 23, 2022

Bump Airbyte version from 0.39.37-alpha to 0.39.38-alpha #14976

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Force legacy if feature flag not set #14694

Force legacy if feature flag not set #14694

benmoriceau commented Jul 13, 2022 •

edited by subodh1810

Loading

benmoriceau commented Jul 13, 2022 •

edited by github-actions bot

Loading

lmossman Jul 13, 2022

benmoriceau Jul 13, 2022

benmoriceau commented Jul 13, 2022 •

edited by github-actions bot

Loading

lmossman Jul 13, 2022

lmossman commented Jul 13, 2022 •

edited by github-actions bot

Loading

benmoriceau commented Jul 14, 2022 •

edited by github-actions bot

Loading

subodh1810 commented Jul 14, 2022

benmoriceau commented Jul 14, 2022 •

edited by github-actions bot

Loading

benmoriceau commented Jul 14, 2022 •

edited by github-actions bot

Loading

benmoriceau commented Jul 14, 2022 •

edited by github-actions bot

Loading

Force legacy if feature flag not set #14694

Force legacy if feature flag not set #14694

Conversation

benmoriceau commented Jul 13, 2022 • edited by subodh1810 Loading

What

benmoriceau commented Jul 13, 2022 • edited by github-actions bot Loading

lmossman Jul 13, 2022

Choose a reason for hiding this comment

benmoriceau Jul 13, 2022

Choose a reason for hiding this comment

benmoriceau commented Jul 13, 2022 • edited by github-actions bot Loading

lmossman Jul 13, 2022

Choose a reason for hiding this comment

lmossman commented Jul 13, 2022 • edited by github-actions bot Loading

Build Failed

benmoriceau commented Jul 14, 2022 • edited by github-actions bot Loading

Build Failed

subodh1810 commented Jul 14, 2022

benmoriceau commented Jul 14, 2022 • edited by github-actions bot Loading

Build Passed

benmoriceau commented Jul 14, 2022 • edited by github-actions bot Loading

benmoriceau commented Jul 14, 2022 • edited by github-actions bot Loading

benmoriceau commented Jul 13, 2022 •

edited by subodh1810

Loading

benmoriceau commented Jul 13, 2022 •

edited by github-actions bot

Loading

benmoriceau commented Jul 13, 2022 •

edited by github-actions bot

Loading

lmossman commented Jul 13, 2022 •

edited by github-actions bot

Loading

benmoriceau commented Jul 14, 2022 •

edited by github-actions bot

Loading

benmoriceau commented Jul 14, 2022 •

edited by github-actions bot

Loading

benmoriceau commented Jul 14, 2022 •

edited by github-actions bot

Loading

benmoriceau commented Jul 14, 2022 •

edited by github-actions bot

Loading