🚨 Add ability to enforce SSL in MongoDB source connector #17590

akashkulk · 2022-10-05T01:00:19Z

What

Currently, the cloud version of MongoDB enforces SSL for Atlas + Replica Set. However, for standalone deployments we can potentially get into a state where SSL could be disabled.

Closes #16278

How

In the check() method of the source-mongodb-strict-encrypt, we enforce tls/ssl param to be true.

🚨 User Impact 🚨

This can potentially break cloud users that somehow got to a stage where "use SSL" was disabled. This is quite hard as the UI stands today, as that is disabled in cloud. However, this could be a problem going forward if the UI were to change or a CLI was exposed which allows the user to do so.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

…ct-encrypt

github-actions · 2022-10-05T01:01:10Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mongodb-v2
source-mongodb-strict-encrypt

akashkulk · 2022-10-05T01:01:43Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/3186542135
❌ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/3186542135
🐛 https://gradle.com/s/obbi23gmct7p6

Build Failed

Test summary info:

Could not find result summary

akashkulk · 2022-10-05T01:02:13Z

/test connector=connectors/source-mongodb-strict-encrypt

🕑 connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3186545215
❌ connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3186545215
🐛 https://gradle.com/s/k3rdeykjnnpik

Build Failed

Test summary info:

Could not find result summary

akashkulk · 2022-10-05T17:20:06Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/3191399590
❌ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/3191399590
🐛 https://gradle.com/s/hl3ktaweznm3q

Build Failed

Test summary info:

Could not find result summary

akashkulk · 2022-10-05T17:20:32Z

/test connector=connectors/source-mongodb-strict-encrypt

🕑 connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3191402047
❌ connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3191402047
🐛 https://gradle.com/s/4mb7af6irwt2k

Build Failed

Test summary info:

Could not find result summary

akashkulk · 2022-10-05T17:44:24Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/3191519877
✅ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/3191519877
No Python unittests run

Build Passed

Test summary info:

All Passed

github-actions · 2022-10-05T17:45:24Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mongodb-v2
source-mongodb-strict-encrypt

akashkulk · 2022-10-05T18:48:49Z

/test connector=connectors/source-mongodb-strict-encrypt

🕑 connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3191917536
✅ connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3191917536
No Python unittests run

Build Passed

Test summary info:

All Passed

github-actions · 2022-10-05T19:11:26Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mongodb-v2
source-mongodb-strict-encrypt

...encrypt/src/main/java/io.airbyte.integrations.source.mongodb/MongodbSourceStrictEncrypt.java

...rs/source-mongodb-v2/src/main/java/io.airbyte.integrations.source.mongodb/MongoDbSource.java

ryankfu · 2022-10-05T19:38:01Z

...urce-mongodb-v2/src/main/java/io.airbyte.integrations.source.mongodb/MongoDbSourceUtils.java

+  /**
+   * Determines whether TLS/SSL should be enabled for a standalone instance of MongoDB.
+   */
+  public static boolean tlsEnabledForStandaloneInstance(final JsonNode config, final JsonNode instanceConfig) {


Should both the javadoc comment and the method name be more generalized? It seems you're passing in an instanceConfig so it appears this can be extended in the future to handle other instance types yet this method is specifically for Standalone instances, right?

It is actually specifically for Standalone instances actually. For Atlas and replica set deployments, we are overrwriting the arguments to always have tls/ssl=true in the parames (see MONGODB_SERVER_URL and MONGODB_CLUSTER_URL above. I was essentially duplicating the logic in the source-mongodb-strict-encrypt's check() method, so pulled it out here. The naming is specifically a warning to not rely on this method for non-standalone instances

ryankfu

Some minor nits, overall looks good. One thing that would be useful is the bullet point that Liren mentioned which is just updating docs to reflect that users are encouraged not to turn off TLS and that can be in the docs since it applies generally to both OSS and Cloud users

…hkulkar/mongo-ssl

github-actions · 2022-10-05T22:27:41Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mongodb-strict-encrypt
source-mongodb-v2

akashkulk · 2022-10-05T22:32:14Z

Some minor nits, overall looks good. One thing that would be useful is the bullet point that Liren mentioned which is just updating docs to reflect that users are encouraged not to turn off TLS and that can be in the docs since it applies generally to both OSS and Cloud users

Thanks Ryan - it seems that the existing documentation mentions this, so will leave this unchanged for now. This is more of a defensive change for the Cloud side of things
https://www.loom.com/i/835efa1e3b9d4490843fdd14bf9cabac

github-actions · 2022-10-05T22:33:37Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mongodb-v2
source-mongodb-strict-encrypt

github-actions · 2022-10-05T23:13:07Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mongodb-v2
source-mongodb-strict-encrypt

akashkulk · 2022-10-05T23:35:58Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/3193427958
❌ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/3193427958
🐛 https://gradle.com/s/wiy7wrpbedkty

Build Failed

Test summary info:

Could not find result summary

github-actions · 2022-10-06T01:34:34Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mongodb-v2
source-mongodb-strict-encrypt

github-actions · 2022-10-06T17:19:20Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mongodb-strict-encrypt
source-mongodb-v2

akashkulk · 2022-10-06T17:54:28Z

/test connector=connectors/source-mongodb-strict-encrypt

🕑 connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3199344202
✅ connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3199344202
No Python unittests run

Build Passed

Test summary info:

All Passed

akashkulk · 2022-10-06T18:08:40Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/3199436520
✅ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/3199436520
No Python unittests run

Build Passed

Test summary info:

All Passed

akashkulk · 2022-10-06T18:36:00Z

/publish connector=connectors/source-mongodb-v2 run-tests=false

🕑 Publishing the following connectors:
connectors/source-mongodb-v2
https://github.com/airbytehq/airbyte/actions/runs/3199603069

Connector	Did it publish?	Were definitions generated?
connectors/source-mongodb-v2	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

github-actions · 2022-10-06T18:36:19Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mongodb-v2
source-mongodb-strict-encrypt

akashkulk · 2022-10-06T18:36:24Z

/publish connector=connectors/source-mongodb-strict-encrypt run-tests=false

🕑 Publishing the following connectors:
connectors/source-mongodb-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/3199604821

Connector	Did it publish?	Were definitions generated?
connectors/source-mongodb-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

github-actions · 2022-10-06T18:41:37Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mongodb-v2
source-mongodb-strict-encrypt

…airbyte into akashkulkar/mongo-ssl

github-actions · 2022-10-06T18:46:47Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mongodb-v2
source-mongodb-strict-encrypt

…7590) * Check and test for if we are disabling SSL/TLS in source-mongodb-strict-encrypt * Update docs * Address comments * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

Check and test for if we are disabling SSL/TLS in source-mongodb-stri…

c883bcb

…ct-encrypt

github-actions bot added the area/connectors Connector related issues label Oct 5, 2022

octavia-squidington-iv added connectors/source/mongodb-v2 connectors/source/mongodb-strict-encrypt labels Oct 5, 2022

Merge branch 'master' into akashkulkar/mongo-ssl

da179fe

akashkulk changed the title ~~Check and test for if we are disabling SSL/TLS in source-mongodb-stri…~~ Add ability to enforce SSL in MongoDB source connector Oct 5, 2022

akashkulk changed the title ~~Add ability to enforce SSL in MongoDB source connector~~ 🚨 Add ability to enforce SSL in MongoDB source connector Oct 5, 2022

akashkulk and others added 3 commits October 5, 2022 08:56

Update docs

8e43f75

Merge branch 'master' into akashkulkar/mongo-ssl

19e4fb0

Merge branch 'master' into akashkulkar/mongo-ssl

333b4e9

Merge branch 'master' into akashkulkar/mongo-ssl

ea6e3ca

github-actions bot added the area/documentation Improvements or additions to documentation label Oct 5, 2022

Merge branch 'master' into akashkulkar/mongo-ssl

57874e6

akashkulk requested review from subodh1810, grishick and ryankfu October 5, 2022 19:26

akashkulk marked this pull request as ready for review October 5, 2022 19:26

ryankfu reviewed Oct 5, 2022

View reviewed changes

...encrypt/src/main/java/io.airbyte.integrations.source.mongodb/MongodbSourceStrictEncrypt.java Outdated Show resolved Hide resolved

ryankfu reviewed Oct 5, 2022

View reviewed changes

...rs/source-mongodb-v2/src/main/java/io.airbyte.integrations.source.mongodb/MongoDbSource.java Outdated Show resolved Hide resolved

ryankfu reviewed Oct 5, 2022

View reviewed changes

ryankfu approved these changes Oct 5, 2022

View reviewed changes

Merge remote-tracking branch 'origin/akashkulkar/mongo-ssl' into akas…

3f718d0

…hkulkar/mongo-ssl

Merge branch 'master' into akashkulkar/mongo-ssl

ce2c357

Merge branch 'master' into akashkulkar/mongo-ssl

a2052e9

Merge branch 'master' into akashkulkar/mongo-ssl

c5803eb

Merge branch 'master' into akashkulkar/mongo-ssl

209f1d5

Merge branch 'master' into akashkulkar/mongo-ssl

b9dd838

Merge branch 'master' into akashkulkar/mongo-ssl

5868e29

octavia-squidington-iii added 2 commits October 6, 2022 18:45

auto-bump connector version [ci skip]

2db1d2f

Merge branch 'akashkulkar/mongo-ssl' of https://github.com/airbytehq/…

e567818

…airbyte into akashkulkar/mongo-ssl

octavia-squidington-iii temporarily deployed to more-secrets October 6, 2022 18:48 Inactive

akashkulk merged commit 4b990d0 into master Oct 6, 2022

akashkulk deleted the akashkulkar/mongo-ssl branch October 6, 2022 18:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🚨 Add ability to enforce SSL in MongoDB source connector #17590

🚨 Add ability to enforce SSL in MongoDB source connector #17590

akashkulk commented Oct 5, 2022 •

edited

Loading

github-actions bot commented Oct 5, 2022

akashkulk commented Oct 5, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 5, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 5, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 5, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 5, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Oct 5, 2022

akashkulk commented Oct 5, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Oct 5, 2022

ryankfu Oct 5, 2022 •

edited

Loading

akashkulk Oct 5, 2022

ryankfu left a comment

github-actions bot commented Oct 5, 2022

akashkulk commented Oct 5, 2022

github-actions bot commented Oct 5, 2022

github-actions bot commented Oct 5, 2022

akashkulk commented Oct 5, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Oct 6, 2022

github-actions bot commented Oct 6, 2022

akashkulk commented Oct 6, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 6, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 6, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Oct 6, 2022

akashkulk commented Oct 6, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Oct 6, 2022

github-actions bot commented Oct 6, 2022

🚨 Add ability to enforce SSL in MongoDB source connector #17590

🚨 Add ability to enforce SSL in MongoDB source connector #17590

Conversation

akashkulk commented Oct 5, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

github-actions bot commented Oct 5, 2022

akashkulk commented Oct 5, 2022 • edited by github-actions bot Loading

Build Failed

akashkulk commented Oct 5, 2022 • edited by github-actions bot Loading

Build Failed

akashkulk commented Oct 5, 2022 • edited by github-actions bot Loading

Build Failed

akashkulk commented Oct 5, 2022 • edited by github-actions bot Loading

Build Failed

akashkulk commented Oct 5, 2022 • edited by github-actions bot Loading

Build Passed

github-actions bot commented Oct 5, 2022

akashkulk commented Oct 5, 2022 • edited by github-actions bot Loading

Build Passed

github-actions bot commented Oct 5, 2022

ryankfu Oct 5, 2022 • edited Loading

Choose a reason for hiding this comment

akashkulk Oct 5, 2022

Choose a reason for hiding this comment

ryankfu left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 5, 2022

akashkulk commented Oct 5, 2022

github-actions bot commented Oct 5, 2022

github-actions bot commented Oct 5, 2022

akashkulk commented Oct 5, 2022 • edited by github-actions bot Loading

Build Failed

github-actions bot commented Oct 6, 2022

github-actions bot commented Oct 6, 2022

akashkulk commented Oct 6, 2022 • edited by github-actions bot Loading

Build Passed

akashkulk commented Oct 6, 2022 • edited by github-actions bot Loading

Build Passed

akashkulk commented Oct 6, 2022 • edited by github-actions bot Loading

github-actions bot commented Oct 6, 2022

akashkulk commented Oct 6, 2022 • edited by github-actions bot Loading

github-actions bot commented Oct 6, 2022

github-actions bot commented Oct 6, 2022

akashkulk commented Oct 5, 2022 •

edited

Loading

akashkulk commented Oct 5, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 5, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 5, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 5, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 5, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 5, 2022 •

edited by github-actions bot

Loading

ryankfu Oct 5, 2022 •

edited

Loading

akashkulk commented Oct 5, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 6, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 6, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 6, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 6, 2022 •

edited by github-actions bot

Loading