🐛 Updated source-mongodb-v2 performance #7982

andriikorotkov · 2021-11-15T15:30:22Z

What

Updated the method for reading unique document fields in a collection and their types. The task indicates that the processing of one stream with 10 thousand records took 20 minutes. After the changes have been made, the processing of the same stream takes 2 minutes and 30 seconds.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

…6544-mongo-performance-issues

andriikorotkov · 2021-11-15T16:25:27Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1463109456
✅ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1463109456
No Python unittests run

andriikorotkov · 2021-11-15T17:19:02Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1463308937
✅ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1463308937
No Python unittests run

alexandr-shegeda

LGTM. we can proceed with publish/merge without airbyte review

andriikorotkov · 2021-11-18T08:56:46Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1475607799
✅ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1475607799
No Python unittests run

andriikorotkov · 2021-11-18T08:57:24Z

/test connector=connectors/source-mongodb-strict-encrypt

🕑 connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1475609577
✅ connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1475609577
No Python unittests run

andriikorotkov · 2021-11-18T09:06:54Z

/publish connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1475650132
✅ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1475650132

andriikorotkov · 2021-11-18T09:07:17Z

/publish connector=connectors/source-mongodb-strict-encrypt

🕑 connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1475651458
✅ connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1475651458

JCWahoo · 2021-11-18T12:53:05Z

I don't believe this resolves the issue. 0.1.5 is showing no difference in performance vs the previous version

andriikorotkov · 2021-11-18T13:01:00Z

@JCWahoo, I opened this task again and see how else I can improve the performance.

JCWahoo · 2021-11-18T13:06:36Z

Thanks - Looking at the Mongo side it appears during schema discovery is trying to parse the entire collection and not a sample of records

…

On Thu, Nov 18, 2021 at 8:01 AM andriikorotkov ***@***.***> wrote: @JCWahoo <https://github.com/JCWahoo>, I opened this task again and see how else I can improve the performance. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7982 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUAW7TDXKC5UHU6OFFRG5NLUMT2JPANCNFSM5IB7GBWA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

* updated source-mongodb-v2 performance * updated code style * fixed remarks * fixed remarks * fixed remarks * updated strict encrypt source mongodb version

andriikorotkov added 3 commits November 15, 2021 16:47

updated source-mongodb-v2 performance

f77238f

updated code style

6ac8c93

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

6e3f202

…6544-mongo-performance-issues

andriikorotkov requested review from alexandr-shegeda, sashaNeshcheret and VitaliiMaltsev November 15, 2021 15:30

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Nov 15, 2021

andriikorotkov linked an issue Nov 15, 2021 that may be closed by this pull request

Performance issues in new Mongo Source Connector #6544

Closed

andriikorotkov temporarily deployed to more-secrets November 15, 2021 15:32 Inactive

jrhizor temporarily deployed to more-secrets November 15, 2021 16:27 Inactive

fixed remarks

9ff2bfd

andriikorotkov temporarily deployed to more-secrets November 15, 2021 16:55 Inactive

fixed remarks

4eb9d09

andriikorotkov temporarily deployed to more-secrets November 15, 2021 17:07 Inactive

fixed remarks

451fe71

andriikorotkov temporarily deployed to more-secrets November 15, 2021 17:14 Inactive

jrhizor temporarily deployed to more-secrets November 15, 2021 17:21 Inactive

VitaliiMaltsev approved these changes Nov 15, 2021

View reviewed changes

andriikorotkov requested a review from subodh1810 November 16, 2021 08:48

andriikorotkov marked this pull request as ready for review November 16, 2021 08:49

sashaNeshcheret approved these changes Nov 16, 2021

View reviewed changes

andriikorotkov requested review from sherifnada and removed request for subodh1810 November 17, 2021 09:10

alexandr-shegeda approved these changes Nov 17, 2021

View reviewed changes

andriikorotkov removed the request for review from sherifnada November 17, 2021 16:53

updated strict encrypt source mongodb version

ae3c06e

andriikorotkov temporarily deployed to more-secrets November 18, 2021 08:48 Inactive

jrhizor temporarily deployed to more-secrets November 18, 2021 08:59 Inactive

jrhizor temporarily deployed to more-secrets November 18, 2021 09:09 Inactive

andriikorotkov temporarily deployed to more-secrets November 18, 2021 09:37 Inactive

andriikorotkov merged commit 8e0e0b4 into master Nov 18, 2021

andriikorotkov deleted the akorotkov/6544-mongo-performance-issues branch November 18, 2021 10:05

This was referenced Nov 18, 2021

Bump Airbyte version from 0.32.2-alpha to 0.32.3-alpha #8105

Merged

Bump Airbyte version from 0.32.2-alpha to 0.32.3-alpha #8109

Closed

joelluijmes mentioned this pull request Dec 3, 2021

🐛 fix(mongodb-v2): use max 10k documents while discovering MongoDB schema #8491

Merged

40 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Updated source-mongodb-v2 performance #7982

🐛 Updated source-mongodb-v2 performance #7982

andriikorotkov commented Nov 15, 2021

andriikorotkov commented Nov 15, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 15, 2021 •

edited by github-actions bot

Loading

alexandr-shegeda left a comment

andriikorotkov commented Nov 18, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 18, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 18, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 18, 2021 •

edited by github-actions bot

Loading

JCWahoo commented Nov 18, 2021

andriikorotkov commented Nov 18, 2021

JCWahoo commented Nov 18, 2021 via email

🐛 Updated source-mongodb-v2 performance #7982

🐛 Updated source-mongodb-v2 performance #7982

Conversation

andriikorotkov commented Nov 15, 2021

What

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

andriikorotkov commented Nov 15, 2021 • edited by github-actions bot Loading

andriikorotkov commented Nov 15, 2021 • edited by github-actions bot Loading

alexandr-shegeda left a comment

Choose a reason for hiding this comment

andriikorotkov commented Nov 18, 2021 • edited by github-actions bot Loading

andriikorotkov commented Nov 18, 2021 • edited by github-actions bot Loading

andriikorotkov commented Nov 18, 2021 • edited by github-actions bot Loading

andriikorotkov commented Nov 18, 2021 • edited by github-actions bot Loading

JCWahoo commented Nov 18, 2021

andriikorotkov commented Nov 18, 2021

JCWahoo commented Nov 18, 2021 via email

andriikorotkov commented Nov 15, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 15, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 18, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 18, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 18, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 18, 2021 •

edited by github-actions bot

Loading