Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source FB Marketing: improve check connection performance #10406

Closed
1 task done
Tracked by #10154
alexandr-shegeda opened this issue Feb 17, 2022 · 2 comments
Closed
1 task done
Tracked by #10154

Source FB Marketing: improve check connection performance #10406

alexandr-shegeda opened this issue Feb 17, 2022 · 2 comments
Assignees
Labels
area/connectors Connector related issues lang/python type/enhancement New feature or request

Comments

@alexandr-shegeda
Copy link
Contributor

alexandr-shegeda commented Feb 17, 2022

Relates to: #10149

Tell us about the problem you're trying to solve

As part of the investigation into speeding up check connection, our testing found that the check connection method (and discover schema method) for some connectors is slow (even after removing platform overhead)

Ideally, our 99 percentile for performing check connection is < 10 seconds. Given that the Kube overhead is ~5 seconds, we want the connector to finish in something like 3 seconds. This gives us 2 seconds of buffer to play with.

Results (does not include platform overhead):

  • destination Postgres - ~ 4 seconds (I was using this as a generic RDS connector)
  • destination Snowflake - ~ 14 seconds
  • source salesforce - ~ 3 seconds

Acceptance Criteria

  • Source FB Marketing check connection takes < 5 seconds

If it is necessary to remove some thoroughness in the check connection (i.e: do fewer operations and get a less granular failure message)

@alexandr-shegeda alexandr-shegeda added area/connectors Connector related issues lang/python type/enhancement New feature or request labels Feb 17, 2022
@alexandr-shegeda alexandr-shegeda moved this to Backlog (unscoped) in GL Roadmap Feb 17, 2022
@antixar antixar moved this from Backlog (unscoped) to Ready for implementation (prioritized) in GL Roadmap Feb 17, 2022
@midavadim midavadim self-assigned this Feb 18, 2022
@midavadim
Copy link
Contributor

  1. I have tested 'check' and 'discover' methods via UI and CLI and I have not seen any significant delays ( about 1s)

  2. I have also checked connector code and I have not seen any places where the problem can happen

  3. I have reviewed related issues and spreadsheets and I actually was not able to find any evidence (and numbers) which could indicate slowness of check and discover methods.

@misteryeo, @davinchia is it enough to close this issue? or could you help me to find more information about circumstances in which I can re-produce the problem?

@midavadim midavadim moved this from Ready for implementation (prioritized) to On Hold in GL Roadmap Feb 23, 2022
@sherifnada
Copy link
Contributor

@midavadim that should be sufficient to close!

@bazarnov bazarnov moved this from On Hold to Done in GL Roadmap Mar 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues lang/python type/enhancement New feature or request
Projects
No open projects
Archived in project
Development

No branches or pull requests

3 participants