Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Source Salesforce: Fix error during generating schema #9478

Merged

Conversation

lazebnyi
Copy link
Collaborator

@lazebnyi lazebnyi commented Jan 13, 2022

What

After collecting all available streams we try to get the schema for the stream, but some of the streams can have flags "queriable": false that depends on permissions of users roles.

How

Filter streams by queriable flags during collecting all available streams.

Recommended reading order

  1. x.java
  2. y.python

Pre-merge Checklist

New Connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions


This change is Reviewable

@github-actions github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Jan 13, 2022
@lazebnyi
Copy link
Collaborator Author

/test connector=connectors/source-salesforce

@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets January 13, 2022 14:37 Inactive
@lazebnyi lazebnyi temporarily deployed to more-secrets January 13, 2022 14:44 Inactive
@lazebnyi lazebnyi requested a review from antixar January 13, 2022 15:00
@annalvova05 annalvova05 self-requested a review January 17, 2022 16:01
@antixar antixar temporarily deployed to more-secrets January 20, 2022 00:38 Inactive
@antixar
Copy link
Contributor

antixar commented Jan 20, 2022

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1720997245
❌ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1720997245
🐛 https://gradle.com/s/bvgn6lhbkyqlw

@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets January 20, 2022 00:44 Inactive
Copy link
Contributor

@antixar antixar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filtering by the "queryable" option is assumption. Can we print all stream options when SF returns 404 error?

@lazebnyi
Copy link
Collaborator Author

Filtering by the "queryable" option is assumption. Can we print all stream options when SF returns 404 error?

We did not save all options for streams (only names). And that user data (need approve from Sherif).

@lazebnyi lazebnyi temporarily deployed to more-secrets January 20, 2022 16:59 Inactive
@lazebnyi lazebnyi temporarily deployed to more-secrets January 20, 2022 17:06 Inactive
@lazebnyi
Copy link
Collaborator Author

lazebnyi commented Jan 20, 2022

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1724706997
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1724706997
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38      0   100%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     17    69%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  979    404    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                 Stmts   Miss  Cover
	 --------------------------------------------------------
	 source_salesforce/__init__.py            2      0   100%
	 source_salesforce/api.py               122     32    74%
	 source_salesforce/exceptions.py          1      0   100%
	 source_salesforce/rate_limiting.py      22      6    73%
	 source_salesforce/source.py             60     20    67%
	 source_salesforce/streams.py           239    144    40%
	 source_salesforce/utils.py               8      7    12%
	 --------------------------------------------------------
	 TOTAL                                  454    209    54%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                 Stmts   Miss  Cover
	 --------------------------------------------------------
	 source_salesforce/__init__.py            2      0   100%
	 source_salesforce/api.py               122     44    64%
	 source_salesforce/exceptions.py          1      0   100%
	 source_salesforce/rate_limiting.py      22      6    73%
	 source_salesforce/source.py             60     25    58%
	 source_salesforce/streams.py           239     42    82%
	 source_salesforce/utils.py               8      0   100%
	 --------------------------------------------------------
	 TOTAL                                  454    117    74%

@lazebnyi lazebnyi temporarily deployed to more-secrets January 20, 2022 17:27 Inactive
@lazebnyi lazebnyi temporarily deployed to more-secrets January 20, 2022 17:30 Inactive
@lazebnyi lazebnyi removed the request for review from midavadim January 20, 2022 17:40
Copy link
Contributor

@antixar antixar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@antixar antixar temporarily deployed to more-secrets January 20, 2022 19:36 Inactive
@lazebnyi lazebnyi marked this pull request as ready for review January 21, 2022 13:11
@lazebnyi
Copy link
Collaborator Author

lazebnyi commented Jan 21, 2022

/publish connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1728833161
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1728833161

@lazebnyi lazebnyi temporarily deployed to more-secrets January 21, 2022 13:13 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets January 21, 2022 13:15 Inactive
@lazebnyi lazebnyi merged commit 0ed20bc into master Jan 24, 2022
@lazebnyi lazebnyi deleted the lazebnyi/61-salesforce-fix-filtering-by-queryable-flag branch January 24, 2022 08:54
@lazebnyi lazebnyi temporarily deployed to more-secrets January 24, 2022 08:55 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants