Source Salesforce: Deprecate API Type parameter #9302

augan-rymkhan · 2022-01-05T05:23:38Z

What

Resolves issue

How

For the initial sync (without state), always use BULK API, where it's possible (Some streams don't support BULK API, for them always use REST). For the subsequent incremental syncs (when state is provided), use REST API.

Recommended reading order

This change is

augan-rymkhan · 2022-01-11T06:25:42Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1681121510
❌ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1681121510
🐛 https://gradle.com/s/qsepmnhhhm4dm

augan-rymkhan · 2022-01-12T06:18:47Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1686226583
❌ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1686226583
🐛 https://gradle.com/s/p6a4hwleaopfy

augan-rymkhan · 2022-01-12T08:35:28Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1686646717
❌ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1686646717
🐛 https://gradle.com/s/vpxrizvmehr5e
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_match_expected[inputs0] - jsonschema.exce...
FAILED test_core.py::TestConnection::test_check[inputs0] - docker.errors.Cont...
FAILED test_core.py::TestDiscovery::test_discover[inputs0] - docker.errors.Co...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs0]
ERROR test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contain...
ERROR test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
ERROR test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
ERROR test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
==================== 3 failed, 7 passed, 6 errors in 10.87s ====================

augan-rymkhan · 2022-01-14T12:10:07Z

airbyte-integrations/connectors/source-salesforce/source_salesforce/spec.json

@@ -5,7 +5,7 @@
    "title": "Salesforce Source Spec",
    "type": "object",
    "required": ["client_id", "client_secret", "refresh_token"],
-    "additionalProperties": false,
+    "additionalProperties": true,


Set it to true because api_type parameter is removed from the spec but it is provided on config files. It's temporary solution not to fail checks.

augan-rymkhan · 2022-01-17T09:06:34Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1707115619
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1707115619
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38      0   100%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     17    69%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  979    404    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                 Stmts   Miss  Cover
	 --------------------------------------------------------
	 source_salesforce/__init__.py            2      0   100%
	 source_salesforce/api.py               118     30    75%
	 source_salesforce/exceptions.py          1      0   100%
	 source_salesforce/rate_limiting.py      22      6    73%
	 source_salesforce/source.py             60     20    67%
	 source_salesforce/streams.py           218    126    42%
	 source_salesforce/utils.py               8      7    12%
	 --------------------------------------------------------
	 TOTAL                                  429    189    56%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                 Stmts   Miss  Cover
	 --------------------------------------------------------
	 source_salesforce/__init__.py            2      0   100%
	 source_salesforce/api.py               118     42    64%
	 source_salesforce/exceptions.py          1      0   100%
	 source_salesforce/rate_limiting.py      22      6    73%
	 source_salesforce/source.py             60     25    58%
	 source_salesforce/streams.py           218     44    80%
	 source_salesforce/utils.py               8      0   100%
	 --------------------------------------------------------
	 TOTAL                                  429    117    73%

augan-rymkhan · 2022-01-19T09:10:01Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1717140074
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1717140074
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38      0   100%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     17    69%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  979    404    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                 Stmts   Miss  Cover
	 --------------------------------------------------------
	 source_salesforce/__init__.py            2      0   100%
	 source_salesforce/api.py               118     30    75%
	 source_salesforce/exceptions.py          1      0   100%
	 source_salesforce/rate_limiting.py      22      6    73%
	 source_salesforce/source.py             60     20    67%
	 source_salesforce/streams.py           235    142    40%
	 source_salesforce/utils.py               8      7    12%
	 --------------------------------------------------------
	 TOTAL                                  446    205    54%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                 Stmts   Miss  Cover
	 --------------------------------------------------------
	 source_salesforce/__init__.py            2      0   100%
	 source_salesforce/api.py               118     42    64%
	 source_salesforce/exceptions.py          1      0   100%
	 source_salesforce/rate_limiting.py      22      6    73%
	 source_salesforce/source.py             60     25    58%
	 source_salesforce/streams.py           235     41    83%
	 source_salesforce/utils.py               8      0   100%
	 --------------------------------------------------------
	 TOTAL                                  446    114    74%

augan-rymkhan · 2022-01-19T10:00:56Z

/publish connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1717351650
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/1717351650

use BULK for the first sync, REST for incremental sync

7fbc2bc

github-actions bot added the area/connectors Connector related issues label Jan 5, 2022

augan-rymkhan temporarily deployed to more-secrets January 5, 2022 05:25 Inactive

if stream contains compound data or/and base64 use always REST

a524688

augan-rymkhan temporarily deployed to more-secrets January 8, 2022 06:19 Inactive

fix get stream state from connector state

9e66e63

augan-rymkhan temporarily deployed to more-secrets January 10, 2022 10:21 Inactive

fix integration test

395071a

augan-rymkhan temporarily deployed to more-secrets January 10, 2022 12:04 Inactive

Augan93 added 2 commits January 11, 2022 12:09

refactor catalog name

413f8d5

format code

eeddc36

augan-rymkhan temporarily deployed to more-secrets January 11, 2022 06:22 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 11, 2022 06:28 Inactive

Augan93 added 2 commits January 12, 2022 11:36

refactor unit tests

c6ad3dc

refactor unit tests 2

bbc32b1

augan-rymkhan temporarily deployed to more-secrets January 12, 2022 06:15 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 12, 2022 06:21 Inactive

format code 2

3370f29

augan-rymkhan temporarily deployed to more-secrets January 12, 2022 08:32 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 12, 2022 08:38 Inactive

augan-rymkhan mentioned this pull request Jan 13, 2022

Source Salesforce: REST API reads only 1000 records from some streams #9136

Closed

augan-rymkhan changed the title ~~use BULK for the first sync, REST for incremental sync~~ Source Salesforce: Deprecating API Type parameter Jan 13, 2022

augan-rymkhan changed the title ~~Source Salesforce: Deprecating API Type parameter~~ Source Salesforce: Deprecate API Type parameter Jan 13, 2022

Set additionalProperties to true not to break test temporarily

f394fd5

augan-rymkhan temporarily deployed to more-secrets January 14, 2022 12:09 Inactive

augan-rymkhan commented Jan 14, 2022

View reviewed changes

Merge branch 'master' into arymkhan/salesforce-deprecate-api-type

c531dc5

octavia-squidington-iii temporarily deployed to more-secrets January 14, 2022 18:02 Inactive

augan-rymkhan marked this pull request as ready for review January 17, 2022 08:50

octavia-squidington-iii temporarily deployed to more-secrets January 17, 2022 08:55 Inactive

Merge branch 'master' into arymkhan/salesforce-deprecate-api-type

ca9afc4

augan-rymkhan temporarily deployed to more-secrets January 17, 2022 09:04 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 17, 2022 09:08 Inactive

augan-rymkhan requested review from sergei-solonitcyn, vitaliizazmic, keu and yevhenii-ldv January 17, 2022 09:19

vitaliizazmic approved these changes Jan 18, 2022

View reviewed changes

sergei-solonitcyn approved these changes Jan 19, 2022

View reviewed changes

Augan93 added 2 commits January 19, 2022 14:29

Merge branch 'master' into arymkhan/salesforce-deprecate-api-type

04166e0

fix unit test and remove unnecessary filtering fields

75e22c2

augan-rymkhan temporarily deployed to more-secrets January 19, 2022 09:08 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 19, 2022 09:12 Inactive

bump version

ceac368

github-actions bot added the area/documentation Improvements or additions to documentation label Jan 19, 2022

augan-rymkhan temporarily deployed to more-secrets January 19, 2022 09:30 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 19, 2022 10:03 Inactive

updated spec and def yaml

fbcc063

augan-rymkhan temporarily deployed to more-secrets January 19, 2022 10:22 Inactive

augan-rymkhan merged commit 0a3713a into master Jan 19, 2022

augan-rymkhan deleted the arymkhan/salesforce-deprecate-api-type branch January 19, 2022 10:23

octavia-squidington-iii mentioned this pull request Jan 20, 2022

Bump Airbyte version from 0.35.5-alpha to 0.35.6-alpha #9630

Merged

jkaelin mentioned this pull request Feb 4, 2022

Source Salesforce: BULK API is only used for initial sync even when supported #10111

Closed

vincentkoc mentioned this pull request Dec 20, 2022

Salesforce Stream "Account" cannot be processed by REST or BULK API #20703

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Salesforce: Deprecate API Type parameter #9302

Source Salesforce: Deprecate API Type parameter #9302

augan-rymkhan commented Jan 5, 2022 •

edited by davinchia

Loading

augan-rymkhan commented Jan 11, 2022 •

edited by github-actions bot

Loading

augan-rymkhan commented Jan 12, 2022 •

edited by github-actions bot

Loading

augan-rymkhan commented Jan 12, 2022 •

edited by github-actions bot

Loading

augan-rymkhan Jan 14, 2022 •

edited

Loading

augan-rymkhan commented Jan 17, 2022 •

edited by github-actions bot

Loading

augan-rymkhan commented Jan 19, 2022 •

edited by github-actions bot

Loading

augan-rymkhan commented Jan 19, 2022 •

edited by github-actions bot

Loading

Source Salesforce: Deprecate API Type parameter #9302

Source Salesforce: Deprecate API Type parameter #9302

Conversation

augan-rymkhan commented Jan 5, 2022 • edited by davinchia Loading

What

How

Recommended reading order

augan-rymkhan commented Jan 11, 2022 • edited by github-actions bot Loading

augan-rymkhan commented Jan 12, 2022 • edited by github-actions bot Loading

augan-rymkhan commented Jan 12, 2022 • edited by github-actions bot Loading

augan-rymkhan Jan 14, 2022 • edited Loading

Choose a reason for hiding this comment

augan-rymkhan commented Jan 17, 2022 • edited by github-actions bot Loading

augan-rymkhan commented Jan 19, 2022 • edited by github-actions bot Loading

augan-rymkhan commented Jan 19, 2022 • edited by github-actions bot Loading

augan-rymkhan commented Jan 5, 2022 •

edited by davinchia

Loading

augan-rymkhan commented Jan 11, 2022 •

edited by github-actions bot

Loading

augan-rymkhan commented Jan 12, 2022 •

edited by github-actions bot

Loading

augan-rymkhan commented Jan 12, 2022 •

edited by github-actions bot

Loading

augan-rymkhan Jan 14, 2022 •

edited

Loading

augan-rymkhan commented Jan 17, 2022 •

edited by github-actions bot

Loading

augan-rymkhan commented Jan 19, 2022 •

edited by github-actions bot

Loading

augan-rymkhan commented Jan 19, 2022 •

edited by github-actions bot

Loading