🐛 Source Bigcommerce: fix infinite loop in Page stream #14940

TimothyZhang7 · 2022-07-21T17:58:01Z

What

Describe what the change is solving
When syncing the Bigcommerce source connector, the Page stream is stucked at first page and loops indefinitely.
Related issue: #14861

How

Describe the solution
It seems that the overriding request_params() in Class Page is blocking the stream reading by not passing next_page_token.
Remove the override request_params() function and use the inherited request_params() from Class IncrementalBigcommerceStream will solve the issue.

🚨 User Impact 🚨

It shouldn't break anything. Instead the bug to be fixed could eat up too much memory and throttle the hosting machine if deployment is not defensively configured.

Pre-merge Checklist

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Acceptance

python3 -m pytest integration_tests -p integration_tests.acceptance
Test session starts (platform: linux, Python 3.9.5, pytest 6.2.5, pytest-sugar 0.9.5)
cachedir: .pytest_cache
rootdir: /home/jzhang91/acho/airbyte-fork/airbyte, configfile: pytest.ini
plugins: timeout-1.4.2, sugar-0.9.5
collecting ...
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_config_match_spec[inputs0] ✓4% ▌
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_match_expected[inputs0] ✓8% ▊
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_docker_env[inputs0] ✓12% █▎
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oneof_usage[inputs0] ✓16% █▋
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_required[inputs0] ✓20% ██
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_optional[inputs0] ✓24% ██▌
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_has_secret[inputs0] ✓28% ██▊
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_secret_never_in_the_output[inputs0] ✓32% ███▎
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_defined_refs_exist_in_json_spec_file[inputs0] ✓36% ███▋
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oauth_flow_parameters[inputs0] ✓40% ████
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs0] ✓44% ████▌
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs1] ✓48% ████▊
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_discover[inputs0] ✓52% █████▎
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_cursors_exist_in_schema[inputs0] ✓56% █████▋
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_refs_exist_in_schema[inputs0] ✓60% ██████
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_keyword_exist_in_schema[inputs0-allOf] ✓64% ██████▌
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_keyword_exist_in_schema[inputs0-not] ✓68% ██████▊
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_primary_keys_exist_in_schema[inputs0] ✓72% ███████▎
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_streams_has_sync_modes[inputs0] ✓76% ███████▋
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs0] ✓80% ████████ {"type": "LOG", "log": {"level": "ERROR", "message": "Docker container was failed, code 1, error:\n{"type": "TRACE", "trace": {"type": "ERROR", "emitted_at": 1658788191343.662, "error": {"message": "Something went wrong in the connector. See the logs for more details.", "internal_message": "2 validation errors for ConfiguredAirbyteCatalog\nstreams -> 0 -> sync_mode\n value is not a valid enumeration member; permitted: 'full_refresh', 'incremental' (type=type_error.enum; enum_values=[<SyncMode.full_refresh: 'full_refresh'>, <SyncMode.incremental: 'incremental'>])\nstreams -> 0 -> destination_sync_mode\n value is not a valid enumeration member; permitted: 'append', 'overwrite', 'append_dedup' (type=type_error.enum; enum_values=[<DestinationSyncMode.append: 'append'>, <DestinationSyncMode.overwrite: 'overwrite'>, <DestinationSyncMode.append_dedup: 'append_dedup'>])", "stack_trace": "Traceback (most recent call last):\n File \"/airbyte/integration_code/main.py\", line 13, in \n launch(source, sys.argv[1:])\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 123, in launch\n for message in source_entrypoint.run(parsed_args):\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 111, in run\n config_catalog = self.source.read_catalog(parsed_args.catalog)\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/source.py\", line 54, in read_catalog\n return ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))\n File \"pydantic/main.py\", line 521, in pydantic.main.BaseModel.parse_obj\n File \"pydantic/main.py\", line 341, in pydantic.main.BaseModel.init\npydantic.error_wrappers.ValidationError: 2 validation errors for ConfiguredAirbyteCatalog\nstreams -> 0 -> sync_mode\n value is not a valid enumeration member; permitted: 'full_refresh', 'incremental' (type=type_error.enum; enum_values=[<SyncMode.full_refresh: 'full_refresh'>, <SyncMode.incremental: 'incremental'>])\nstreams -> 0 -> destination_sync_mode\n value is not a valid enumeration member; permitted: 'append', 'overwrite', 'append_dedup' (type=type_error.enum; enum_values=[<DestinationSyncMode.append: 'append'>, <DestinationSyncMode.overwrite: 'overwrite'>, <DestinationSyncMode.append_dedup: 'append_dedup'>])\n", "failure_type": "system_error"}}}\n"}}

airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_airbyte_trace_message_on_failure[inputs0] ✓84% ████████▌
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs0] ✓88% ████████▊
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_two_sequential_reads[inputs0] ✓92% █████████▎
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_read_sequential_slices[inputs0] ✓96% █████████▋
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_state_with_abnormally_large_values[inputs0] ✓100% ██████████
{"type": "LOG", "log": {"level": "INFO", "message": "/home/jzhang91/acho/airbyte-fork/airbyte/airbyte-integrations/connectors/source-bigcommerce - SAT run - d5507d8 - PASSED"}}

Results (297.34s):
25 passed

… in class Page

CLAassistant · 2022-07-21T17:58:06Z

All committers have signed the CLA.

harshithmullapudi · 2022-08-04T13:31:03Z

Hey thanks for the contribution can you check this doc https://docs.airbyte.io/connector-development#updating-an-existing-connector and do the needful

TimothyZhang7 · 2022-08-04T23:54:35Z

Hey thanks for the contribution can you check this doc https://docs.airbyte.io/connector-development#updating-an-existing-connector and do the needful

I think I've already done the viable parts within my privileges. Would you mind specifying if there's anything missing?

harshithmullapudi · 2022-08-11T07:54:22Z

/test connector=connectors/source-bigcommerce

🕑 connectors/source-bigcommerce https://github.com/airbytehq/airbyte/actions/runs/2838252864
❌ connectors/source-bigcommerce https://github.com/airbytehq/airbyte/actions/runs/2838252864
🐛 https://gradle.com/s/z6c6o3edezto6

Build Failed

Test summary info:

Could not find result summary

harshithmullapudi · 2022-08-11T08:04:42Z

/test connector=connectors/source-bigcommerce

🕑 connectors/source-bigcommerce https://github.com/airbytehq/airbyte/actions/runs/2838307488
❌ connectors/source-bigcommerce https://github.com/airbytehq/airbyte/actions/runs/2838307488
🐛 https://gradle.com/s/5jsdrgygpy3pe

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_additional_properties_is_true[inputs0] - ...
======================== 1 failed, 28 passed in 42.21s =========================

harshithmullapudi · 2022-08-11T09:53:16Z

/test connector=connectors/source-bigcommerce

🕑 connectors/source-bigcommerce https://github.com/airbytehq/airbyte/actions/runs/2838927832
❌ connectors/source-bigcommerce https://github.com/airbytehq/airbyte/actions/runs/2838927832
🐛 https://gradle.com/s/g2b3tclrtp7vk

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_backward_compatibility[inputs0] - IndexEr...
======================== 1 failed, 28 passed in 39.84s =========================

harshithmullapudi · 2022-08-13T04:07:36Z

/test connector=connectors/source-bigcommerce

🕑 connectors/source-bigcommerce https://github.com/airbytehq/airbyte/actions/runs/2850666486
✅ connectors/source-bigcommerce https://github.com/airbytehq/airbyte/actions/runs/2850666486
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1321    463    65%

Build Passed

Test summary info:

All Passed

…ianhaoZhang/airbyte into fix-bigcommerce-pagestream-termination

harshithmullapudi · 2022-08-15T05:54:57Z

/test connector=connectors/source-bigcommerce

🕑 connectors/source-bigcommerce https://github.com/airbytehq/airbyte/actions/runs/2858739433
✅ connectors/source-bigcommerce https://github.com/airbytehq/airbyte/actions/runs/2858739433
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1321    463    65%

Build Passed

Test summary info:

All Passed

harshithmullapudi · 2022-08-15T06:09:38Z

/publish connector=connectors/source-bigcommerce

🕑 Publishing the following connectors:
connectors/source-bigcommerce
https://github.com/airbytehq/airbyte/actions/runs/2858800123

Connector	Did it publish?	Were definitions generated?
connectors/source-bigcommerce	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

* fix(source-bigcommerce): infinite loop by overloaded request_params() in class Page * bump connector version to 0.1.6 in Dockerfile * update docs/integrations/sources/bigcommerce.md * fix: tests are failing * docs: updated connector definitions * auto-bump connector version [ci skip] Co-authored-by: Harshith Mullapudi <harshithmullapudi@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

fix(source-bigcommerce): infinite loop by overloaded request_params()…

d5507d8

… in class Page

github-actions bot added the area/connectors Connector related issues label Jul 21, 2022

octavia-squidington-iii added community connectors/source/bigcommerce labels Jul 21, 2022

TimothyZhang7 changed the title ~~fix(source-bigcommerce): infinite loop by overloaded request_params()…~~ 🐛 Source Bigcommerce: infinite loop in Page stream Jul 25, 2022

TimothyZhang7 changed the title ~~🐛 Source Bigcommerce: infinite loop in Page stream~~ 🐛 Source Bigcommerce: fix infinite loop in Page stream Jul 25, 2022

bump connector version to 0.1.6 in Dockerfile

b28592d

TimothyZhang7 marked this pull request as ready for review July 25, 2022 23:15

octavia-squidington-iii added the bounty label Jul 25, 2022

update docs/integrations/sources/bigcommerce.md

a297333

github-actions bot added the area/documentation Improvements or additions to documentation label Jul 27, 2022

harshithmullapudi self-requested a review August 1, 2022 04:24

harshithmullapudi self-assigned this Aug 1, 2022

Merge branch 'master' into fix-bigcommerce-pagestream-termination

b3ebcdf

fix: tests are failing

665143d

Merge branch 'master' into fix-bigcommerce-pagestream-termination

6c0185b

harshithmullapudi added 2 commits August 15, 2022 11:24

docs: updated connector definitions

fd598d3

Merge branch 'fix-bigcommerce-pagestream-termination' of github.com:J…

3cd5f8e

…ianhaoZhang/airbyte into fix-bigcommerce-pagestream-termination

auto-bump connector version [ci skip]

342915d

harshithmullapudi merged commit ab65ef5 into airbytehq:master Aug 15, 2022

octavia-squidington-iii mentioned this pull request Aug 19, 2022

Bump Airbyte version from 0.40.0-alpha to 0.40.1-alpha #15775

Closed

evantahler mentioned this pull request Aug 22, 2022

Bump Airbyte version from 0.40.0-alpha to 0.40.1 #15857

Merged

marcosmarxm mentioned this pull request Jun 27, 2023

Source BigCommerce: page stream stuck and copying same id indefinitely #14861

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Bigcommerce: fix infinite loop in Page stream #14940

🐛 Source Bigcommerce: fix infinite loop in Page stream #14940

TimothyZhang7 commented Jul 21, 2022 •

edited

Loading

CLAassistant commented Jul 21, 2022 •

edited

Loading

harshithmullapudi commented Aug 4, 2022

TimothyZhang7 commented Aug 4, 2022

harshithmullapudi commented Aug 11, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented Aug 11, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented Aug 11, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented Aug 13, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented Aug 15, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented Aug 15, 2022 •

edited by github-actions bot

Loading

🐛 Source Bigcommerce: fix infinite loop in Page stream #14940

🐛 Source Bigcommerce: fix infinite loop in Page stream #14940

Conversation

TimothyZhang7 commented Jul 21, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

CLAassistant commented Jul 21, 2022 • edited Loading

harshithmullapudi commented Aug 4, 2022

TimothyZhang7 commented Aug 4, 2022

harshithmullapudi commented Aug 11, 2022 • edited by github-actions bot Loading

Build Failed

harshithmullapudi commented Aug 11, 2022 • edited by github-actions bot Loading

Build Failed

harshithmullapudi commented Aug 11, 2022 • edited by github-actions bot Loading

Build Failed

harshithmullapudi commented Aug 13, 2022 • edited by github-actions bot Loading

Build Passed

harshithmullapudi commented Aug 15, 2022 • edited by github-actions bot Loading

Build Passed

harshithmullapudi commented Aug 15, 2022 • edited by github-actions bot Loading

TimothyZhang7 commented Jul 21, 2022 •

edited

Loading

CLAassistant commented Jul 21, 2022 •

edited

Loading

harshithmullapudi commented Aug 11, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented Aug 11, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented Aug 11, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented Aug 13, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented Aug 15, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented Aug 15, 2022 •

edited by github-actions bot

Loading