🐛 Source Zendesk Support: Fix schema User stream to pull all users #12122

danieldiamond · 2022-04-19T02:51:37Z

What

Fix #11895
Fix #12059
Fix #11899 (for user stream only!)
Fix #12060 (TBC)

How

Add SourceZendeskUserExportStream similar to ticket and ticket comment streams

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

harshithmullapudi · 2022-04-19T04:29:04Z

Thanks for the contribution @danieldiamond. Can you also share the acceptance tests screenshot here?

danieldiamond · 2022-04-19T04:48:42Z

airbyte/airbyte-integrations/connectors/source-zendesk-support on  danieldiamond/fix-zendesk-support-user-limit on 🐳 v20.10.8 via airbyte
➜ python -m pytest unit_tests
Test session starts (platform: darwin, Python 3.9.9, pytest 6.2.4, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/danieldiamond/repos/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, requests-mock-1.9.3, mock-3.7.0, timeout-1.4.2
collecting ...
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_proper_number_of_future_requests_generated[1000-100-10] ✓5% ▌
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_proper_number_of_future_requests_generated[1000-10-100] ✓9% ▉
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_proper_number_of_future_requests_generated[0-100-0] ✓14% █▍
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_proper_number_of_future_requests_generated[1-100-1] ✓18% █▊
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_proper_number_of_future_requests_generated[101-100-2] ✓23% ██▍
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_parse_future_records[10-10-10] ✓27% ██▊
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_parse_future_records[10-100-10] ✓32% ███▎
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_parse_future_records[10-10-0] ✓36% ███▋
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_read_records[1000-100-10-True] ✓                                                                                         41% ████▏
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_read_records[1000-10-100-True] ✓                                                                                         45% ████▋
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_str2datetime ✓                                                                                                              50% █████
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_datetime2str ✓                                                                                                              55% █████▌
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_str2unixtime ✓                                                                                                              59% █████▉
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_check_start_time_param ✓                                                                                                    64% ██████▍
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_parse_next_page_number ✓                                                                                                    68% ██████▊
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_next_page_token ✓                                                                                                           73% ███████▍
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_check_stream_state[state present] ✓                                                                                         77% ███████▊
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_check_stream_state[empty string in state] ✓                                                                                 82% ████████▎
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_check_stream_state[state is None] ✓                                                                                         86% ████████▋
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_check_stream_state[cursor is not in the state object] ✓                                                                     91% █████████▏
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_request_params ✓                                                                                                            95% █████████▋
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_parse_response ✓                                                                                                           100% ██████████
======================================================================================================= warnings summary =======================================================================================================
../../../../../.virtualenvs/airbyte/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py:42: 1 warning
airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py: 10 warnings
airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py: 4 warnings
  /Users/danieldiamond/.virtualenvs/airbyte/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py:42: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
    self._authenticator: HttpAuthenticator = NoAuth()

../../../../../.virtualenvs/airbyte/lib/python3.9/site-packages/deprecated/classic.py:173: 1 warning
airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py: 10 warnings
airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py: 4 warnings
  /Users/danieldiamond/.virtualenvs/airbyte/lib/python3.9/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
    return old_new1(cls, *args, **kwargs)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (96.82s):
      22 passed

danieldiamond · 2022-04-19T05:49:32Z

@harshithmullapudi added acceptance tests screenshot and updated description as it fixes a range of issues.

bazarnov · 2022-04-19T16:57:09Z

@danieldiamond
Please check my suggestions in the latest commit, you could have done it with less code, I believe)
Did some typo corrections as well (my bad).
Please pull the latest changes from this branch and run the acceptance test to make sure everything works as expected. Please provide me with the feedback afterwards. Thank you.

danieldiamond · 2022-04-19T23:08:20Z

@harshithmullapudi this should be good to merge, the change was minor

danieldiamond · 2022-04-20T00:41:47Z

➜ python -m pytest unit_tests
Test session starts (platform: darwin, Python 3.9.9, pytest 6.2.4, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/danieldiamond/repos/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, requests-mock-1.9.3, mock-3.7.0, timeout-1.4.2
collecting ...
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_proper_number_of_future_requests_generated[1000-100-10] ✓5% ▌
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_proper_number_of_future_requests_generated[1000-10-100] ✓9% ▉
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_proper_number_of_future_requests_generated[0-100-0] ✓14% █▍
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_proper_number_of_future_requests_generated[1-100-1] ✓18% █▊
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_proper_number_of_future_requests_generated[101-100-2] ✓23% ██▍
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_parse_future_records[10-10-10] ✓27% ██▊
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_parse_future_records[10-100-10] ✓32% ███▎
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_parse_future_records[10-10-0] ✓36% ███▋
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_read_records[1000-100-10-True] ✓41% ████▏
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py::test_read_records[1000-10-100-True] ✓45% ████▋
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_str2datetime ✓            50% █████
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_datetime2str ✓            55% █████▌
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_str2unixtime ✓            59% █████▉
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_check_start_time_param ✓  64% ██████▍
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_parse_next_page_number ✓  68% ██████▊
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_next_page_token ✓         73% ███████▍
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_check_stream_state[state present] ✓77% ███████▊
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_check_stream_state[empty string in state] ✓82% ████████▎
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_check_stream_state[state is None] ✓86% ████████▋
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_check_stream_state[cursor is not in the state object] ✓91% █████████▏
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_request_params ✓          95% █████████▋
 airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py::test_parse_response ✓         100% ██████████
====================================================== warnings summary ======================================================
../../../../../.virtualenvs/airbyte/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py:42: 1 warning
airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py: 10 warnings
airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py: 4 warnings
  /Users/danieldiamond/.virtualenvs/airbyte/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py:42: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
    self._authenticator: HttpAuthenticator = NoAuth()

../../../../../.virtualenvs/airbyte/lib/python3.9/site-packages/deprecated/classic.py:173: 1 warning
airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py: 10 warnings
airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py: 4 warnings
  /Users/danieldiamond/.virtualenvs/airbyte/lib/python3.9/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
    return old_new1(cls, *args, **kwargs)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (96.28s):
      22 passed

sherifnada · 2022-04-20T02:55:21Z

/test connector=source-zendesk-support

🕑 source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/2193317725
❌ source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/2193317725
🐛 https://gradle.com/s/pg4czbbseslrs
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_oneof_usage[inputs0] - AssertionError: Th...
=================== 1 failed, 28 passed in 154.13s (0:02:34) ===================

sherifnada · 2022-04-20T03:25:49Z

/test connector=source-zendesk-support

🕑 source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/2193404353
✅ source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/2193404353
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  886    259    71%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_zendesk_support/__init__.py       2      0   100%
source_zendesk_support/streams.py      293     56    81%
source_zendesk_support/source.py        45     25    44%
--------------------------------------------------------
TOTAL                                  340     81    76%

…12122) * Add SourceZendeskUserExportStream * Format streams * Update zendesk-support README * Update dockerfile * Update changelog to reflect additional fixes * Review updates * Fix typo from previous contribution * Update unit tests * Remove enum and default from spec Co-authored-by: Baz <bazarnov@me.com>

danieldiamond added 2 commits April 19, 2022 12:39

Add SourceZendeskUserExportStream

c4cbae4

Format streams

abab552

github-actions bot added the area/connectors Connector related issues label Apr 19, 2022

octavia-squidington-iii added the community label Apr 19, 2022

Update zendesk-support README

c41737b

github-actions bot added the area/documentation Improvements or additions to documentation label Apr 19, 2022

Update dockerfile

a7217c3

harshithmullapudi self-assigned this Apr 19, 2022

Update changelog to reflect additional fixes

88e3377

sherifnada requested a review from bazarnov April 19, 2022 06:34

Review updates

73bb97d

bazarnov approved these changes Apr 19, 2022

View reviewed changes

Fix typo from previous contribution

97e548b

Update unit tests

9efe7c4

Remove enum and default from spec

6e65ec3

harshithmullapudi mentioned this pull request Apr 20, 2022

Harshith/test pr 12122 #12164

Merged

harshithmullapudi merged commit 147f87a into airbytehq:master Apr 20, 2022

This was referenced Apr 20, 2022

Bump Airbyte version from 0.36.1-alpha to 0.36.2-alpha #12194

Closed

Bump Airbyte version from 0.36.1-alpha to 0.36.2-alpha #12258

Merged

jablonskijakub mentioned this pull request Jun 28, 2022

Zendesk Support - missing entries, does not sync all ticket_metrics #13951

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Zendesk Support: Fix schema User stream to pull all users #12122

🐛 Source Zendesk Support: Fix schema User stream to pull all users #12122

danieldiamond commented Apr 19, 2022 •

edited

Loading

harshithmullapudi commented Apr 19, 2022

danieldiamond commented Apr 19, 2022

danieldiamond commented Apr 19, 2022

bazarnov commented Apr 19, 2022

danieldiamond commented Apr 19, 2022

danieldiamond commented Apr 20, 2022

sherifnada commented Apr 20, 2022 •

edited by github-actions bot

Loading

sherifnada commented Apr 20, 2022 •

edited by github-actions bot

Loading

🐛 Source Zendesk Support: Fix schema User stream to pull all users #12122

🐛 Source Zendesk Support: Fix schema User stream to pull all users #12122

Conversation

danieldiamond commented Apr 19, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

harshithmullapudi commented Apr 19, 2022

danieldiamond commented Apr 19, 2022

danieldiamond commented Apr 19, 2022

bazarnov commented Apr 19, 2022

danieldiamond commented Apr 19, 2022

danieldiamond commented Apr 20, 2022

sherifnada commented Apr 20, 2022 • edited by github-actions bot Loading

sherifnada commented Apr 20, 2022 • edited by github-actions bot Loading

danieldiamond commented Apr 19, 2022 •

edited

Loading

sherifnada commented Apr 20, 2022 •

edited by github-actions bot

Loading

sherifnada commented Apr 20, 2022 •

edited by github-actions bot

Loading