🎉 New Source: Netsuite #13086

wjwatkinson · 2022-05-23T12:19:51Z

What

Netsuite Connector

How

Add Netsuite Connector using python cdk

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Tests

Unit

pipenv run pytest -s unit_tests/
Test session starts (platform: darwin, Python 3.9.12, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/willwatkinson/src/github.com/airbytehq/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, requests-mock-1.9.3, mock-3.6.1, timeout-1.4.2
collecting ...
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_incremental_streams.py::test_cursor_field ✓5% ▌
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_incremental_streams.py::test_get_updated_state ✓10% ▉
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_incremental_streams.py::test_supports_incremental ✓14% █▌
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_incremental_streams.py::test_source_defined_cursor ✓19% █▉
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_incremental_streams.py::test_request_params ✓24% ██▍
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_incremental_streams.py::test_stream_slices_start_datetime ✓29% ██▉
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_incremental_streams.py::test_stream_slices_stream_state ✓33% ███▍
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_source.py::test_check_connection ✓  38% ███▊
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_source.py::test_check_connection_record_types ✓43% ████▍
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_source.py::test_streams ✓           48% ████▊
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_streams.py::test_request_params ✓   52% █████▎
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_streams.py::test_next_page_token ✓  57% █████▊
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_streams.py::test_format_date ✓      62% ██████▎
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_streams.py::test_fetch_record ✓     67% ██████▋
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_streams.py::test_parse_response ✓   71% ███████▎
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_streams.py::test_http_method ✓      76% ███████▋
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_streams.py::test_should_retry[HTTPStatus.OK-False] ✓81% ████████▏
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_streams.py::test_should_retry[HTTPStatus.BAD_REQUEST-False] ✓86% ████████▋
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_streams.py::test_should_retry[HTTPStatus.TOO_MANY_REQUESTS-True] ✓90% █████████▏
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_streams.py::test_should_retry[HTTPStatus.INTERNAL_SERVER_ERROR-True] ✓95% █████████▌
 airbyte-integrations/connectors/source-netsuite/unit_tests/test_streams.py::test_backoff_time ✓    100% ██████████
================================================ warnings summary =================================================
airbyte-integrations/connectors/source-netsuite/unit_tests/test_incremental_streams.py: 7 warnings
airbyte-integrations/connectors/source-netsuite/unit_tests/test_source.py: 2 warnings
airbyte-integrations/connectors/source-netsuite/unit_tests/test_streams.py: 11 warnings
  /Users/willwatkinson/.local/share/virtualenvs/source-netsuite-AEapHDjm/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py:43: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
    self._authenticator: HttpAuthenticator = NoAuth()

airbyte-integrations/connectors/source-netsuite/unit_tests/test_incremental_streams.py: 7 warnings
airbyte-integrations/connectors/source-netsuite/unit_tests/test_source.py: 2 warnings
airbyte-integrations/connectors/source-netsuite/unit_tests/test_streams.py: 11 warnings
  /Users/willwatkinson/.local/share/virtualenvs/source-netsuite-AEapHDjm/lib/python3.9/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
    return old_new1(cls, *args, **kwargs)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (0.13s):
      21 passed

Integration

Put your integration tests output here.

Acceptance

docker  build . -t airbyte/source-netsuite:dev && pipenv run python -m pytest -p integration_tests.acceptance
[+] Building 0.9s (16/16) FINISHED
 => [internal] load build definition from Dockerfile                                                          0.0s
 => => transferring dockerfile: 37B                                                                           0.0s
 => [internal] load .dockerignore                                                                             0.0s
 => => transferring context: 34B                                                                              0.0s
 => [internal] load metadata for docker.io/library/python:3.9.11-alpine3.15                                   0.7s
 => [base 1/1] FROM docker.io/library/python:3.9.11-alpine3.15@sha256:45ddd216e6b4efee0617e15d541e9148ffd689  0.0s
 => [internal] load build context                                                                             0.0s
 => => transferring context: 9.42kB                                                                           0.0s
 => CACHED [builder 1/4] WORKDIR /airbyte/integration_code                                                    0.0s
 => CACHED [builder 2/4] RUN apk --no-cache upgrade     && pip install --upgrade pip     && apk --no-cache a  0.0s
 => CACHED [builder 3/4] COPY setup.py ./                                                                     0.0s
 => CACHED [builder 4/4] RUN pip install --prefix=/install .                                                  0.0s
 => CACHED [stage-2 2/7] COPY --from=builder /install /usr/local                                              0.0s
 => CACHED [stage-2 3/7] COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime                       0.0s
 => CACHED [stage-2 4/7] RUN echo "Etc/UTC" > /etc/timezone                                                   0.0s
 => CACHED [stage-2 5/7] RUN apk --no-cache add bash                                                          0.0s
 => CACHED [stage-2 6/7] COPY main.py ./                                                                      0.0s
 => CACHED [stage-2 7/7] COPY source_netsuite ./source_netsuite                                               0.0s
 => exporting to image                                                                                        0.0s
 => => exporting layers                                                                                       0.0s
 => => writing image sha256:64c471617ac2adc40d4c89da006bb7a2f61d1a7f7337640d3cc61d9e9911a78d                  0.0s
 => => naming to docker.io/airbyte/source-netsuite:dev                                                        0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Test session starts (platform: darwin, Python 3.9.12, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/willwatkinson/src/github.com/airbytehq/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, requests-mock-1.9.3, mock-3.6.1, timeout-1.4.2
collecting ...
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_config_match_spec[inputs0] ✓5% ▌
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_match_expected[inputs0] ✓9% ▉
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_docker_env[inputs0] ✓14% █▍
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oneof_usage[inputs0] ✓18% █▊
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_required[inputs0] ✓23% ██▍
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_optional[inputs0] ✓27% ██▊
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_has_secret[inputs0] ✓32% ███▎
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_secret_never_in_the_output[inputs0] ✓36% ███▋
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_defined_refs_exist_in_json_spec_file[inputs0] ✓41% ████▏
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oauth_flow_parameters[inputs0] ✓45% ████▋
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs0] ✓50% █████
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs1] ✓55% █████▌
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_discover[inputs0] ✓59% █████▉
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_cursors_exist_in_schema[inputs0] ✓64% ██████▍
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_refs_exist_in_schema[inputs0] ✓68% ██████▊
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_keyword_exist_in_schema[inputs0-allOf] ✓73% ███████▍
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_keyword_exist_in_schema[inputs0-not] ✓77% ███████▊
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_primary_keys_exist_in_schema[inputs0] ✓82% ████████▎
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs0] ✓86% ████████▋
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs0] ✓91% █████████▏
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_two_sequential_reads[inputs0] ✓95% █████████▋
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_state_with_abnormally_large_values[inputs0] ✓100% ██████████
{"type": "LOG", "log": {"level": "INFO", "message": "/Users/willwatkinson/src/github.com/airbytehq/airbyte/airbyte-integrations/connectors/source-netsuite - SAT run - 31fb32ad5689700f182a1703c832903a27d91347 - PASSED"}}

================================================ warnings summary =================================================
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py: 16 warnings
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py: 1 warning
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py: 2 warnings
  /Users/willwatkinson/.local/share/virtualenvs/source-netsuite-AEapHDjm/lib/python3.9/site-packages/docker/utils/utils.py:52: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    s1 = StrictVersion(v1)

airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py: 16 warnings
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py: 1 warning
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py: 2 warnings
  /Users/willwatkinson/.local/share/virtualenvs/source-netsuite-AEapHDjm/lib/python3.9/site-packages/docker/utils/utils.py:53: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    s2 = StrictVersion(v2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (567.88s):
      22 passed

wjwatkinson · 2022-05-23T12:22:53Z

The acceptance tests are failing because of timeouts. I think this is due to the connector dynamically pulling all of the objects and schemas for Netsuite, which takes a long time. This is necessary since the schemas are Netsuite instance specific. Netsuite allows the addition of custom fields and record types, which are captured in the dynamic schemas generated by Netsuite, but couldn't be with hard coded schemas.

Any ideas on how to fix the timeout issues while still pulling the schemas dynamically?

wjwatkinson · 2022-05-23T14:49:44Z

I added an array property where users can specify the record types they want. This leads to a better user experience as it speeds up the source retrieval process dramatically and makes navigating the available streams much simpler. Setting this before running the integration tests resolves the timeout issue.

Closing this PR for now, though, as there are some other issues with the source that need to be addressed.

wjwatkinson · 2022-05-25T11:41:41Z

All the issues with the acceptance tests have been addressed.

marcosmarxm · 2022-05-26T17:45:55Z

Thanks for this amazing contributon @wjwatkinson ! I'll ask the team to review it.

marcosmarxm · 2022-05-26T17:47:37Z

airbyte-integrations/connectors/source-netsuite/source_netsuite/spec.yaml

+    record_types:
+      type: array
+      items:
+        type: string
+      description: The API names of the Netsuite objects you want to sync. Setting this speeds up the connection setup process by limiting the number of schemas that need to be retrieved from Netsuite.


so you need to specify what schema/streams do you want to sync? maybe give the option to sync aditional streams BUT the connector need the default ones.

If you do not specify anything the connector will pull all objects present in the user's Netsuite instance. This takes a really long time, though, which is not a great experience and caused acceptance tests to fail.

wjwatkinson · 2022-07-01T19:41:59Z

FWIW I realized recently that there is an Airbyte JDBC source framework (I missed it because it's not in the CDK documentatin). This is likely a cleaner and faster way to retrieve Netsuite data.

I'm open to renaming this connector to Netsuite REST, or something like that to leave the Netsuite name open for a JDBC connector in the future.

marcosmarxm · 2022-07-27T21:04:51Z

Hello @wjwatkinson sorry the delay in return to you. I'll ask the connector team to review and validate your contribution.

misteryeo · 2022-08-10T20:36:18Z

Thanks @marcosmarxm!

Hi @wjwatkinson, thank you so much for your contribution. Our team was actually also looking at creating this connector so I'm tagging in @drrest who was looking at it so he can not only review your contribution but can also make suggested updates based on the research we've done on our end.

swatosh · 2022-09-21T18:48:53Z

Has this been overtaken by #16093?

bazarnov · 2022-09-22T14:13:15Z

Closing this, as of #16093 was merged already.

wjwatkinson · 2022-09-26T10:11:36Z

Unfortunate that #16093 has the same architecture as I used here and therefore the same limitations in performance and schema retrieval.

misteryeo · 2022-09-26T13:38:37Z

Are you able to specify the performance and schema retrieval limitations you ran into? This is helpful feedback for us to keep improving the connector during the alpha period @wjwatkinson

cc: @YowanR

wjwatkinson · 2022-09-26T14:01:42Z

I mentioned this to @YowanR and others in a slack thread where I was helping with Netsuite auth. The Netsuite REST API query returns only record ids, so you have to make 1 request per record to Netsuite to get the data. This results in very slow extraction.

There are further performance issues mentioned in this PR to do with retrieving schemas. I resolved these by passing in an array of object names, so fewer schemas are retrieved, but it's annoying. It looks like this same approach was taken in the merged source.

Netsuite offers a JDBC connector and it looks like Airbyte has a JDBC source framework, which I mentioned in a comment here, that seems like it would have better performance, but I am not very familiar with the technology.

bazarnov · 2022-09-26T16:57:47Z

I agree the SOAP or REST approaches are weak in terms of performance in general, REST allows us to fetch all possible objects and use them as streams of data. Probably the JDBC approach will give us the opportunity to work with Netsuite as Database, if so - the schema retrieval should be much faster, not sure about the streaming performance, but definitely worth trying.

wjwatkinson · 2022-09-30T12:57:29Z

Generally I have found the performance of JSON APIs performant enough for us, but they return 100+ records per call making them 100+x more efficient than the Netsuite API.

bazarnov · 2022-09-30T13:10:46Z

JSON APIs

@wjwatkinson
Could you please give more context about JSON APIs ?

wjwatkinson · 2022-09-30T13:44:45Z

Netsuite's queries only return record ids, so you have to make an additional request per record. 1000 records = 1001 requests.

All of the other APIs I work with through Airbyte return the data directly in the query, so one request returns 100 records. 1000 records = 10 requests.

If you're asking what I mean by JSON APIs I just mean APIs that return JSON data. REST has a specific definition JSON APIs generally don't meet, although I think Netsuite's REST API does.

wjwatkinson added 8 commits May 12, 2022 16:03

created netsuite source

c6382e9

format date for queries, use stream slices, parallel requests

9eda758

generate and close pool while parsing response

6d51425

yield parsed response, fix test

a657735

removed Pipfile.lock

1c711cd

use airbyte request/retry logic and expand sub resources

e09fed5

formatted

2a596bd

retrieve subobjects, schema load time, retrieve beta objects settings.

536aa7f

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels May 23, 2022

octavia-squidington-iii added the community label May 23, 2022

wjwatkinson added 3 commits May 23, 2022 15:25

add record types config to improve setup time.

ce9b4f0

check record types are valid when checking the connection

01541e5

require start_datetime

75cb328

wjwatkinson force-pushed the source-netsuite branch from 70a43f9 to 75cb328 Compare May 23, 2022 14:07

wjwatkinson added 2 commits May 23, 2022 16:47

require start_datetime

7bdc1aa

fix thread prop init and close pool

4dcaa2e

wjwatkinson closed this May 23, 2022

wjwatkinson added 3 commits May 23, 2022 16:57

clean up pool

82c2400

use with instead of pool

31fb32a

fix integration tests, add subschemas to schema.

15f8b3e

wjwatkinson reopened this May 25, 2022

marcosmarxm requested changes May 26, 2022

View reviewed changes

bleonard added bounty connectors/source/netsuite labels Jun 2, 2022

alafanechere removed the bounty label Jun 7, 2022

wjwatkinson added 4 commits June 16, 2022 14:06

test record type access

8213ec4

remove enums from schemas as they are inaccurate

074930a

remove unnecessary parts of schema, fix nsLink schema

fc21874

treat the netsuite object type property differntly than json schema type

4974465

wjwatkinson force-pushed the source-netsuite branch 2 times, most recently from 35e1b1c to 4974465 Compare June 22, 2022 15:30

wjwatkinson added 2 commits July 1, 2022 15:28

static checks

7f2c5d9

added netsuite source definition

7358ff0

wjwatkinson marked this pull request as ready for review July 1, 2022 19:33

fixed conflict

b5cf2c5

wjwatkinson added 3 commits July 1, 2022 15:47

correct PR link

ccc495a

fix tests

ddfc478

remove concurrency param

f0f92ea

Merge branch 'master' into source-netsuite

1637999

marcosmarxm changed the title ~~New Source: Netsuite~~ 🎉 New Source: Netsuite Aug 8, 2022

misteryeo assigned drrest Aug 10, 2022

bazarnov closed this Sep 22, 2022

swatosh mentioned this pull request Nov 28, 2022

🐛 Source Netsuite: Determine request date format #17470

Closed

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: Netsuite #13086

🎉 New Source: Netsuite #13086

wjwatkinson commented May 23, 2022 •

edited

Loading

wjwatkinson commented May 23, 2022

wjwatkinson commented May 23, 2022

wjwatkinson commented May 25, 2022

marcosmarxm commented May 26, 2022

marcosmarxm May 26, 2022

wjwatkinson Jun 6, 2022

wjwatkinson commented Jul 1, 2022

marcosmarxm commented Jul 27, 2022

misteryeo commented Aug 10, 2022

swatosh commented Sep 21, 2022

bazarnov commented Sep 22, 2022

wjwatkinson commented Sep 26, 2022

misteryeo commented Sep 26, 2022

wjwatkinson commented Sep 26, 2022

bazarnov commented Sep 26, 2022

wjwatkinson commented Sep 30, 2022

bazarnov commented Sep 30, 2022

wjwatkinson commented Sep 30, 2022

🎉 New Source: Netsuite #13086

🎉 New Source: Netsuite #13086

Conversation

wjwatkinson commented May 23, 2022 • edited Loading

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

wjwatkinson commented May 23, 2022

wjwatkinson commented May 23, 2022

wjwatkinson commented May 25, 2022

marcosmarxm commented May 26, 2022

marcosmarxm May 26, 2022

Choose a reason for hiding this comment

wjwatkinson Jun 6, 2022

Choose a reason for hiding this comment

wjwatkinson commented Jul 1, 2022

marcosmarxm commented Jul 27, 2022

misteryeo commented Aug 10, 2022

swatosh commented Sep 21, 2022

bazarnov commented Sep 22, 2022

wjwatkinson commented Sep 26, 2022

misteryeo commented Sep 26, 2022

wjwatkinson commented Sep 26, 2022

bazarnov commented Sep 26, 2022

wjwatkinson commented Sep 30, 2022

bazarnov commented Sep 30, 2022

wjwatkinson commented Sep 30, 2022

wjwatkinson commented May 23, 2022 •

edited

Loading