Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Source Netsuite: Determine request date format #17470

Closed
wants to merge 13 commits into from

Conversation

swatosh
Copy link

@swatosh swatosh commented Sep 30, 2022

What

We'd cloned and used the version at wjwatkinson:source-netsuite and were delighted to see one appear in the airbyte repo. However, when we cloned and tried to use it, it didn't work. After brief debugging we determined that Netsuite had changed the format of the date input

How

read_records is overridden so that the first time it calls the new _determine_input_datetime_format method.
The new _determine_input_datetime_format method sets the input_datetime_format to the original format and ties to hit the URL (with a limit of one record to be returned), if it fails for a bad date format, the input_datetime_format is updated to the format that worked for us. Succeed or fail it has set input_datetime_format.
read_records continues by forwarding to the overridden method to try to get the actual data. Presumably if it fails, at this point it won’t be due to a bad date format. All subsequent calls to read_records will use the format determined in the first call for the lifetime of the connector.

Recommended reading order

To make it work, we deferred the formatting of the date from stream_slices to request_params

🚨 User Impact 🚨

Connector works for either format of date that Netsuite might be using

Pre-merge Checklist

Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

@CLAassistant
Copy link

CLAassistant commented Sep 30, 2022

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions bot added the area/connectors Connector related issues label Sep 30, 2022
@sajarin sajarin added the bounty-S Maintainer program: claimable small bounty PR label Oct 3, 2022
@natalyjazzviolin
Copy link
Contributor

Hey @swatosh, the integration tests are failing for your PR. Would you be able to debug them?

I tested it locally and they're failing in the TestIncremental.test_read_sequential_slices test. Here's the documentation for this one, I think the problem is probably in how configured_catalog.json is written.
https://docs.airbyte.com/connector-development/testing-connectors/source-acceptance-tests-reference/#testreadsequentialslices

Another thing you'll need to do before merging this is bump up the connector version in Dockerfile!

Thanks for your contribution :)

@swatosh
Copy link
Author

swatosh commented Oct 4, 2022

Sure I'll have a look. I didn't have much luck getting the tests to run while I was working on this so I was just hoping for the best because I tried to use the original format first. I did finally get the tests running (they take FOREVER) so if I can recreate the failure I hope to be able to fix it.

@swatosh
Copy link
Author

swatosh commented Oct 4, 2022

I must be doing something wrong. I get 5 errors when running against master at aedd535

======================================================================================================================================================================== short test summary info =========================================================================================================================================================================
FAILED ../../bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs0] - AssertionError: Found unresolved $refs values for selected streams: ({'customrecord_advpromo_discount': ['/services/rest/record/v1/metadata-catalog/inventoryitem', '/services/rest/record/v1/metadata-catalog/assem...
FAILED ../../bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.ContainerError: Command 'read --config /data/tap_config.json --catalog /data/catalog.json' in image '<Image: 'airbyte/source-netsuite:dev'>' returned non-zero exit status 1: {"type": "TRACE", "trace": {"type": "ERROR", "emit...
FAILED ../../bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0] - docker.errors.ContainerError: Command 'read --config /data/tap_config.json --catalog /data/catalog.json' in image '<Image: 'airbyte/source-netsuite:dev'>' returned non-zero exit status 1: {"type": "TRACE", "trace": {...
FAILED ../../bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0] - AssertionError: Should produce at least one record
FAILED ../../bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0] - AssertionError: Should produce at least one record

@natalyjazzviolin any suggestions?

Just those 5 tests take 45 minutes to run, so turn around on this isn't going to be lightening fast....

@swatosh
Copy link
Author

swatosh commented Oct 10, 2022

From what I can tell, this should address #17357

Can anyone help me figure out how to get the source acceptance tests passing on master?

@natalyjazzviolin
Copy link
Contributor

natalyjazzviolin commented Oct 18, 2022

Sorry for the long delay - I've been trying to run the tests locally, but they are taking 1 hour +. From the info we have so far I think there is a problem in how stream slices are implemented, hence TestIncremental.test_read_sequential_slices is failing. The discovery and full refresh tests are passing for me, I wonder if there is something different in your setup that is making those tests fail.

Could you debug the stream_slices function on 217? I'm waiting for tests to run on master to see if they are passing, but form the PR it looks like the incremental stream test was passing before:
#16093

P.S. you can comment out the tests you don't want to run while working on this in acceptance-test-config.yml, that will cut down on the time it takes.

@natalyjazzviolin
Copy link
Contributor

natalyjazzviolin commented Oct 18, 2022

/test connector=connectors/source-netsuite

🕑 connectors/source-netsuite https://github.com/airbytehq/airbyte/actions/runs/3276052245
❌ connectors/source-netsuite https://github.com/airbytehq/airbyte/actions/runs/3276052245
🐛 https://gradle.com/s/vfbxwq33vulo4

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestDiscovery::test_discover[inputs0] - docker.errors.Co...
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
============== 3 failed, 25 passed, 1 error in 3734.69s (1:02:14) ==============

@marcosmarxm
Copy link
Member

@sajarin are this going to be assign to a maintainer?

@swatosh
Copy link
Author

swatosh commented Oct 19, 2022

I did modify stream_slices to not format the dates so that we could try different date formats in request_parameters lines 212 and 213. Not clear to me why that might be a problem.

I'm still not able to get the tests passing on master so I'm not certain that I'm seeing the same failure you are on the branch.

@trowacat
Copy link
Contributor

I can have a look at this @sajarin

@trowacat
Copy link
Contributor

trowacat commented Nov 6, 2022

Hi @swatosh sorry for the delay. I was able to run all the integration tests successfully on my local and am wondering if you can simply do a version upgrade in the Dockerfile to version 0.1.2.

Also if possible could you share a screenshot of your acceptance tests. Thank you!

@trowacat
Copy link
Contributor

trowacat commented Nov 6, 2022

/test connector=connectors/source-netsuite

@sajarin
Copy link
Contributor

sajarin commented Nov 7, 2022

/test connector=connectors/source-netsuite

🕑 connectors/source-netsuite https://github.com/airbytehq/airbyte/actions/runs/3412234856
❌ connectors/source-netsuite https://github.com/airbytehq/airbyte/actions/runs/3412234856
🐛 https://gradle.com/s/ynltvkkm3wrr2

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
============ 2 failed, 27 passed, 29 warnings in 6058.58s (1:40:58) ============

@sajarin
Copy link
Contributor

sajarin commented Nov 10, 2022

/test connector=connectors/source-netsuite

🕑 connectors/source-netsuite https://github.com/airbytehq/airbyte/actions/runs/3440646358
❌ connectors/source-netsuite https://github.com/airbytehq/airbyte/actions/runs/3440646358
🐛 https://gradle.com/s/zfpp6vpugupxq

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
============ 1 failed, 28 passed, 29 warnings in 6517.11s (1:48:37) ============

@swatosh
Copy link
Author

swatosh commented Nov 11, 2022

/test connector=connectors/source-netsuite

🕑 connectors/source-netsuite https://github.com/airbytehq/airbyte/actions/runs/3440646358
❌ connectors/source-netsuite https://github.com/airbytehq/airbyte/actions/runs/3440646358
🐛 https://gradle.com/s/zfpp6vpugupxq

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
============ 1 failed, 28 passed, 29 warnings in 6517.11s (1:48:37) ============

Appears to me to still be a employee/contact issue (this time buried in the href):


"links": [


"rel": "self",
"href": "https://7111959-sb2.suitetalk.api.netsuite.com/services/rest/record/v1/task/37?expandSubResources=True"

],
"accessLevel": true,
"assigned": ***
"links": [
***
"rel": "self",
"href": "https://7111959-sb2.suitetalk.api.netsuite.com/services/rest/record/v1/employee/4"
***
],
"id": "4",
"refName": "Integration User"
***,

vs


"links": [


"rel": "self",
"href": "https://7111959-sb2.suitetalk.api.netsuite.com/services/rest/record/v1/task/37?expandSubResources=True"

],
"accessLevel": true,
"assigned": ***
"links": [
***
"rel": "self",
"href": "https://7111959-sb2.suitetalk.api.netsuite.com/services/rest/record/v1/contact/4"
***
],
"id": "4",
"refName": "Integration User"
***,

Do these tests reliably pass on master?

@trowacat
Copy link
Contributor

/test connector=connectors/source-netsuite

🕑 connectors/source-netsuite https://github.com/airbytehq/airbyte/actions/runs/3440646358

❌ connectors/source-netsuite https://github.com/airbytehq/airbyte/actions/runs/3440646358

🐛 https://gradle.com/s/zfpp6vpugupxq

Build Failed

Test summary info:

=========================== short test summary info ============================

FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]

============ 1 failed, 28 passed, 29 warnings in 6517.11s (1:48:37) ============

Appears to me to still be a employee/contact issue (this time buried in the href):


"links": [


"rel": "self",
"href": "https://7111959-sb2.suitetalk.api.netsuite.com/services/rest/record/v1/task/37?expandSubResources=True"

],

"accessLevel": true,

"assigned": ***

"links": [

***
 "rel": "self",
 "href": "https://7111959-sb2.suitetalk.api.netsuite.com/services/rest/record/v1/employee/4"
***

],

"id": "4",

"refName": "Integration User"

***,

vs


"links": [


"rel": "self",
"href": "https://7111959-sb2.suitetalk.api.netsuite.com/services/rest/record/v1/task/37?expandSubResources=True"

],

"accessLevel": true,

"assigned": ***

"links": [

***
 "rel": "self",
 "href": "https://7111959-sb2.suitetalk.api.netsuite.com/services/rest/record/v1/contact/4"
***

],

"id": "4",

"refName": "Integration User"

***,

Do these tests reliably pass on master?

No, I am getting the same behaviour in master after running a few times. My previous two ignores were in regards to href as well located in owner and assigned. I did not notice this one swapping also but I suppose we can add links to be ignored for these two feeds also. Assuming thats the last field that is having this behaviour we should be OK after. Could you give that a try?

@swatosh
Copy link
Author

swatosh commented Nov 14, 2022

Do these tests reliably pass on master?

No, I am getting the same behaviour in master after running a few times. My previous two ignores were in regards to href as well located in owner and assigned. I did not notice this one swapping also but I suppose we can add links to be ignored for these two feeds also. Assuming thats the last field that is having this behaviour we should be OK after. Could you give that a try?

It really doesn't feel like this branch is the best place to fix failures that happen on master, but if you send me changes to the acceptance test yml file, I'm willing to try them

@marcosmarxm
Copy link
Member

Hello 👋, first thank you for this amazing contribution.

We really appreciate the effort you've made to improve the project.
We ask you patience for the code review. Last month our team was focused on Hacktoberfest event and that probably left some PR without the proper feedback. And this week, due to the Thanksgiving US Holiday, most our team is out of office with their families. Another important piece of information why code won't be merge this week is: as a safety measure the core team has decided to freeze merging code to main branch to keep the release stable. Next week we'll return to you with the proper code review and update the status of your contribution.

If you have any questions feel free to send me a message in Slack!
Thanks!

@bazarnov
Copy link
Collaborator

bazarnov commented Nov 24, 2022

@swatosh Hello, could you please give a reference where you saw the Netsuite input date format has been changed, based on the documentation? Thank you!

I've just run the local tests using the old date format as input_datetime_format = "%m/%d/%Y" with no issues.

Could the date issue be related to your account setting directly? If so, we would probably need to expose this setting up to the user while setting up the connector.

@edgarvaldez
Copy link

@swatosh Hello, could you please give a reference where you saw the Netsuite input date format has been changed, based on the documentation? Thank you!

I've just run the local tests using the old date format as input_datetime_format = "%m/%d/%Y" with no issues.

Could the date issue be related to your account setting directly? If so, we would probably need to expose this setting up to the user while setting up the connector.

Hi there, I can confirm this behaviour, I had a setting DD/MM/YYYY then changed it to MM/DD/YYYY then ran the pipeline and it was completed successfully

agreed that the setting may be a config setting from the UI

Cheers!

@bazarnov
Copy link
Collaborator

@swatosh @edgarvaldez @marcosmarxm
The reason this change is not a silver bullet is that we should probably have the possibility to fetch the actual date format set for the account, for the authenticated user (and related account used) directly using the API call. I'm working on it, updates soon.

@swatosh
Copy link
Author

swatosh commented Nov 28, 2022

@swatosh Hello, could you please give a reference where you saw the Netsuite input date format has been changed, based on the documentation? Thank you!

I've just run the local tests using the old date format as input_datetime_format = "%m/%d/%Y" with no issues.

Could the date issue be related to your account setting directly? If so, we would probably need to expose this setting up to the user while setting up the connector.

In a word, "No." I never found anything documenting this change. I haven't tried looking again and it's been several months now, so things may have changed.

We had Airbyte running, successfully collecting data from our Netsuite sandbox (this was before #16093 was merged, and to be honest, I can't remember if we were using it or #13086). When our sandbox instance was updated the connector stopped working. I found a failing request, made a similar one with postman, got an extraordinarily useful error message saying that the date format was bad and what it should be, when I tried that new date format, it started working.

I made several local branches with different approaches, mostly using some sort of config, but settled on this one to submit because it didn't make changes to the interface. Obviously, that rested on several assumptions any of which being invalid would invalidate the approach.

@swatosh
Copy link
Author

swatosh commented Nov 28, 2022

@swatosh @edgarvaldez @marcosmarxm The reason this change is not a silver bullet is that we should probably have the possibility to fetch the actual date format set for the account, for the authenticated user (and related account used) directly using the API call. I'm working on it, updates soon.

Great! I'll happily close this PR! We were just trying to help an open source project that helped us. We'll look forward to seeing your fixes committed

@bazarnov
Copy link
Collaborator

Great! I'll happily close this PR! We were just trying to help an open source project that helped us. We'll look forward to seeing your fixes committed

@swatosh this is under research, your PR is a good reference at some places, let's keep it, if we could modify it easily - we'll go with it

@swatosh
Copy link
Author

swatosh commented Nov 28, 2022

Great! I'll happily close this PR! We were just trying to help an open source project that helped us. We'll look forward to seeing your fixes committed

@swatosh this is under research, your PR is a good reference at some places, let's keep it, if we could modify it easily - we'll go with it

It just seems like completely the wrong approach if my understanding that you want to add a new configuration for the data format is correct?

@bazarnov
Copy link
Collaborator

Great! I'll happily close this PR! We were just trying to help an open source project that helped us. We'll look forward to seeing your fixes committed

@swatosh this is under research, your PR is a good reference at some places, let's keep it, if we could modify it easily - we'll go with it

It just seems like completely the wrong approach if my understanding that you want to add a new configuration for the data format is correct?

It's not, this PR covers your case, specifically, but to cover 95% of all customers with their possible format deviations we should have a generic wide approach here, since with introducing the new field for format input we might have even more bugs, this time with user's understanding of the format correctness.

@swatosh
Copy link
Author

swatosh commented Nov 29, 2022

It's not, this PR covers your case, specifically, but to cover 95% of all customers with their possible format deviations we should have a generic wide approach here, since with introducing the new field for format input we might have even more bugs, this time with user's understanding of the format correctness.

@bazarnov can you provide a link the the netsuite documentation describing how to configure the date formats? That'd be great! (this is the only place that I've heard that they might be configurable)

I had a different version of this PR that took a boolean setting to switch between the formats, but I really didn't feel up to trying to document that to a level that would make sense to anyone besides me. It sounds to me like you're talking about a setting that is just a date format string, that seems even harder to document comprehensibly.

In any event, the particulars of the implementation here was to minimize changes for the user -- we don't have any particular need for this solution to the date formatting problem to be the one selected. Like everyone else, we just want the problem solved! :-)

@bazarnov
Copy link
Collaborator

bazarnov commented Dec 1, 2022

@swatosh

In any event, the particulars of the implementation here was to minimize changes for the user -- we don't have any particular need for this solution to the date formatting problem to be the one selected. Like everyone else, we just want the problem solved! :-)

In order to minimize this problem even further the best way here, is to make the connector determine the date format dynamically based on authenticated user preferences, it's possible using SuiteScript, but I'm not seeing the same for the API calls within the official docs.

It sounds to me like you're talking about a setting that is just a date format string, that seems even harder to document comprehensible.

The setting you're looking for might be by the following Netsuite web page, could you please check it and the actual timezone set nearby that place:

From the Netsuite Web UI go to:
timezone: HOME Icon > Set Preferences > Localization section > TIMEZONE
date format: HOME Icon > Set Preferences > Formating section > DATE FORMAT

@bazarnov
Copy link
Collaborator

bazarnov commented Dec 1, 2022

@swatosh Could you please also check if this issue occurs for all the streams (Netsuite objects) in your case or for some specific ones? Thanks.

@bazarnov
Copy link
Collaborator

bazarnov commented Dec 1, 2022

@swatosh
Copy link
Author

swatosh commented Dec 1, 2022

Some more light on the problem: https://www.reddit.com/r/Netsuite/comments/kxtdgw/comment/gjczoms/?utm_source=share&utm_medium=web2x&context=3

This is interesting. It's basically the opposite of what is happening now since the original implementation used slashes, and my implementation retries the ISO format if there is a particular failure

I'm currently locked out of our instance, so answers to the other questions will have to wait a bit. Sorry about that

@bazarnov bazarnov self-requested a review December 1, 2022 21:49
@bazarnov
Copy link
Collaborator

bazarnov commented Dec 5, 2022

Some more light on the problem: https://www.reddit.com/r/Netsuite/comments/kxtdgw/comment/gjczoms/?utm_source=share&utm_medium=web2x&context=3

This is interesting. It's basically the opposite of what is happening now since the original implementation used slashes, and my implementation retries the ISO format if there is a particular failure

I'm currently locked out of our instance, so answers to the other questions will have to wait a bit. Sorry about that

@swatosh Do you have any updates?

@sajarin
Copy link
Contributor

sajarin commented Dec 12, 2022

Hey @swatosh can you address the above comments ASAP? Thanks!

@swatosh
Copy link
Author

swatosh commented Dec 13, 2022

Hey @swatosh can you address the above comments ASAP? Thanks!

It does seem to depend on the state of my date formatting preferences. Which as I think about it makes sense, but I was thinking of it as a separate account accessing the data for M2M, not as a token to access the data as me.

Given that, I suspect the best solution to this is to add the date format setting that was talked about earlier, and as part of the documentation saying that its value should be copied from Netsuite UI HOME Icon > Set Preferences > Formating section > DATE FORMAT and just closing this PR

@bazarnov
Copy link
Collaborator

Hey @swatosh can you address the above comments ASAP? Thanks!

It does seem to depend on the state of my date formatting preferences. Which as I think about it makes sense, but I was thinking of it as a separate account accessing the data for M2M, not as a token to access the data as me.

Given that, I suspect the best solution to this is to add the date format setting that was talked about earlier, and as part of the documentation saying that its value should be copied from Netsuite UI HOME Icon > Set Preferences > Formating section > DATE FORMAT and just closing this PR

If all those assumptions mentioned above and your experience are the same now you did a great job @swatosh. Thank you for your time reproducing the problem and this PR as well. The only issue here is not a date format even, but the way a user interacts with this setting, it should be convenient, ideally - automatically for the user. I'm not seeing the API reference for this elsewhere.

@swatosh Could you please provide your format so we could try yours on our account?

@marcosmarxm
Copy link
Member

Hello 👋:skin-tone-2: and thank you for your contribution!

Airbyte has instituted a code freeze between 19 and 30 December, to make sure there are no disruptions during the holidays.
Because of this, reviewing and merging your contribution may take longer than usual.
We apologize for the delay, but we want everyone to have a quiet and happy holiday.

If you have any questions or need further clarification, please don't hesitate to ping via Slack.

@bazarnov
Copy link
Collaborator

bazarnov commented Jan 4, 2023

Closed as duplicate to: #19798.

@swatosh thank you for your time and patience!

@bazarnov bazarnov closed this Jan 4, 2023
@swatosh swatosh deleted the fix-date-format branch January 6, 2023 00:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Development

Successfully merging this pull request may close these issues.