-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🐛 Source Netsuite: Determine request date format #17470
Conversation
So that the first call can figure out which date format Netsuite wants to use
Hey @swatosh, the integration tests are failing for your PR. Would you be able to debug them? I tested it locally and they're failing in the Another thing you'll need to do before merging this is bump up the connector version in Thanks for your contribution :) |
Sure I'll have a look. I didn't have much luck getting the tests to run while I was working on this so I was just hoping for the best because I tried to use the original format first. I did finally get the tests running (they take FOREVER) so if I can recreate the failure I hope to be able to fix it. |
I must be doing something wrong. I get 5 errors when running against master at aedd535 ======================================================================================================================================================================== short test summary info ========================================================================================================================================================================= @natalyjazzviolin any suggestions? Just those 5 tests take 45 minutes to run, so turn around on this isn't going to be lightening fast.... |
From what I can tell, this should address #17357 Can anyone help me figure out how to get the source acceptance tests passing on master? |
Sorry for the long delay - I've been trying to run the tests locally, but they are taking 1 hour +. From the info we have so far I think there is a problem in how stream slices are implemented, hence Could you debug the P.S. you can comment out the tests you don't want to run while working on this in |
/test connector=connectors/source-netsuite
Build FailedTest summary info:
|
@sajarin are this going to be assign to a maintainer? |
I did modify I'm still not able to get the tests passing on master so I'm not certain that I'm seeing the same failure you are on the branch. |
I can have a look at this @sajarin |
Hi @swatosh sorry for the delay. I was able to run all the integration tests successfully on my local and am wondering if you can simply do a version upgrade in the Dockerfile to version 0.1.2. Also if possible could you share a screenshot of your acceptance tests. Thank you! |
/test connector=connectors/source-netsuite |
/test connector=connectors/source-netsuite
Build FailedTest summary info:
|
/test connector=connectors/source-netsuite
Build FailedTest summary info:
|
Appears to me to still be a employee/contact issue (this time buried in the href):
vs
Do these tests reliably pass on master? |
No, I am getting the same behaviour in master after running a few times. My previous two ignores were in regards to href as well located in owner and assigned. I did not notice this one swapping also but I suppose we can add links to be ignored for these two feeds also. Assuming thats the last field that is having this behaviour we should be OK after. Could you give that a try? |
It really doesn't feel like this branch is the best place to fix failures that happen on master, but if you send me changes to the acceptance test yml file, I'm willing to try them |
Hello 👋, first thank you for this amazing contribution. We really appreciate the effort you've made to improve the project. If you have any questions feel free to send me a message in Slack! |
@swatosh Hello, could you please give a reference where you saw the Netsuite input date format has been changed, based on the documentation? Thank you! I've just run the local tests using the Could the date issue be related to your account setting directly? If so, we would probably need to expose this setting up to the user while setting up the connector. |
Hi there, I can confirm this behaviour, I had a setting DD/MM/YYYY then changed it to MM/DD/YYYY then ran the pipeline and it was completed successfully agreed that the setting may be a config setting from the UI Cheers! |
@swatosh @edgarvaldez @marcosmarxm |
In a word, "No." I never found anything documenting this change. I haven't tried looking again and it's been several months now, so things may have changed. We had Airbyte running, successfully collecting data from our Netsuite sandbox (this was before #16093 was merged, and to be honest, I can't remember if we were using it or #13086). When our sandbox instance was updated the connector stopped working. I found a failing request, made a similar one with postman, got an extraordinarily useful error message saying that the date format was bad and what it should be, when I tried that new date format, it started working. I made several local branches with different approaches, mostly using some sort of config, but settled on this one to submit because it didn't make changes to the interface. Obviously, that rested on several assumptions any of which being invalid would invalidate the approach. |
Great! I'll happily close this PR! We were just trying to help an open source project that helped us. We'll look forward to seeing your fixes committed |
@swatosh this is under research, your PR is a good reference at some places, let's keep it, if we could modify it easily - we'll go with it |
It just seems like completely the wrong approach if my understanding that you want to add a new configuration for the data format is correct? |
It's not, this PR covers your case, specifically, but to cover 95% of all customers with their possible format deviations we should have a generic wide approach here, since with introducing the new field for format input we might have even more bugs, this time with user's understanding of the format correctness. |
@bazarnov can you provide a link the the netsuite documentation describing how to configure the date formats? That'd be great! (this is the only place that I've heard that they might be configurable) I had a different version of this PR that took a boolean setting to switch between the formats, but I really didn't feel up to trying to document that to a level that would make sense to anyone besides me. It sounds to me like you're talking about a setting that is just a date format string, that seems even harder to document comprehensibly. In any event, the particulars of the implementation here was to minimize changes for the user -- we don't have any particular need for this solution to the date formatting problem to be the one selected. Like everyone else, we just want the problem solved! :-) |
In order to minimize this problem even further the best way here, is to make the connector determine the date format dynamically based on authenticated user preferences, it's possible using SuiteScript, but I'm not seeing the same for the API calls within the official docs.
The setting you're looking for might be by the following Netsuite web page, could you please check it and the actual timezone set nearby that place: From the Netsuite Web UI go to: |
@swatosh Could you please also check if this issue occurs for all the streams (Netsuite objects) in your case or for some specific ones? Thanks. |
Some more light on the problem: https://www.reddit.com/r/Netsuite/comments/kxtdgw/comment/gjczoms/?utm_source=share&utm_medium=web2x&context=3 |
This is interesting. It's basically the opposite of what is happening now since the original implementation used slashes, and my implementation retries the ISO format if there is a particular failure I'm currently locked out of our instance, so answers to the other questions will have to wait a bit. Sorry about that |
@swatosh Do you have any updates? |
Hey @swatosh can you address the above comments ASAP? Thanks! |
It does seem to depend on the state of my date formatting preferences. Which as I think about it makes sense, but I was thinking of it as a separate account accessing the data for M2M, not as a token to access the data as me. Given that, I suspect the best solution to this is to add the date format setting that was talked about earlier, and as part of the documentation saying that its value should be copied from Netsuite |
If all those assumptions mentioned above and your experience are the same now you did a great job @swatosh. Thank you for your time reproducing the problem and this PR as well. The only issue here is not a date format even, but the way a user interacts with this setting, it should be convenient, ideally - automatically for the user. I'm not seeing the API reference for this elsewhere. @swatosh Could you please provide your format so we could try yours on our account? |
Hello 👋:skin-tone-2: and thank you for your contribution! Airbyte has instituted a code freeze between 19 and 30 December, to make sure there are no disruptions during the holidays. If you have any questions or need further clarification, please don't hesitate to ping via Slack. |
What
We'd cloned and used the version at wjwatkinson:source-netsuite and were delighted to see one appear in the airbyte repo. However, when we cloned and tried to use it, it didn't work. After brief debugging we determined that Netsuite had changed the format of the date input
How
read_records
is overridden so that the first time it calls the new_determine_input_datetime_format
method.The new
_determine_input_datetime_format
method sets theinput_datetime_format
to the original format and ties to hit the URL (with a limit of one record to be returned), if it fails for a bad date format, theinput_datetime_format
is updated to the format that worked for us. Succeed or fail it has setinput_datetime_format
.read_records
continues by forwarding to the overridden method to try to get the actual data. Presumably if it fails, at this point it won’t be due to a bad date format. All subsequent calls toread_records
will use the format determined in the first call for the lifetime of the connector.Recommended reading order
To make it work, we deferred the formatting of the date from
stream_slices
torequest_params
🚨 User Impact 🚨
Connector works for either format of date that Netsuite might be using
Pre-merge Checklist
Updating a connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereTests
Unit
Put your unit tests output here.
Integration
Put your integration tests output here.
Acceptance
Put your acceptance tests output here.