🐛 Source Delighted: Survey_Responses stream limited to 100 rows per sync - Date Range being Skipped

[pprithvi]

Airbyte version: 0.30.18-alpha
OS Version / Instance: Linux , AWS EC2
Deployment: Docker
Source Connector and version: Delighted 0.1.0
Destination Connector and version: AWS S3 0.2.0
Severity: Critical
Step where error happened: Sync Job

Current Behavior

While syncing survey_responses table, each sync is only pushing 100 rows while there is more data to be pushed. The first sync(historical sync) is also only pushing 100 records. Also it is seen that data between certain time period is skipped. For ex: Data between november 2020 to august 2021 is skipped in the syncs While connecting to the API directly, was able to confirm data do exists for that time frame.

Expected Behavior

First sync after reset should move entire historical data from start dat but does not. Data with a certain Created date range is being skipped which it should not.

logs-5968-0.txt

[alafanechere]

Hi @tredencegithub, could you please check if tweaking the since parameters changes the number of survey responses you get?

[pprithvi]

@alafanechere I tried with a different date but it still put out only 100 rows.

[Zirochkaa]

Scoping

I created 2 subtasks since there are two issues described in this ticket:

• 🐛 Source Delighted: incorrect pagination handling in survey_responses, bounces and unsubscribes streams #9271
• 🐛 Source Delighted: Date Range being Skipped #9273

Next investigations will be posted in those tickets.

[pprithvi]

@Zirochkaa This issue still exists. The date range skip has been fixed but the row limit still exists.
logs-9135-0.txt

As u can see, 139 rows are emitting every single time.

[Zirochkaa]

@tredencegithub I have few questions:

1. How many records do you have for each stream?
2. What is your initial config? If it's ok with you could you please send it to me in slack because I can't reproduce this issue with our config? I just run delighted source again with our account where each streams have 200+, 200+, 300+ and 100+ records and all of them were read. I then run incremental sync and again only new records were synced. So as you can see i can't reproduce this in our delighted account. That's why I asked you if it's ok for you to share with me in slack your config.

[pprithvi]

@Zirochkaa Unfortunately that is account specific information and am not authorized to share it. Do we have an alternative way?
My question is: The first sync that happens , should be a full refresh, hence all historic records must be read right? Th following syncs should be incremental based on live updates. Hence even as per your example also the first sync must be 800 records( 200+200+300+100) post which it would be the new records.

[Zirochkaa]

@tredencegithub could you please create new 0.1.2 delighted source and run full_refresh sync and incremental sync few times and send logs from all runs here? We need logs for investigation.
Also, could you please provide following info:

1. How many records does your delighted account contain by each stream?
2. When were the earliest/latest records created by each stream? For example: for stream "people" the earliest record was created at 1641289583 and the latest was at 1641455286.

[pprithvi]

@Zirochkaa Here is the sync logs: 1 full refresh and 4 subsequent syncs. I dont have access to the UI. The marketing team handles the same. I can try getting that information.

logs-9470-0.txt
logs-9469-0.txt
logs-9468-0.txt
logs-9467-0.txt
logs-9466-0.txt
logs-9465-0.txt

[Zirochkaa]

@tredencegithub thanks for your logs. I investigated them and I also investigated sync logs for our delighted account and I can tell that there is no issue about row limit which you mentioned here. The Delighted API supports since or updated_since parameters and they require Unix timestamp in order to restrict responses to those updated on or after this time. So, for example, if we send since=1641289583 this means that records which were created at 1641289583 or later will be returned. In your case it was:

• 1 record for bounces stream (because only one record has created_at >= 1632420528);
• 136 records for people stream (because 136 record have created_at >= 1614731699);
• 1 record for survey_responses stream (because only one record has updated_at >= 1616167778);
• 1 record for unsubscribes stream (because only one record has unsubscribed_at >= 1614783627).

I created a PR in which we output only new records. So for example, if state has 1641289583 value, then we will output only records which are greater or equal to 1641289584.