Zendesk Source incremental sync is very slow

[olivier-rvezy]

Enviroment

Is this your first time deploying Airbyte: (I have upgraded the original install which was my first one, so not sure if I should post a Yes or No here)

• OS Version / Instance: Amzon Linux AMI
• Memory / Disk: 8 GB / ~ 100 GB
• Deployment: Docker
• Airbyte Version: v0.29.18-alpha
• Source name/version: Zendesk Support (0.1.1)
• Destination name/version: MSSQL Destination (0.1.8)
• Step where error happened: Sync job

Current Behavior

I have an incremental sync of my zendesk support source of all the which seems really slow. almost 2hrs for about 1000 records. Having the same problem syncing to BigQuery. From my logs, there seems to be a lot of sleeping in between setting up the tmp tables and starting the source sync. Then, in the middle of the process, it catches an exception and starts to sleep a lot.

Expected Behavior

Sync should be much faster, probably around 5 minutes for such a small update

Logs

LOG

Attached log file: [logs-314-0.txt](https://github.com/airbytehq/airbyte/files/7217808/logs-314-0.txt)

Steps to Reproduce

1. Setup a Zendesk Source
2. Setup a MSSQL Destination (had similar problems with a BigQuery destination so destination may not matter)
3. Run a sync

Are you willing to submit a PR?

No

[sherifnada]

@olivier-rvezy this looks like rate limiting from zendesk. We should do a better job communicating that this is the case. It doesn't look like normalization is the problem here. I've created an issue to address the logging here. The improved logs should give us a better idea of whether this is just rate limiting or something else.

[olivier-rvezy]

@sherifnada I don't see why the rate-limiting would happen. From the logs, it looks like it only starts happening when syncing satisfaction_ratings. Delved a bit deeper into the airbyte code, I believe that the logic for the 'Retrying' log handles both the rate limit and 500 errors and that the satisfaction_ratings integration is getting a server error there. Once I removed the satisfaction_ratings from the integration, and it looks like it succeeds in 6 minutes. I wasn't able to get the rate limit error when calling the satisfaction_ratings API from a local python script I wrote to confirm my suspicion