🐛 Source Pinterest: no backoff strategy when rate limit is exceeded.

[alafanechere]

Environment

• Airbyte version: 0.32.0-alpha
• OS Version / Instance:Google cloud VM
• Deployment: Docker
• Source Connector and version: Pinterest 0.1.0
• Severity: Critical
• Step where error happened: SyncJob

Current Behavior

When rate limit is exceeded (on CampaignAnalytics stream) the sync stops for this stream and there's no retry nor backoff attempts made.

The should_retry method handles the 429 HTTP code only for logging purpose but does not retry on it which looks strange to me:

    def should_retry(self, response: requests.Response) -> bool:
        if response.status_code == 429:
            self.logger.error(f"For stream {self.name} rate limit exceeded.")
            setattr(self, "raise_on_http_errors", False)
        return 500 <= response.status_code < 600

Expected Behavior

Exponential backoff to handle rate limits on Pinterest API

Logs

Logs are provided by Matt Fleshman who raised this error on Slack.

logs-904-0.txt

Steps to Reproduce

1. Create a connection with Pinterest as source
2. Sync a CampaignAnalytics stream that makes more than 1000 calls (the limit is 1000calls / day).

[mattfleshman]

Thank you! Two more details: 1. I believe the rate limiting also happens on the AdGroupAnalytics stream. 2. We are using Advanced API Access and our rate limit should be 1000 calls a minute.

[lgomezm]

Here's a PR to fix this issue: #16161