🐛 Source Amazon Seller Partner: only 100 orders are read.

[alafanechere]

Environment

• Airbyte version: 0.29.22-alpha
• OS Version / Instance:Ubuntu 20.04/ Ec2 t3a.xlarge
• Deployment: Docker
• Source Connector and version:Amazon Seller Partner/0.2.6
• Destination Connector and version: Snowflake
• Severity: Critical
• Step where error happened: Sync job

Current Behavior

This problem was raised by Sastry Mantravadi on Slack
In full refresh mode, the connector only reads 100 orders from the order stream. It's the max page size according to the API so this problem might be related to the pagination logic.
No error appears during the sync.

Expected Behavior

All orders should be read by the connector in a full refresh mode.

Logs

Amazon Seller Partner Full refresh logs

Steps to Reproduce

1. Have an Amazon Seller partner account with more than 100 orders.
2. Create a connection with a Amazon Seller Partner source
3. Select the order stream
4. Run the sync

[prudhvi85]

Hi,

There is an issue in the next_page_token function. It always returns None because stream_data does not have "NextToken" key in the root level.

def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
        stream_data = response.json()
        #NextToken is nested inside payload
        next_page_token = stream_data.get("payload").get(self.next_page_token_field)
        if next_page_token:
            return {self.next_page_token_field: next_page_token}

in request_params method( Class Orders) MarketplaceIds should be part of params irrespective of whether there is a next_page _token. Removing the redundant if condition solves the issue.

def request_params(
        self, stream_state: Mapping[str, Any], next_page_token: Mapping[str, Any] = None, **kwargs
    ) -> MutableMapping[str, Any]:
        params = super().request_params(stream_state=stream_state, next_page_token=next_page_token, **kwargs)
        #redudant if check
        #if not next_page_token: 
        params.update({"MarketplaceIds": ",".join(self.marketplace_ids)})
        return params

[alafanechere]

Hi @prudhvi85 thank you very much for the investigation, do you plan to open a PR to patch this?

[prudhvi85]

@alafanechere I have raised a PR with the fixes. Can you please merge the pull request?

[htrueman]

There is PR for the issue #9115. It's approved, so may be merged any time soon.

[Zirochkaa]

Fix for this issue was already published in #9294.