🐛 Source Sendgrid: Skip the stream on authorization required permis…

…sion error `[400, 401]` (#18847)

airbyte-config/init/src/main/resources/seed/source_definitions.yaml

@@ -1103,7 +1103,7 @@
 - name: Sendgrid
   sourceDefinitionId: fbb5fbe2-16ad-4cf4-af7d-ff9d9c316c87
   dockerRepository: airbyte/source-sendgrid
-  dockerImageTag: 0.2.15
+  dockerImageTag: 0.2.16
   documentationUrl: https://docs.airbyte.com/integrations/sources/sendgrid
   icon: sendgrid.svg
   sourceType: api

airbyte-config/init/src/main/resources/seed/source_specs.yaml

@@ -10954,7 +10954,7 @@
     supportsNormalization: false
     supportsDBT: false
     supported_destination_sync_modes: []
-- dockerImage: "airbyte/source-sendgrid:0.2.15"
+- dockerImage: "airbyte/source-sendgrid:0.2.16"
   spec:
     documentationUrl: "https://docs.airbyte.com/integrations/sources/sendgrid"
     connectionSpecification:

airbyte-integrations/connectors/source-sendgrid/Dockerfile

@@ -12,5 +12,5 @@ RUN pip install .
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.2.15
+LABEL io.airbyte.version=0.2.16
 LABEL io.airbyte.name=airbyte/source-sendgrid

airbyte-integrations/connectors/source-sendgrid/README.md

@@ -82,7 +82,7 @@ docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integrat
 Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.
 First install test dependencies into your virtual environment:
 ```
-pip install .[tests]
+pip install '.[tests]'
 ```
 ### Unit Tests
 To run unit tests locally, from the connector directory run:
@@ -102,7 +102,8 @@ Customize `acceptance-test-config.yml` file to configure tests. See [Source Acce
 If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
 To run your integration tests with acceptance tests, from the connector root, run
 ```
-python -m pytest integration_tests -p integration_tests.acceptance
+docker build . --no-cache -t airbyte/source-sendgrid:dev \
+&& python -m pytest -p source_acceptance_test.plugin
 ```
 To run your integration tests with docker
 

airbyte-integrations/connectors/source-sendgrid/source_sendgrid/streams.py

@@ -17,6 +17,11 @@ class SendgridStream(HttpStream, ABC):
     primary_key = "id"
     limit = 50
     data_field = None
+    raise_on_http_errors = True
+    permission_error_codes = {
+        400: "authorization required",
+        401: "authorization required",
+    }
 
     def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
         pass
@@ -48,6 +53,20 @@ def parse_response(
             # do NOT print request headers as it contains auth token
             self.logger.info(err_msg)
 
+    def should_retry(self, response: requests.Response) -> bool:
+        """Override to provide skip the stream possibility"""
+
+        status = response.status_code
+        if status in self.permission_error_codes.keys():
+            for message in response.json().get("errors", []):
+                if message.get("message") == self.permission_error_codes.get(status):
+                    self.logger.error(
+                        f"Stream `{self.name}` is not available, due to subscription plan limitations or perrmission issues. Skipping."
+                    )
+                    setattr(self, "raise_on_http_errors", False)
+                    return False
+        return 500 <= response.status_code < 600
+
 
 class SendgridStreamOffsetPagination(SendgridStream):
     offset = 0
@@ -75,8 +94,6 @@ class SendgridStreamIncrementalMixin(HttpStream, ABC):
     def __init__(self, start_time: Optional[Union[int, str]], **kwargs):
         super().__init__(**kwargs)
         self._start_time = start_time or 0
-        # for backward compatibility
-        self._start_time = start_time
         if isinstance(self._start_time, str):
             self._start_time = int(pendulum.parse(self._start_time).timestamp())
 

airbyte-integrations/connectors/source-sendgrid/unit_tests/unit_test.py

@@ -3,7 +3,7 @@
 #
 
 import unittest
-from unittest.mock import MagicMock, patch
+from unittest.mock import MagicMock, Mock, patch
 
 import pendulum
 import pytest
@@ -124,3 +124,36 @@ def test_read_records(
     records = list(stream.read_records(sync_mode=SyncMode))
 
     assert records == expected
+
+
+@pytest.mark.parametrize(
+    "stream_class, expected",
+    (
+        [Templates, "templates"],
+        [Lists, "marketing/lists"],
+        [Campaigns, "marketing/campaigns"],
+        [Contacts, "marketing/contacts"],
+        [Segments, "marketing/segments"],
+        [Blocks, "suppression/blocks"],
+        [SuppressionGroupMembers, "asm/suppressions"],
+        [SuppressionGroups, "asm/groups"],
+        [GlobalSuppressions, "suppression/unsubscribes"],
+    ),
+)
+def test_path(stream_class, expected):
+    stream = stream_class(Mock())
+    assert stream.path() == expected
+
+
+@pytest.mark.parametrize(
+    "stream_class, status, expected",
+    (
+        (Messages, 400, False),
+        (SuppressionGroupMembers, 401, False),
+    ),
+)
+def test_should_retry_on_permission_error(requests_mock, stream_class, status, expected):
+    stream = stream_class(Mock())
+    response_mock = MagicMock()
+    response_mock.status_code = status
+    assert stream.should_retry(response_mock) == expected

docs/integrations/sources/sendgrid.md

@@ -78,6 +78,7 @@ The connector is restricted by normal Sendgrid [requests limitation](https://sen
 
 | Version | Date       | Pull Request                                             | Subject                                               |
 |:--------|:-----------|:---------------------------------------------------------|:------------------------------------------------------|
+| 0.2.16  | 2022-11-02 | [18847](https://github.com/airbytehq/airbyte/pull/18847) | Skip the stream on `400, 401 - authorization required` with log message |
 | 0.2.15  | 2022-10-19 | [18182](https://github.com/airbytehq/airbyte/pull/18182) | Mark the sendgrid api key secret in the spec          |
 | 0.2.14  | 2022-09-07 | [16400](https://github.com/airbytehq/airbyte/pull/16400) | Change Start Time config parameter to datetime string |
 | 0.2.13  | 2022-08-29 | [16112](https://github.com/airbytehq/airbyte/pull/16112) | Revert back to Python CDK                             |