🐛 Source FB Marketing: deprecate INSIGHTS_DAYS_PER_JOB from connector's specification.

airbyte-config/init/src/main/resources/config/STANDARD_SOURCE_DEFINITION/e7778cfc-e97c-4458-9ecb-b4f2bba8946c.json

@@ -2,7 +2,7 @@
   "sourceDefinitionId": "e7778cfc-e97c-4458-9ecb-b4f2bba8946c",
   "name": "Facebook Marketing",
   "dockerRepository": "airbyte/source-facebook-marketing",
-  "dockerImageTag": "0.2.26",
+  "dockerImageTag": "0.2.27",
   "documentationUrl": "https://docs.airbyte.io/integrations/sources/facebook-marketing",
   "icon": "facebook.svg"
 }

airbyte-config/init/src/main/resources/seed/source_definitions.yaml

@@ -169,7 +169,7 @@
 - name: Facebook Marketing
   sourceDefinitionId: e7778cfc-e97c-4458-9ecb-b4f2bba8946c
   dockerRepository: airbyte/source-facebook-marketing
-  dockerImageTag: 0.2.26
+  dockerImageTag: 0.2.27
   documentationUrl: https://docs.airbyte.io/integrations/sources/facebook-marketing
   icon: facebook.svg
   sourceType: api

airbyte-integrations/connectors/source-facebook-marketing/Dockerfile

@@ -12,5 +12,5 @@ RUN pip install .
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.2.26
+LABEL io.airbyte.version=0.2.27
 LABEL io.airbyte.name=airbyte/source-facebook-marketing

airbyte-integrations/connectors/source-facebook-marketing/README.md

@@ -39,7 +39,7 @@ From the Airbyte repository root, run:
 **If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/facebook-marketing)
 to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_facebook_marketing/spec.json` file.
 Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
-See `sample_files/sample_config.json` for a sample config file.
+See `integration_tests/sample_config.json` for a sample config file.
 
 **If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source facebook-marketing test creds`
 and place them into `secrets/config.json`.
@@ -49,7 +49,7 @@ and place them into `secrets/config.json`.
 python main.py spec
 python main.py check --config secrets/config.json
 python main.py discover --config secrets/config.json
-python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json
+python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
 ```
 
 ### Locally running the connector docker image
@@ -73,7 +73,7 @@ Then run any of the connector commands as follows:
 docker run --rm airbyte/source-facebook-marketing:dev spec
 docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-facebook-marketing:dev check --config /secrets/config.json
 docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-facebook-marketing:dev discover --config /secrets/config.json
-docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/source-facebook-marketing:dev read --config /secrets/config.json --catalog /sample_files/configured_catalog.json
+docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/source-facebook-marketing:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
 ```
 ## Testing
    Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.

airbyte-integrations/connectors/source-facebook-marketing/acceptance-test-config.yml

@@ -14,14 +14,15 @@ tests:
   basic_read:
     - config_path: "secrets/config.json"
       configured_catalog_path: "integration_tests/configured_catalog.json"
-      timeout_seconds: 600
+      timeout_seconds: 1200
   incremental:
     - config_path: "secrets/config.json"
       configured_catalog_path: "integration_tests/configured_catalog_without_insights.json"
       future_state_path: "integration_tests/future_state.json"
   full_refresh:
     - config_path: "secrets/config.json"
       configured_catalog_path: "integration_tests/configured_catalog.json"
+      timeout_seconds: 1200
       # Ad Insights API has estimated metrics in response, which is calculated based on another metrics.
       # Sometimes API doesn't return estimated metrics. E.g, cost_per_estimated_ad_recallers is calculated
       # as total amount spent divided by estimated ad recall lift rate. When second metric is equal to zero

airbyte-integrations/connectors/source-facebook-marketing/integration_tests/spec.json

@@ -5,37 +5,43 @@
     "title": "Source Facebook Marketing",
     "type": "object",
     "properties": {
-      "account_id": {
-        "title": "Account Id",
-        "description": "The Facebook Ad account ID to use when pulling data from the Facebook Marketing API.",
-        "type": "string"
-      },
-      "access_token": {
-        "title": "Access Token",
-        "description": "The value of the access token generated. See the <a href=\"https://docs.airbyte.io/integrations/sources/facebook-marketing\">docs</a> for more information",
-        "airbyte_secret": true,
-        "type": "string"
-      },
       "start_date": {
         "title": "Start Date",
         "description": "The date from which you'd like to replicate data for AdCreatives and AdInsights APIs, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated.",
         "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$",
         "examples": ["2017-01-25T00:00:00Z"],
+        "order": 0,
         "type": "string",
         "format": "date-time"
       },
       "end_date": {
-        "title": "End Date",
+        "title": "End Date (Optional)",
         "description": "The date until which you'd like to replicate data for AdCreatives and AdInsights APIs, in the format YYYY-MM-DDT00:00:00Z. All data generated between start_date and this date will be replicated. Not setting this option will result in always syncing the latest data.",
         "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$",
         "examples": ["2017-01-26T00:00:00Z"],
+        "order": 1,
         "type": "string",
         "format": "date-time"
       },
+      "account_id": {
+        "title": "Account ID",
+        "description": "The Facebook Ad account ID to use when pulling data from the Facebook Marketing API.",
+        "examples": ["111111111111111"],
+        "order": 2,
+        "type": "string"
+      },
+      "access_token": {
+        "title": "Access Token",
+        "description": "The value of the access token generated. See the <a href=\"https://docs.airbyte.io/integrations/sources/facebook-marketing\">docs</a> for more information",
+        "airbyte_secret": true,
+        "order": 3,
+        "type": "string"
+      },
       "include_deleted": {
         "title": "Include Deleted",
         "description": "Include data from deleted campaigns, ads, and adsets.",
         "default": false,
+        "order": 4,
         "type": "boolean"
       },
       "insights_lookback_window": {
@@ -44,19 +50,13 @@
         "default": 28,
         "minimum": 0,
         "maximum": 28,
-        "type": "integer"
-      },
-      "insights_days_per_job": {
-        "title": "Insights Days Per Job",
-        "description": "Number of days to sync in one job. The more data you have - the smaller you want this parameter to be.",
-        "default": 7,
-        "minimum": 1,
-        "maximum": 30,
+        "order": 5,
         "type": "integer"
       },
       "custom_insights": {
-        "title": "Custom Insights",
+        "title": "Custom Insights (Optional)",
         "description": "A list wich contains insights entries, each entry must have a name and can contains fields, breakdowns or action_breakdowns)",
+        "order": 6,
         "type": "array",
         "items": {
           "title": "InsightConfig",
@@ -99,7 +99,7 @@
         }
       }
     },
-    "required": ["account_id", "access_token", "start_date"],
+    "required": ["start_date", "account_id", "access_token"],
     "definitions": {
       "InsightConfig": {
         "title": "InsightConfig",

airbyte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/source.py

@@ -40,59 +40,88 @@
 
 logger = logging.getLogger("airbyte")
 
+# deprecated from ConnectorSpec, using default value of 5 = 5 days.
+# Related Issue here: https://github.com/airbytehq/airbyte/issues/8027
+INSIGHTS_DAYS_PER_JOB: int = 5
+
 
 class InsightConfig(BaseModel):
 
-    name: str = Field(description="The name value of insight")
+    name: str = Field(
+        description="The name value of insight",
+    )
 
-    fields: Optional[List[str]] = Field(description="A list of chosen fields for fields parameter", default=[])
+    fields: Optional[List[str]] = Field(
+        description="A list of chosen fields for fields parameter",
+        default=[],
+    )
 
-    breakdowns: Optional[List[str]] = Field(description="A list of chosen breakdowns for breakdowns", default=[])
+    breakdowns: Optional[List[str]] = Field(
+        description="A list of chosen breakdowns for breakdowns",
+        default=[],
+    )
 
-    action_breakdowns: Optional[List[str]] = Field(description="A list of chosen action_breakdowns for action_breakdowns", default=[])
+    action_breakdowns: Optional[List[str]] = Field(
+        description="A list of chosen action_breakdowns for action_breakdowns",
+        default=[],
+    )
 
 
 class ConnectorConfig(BaseModel):
     class Config:
         title = "Source Facebook Marketing"
 
-    account_id: str = Field(description="The Facebook Ad account ID to use when pulling data from the Facebook Marketing API.")
-
-    access_token: str = Field(
-        description='The value of the access token generated. See the <a href="https://docs.airbyte.io/integrations/sources/facebook-marketing">docs</a> for more information',
-        airbyte_secret=True,
-    )
-
     start_date: datetime = Field(
+        title="Start Date",
         description="The date from which you'd like to replicate data for AdCreatives and AdInsights APIs, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated.",
         pattern="^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$",
         examples=["2017-01-25T00:00:00Z"],
+        order=0,
     )
 
     end_date: Optional[datetime] = Field(
+        title="End Date (Optional)",
         description="The date until which you'd like to replicate data for AdCreatives and AdInsights APIs, in the format YYYY-MM-DDT00:00:00Z. All data generated between start_date and this date will be replicated. Not setting this option will result in always syncing the latest data.",
         pattern="^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$",
         examples=["2017-01-26T00:00:00Z"],
         default_factory=pendulum.now,
+        order=1,
+    )
+
+    account_id: str = Field(
+        title="Account ID",
+        description="The Facebook Ad account ID to use when pulling data from the Facebook Marketing API.",
+        examples=["111111111111111"],
+        order=2,
+    )
+
+    access_token: str = Field(
+        title="Access Token",
+        description='The value of the access token generated. See the <a href="https://docs.airbyte.io/integrations/sources/facebook-marketing">docs</a> for more information',
+        airbyte_secret=True,
+        order=3,
     )
 
-    include_deleted: bool = Field(default=False, description="Include data from deleted campaigns, ads, and adsets.")
+    include_deleted: bool = Field(
+        title="Include Deleted", 
+        default=False, 
+        description="Include data from deleted campaigns, ads, and adsets.", 
+        order=4,
+    )
 
     insights_lookback_window: int = Field(
+        title="Insights Lookback Window",
         default=28,
         description="The attribution window for the actions",
         minimum=0,
         maximum=28,
+        order=5,
     )
 
-    insights_days_per_job: int = Field(
-        default=7,
-        description="Number of days to sync in one job. The more data you have - the smaller you want this parameter to be.",
-        minimum=1,
-        maximum=30,
-    )
     custom_insights: Optional[List[InsightConfig]] = Field(
-        description="A list wich contains insights entries, each entry must have a name and can contains fields, breakdowns or action_breakdowns)"
+        title="Custom Insights (Optional)",
+        description="A list wich contains insights entries, each entry must have a name and can contains fields, breakdowns or action_breakdowns)",
+        order=6,
     )
 
 
@@ -130,7 +159,7 @@ def streams(self, config: Mapping[str, Any]) -> List[Type[Stream]]:
             start_date=config.start_date,
             end_date=config.end_date,
             buffer_days=config.insights_lookback_window,
-            days_per_job=config.insights_days_per_job,
+            days_per_job=INSIGHTS_DAYS_PER_JOB,
         )
 
         streams = [

airbyte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/streams.py

@@ -361,6 +361,7 @@ def stream_slices(self, stream_state: Mapping[str, Any] = None, **kwargs) -> Ite
                 yield {"job": running_jobs.popleft()}
 
         while running_jobs:
+            #print(f"\nTEST: 'job': {running_jobs.popleft()}\n")
             yield {"job": running_jobs.popleft()}
 
     @retry_pattern(backoff.expo, JobException, max_tries=10, factor=5)

docs/integrations/sources/facebook-marketing.md

@@ -96,6 +96,7 @@ As a summary, custom insights allows to replicate only some fields, resulting in
 
 | Version | Date | Pull Request | Subject |
 | :--- | :--- | :--- | :--- |
+| 0.2.27  | 2021-11-25 | [8234](https://github.com/airbytehq/airbyte/pull/8234) | Reordered setup fields, deprecated `insigths days per job` from setup configuration |
 | 0.2.26  | 2021-11-19 | [7855](https://github.com/airbytehq/airbyte/pull/7855) | Add Video stream |
 | 0.2.25  | 2021-11-12 | [7904](https://github.com/airbytehq/airbyte/pull/7904) | Implement retry logic for async jobs |
 | 0.2.24  | 2021-11-09 | [7744](https://github.com/airbytehq/airbyte/pull/7744) | Fix fail when async job takes too long |