Source Facebook Marketing: remove unsupported fields #20402

artem1205 · 2022-12-12T23:25:59Z

What

Resolving https://github.com/airbytehq/oncall/issues/1137

How

Exclude fields from request

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

artem1205 · 2022-12-12T23:26:30Z

/test connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/3680719172
✅ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/3680719172
Python tests coverage:

Name                                                        Stmts   Miss  Cover
-------------------------------------------------------------------------------
source_facebook_marketing/streams/__init__.py                   2      0   100%
source_facebook_marketing/__init__.py                           2      0   100%
source_facebook_marketing/spec.py                              41      1    98%
source_facebook_marketing/api.py                               96     12    88%
source_facebook_marketing/streams/base_streams.py             138     27    80%
source_facebook_marketing/streams/common.py                    44     16    64%
source_facebook_marketing/utils.py                             24      9    62%
source_facebook_marketing/source.py                            58     22    62%
source_facebook_marketing/streams/streams.py                  119     46    61%
source_facebook_marketing/streams/base_insight_streams.py     145     64    56%
source_facebook_marketing/streams/async_job.py                217    136    37%
source_facebook_marketing/streams/async_job_manager.py         78     60    23%
-------------------------------------------------------------------------------
TOTAL                                                         964    393    59%
Name                                                        Stmts   Miss  Cover
-------------------------------------------------------------------------------
source_facebook_marketing/utils.py                             24      0   100%
source_facebook_marketing/streams/common.py                    44      0   100%
source_facebook_marketing/streams/async_job.py                217      0   100%
source_facebook_marketing/streams/__init__.py                   2      0   100%
source_facebook_marketing/spec.py                              41      0   100%
source_facebook_marketing/__init__.py                           2      0   100%
source_facebook_marketing/streams/async_job_manager.py         78      3    96%
source_facebook_marketing/api.py                               96      9    91%
source_facebook_marketing/streams/base_insight_streams.py     145     16    89%
source_facebook_marketing/source.py                            58      7    88%
source_facebook_marketing/streams/base_streams.py             138     30    78%
source_facebook_marketing/streams/streams.py                  119     27    77%
-------------------------------------------------------------------------------
TOTAL                                                         964     92    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1599    332    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:200: Skipping new incremental test based on acceptance-test-config.yml
============ 30 passed, 1 skipped, 31 warnings in 767.45s (0:12:47) ============

grubberr · 2022-12-13T14:49:22Z

airbyte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/source.py

+                # https://github.com/airbytehq/oncall/issues/1137
+                logger.info(f"Following fields were removed from the Custom{insight.name} fields list due to error"
+                            f" on Facebook side: {insight_fields.intersection(UNSUPPORTED_FIELDS)}")
+                insight_fields -= UNSUPPORTED_FIELDS


I am not sure how it works, I see you just removed this fields from some temp variable
insight_fields and that's all ?

is it ok?

davydov-d · 2022-12-13T15:03:58Z

airbyte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/source.py

+                # https://github.com/airbytehq/oncall/issues/1137
+                logger.info(f"Following fields were removed from the Custom{insight.name} fields list due to error"
+                            f" on Facebook side: {insight_fields.intersection(UNSUPPORTED_FIELDS)}")
+                insight_fields -= UNSUPPORTED_FIELDS


can we validate presence of these fields in the check_connection to prevent user from providing an invalid config and sooner highlighting the issue?

check_connection already uses this function, do you want to raise an error here instead of quiet deleting?

Yes, I think it's a better UX

davydov-d · 2022-12-14T07:24:56Z

airbyte-integrations/connectors/source-facebook-marketing/Dockerfile

@@ -13,5 +13,5 @@ ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]


-LABEL io.airbyte.version=0.2.78


You also need to update changelog in the doc file please

artem1205 · 2022-12-15T13:20:20Z

/publish connector=connectors/source-facebook-marketing

🕑 Publishing the following connectors:
connectors/source-facebook-marketing
https://github.com/airbytehq/airbyte/actions/runs/3704484918

Connector	Did it publish?	Were definitions generated?
connectors/source-facebook-marketing	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Source Facebook Marketing: remove unsupported fields

366c6b4

octavia-squidington-iv added area/connectors Connector related issues connectors/source/facebook-marketing labels Dec 12, 2022

artem1205 requested review from grubberr and davydov-d December 13, 2022 14:33

grubberr reviewed Dec 13, 2022

View reviewed changes

davydov-d reviewed Dec 13, 2022

View reviewed changes

artem1205 added 2 commits December 13, 2022 17:40

Source Facebook Marketing: fix

2abfbd9

Source Facebook Marketing: refactor; raise error

f42853f

artem1205 requested a review from davydov-d December 13, 2022 23:23

davydov-d requested changes Dec 14, 2022

View reviewed changes

Source Facebook marketing: docs update

21b4bc2

octavia-squidington-iv added the area/documentation Improvements or additions to documentation label Dec 15, 2022

artem1205 requested a review from davydov-d December 15, 2022 13:16

davydov-d approved these changes Dec 15, 2022

View reviewed changes

auto-bump connector version

1066035

octavia-squidington-iii temporarily deployed to more-secrets December 15, 2022 13:59 — with GitHub Actions Inactive

artem1205 merged commit 6bf2448 into master Dec 15, 2022

artem1205 deleted the artem1205/source-facebook-marketing-JobException-OC-1137 branch December 15, 2022 13:59

octavia-squidington-iii temporarily deployed to more-secrets December 15, 2022 14:00 — with GitHub Actions Inactive

github-actions bot mentioned this pull request Dec 15, 2022

Bump helm chart version reference to 0.43.5 #20526

Closed

octavia-squidington-iii mentioned this pull request Dec 18, 2022

Bump Airbyte version from 0.40.25 to 0.40.26 #20630

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Facebook Marketing: remove unsupported fields #20402

Source Facebook Marketing: remove unsupported fields #20402

artem1205 commented Dec 12, 2022

artem1205 commented Dec 12, 2022 •

edited by github-actions bot

Loading

grubberr Dec 13, 2022

artem1205 Dec 13, 2022

davydov-d Dec 13, 2022

artem1205 Dec 13, 2022 •

edited

Loading

davydov-d Dec 13, 2022

artem1205 Dec 13, 2022

davydov-d Dec 14, 2022

artem1205 Dec 15, 2022

artem1205 commented Dec 15, 2022 •

edited by github-actions bot

Loading

		@@ -13,5 +13,5 @@ ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
		ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]


		LABEL io.airbyte.version=0.2.78

Source Facebook Marketing: remove unsupported fields #20402

Source Facebook Marketing: remove unsupported fields #20402

Conversation

artem1205 commented Dec 12, 2022

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

artem1205 commented Dec 12, 2022 • edited by github-actions bot Loading

Build Passed

grubberr Dec 13, 2022

Choose a reason for hiding this comment

artem1205 Dec 13, 2022

Choose a reason for hiding this comment

davydov-d Dec 13, 2022

Choose a reason for hiding this comment

artem1205 Dec 13, 2022 • edited Loading

Choose a reason for hiding this comment

davydov-d Dec 13, 2022

Choose a reason for hiding this comment

artem1205 Dec 13, 2022

Choose a reason for hiding this comment

davydov-d Dec 14, 2022

Choose a reason for hiding this comment

artem1205 Dec 15, 2022

Choose a reason for hiding this comment

artem1205 commented Dec 15, 2022 • edited by github-actions bot Loading

artem1205 commented Dec 12, 2022 •

edited by github-actions bot

Loading

artem1205 Dec 13, 2022 •

edited

Loading

artem1205 commented Dec 15, 2022 •

edited by github-actions bot

Loading