Fix `source-google-ads` on M1 Macs by pinning `protobuf==3.14` #13624

evantahler · 2022-06-08T20:52:52Z

Closes #13580. Closes https://github.com/airbytehq/oncall/issues/263.

Looks like newer versions of Protobuf have some problems (protocolbuffers/protobuf#10075). We can likely remove this version pin in the future.

evantahler · 2022-06-08T21:17:36Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/2464433264
❌ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/2464433264
🐛 https://gradle.com/s/b5ro2ks3r5fc2

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Timeout >600.0s
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
============= 1 failed, 21 passed, 2 skipped in 642.27s (0:10:42) ==============

sherifnada

do you know how did this pass tests before? 🤔

evantahler · 2022-06-08T21:21:33Z

@sherifnada no clue. The problem only appears on M1 macs (which likely weren't used for development of this connector until now). It's also a transitive dependency (a dep of a dep) so the version was drifting under us without us knowing. Older versions did work.

evantahler · 2022-06-08T22:04:23Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/2464647191
❌ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/2464647191
🐛 https://gradle.com/s/olhdig6ex5qpc

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Timeout >600.0s
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
============= 1 failed, 21 passed, 2 skipped in 638.12s (0:10:38) ==============

evantahler · 2022-06-08T23:21:04Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/2464951842
❌ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/2464951842
🐛 https://gradle.com/s/kkhlayunss7qk

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Timeout >600.0s
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
============= 1 failed, 21 passed, 2 skipped in 640.08s (0:10:40) ==============

davydov-d · 2022-06-09T07:40:18Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/2466757067
❌ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/2466757067
🐛 https://gradle.com/s/k3m55fucvtcqe

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Timeout >600.0s
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
============= 1 failed, 21 passed, 2 skipped in 640.67s (0:10:40) ==============

codecov · 2022-06-09T07:47:03Z

Codecov Report

❗ No coverage uploaded for pull request base (master@922bf46). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #13624   +/-   ##
=========================================
  Coverage          ?   92.61%           
=========================================
  Files             ?        6           
  Lines             ?      406           
  Branches          ?        0           
=========================================
  Hits              ?      376           
  Misses            ?       30           
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 922bf46...46a2ab2. Read the comment docs.

davydov-d · 2022-06-09T09:37:04Z

I guess we need to make further investigation as protobuff==3.14 slows down the connector more than 3 times!

davydov-d · 2022-06-09T16:36:32Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/2469833645
✅ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/2469833645
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
source_google_ads/models.py                   18      0   100%
source_google_ads/__init__.py                  2      0   100%
source_google_ads/google_ads.py               68     10    85%
source_google_ads/streams.py                 163     26    84%
source_google_ads/source.py                   80     23    71%
source_google_ads/custom_query_stream.py      75     46    39%
--------------------------------------------------------------
TOTAL                                        406    105    74%
Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
source_google_ads/models.py                   18      0   100%
source_google_ads/__init__.py                  2      0   100%
source_google_ads/streams.py                 163      8    95%
source_google_ads/source.py                   80      4    95%
source_google_ads/custom_query_stream.py      75      6    92%
source_google_ads/google_ads.py               68     12    82%
--------------------------------------------------------------
TOTAL                                        406     30    93%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
================== 22 passed, 2 skipped in 245.81s (0:04:05) ===================

davydov-d · 2022-06-09T16:53:24Z

Finally, this looks much better. Docker run spec also works on Mac M1. @evantahler take a look please

evantahler · 2022-06-09T18:11:44Z

Thanks @davydov-d! I can't approve this PR because I started it, but approved by me!

On my M1 mac:

docker pull airbyte/source-google-ads:latest
docker run airbyte/source-google-ads:latest spec
# (error)

./gradlew :airbyte-integrations:connectors:source-google-ads:build
docker run airbyte/source-google-ads:dev spec
# (works!)

SPEC works!

As a note, I also tried to run integration tests locally (./acceptance-test-docker.sh), and they don't work... but they never did. Things seem OK on CI, so go for it!

evantahler · 2022-06-09T18:12:22Z

/publish connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/2470320745
🚀 Successfully published connectors/source-google-ads
🚀 Auto-bumped version for connectors/source-google-ads
✅ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/2470320745

Bump source-google-ads to build for both AMD and ARM

7baa9b5

evantahler mentioned this pull request Jun 8, 2022

airbyte/source-google-ads connector's SPEC command does not succeed #13580

Closed

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Jun 8, 2022

octavia-squidington-iii added the connectors/source/google-ads label Jun 8, 2022

pin protobuf==3.14

3135c3c

evantahler changed the title ~~Bump source-google-ads to build for both AMD and ARM~~ Fix source-google-ads by pinning protobuf==3.14 Jun 8, 2022

update readme

be1d1ea

evantahler requested review from pedroslopez and sherifnada and removed request for pedroslopez June 8, 2022 21:15

evantahler marked this pull request as ready for review June 8, 2022 21:15

evantahler changed the title ~~Fix source-google-ads by pinning protobuf==3.14~~ Fix source-google-ads on M1 by pinning protobuf==3.14 Jun 8, 2022

evantahler added the team/extensibility label Jun 8, 2022

evantahler changed the title ~~Fix source-google-ads on M1 by pinning protobuf==3.14~~ Fix source-google-ads on M1 Macs by pinning protobuf==3.14 Jun 8, 2022

sherifnada approved these changes Jun 8, 2022

View reviewed changes

davydov-d approved these changes Jun 9, 2022

View reviewed changes

Merge branch 'master' into evan/build-source-google-ads-m1

9a5190f

#263 oncall: bump google ads version 15.1.1, protobuf 3.20.0

e9cf5e8

auto-bump connector version

46a2ab2

octavia-squidington-iii temporarily deployed to more-secrets June 9, 2022 18:34 Inactive

evantahler merged commit db67936 into master Jun 9, 2022

evantahler deleted the evan/build-source-google-ads-m1 branch June 9, 2022 18:55

octavia-squidington-iii mentioned this pull request Jun 14, 2022

Bump Airbyte version from 0.39.17-alpha to 0.39.18-alpha #13759

Merged

liuverta mentioned this pull request Jul 28, 2022

chore: Allow protobuf < 4.0 to support M1 Macs VertaAI/modeldb#3070

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `source-google-ads` on M1 Macs by pinning `protobuf==3.14` #13624

Fix `source-google-ads` on M1 Macs by pinning `protobuf==3.14` #13624

evantahler commented Jun 8, 2022 •

edited

Loading

evantahler commented Jun 8, 2022 •

edited by github-actions bot

Loading

sherifnada left a comment

evantahler commented Jun 8, 2022

evantahler commented Jun 8, 2022 •

edited by github-actions bot

Loading

evantahler commented Jun 8, 2022 •

edited by github-actions bot

Loading

davydov-d commented Jun 9, 2022 •

edited by github-actions bot

Loading

codecov bot commented Jun 9, 2022 •

edited

Loading

davydov-d commented Jun 9, 2022

davydov-d commented Jun 9, 2022 •

edited by github-actions bot

Loading

davydov-d commented Jun 9, 2022

evantahler commented Jun 9, 2022

evantahler commented Jun 9, 2022 •

edited by github-actions bot

Loading

Fix source-google-ads on M1 Macs by pinning protobuf==3.14 #13624

Fix source-google-ads on M1 Macs by pinning protobuf==3.14 #13624

Conversation

evantahler commented Jun 8, 2022 • edited Loading

evantahler commented Jun 8, 2022 • edited by github-actions bot Loading

Build Failed

sherifnada left a comment

Choose a reason for hiding this comment

evantahler commented Jun 8, 2022

evantahler commented Jun 8, 2022 • edited by github-actions bot Loading

Build Failed

evantahler commented Jun 8, 2022 • edited by github-actions bot Loading

Build Failed

davydov-d commented Jun 9, 2022 • edited by github-actions bot Loading

Build Failed

codecov bot commented Jun 9, 2022 • edited Loading

Codecov Report

davydov-d commented Jun 9, 2022

davydov-d commented Jun 9, 2022 • edited by github-actions bot Loading

Build Passed

davydov-d commented Jun 9, 2022

evantahler commented Jun 9, 2022

evantahler commented Jun 9, 2022 • edited by github-actions bot Loading

Fix `source-google-ads` on M1 Macs by pinning `protobuf==3.14` #13624

Fix `source-google-ads` on M1 Macs by pinning `protobuf==3.14` #13624

evantahler commented Jun 8, 2022 •

edited

Loading

evantahler commented Jun 8, 2022 •

edited by github-actions bot

Loading

evantahler commented Jun 8, 2022 •

edited by github-actions bot

Loading

evantahler commented Jun 8, 2022 •

edited by github-actions bot

Loading

davydov-d commented Jun 9, 2022 •

edited by github-actions bot

Loading

codecov bot commented Jun 9, 2022 •

edited

Loading

davydov-d commented Jun 9, 2022 •

edited by github-actions bot

Loading

evantahler commented Jun 9, 2022 •

edited by github-actions bot

Loading