🚨🚨 🐛 Source TikTok Marketing: fix error while converting `mobile_app_id` column to int in AudienceReports streams #19758

achaussende · 2022-11-23T15:46:22Z

What

Fixing the audience_reports schema only for the mobile_app_id that should be a string instead of an integer.

Closes #19757

How

Update mobile_app_id type from integer to string in schemas/audience_reports.json file.

🚨 User Impact 🚨

The impact is a change in column type in the destination: from integer to string.

The AudienceReports streams only works if the mobile_app_id is an integer value. From TikTok Marketing API documentation, an integer value in mobile_app_id means that the app is only coming from Apple's App Store. But it's possible to have string values in this column for Android devices. Also according to the API spec, the column is a string.

So this PR is a breaking change if users have setup the replication of audience_reports streams. But the replication is faulty so it may actually fix the connection.

Pre-merge Checklist

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

… instead of int

CLAassistant · 2022-11-23T15:46:29Z

All committers have signed the CLA.

alafanechere · 2022-11-24T17:14:31Z

/test connector=connectors/source-tiktok-marketing

🕑 connectors/source-tiktok-marketing https://github.com/airbytehq/airbyte/actions/runs/3542577681
❌ connectors/source-tiktok-marketing https://github.com/airbytehq/airbyte/actions/runs/3542577681
🐛 https://gradle.com/s/tkl7ufscpxylg

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - so...
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs1] - so...
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs2] - so...
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs3] - so...
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs4] - so...
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs5] - so...
ERROR test_core.py::TestBasicRead::test_read[inputs3] - Failed: The ads_repor...
ERROR test_core.py::TestBasicRead::test_read[inputs4] - Failed: The ads_repor...
======= 6 failed, 87 passed, 31 warnings, 2 errors in 5720.41s (1:35:20) =======

> Task :airbyte-integrations:connectors:source-tiktok-marketing:sourceAcceptanceTest FAILED

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings

Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.
48 actionable tasks: 35 executed, 13 up-to-date

Publishing build scan...
https://gradle.com/s/tkl7ufscpxylg

The remote build cache was disabled during the build due to errors.
S3 cache 717ms wasted on misses, reads: 1, elapsed: 717ms
S3 cache writes: 1, elapsed: 55ms, sent to cache: 90 KiB

alafanechere · 2022-11-25T18:28:13Z

The failing test that is not related to the backward incompatibility should be resolved by this PR: #19806

@achaussende feel free to rebase your branch once it's merged

alafanechere · 2022-11-29T12:42:41Z

/test connector=connectors/source-tiktok-marketing

🕑 connectors/source-tiktok-marketing https://github.com/airbytehq/airbyte/actions/runs/3574173373
❌ connectors/source-tiktok-marketing https://github.com/airbytehq/airbyte/actions/runs/3574173373
🐛 https://gradle.com/s/osmqx5wnqb3js

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - so...
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs1] - so...
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs2] - so...
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs3] - so...
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs4] - so...
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs5] - so...
============ 6 failed, 89 passed, 31 warnings in 6688.48s (1:51:28) ============

alafanechere · 2022-11-29T12:43:17Z

@achaussende I updated your branch an re-ran the tests. The failing test_read due to a missing stream in the catalog should be gone.

alafanechere · 2022-12-01T17:06:35Z

/test connector=connectors/source-tiktok-marketing

🕑 connectors/source-tiktok-marketing https://github.com/airbytehq/airbyte/actions/runs/3594841124

alafanechere · 2022-12-05T08:55:54Z

/test connector=connectors/source-tiktok-marketing

🕑 connectors/source-tiktok-marketing https://github.com/airbytehq/airbyte/actions/runs/3618442262

alafanechere · 2022-12-05T16:07:43Z

@davydov-d / @YowanR may I ask you for a review and balance for our cloud users on the benefit of this fix vs the backward incompatibility that introduces the schema change?

YowanR · 2022-12-06T14:55:50Z

@davydov-d Before we move ahead here, let's do a quick audit on the impact on customers please. Are you able to pull that data? We don't want to break any customer syncs without giving them a heads-up.
cc @erica-airbyte as FYI

erica-airbyte · 2022-12-06T15:10:58Z

@YowanR and @davydov-d ideally if we can get around having to have customers run a schema reset that would be idea but please loop me in if there is customer impact and who we need to reach out to! Thanks.

davydov-d · 2022-12-06T19:01:35Z

erica-airbyte · 2022-12-06T19:49:48Z

@davydov-d one this fixes goes out the customer needs to refresh their schema and run a reset for the AudienceReports streams?

davydov-d · 2022-12-06T20:11:21Z

@erica-airbyte yes! but resync should be made for all of the AudienceReports streams (there may be multiple streams of this type, see first connection link for example)

erica-airbyte · 2022-12-07T14:57:25Z

@davydov-d we are reaching out to customer today. When is this going to cloud?

davydov-d · 2022-12-07T16:31:37Z

@erica-airbyte does tomorrow 12-16 UTC work?

erica-airbyte · 2022-12-07T17:24:59Z

@davydov-d that should work thanks!

erica-airbyte · 2022-12-07T19:55:45Z

Customer comms have been sent out to [those noted in this comment] (#19758 (comment)) letting know the update that will go out tomorrow 12-16 UTC

davydov-d · 2022-12-08T13:30:03Z

/publish connector=connectors/source-tiktok-marketing

🕑 Publishing the following connectors:
connectors/source-tiktok-marketing
https://github.com/airbytehq/airbyte/actions/runs/3648735528

Connector	Did it publish?	Were definitions generated?
connectors/source-tiktok-marketing	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

davydov-d · 2022-12-08T13:55:29Z

/publish connector=connectors/source-tiktok-marketing

🕑 Publishing the following connectors:
connectors/source-tiktok-marketing
https://github.com/airbytehq/airbyte/actions/runs/3648925714

Connector	Did it publish?	Were definitions generated?
connectors/source-tiktok-marketing	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

davydov-d · 2022-12-08T16:19:53Z

@erica-airbyte I'm sorry but delivery of this change is delayed. For some reason publish has already been running for almost 2.5 hours when previous one finished in 1.5 hours. I'll investigate and keep you updated

erica-airbyte · 2022-12-08T18:53:23Z

@davydov-d any update when this might go out?

davydov-d · 2022-12-08T19:59:25Z

I missed that the integration tests are failing 😕
Combined with our new workflow retry strategy it leads to endless builds..

davydov-d · 2022-12-08T20:01:05Z

@achaussende could you please also make the change like this? I'm not able to push the changes to your branch by myself

(this is acceptance-test-config.yml file)

davydov-d · 2022-12-08T20:05:30Z

@erica-airbyte well I have figured out what needs to be done but I can not do it on my own ☝️
as soon as we fix the acceptance tests, things should start working. Hope for assistance from @achaussende

…r this version

achaussende · 2022-12-08T21:52:49Z

@davydov-d Done in e9358f9 ✅

erica-airbyte · 2022-12-08T22:55:25Z

@davydov-d please lmk when this is on cloud and then we can circle back to users!

davydov-d · 2022-12-09T02:32:36Z

/test connector=connectors/source-tiktok-marketing

🕑 connectors/source-tiktok-marketing https://github.com/airbytehq/airbyte/actions/runs/3653927876
✅ connectors/source-tiktok-marketing https://github.com/airbytehq/airbyte/actions/runs/3653927876
Python tests coverage:

Name                                  Stmts   Miss  Cover
---------------------------------------------------------
source_tiktok_marketing/__init__.py       2      0   100%
source_tiktok_marketing/source.py        61      6    90%
source_tiktok_marketing/streams.py      352     37    89%
---------------------------------------------------------
TOTAL                                   415     43    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1599    332    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [6] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:370: Backward compatibility tests are disabled for version 0.1.17.
=========== 89 passed, 6 skipped, 31 warnings in 6690.20s (1:51:30) ============

davydov-d · 2022-12-09T06:27:37Z

/publish connector=connectors/source-tiktok-marketing

🕑 Publishing the following connectors:
connectors/source-tiktok-marketing
https://github.com/airbytehq/airbyte/actions/runs/3655054250

Connector	Did it publish?	Were definitions generated?
connectors/source-tiktok-marketing	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

davydov-d · 2022-12-09T08:52:47Z

@erica-airbyte done!
thanks @achaussende!

fix(sources/tiktok-marketing): column mobile_app_id is now a string…

a60775d

… instead of int

octavia-squidington-iv added community connectors/source/tiktok-marketing area/connectors Connector related issues labels Nov 23, 2022

achaussende changed the title ~~🐛 Source TikTok Marketing: column mobile_app_id is a string instead of int~~ 🐛 Source TikTok Marketing: fix error while converting mobile_app_id column to int in AudienceReports streams Nov 23, 2022

Merge branch 'master' into fix/source-tiktok-audience-reports-schemas

f55e0fc

achaussende marked this pull request as ready for review November 25, 2022 10:22

octavia-squidington-iv added the bounty label Nov 25, 2022

Merge branch 'master' into fix/source-tiktok-audience-reports-schemas

1a2a4cb

Merge branch 'master' into fix/source-tiktok-audience-reports-schemas

5651540

alafanechere assigned davydov-d and YowanR Dec 5, 2022

achaussende added 2 commits December 5, 2022 17:13

chore(sources/tiktok-marketing): set image version to 1.0.0

b773b74

chore(docs): update changelog for source tiktok-marketing, v1.0.0

2fd9405

achaussende changed the title ~~🐛 Source TikTok Marketing: fix error while converting mobile_app_id column to int in AudienceReports streams~~ 🚨🚨 🐛 Source TikTok Marketing: fix error while converting mobile_app_id column to int in AudienceReports streams Dec 5, 2022

octavia-squidington-iv added the area/documentation Improvements or additions to documentation label Dec 5, 2022

fix(docs): wrong date in changelog for source tiktok-marketing

833eb92

davydov-d approved these changes Dec 5, 2022

View reviewed changes

sajarin added internal and removed bounty labels Dec 5, 2022

Merge branch 'master' into fix/source-tiktok-audience-reports-schemas

13bcd48

fix(sources/tiktok-marketing): disable backward compatibility test fo…

e9358f9

…r this version

auto-bump connector version

86e9930

davydov-d merged commit b691841 into airbytehq:master Dec 9, 2022

github-actions bot mentioned this pull request Dec 9, 2022

Bump helm chart version reference to 0.43.2 #20320

Closed

octavia-squidington-iii mentioned this pull request Dec 18, 2022

Bump Airbyte version from 0.40.25 to 0.40.26 #20630

Merged

achaussende deleted the fix/source-tiktok-audience-reports-schemas branch February 10, 2023 09:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🚨🚨 🐛 Source TikTok Marketing: fix error while converting `mobile_app_id` column to int in AudienceReports streams #19758

🚨🚨 🐛 Source TikTok Marketing: fix error while converting `mobile_app_id` column to int in AudienceReports streams #19758

achaussende commented Nov 23, 2022 •

edited

Loading

CLAassistant commented Nov 23, 2022 •

edited

Loading

alafanechere commented Nov 24, 2022 •

edited by github-actions bot

Loading

alafanechere commented Nov 25, 2022

alafanechere commented Nov 29, 2022 •

edited by github-actions bot

Loading

alafanechere commented Nov 29, 2022 •

edited

Loading

alafanechere commented Dec 1, 2022 •

edited by github-actions bot

Loading

alafanechere commented Dec 5, 2022 •

edited by github-actions bot

Loading

alafanechere commented Dec 5, 2022

YowanR commented Dec 6, 2022

erica-airbyte commented Dec 6, 2022

davydov-d commented Dec 6, 2022

erica-airbyte commented Dec 6, 2022

davydov-d commented Dec 6, 2022

erica-airbyte commented Dec 7, 2022

davydov-d commented Dec 7, 2022

erica-airbyte commented Dec 7, 2022

erica-airbyte commented Dec 7, 2022 •

edited

Loading

davydov-d commented Dec 8, 2022 •

edited by github-actions bot

Loading

davydov-d commented Dec 8, 2022 •

edited by github-actions bot

Loading

davydov-d commented Dec 8, 2022

erica-airbyte commented Dec 8, 2022

davydov-d commented Dec 8, 2022

davydov-d commented Dec 8, 2022

davydov-d commented Dec 8, 2022

achaussende commented Dec 8, 2022

erica-airbyte commented Dec 8, 2022

davydov-d commented Dec 9, 2022 •

edited by github-actions bot

Loading

davydov-d commented Dec 9, 2022 •

edited by github-actions bot

Loading

davydov-d commented Dec 9, 2022

🚨🚨 🐛 Source TikTok Marketing: fix error while converting mobile_app_id column to int in AudienceReports streams #19758

🚨🚨 🐛 Source TikTok Marketing: fix error while converting mobile_app_id column to int in AudienceReports streams #19758

Conversation

achaussende commented Nov 23, 2022 • edited Loading

What

How

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

CLAassistant commented Nov 23, 2022 • edited Loading

alafanechere commented Nov 24, 2022 • edited by github-actions bot Loading

Build Failed

alafanechere commented Nov 25, 2022

alafanechere commented Nov 29, 2022 • edited by github-actions bot Loading

Build Failed

alafanechere commented Nov 29, 2022 • edited Loading

alafanechere commented Dec 1, 2022 • edited by github-actions bot Loading

alafanechere commented Dec 5, 2022 • edited by github-actions bot Loading

alafanechere commented Dec 5, 2022

YowanR commented Dec 6, 2022

erica-airbyte commented Dec 6, 2022

davydov-d commented Dec 6, 2022

erica-airbyte commented Dec 6, 2022

davydov-d commented Dec 6, 2022

erica-airbyte commented Dec 7, 2022

davydov-d commented Dec 7, 2022

erica-airbyte commented Dec 7, 2022

erica-airbyte commented Dec 7, 2022 • edited Loading

davydov-d commented Dec 8, 2022 • edited by github-actions bot Loading

davydov-d commented Dec 8, 2022 • edited by github-actions bot Loading

davydov-d commented Dec 8, 2022

erica-airbyte commented Dec 8, 2022

davydov-d commented Dec 8, 2022

davydov-d commented Dec 8, 2022

davydov-d commented Dec 8, 2022

achaussende commented Dec 8, 2022

erica-airbyte commented Dec 8, 2022

davydov-d commented Dec 9, 2022 • edited by github-actions bot Loading

Build Passed

davydov-d commented Dec 9, 2022 • edited by github-actions bot Loading

davydov-d commented Dec 9, 2022

🚨🚨 🐛 Source TikTok Marketing: fix error while converting `mobile_app_id` column to int in AudienceReports streams #19758

🚨🚨 🐛 Source TikTok Marketing: fix error while converting `mobile_app_id` column to int in AudienceReports streams #19758

achaussende commented Nov 23, 2022 •

edited

Loading

CLAassistant commented Nov 23, 2022 •

edited

Loading

alafanechere commented Nov 24, 2022 •

edited by github-actions bot

Loading

alafanechere commented Nov 29, 2022 •

edited by github-actions bot

Loading

alafanechere commented Nov 29, 2022 •

edited

Loading

alafanechere commented Dec 1, 2022 •

edited by github-actions bot

Loading

alafanechere commented Dec 5, 2022 •

edited by github-actions bot

Loading

erica-airbyte commented Dec 7, 2022 •

edited

Loading

davydov-d commented Dec 8, 2022 •

edited by github-actions bot

Loading

davydov-d commented Dec 8, 2022 •

edited by github-actions bot

Loading

davydov-d commented Dec 9, 2022 •

edited by github-actions bot

Loading

davydov-d commented Dec 9, 2022 •

edited by github-actions bot

Loading