Mixpanel: Filtering out individual items based on datetime in state #15091

Kopiczek · 2022-07-27T22:48:34Z

What

Mixpanel export API allow to filter based on DATE (without the time portion).
This means that if a sync is run multiple times a day, we keep duplicating the same data over and over.
This change filters the records after getting them from source.

How

This is done on export stream only by comparing the cursor field of each record to stream state.

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Tests

Unit

Test that verifies that a record is filtered by state.

Integration

N/A

Acceptance

N/A

CLAassistant · 2022-07-27T22:48:39Z

All committers have signed the CLA.

marcosmarxm · 2022-08-01T20:04:33Z

/test connector=connectors/source-mixpanel

🕑 connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2777526749
❌ connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2777526749
🐛 https://gradle.com/s/5dr52h6j75y2i

Build Failed

Test summary info:

Could not find result summary

# Conflicts: # airbyte-integrations/connectors/source-mixpanel/source_mixpanel/streams/export.py

Kopiczek · 2022-08-08T22:15:29Z

/test connector=connectors/source-mixpanel

Kopiczek · 2022-08-09T17:03:45Z

@marcosmarxm am I doing something wrong trying to initialize those test you did?

marcosmarxm · 2022-08-09T17:25:32Z

@Kopiczek sorry the delay here, I'll take a look.

Kopiczek · 2022-08-09T23:41:11Z

/test connector=connectors/source-mixpanel

marcosmarxm · 2022-08-10T13:39:26Z

/test connector=connectors/source-mixpanel

🕑 connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2833069734
❌ connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2833069734
🐛 https://gradle.com/s/ncqdmsxtyb752

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_streams.py::test_export_stream - assert 1 == 0
	 �[31m================== �[31m�[1m1 failed�[0m, �[32m22 passed�[0m, �[33m138 warnings�[0m�[31m in 0.56s�[0m�[31m ==================�[0m

marcosmarxm

Unit test are broken @Kopiczek please implement a new unit test

airbyte-integrations/connectors/source-mixpanel/unit_tests/test_streams.py

Kopiczek · 2022-08-10T17:39:30Z

Unit test are broken @Kopiczek please implement a new unit test

@marcosmarxm This should be working now. All unit tests passing locally

marcosmarxm · 2022-08-11T22:02:20Z

/test connector=connectors/source-mixpanel

🕑 connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2843095431
❌ connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2843095431
🐛 https://gradle.com/s/bkvtammb7ejlc

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_streams.py::test_export_stream_request_params - assert...
	 �[31m================== �[31m�[1m1 failed�[0m, �[32m23 passed�[0m, �[33m140 warnings�[0m�[31m in 0.56s�[0m�[31m ==================�[0m

Kopiczek · 2022-08-11T23:15:15Z

Sorry for all the back and forth.
Since the code treats the long date from source as current timezone we are doing the same when going from cursor to long
https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mixpanel/source_mixpanel/streams/export.py#L122-L124

We do the same, forgot to include that in the test and hardcoded my local timezone into expected result

marcosmarxm · 2022-08-12T12:32:25Z

/test connector=connectors/source-mixpanel

🕑 connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2846757501
✅ connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2846757501
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  77     17    78%
source_acceptance_test/tests/test_core.py              307    106    65%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  970    246    75%
Name                                         Stmts   Miss  Cover
----------------------------------------------------------------
source_mixpanel/streams/revenue.py              14      0   100%
source_mixpanel/streams/funnels.py              56      0   100%
source_mixpanel/streams/annotations.py           6      0   100%
source_mixpanel/streams/__init__.py              9      0   100%
source_mixpanel/property_transformation.py      19      0   100%
source_mixpanel/__init__.py                      2      0   100%
source_mixpanel/source.py                       45      1    98%
source_mixpanel/streams/engage.py               89      5    94%
source_mixpanel/streams/base.py                 71      7    90%
source_mixpanel/streams/cohorts.py              20      3    85%
source_mixpanel/streams/export.py               51      8    84%
source_mixpanel/streams/cohort_members.py       19      5    74%
source_mixpanel/testing.py                      25     11    56%
----------------------------------------------------------------
TOTAL                                          426     40    91%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:23: `future_state_path` not specified, skipping
================== 26 passed, 1 skipped in 402.92s (0:06:42) ===================

Kopiczek · 2022-08-15T17:09:04Z

@marcosmarxm is there anything else you need for this to be merged?

marcosmarxm · 2022-08-15T23:25:06Z

@roman-yermilov-gl can you do the final review here?

Kopiczek · 2022-08-22T16:46:25Z

@marcosmarxm need your approval as well since you requested changes.

marcosmarxm · 2022-08-22T17:24:50Z

@marcosmarxm need your approval as well since you requested changes.

@Kopiczek can you bump the Dockerfile version and documentation, I'll publish after it

Kopiczek · 2022-08-22T18:23:26Z

@marcosmarxm done

marcosmarxm · 2022-08-23T04:09:48Z

/publish connector=connectors/source-mixpanel

🕑 Publishing the following connectors:
connectors/source-mixpanel
https://github.com/airbytehq/airbyte/actions/runs/2908785371

Connector	Did it publish?	Were definitions generated?
connectors/source-mixpanel	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…15091) * Mixpanel: Filtering out individual items based on datetime in state * Mixpanel: Filtering out individual items based on datetime in state * Mixpanel: Use where API option to further filter out events * Fixing unit tests * Fixing unit test timezone issues * Version bump + docs * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

Mixpanel: Filtering out individual items based on datetime in state

6dc9e1e

github-actions bot added the area/connectors Connector related issues label Jul 27, 2022

octavia-squidington-iii added bounty community connectors/source/mixpanel labels Jul 27, 2022

Mixpanel: Filtering out individual items based on datetime in state

47739cb

marcosmarxm force-pushed the mixpanel branch from 6dc9e1e to 47739cb Compare August 1, 2022 20:04

sajarin added the bounty-S Maintainer program: claimable small bounty PR label Aug 5, 2022

Kris added 2 commits August 8, 2022 15:11

Mixpanel: Use where API option to further filter out events

1e2b45b

Merge remote-tracking branch 'origin/mixpanel' into mixpanel

980f461

# Conflicts: # airbyte-integrations/connectors/source-mixpanel/source_mixpanel/streams/export.py

marcosmarxm requested changes Aug 10, 2022

View reviewed changes

airbyte-integrations/connectors/source-mixpanel/unit_tests/test_streams.py Outdated Show resolved Hide resolved

Fixing unit tests

de48823

Kopiczek requested a review from marcosmarxm August 10, 2022 23:58

sajarin added internal and removed bounty bounty-S Maintainer program: claimable small bounty PR labels Aug 11, 2022

Fixing unit test timezone issues

de2b1dc

marcosmarxm requested a review from roman-yermilov-gl August 15, 2022 23:24

roman-yermilov-gl approved these changes Aug 22, 2022

View reviewed changes

Merge branch 'master' into mixpanel

6060c2f

Kris Sikora added 2 commits August 22, 2022 11:17

Merge branch 'master' into mixpanel

c594f91

Version bump + docs

e3ea523

github-actions bot added the area/documentation Improvements or additions to documentation label Aug 22, 2022

auto-bump connector version [ci skip]

3ab29dc

marcosmarxm approved these changes Aug 23, 2022

View reviewed changes

marcosmarxm merged commit a4916a4 into airbytehq:master Aug 23, 2022

Kopiczek deleted the mixpanel branch August 23, 2022 16:10

This was referenced Aug 23, 2022

Bump Airbyte version from 0.40.1 to 0.40.2 #15909

Closed

Bump Airbyte version from 0.40.1 to 0.40.2 #15931

Merged

davydov-d mentioned this pull request Jul 18, 2023

🐛 Source Mixpanel: reimplement backoff strategy #27752

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mixpanel: Filtering out individual items based on datetime in state #15091

Mixpanel: Filtering out individual items based on datetime in state #15091

Kopiczek commented Jul 27, 2022

CLAassistant commented Jul 27, 2022 •

edited

Loading

marcosmarxm commented Aug 1, 2022 •

edited by github-actions bot

Loading

Kopiczek commented Aug 8, 2022 •

edited

Loading

Kopiczek commented Aug 9, 2022

marcosmarxm commented Aug 9, 2022

Kopiczek commented Aug 9, 2022

marcosmarxm commented Aug 10, 2022 •

edited by github-actions bot

Loading

marcosmarxm left a comment

Kopiczek commented Aug 10, 2022 •

edited

Loading

marcosmarxm commented Aug 11, 2022 •

edited by github-actions bot

Loading

Kopiczek commented Aug 11, 2022

marcosmarxm commented Aug 12, 2022 •

edited by github-actions bot

Loading

Kopiczek commented Aug 15, 2022

marcosmarxm commented Aug 15, 2022

Kopiczek commented Aug 22, 2022

marcosmarxm commented Aug 22, 2022

Kopiczek commented Aug 22, 2022

marcosmarxm commented Aug 23, 2022 •

edited by github-actions bot

Loading

Mixpanel: Filtering out individual items based on datetime in state #15091

Mixpanel: Filtering out individual items based on datetime in state #15091

Conversation

Kopiczek commented Jul 27, 2022

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Tests

CLAassistant commented Jul 27, 2022 • edited Loading

marcosmarxm commented Aug 1, 2022 • edited by github-actions bot Loading

Build Failed

Kopiczek commented Aug 8, 2022 • edited Loading

Kopiczek commented Aug 9, 2022

marcosmarxm commented Aug 9, 2022

Kopiczek commented Aug 9, 2022

marcosmarxm commented Aug 10, 2022 • edited by github-actions bot Loading

Build Failed

marcosmarxm left a comment

Choose a reason for hiding this comment

Kopiczek commented Aug 10, 2022 • edited Loading

marcosmarxm commented Aug 11, 2022 • edited by github-actions bot Loading

Build Failed

Kopiczek commented Aug 11, 2022

marcosmarxm commented Aug 12, 2022 • edited by github-actions bot Loading

Build Passed

Kopiczek commented Aug 15, 2022

marcosmarxm commented Aug 15, 2022

Kopiczek commented Aug 22, 2022

marcosmarxm commented Aug 22, 2022

Kopiczek commented Aug 22, 2022

marcosmarxm commented Aug 23, 2022 • edited by github-actions bot Loading

CLAassistant commented Jul 27, 2022 •

edited

Loading

marcosmarxm commented Aug 1, 2022 •

edited by github-actions bot

Loading

Kopiczek commented Aug 8, 2022 •

edited

Loading

marcosmarxm commented Aug 10, 2022 •

edited by github-actions bot

Loading

Kopiczek commented Aug 10, 2022 •

edited

Loading

marcosmarxm commented Aug 11, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 12, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 23, 2022 •

edited by github-actions bot

Loading