feature/sync-with-contacts-preloaded

[chopkinsmade]

Description of change

Using the django DB for individually checking if a contact in the S3 file existed was running slowly on production (200-300 milliseconds per query), with a file that contains 1.3 million rows.

To reduce the number of individual queries being executed against the DB, at the start of the job load all contacts from the DB into a python dictionary. This initial load of users takes around 4-5 minutes, however no further DB reads are required during the processing of the file.

This has been tested on a local environment with 600,000 contacts, and took 6-10 minutes to fully process the file with ENABLE_CONTACT_CONSENT_INGEST setting equal to False. Setting this value to False allows testing of the file processing logic, without making any changes to the contact in the DB.

When ENABLE_CONTACT_CONSENT_INGEST was set to True, the same 600,000 users takes 20-25 minutes to fully process the file and update the contact in the DB with the latest value for that contacts consent. As the only fields this job need to update are the consent_data and consent_data_last_modified, the update django function is used instead of save to avoid triggering the subscribed contact signals that submit additional sync jobs to the short queue

Checklist

• Has this branch been rebased on top of the current main branch?

  Explanation

  The branch should not be stale or have conflicts at the time reviews are requested.

• Is the CircleCI build passing?

General points

Other things to check

• Make sure fixtures/test_data.yaml is maintained when updating models
• Consider the admin site when making changes to models
• Use select-/prefetch-related field lists in views and search apps, and update them when fields are added
• Make sure the README is updated e.g. when adding new environment variables

See docs/CONTRIBUTING.md for more guidelines.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.49%. Comparing base (29c4ceb) to head (20d5e17).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5818      +/-   ##
==========================================
- Coverage   96.49%   96.49%   -0.01%     
==========================================
  Files        1041     1041              
  Lines       24708    24724      +16     
  Branches     1641     1644       +3     
==========================================
+ Hits        23843    23858      +15     
- Misses        709      710       +1     
  Partials      156      156              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests
• JS Bundle Analysis - Avoid shipping oversized bundles