Extract common ingestion tasks

[oliverjwroberts]

Description of change

This PR creates a new ingest sub-app that contains base functionality and utilities for ingesting objects from S3 buckets. The intention is the existing ingestion tasks can be refactored and plugged into this new common functionality in subsequent work.

Notably, it adds:

• A get_s3_client() function that returns a boto3 S3 client, either using the automatic endpoint URL or localstack's if the env variable S3_LOCAL_ENDPOINT_URL is set.
• An S3ObjectProcessor class for processing objects located at a specified prefix within an S3 bucket.
• Several constants for when using S3 buckets.
• A new ingest.IngestedObject model, that mirrors the functionality of company_activity.IngestedFile.
• A BaseObjectIdentificationTask class for identifying new objects in S3 and determining if they should be ingested.
• A BaseObjectIngestionTask class for ingesting a specified object from S3.

The BaseObject...Task classes are intended to be inherited for each dataset that we wish to ingest. The class methods can be overridden to tailor the tasks to the specific dataset; the idea being that only minimal code changes are needed to do so.

In terms of testing, it also modifies/adds:

• An s3_client fixture
• An s3_object_processor fixture
• An test_object_tuples fixture which can be used to generate some object definitions for testing. This fixture also uses the utility function compressed_json_faker to serialize the contents and compress them into a .jsonl.gz object.
• An upload_objects_to_s3 function that can be used to upload objects to a test S3 bucket.

Checklist

• Has this branch been rebased on top of the current main branch?

  Explanation

  The branch should not be stale or have conflicts at the time reviews are requested.

• Is the CircleCI build passing?

General points

Other things to check

• Make sure fixtures/test_data.yaml is maintained when updating models
• Consider the admin site when making changes to models
• Use select-/prefetch-related field lists in views and search apps, and update them when fields are added
• Make sure the README is updated e.g. when adding new environment variables

See docs/CONTRIBUTING.md for more guidelines.

[codecov]

Codecov Report

Attention: Patch coverage is 99.47368% with 1 line in your changes missing coverage. Please review.

Project coverage is 96.62%. Comparing base (84afe4a) to head (d0d65e3).
Report is 34 commits behind head on main.

Files with missing lines	Patch %	Lines
datahub/ingest/tasks.py	99.02%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5847      +/-   ##
==========================================
+ Coverage   96.61%   96.62%   +0.01%     
==========================================
  Files        1046     1053       +7     
  Lines       24904    25094     +190     
  Branches     1647     1663      +16     
==========================================
+ Hits        24060    24248     +188     
- Misses        688      690       +2     
  Partials      156      156              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[baarkerlounger]

Looks good. I know preference is generally for smaller PRs but I'd say in cases like this it would be worth including at least one existing ingestion task to use this mechanism. Helps show it works, and makes it easier to review the new generic code side by side with the old being removed.

[oliverjwroberts]

@baarkerlounger I was very conscious of how large this PR already is, but I agree, it probably would be easier to see a real example. I did attempt to describe an example in the docstring, but appreciate it's not the same. I was planning to plug the EYB ingestion tasks into this common functionality next; I wonder if it would be useful see that first before merging this in?

[baarkerlounger]

@oliverjwroberts my personal preference would be for the PR to be a "functionally complete and useful" unit, so I'd include it, otherwise this new code potentially never ends up doing anything if the rest doesn't get added (aside from being easier to review that way).

[DeanElliott96]

Looks good, just a few suggestions/changes.

[DeanElliott96]

Looks great! Once merged I will try using it with Stova.