Importers: migrate to options class

[Maffooch]

Moves the parsing of kwargs to a central ImporterOptions class where validation can occur and defaults be set

[dryrunsecurity]

Hi there 👋, @DryRunSecurity here, below is a summary of our analysis and findings.

DryRun Security	Status	Findings
Configured Codepaths Analyzer	✅	0 findings
Sensitive Files Analyzer	❕	1 finding
AppSec Analyzer	✅	0 findings
Authn/Authz Analyzer	❕	5 findings
Secrets Analyzer	✅	0 findings

Note

🟡 Please give this pull request extra attention during review.

Change Summary (click to expand)

The following is a summary of changes in this pull request made by me, your security buddy 🤖. Note that this summary is auto-generated and not meant to be a definitive list of security issues but rather a helpful summary from a security perspective.

Summary:

The code changes in this pull request focus on improving the functionality and security of the import and reimport processes in the DefectDojo application security tool. The key improvements include:

1. Enhancing the handling of endpoints, including the ability to process endpoints asynchronously and manage their status (mitigation, reactivation).
2. Improving the deduplication and matching of findings during the reimport process, ensuring that the application maintains an accurate and up-to-date view of the security posture.
3. Introducing more robust input validation and error handling, which helps to prevent potential security vulnerabilities and maintain the integrity of the imported data.
4. Providing better tracking and reporting of security findings, including detailed statistics on new, mitigated, false positive, and other finding statuses.
5. Improving the overall flexibility and maintainability of the importer classes, making it easier to extend and customize the import and reimport functionality.

Files Changed:

1. dojo/engagement/views.py: The changes simplify the import_findings function by passing additional keyword arguments to the DefaultImporter object, which improves the flexibility of the import process.
2. dojo/endpoint/utils.py: The changes improve the handling and management of Endpoint objects, including the ability to create new Endpoints, merge duplicates, and remove broken Endpoint Statuses.
3. dojo/api_v2/serializers.py: The changes enhance the ReImportScanSerializer class, improving the handling of scan data and associated statistics during the reimport process.
4. dojo/importers/auto_create_context.py: The changes introduce the use of database transactions and row-level locking to ensure the consistency and integrity of the resource management functionality during the import and reimport processes.
5. dojo/importers/base_importer.py: The changes improve the flexibility and maintainability of the BaseImporter class, including the introduction of an EndpointManager and the ability to handle both synchronous and asynchronous processing of findings.
6. dojo/importers/default_importer.py: The changes focus on enhancing the DefaultImporter class, which is responsible for the classic import process used by DefectDojo, including improvements to the handling of findings, test metadata, and import history tracking.
7. dojo/importers/endpoint_manager.py: The changes introduce the EndpointManager class, which is responsible for handling the processing and chunking of endpoints associated with findings, including the ability to process endpoints asynchronously.
8. dojo/importers/default_reimporter.py: The changes improve the DefaultReImporter class, which is responsible for the reimport process, including the handling of finding deduplication, matching, and mitigation.
9. dojo/importers/options.py: The changes introduce the ImporterOptions class, which is responsible for managing and validating various options related to the import process, including input validation and data compression/decompression.

Powered by DryRun Security

[kiblik]

@Maffooch, if this kind of rewrite is happening, do you suppose it is possible to solve also #10219?

[Maffooch]

This is more of a restructure/reorg rather and a rewrite. The functionalities are otherwise the same. I can take a look to see if that issue linked has a quick solution

[mtesauro]

Approved

[cneill]

Just a couple questions/comments