Cahaba Codebase Refactor and Cleanup

[nickchadwick-noaa]

This Issue is to track the overall progress of cleaning up the Cahaba Repo. The following line items are to be made into separate issues with their own deeper descriptions and discussions. The goal of this list is to clean out any unused code and files, unify the formatting and languages of all the remaining code, and to organize the repository for future contributions.

Won't Impact fim_4:

• Remove Dockerfile.prod, rename Dockerfile.dev to just Dockerfile, and remove .dockerignore. [9pt] Cahaba Codebase Cleanup - Small Items #432
• Clean up Dockerfile and remove any unused* packages or variables. [9pt] Cahaba Codebase Cleanup - Small Items #432
• Remove any unused* Python packages from the Pipfile. [9pt] Cahaba Codebase Cleanup - Small Items #432
• Move the CHANGELOG.md, SECURITY.md, and TERMS.md files to the /docs folder. [9pt] Cahaba Codebase Cleanup - Small Items #432
• Remove any unused* scripts in the /tools and /src folders. [9pt] Cahaba Codebase Cleanup - Small Items #432
• Move tools/preprocess scripts into tools/ [9pt] Cahaba Codebase Cleanup - Small Items #432
• Ensure all scripts in the /src folder have their code in functions and are being called via a __main__ function (This will help with implementing memory profiling fully). [9pt] Cahaba Codebase Cleanup - Small Items #432
• Make memory profiling run as an option flag only.
• Standardize code [8pt] Standardize code #194
• Validate argument inputs [18pt] Validate argument inputs #197
• Develop unit tests for code Develop unit tests for existing code #195
• Remove unused* imports from Python scripts [3pt] Remove unused imports from Python scripts #52
• Validation, logging of fim_run.sh HUC success / failure
• Address bug where summary log file writes "Exit status = 0" for HUCs with non-zero exit statuses.

*unused by both Cahaba and fim_4.

Might Impact fim_4

• Convert the run_by_unit.sh script from Bash to Python.
• Convert the fim_run.sh script from Bash to Python.
• Finish implementing memory_profiling in the fim_run, run_by_unit, and /src/* scripts.

[BradfordBates-NOAA]

@nickchadwick-noaa I've noted for each item whether or not it might harm the fim_4 forked repo, please double check it. Two lists, one called "Won't Impact fim_4" and another list called "Might Impact fim_4". For example, "Converting the run_by_unit.sh script from Bash to Python" might impact fim_4. This is just a best guess. We'll coordinate with DS later to confirm.

[fernando-aristizabal]

@BradfordBates-NOAA Good point. I would avoid changing fim_run and run_by_unit to Python for now. Come the first quarter of the new fiscal year, FIM 4 will release GMS and then should be a good time to do major refactoring.

@nickchadwick-noaa Could you post any scripts, Python, or apt dependencies that you propose to remove prior to?

Also, what's the Dockerfile consolidation about?

[BradfordBates-NOAA]

@fernando-aristizabal Thanks for checking in on this. After July 4 we plan to hold a session with the DS team to iron out a refactor/clean-up plan for fim_4 sensitive items. Until then, we plan to knock out some of the harmless stuff. If you see something on the list, like the Dockerfile refactoring plans, that causes concern, please let us know.

[nickchadwick-noaa]

@fernando-aristizabal I will of course post any scripts we plan to remove, to ensure that they are not being used by anyone.

Initially, I had created a separate Dockerfile.prod to be used in a production environment. This Dockerfile is very similar to the Dev, except that it includes the codebase directly in the image instead of using a docker volume to link the codebase. We ended up not using this production Docker image when we moved to a "production" environment. To my knowledge this Dockerfile.prod is not used (unless you guys are using it), so I was just going to remove it.

[fernando-aristizabal]

@WinstonHoyle-NOAA Do you need a Docker image with the source code embedded within for AWS Batch?

[WinstonHoyle-NOAA]

I do not think so, we already have two in ECS that have the application built and models ready to run when the container starts.

…

On Tue, Jun 22, 2021 at 7:16 AM FernandoAristizabal-NOAA < ***@***.***> wrote: @WinstonHoyle-NOAA <https://github.com/WinstonHoyle-NOAA> Do you need a Docker image with the source code embedded within for AWS Batch? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#422 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUHA5EE4F3J37FIODQR4MJ3TUCLMLANCNFSM47CKFM2A> .

[RobHanna-NOAA]

In light of recent changes (Fim 4.1.0.0, fim 4.2.0.0 and fim 4.3.0.0), this issue is not longer applicable as it has been covered.