Areas for improving our processes

[Jennit07]

Following the September update, we have a few ideas for improving our processes.

• Use the package Polars in R version 4.2 (R updated required for access). This will improve speed efficiency.
• Use of temporary files for improving de-bugging. Currently to find a bug we must run the whole process from start to the function. If we have temporary files this will allow us to de-bug faster.
• Following the use of temporary files, create test_mode = TRUE for only testing one year of data. This will only create temporary files for one year to save space in sourcedev. The idea is to test one year and if everything is working then we can proceed to running all years as the assumption is that the code will work because it should be identical.
• Update documentation. We can create flow charts to help visualise the process. This was particularly helpful for understanding the deaths processing due to the complexities.
• Time consumption documentation. How long does each function take to process? How long is the process for ep file start to finish? Same for individual file.
• Update/create more tests for functions. For example, the deaths lookup had a duplicate in the September update. Create tests for similar lookups to prevent this in the future. (low priority)
• Make check_year_valid function more visible for updating every quarter. Possibly add to 00_update_refs
• Investigate processing for 1819 SLFs. Ran into issues with memory and this year will only run on the BIG node