Road data script troubleshooting and fixes (2024 Q3)

[MichaelAGRC]

Benefit

As road data comes in from the counties, occasionally they change their schema or add values to the data that is outside the expected domain. This causes our scripts to fail. Troubleshooting these error messages and fixing the issues will speed up the monthly road update process.

Acceptance Criteria

Getting the scripts to run after they throw an error message. This will require either altering the scripts a bit or the data.

Notes

Notes on troubleshooting script errors
The majority of the time, when a script throws an error it's because the county changed the schema. This could be a change in expected domain values or a field name change. Our county ETL scripts are hard-coded to expect certain values and when they change, the script throws an error.

The troubleshooting part is determining what changed and how to proceed. Here are some steps to take when troubleshooting and then fixing the issue:

Check the error message. It will often inform you of the unexpected value (domain change) or a missing or renamed field name.
Based on the error message, determine if it's easier to fix/update the script or the county data. Oftentimes, it's quicker to alter the county's data (fixing the wonky field values or the missing field). Other times, it's easier to alter the script to accommodate their new schema. It's a case-by-case situation.
When updating scripts, if the issue occurs in Step 1 or Step 3 you can update the script directly. If the issue occurs in Step 2, feel free to work with Greg the first couple times as these scripts are a slightly more complex, but not out of scope. Essentially, if it occurs in Step 2, you'll need to look for the county's name in UtransETL_FieldMappingFunctions.py and make the change there.

Risks

No response

Issue Reference

refs #

[MichaelAGRC]

Sprint 1

• Sevier
• Carbon

[MichaelAGRC]

All counties processed for sprint 1. Davis was not processed - waiting on feature update on arcmap tool. Ref #316

[steveoh]

@gregbunce are you planning on updating the arcmap tool quickly to allow for the counties to get processed if michael is waiting on it?

[gregbunce]

yeah, my plan is to have it updated before michael processes the roads next month.

[MichaelAGRC]

Code wasn't quite ready and required further testing and installing. As such, Grand will be pushed to February processing.

[MichaelAGRC]

Also, Davis will be processed using updated tool during sprint 3/4.

[MichaelAGRC]

Sprint 4 Notes

Grand county is throwing an unknown error during the first script. Will need to work with @gregbunce to resolve the issue.

[MichaelAGRC]

This is closed. New issue created for Q4.