[5pt] Preprocess MS and FR stream networks

[BrianAvant]

A new method to moves the contents of snap_and_clip_to_nhd.py to aggregate_vector_inputs.py as preprocessing step.
addresses issues #58, #12, #118, #73, and partial resolution of #4.

Major Changes

In the current FIM 3.0.0.2 workflow, NHDPlus HR is subset and traversed on an individual HUC scale. This creates a problem in HUCs with incoming streams due to incomplete knowledge of the upstream network. This feature branch attempts to resolve this issue by preprocessing the FR and MS network for the entire NWM domain in aggregate_vector_inputs.py. In addition, the new workflow aims to resolve several recurring issues:

• headwater stream segments geometries are adjusted to align with with NWM streams
• snapping NWM headwater points to NHDPlus HR is confined within HUC8 boundaries to ensure catchment preservation where major changes have been made from MR to HR stream network. In the current FIM workflow, a few instances have been observed where the closest HR stream segment is actually outside the HUC boundary of the NWM headwater point.
• Incoming streams are selected using intersection points between NWM streams and HUC4 boundaries. Tributary stream segments enchroaching HUC boundaries occurs in both NHDPlus HR and NWM stream networks. This creates an issue when trying to identify incoming streams by intersection.
• clip_vectors_to_wbd.py handles cases where no headwater points lie within the buffered HUC boundary but incoming streams exist. In this case a local headwater is designated and the downstream network is retained for further processing.
• removes NHDPlus features categorized as coastline and underground conduit. There is room for improvement here if the HR dataset becomes more consistent in categorizing FType.

Added Files

• lib/adjust_headwater_streams.py
• lib/reduce_nhd_stream_density.py
• replaced snap_and_clip.py with clip_vectors_to_wbd.py

Additional Minor Changes

• added streams layer to production whitelist
• consistent variable types in parameter environment files
• fixed progress bar bug in lib/acquire_and_preprocess_inputs.py
• added getDriver to shared functions.py
• cleaned up variable names in buildstreamtraversal.py

Testing

Aggregate CSI score comparison without headwater adjustment

FIM 3.0.0.2	Feature branch	Extent	Event
0.389	0.386	MS	100 yr
0.556	0.556	FR	100 yr
0.410	0.407	MS	500 yr
0.582	0.582	FR	500 yr

Aggregate CSI score comparison with headwater adjustment

FIM 3.0.0.2	Feature branch	Extent	Event
0.389	0.386	MS	100 yr
0.556	0.557	FR	100 yr
0.410	0.407	MS	500 yr
0.582	0.584	FR	500 yr

FR CSI scores are very similar to the current development branch before adjusting headwater streams. This is expected as there are currently no intersection operations for this extent. However, this new method should provide a better FIM network overall beyond just the 49 BLE sites used for evaluation. Adjusting the FR headwaters shows a slight overall improvement in CSI scores.

It has been noted in a previous PR that BLE observation data is not ideal for comparing MS outputs. Although there is a slight regression in aggregate CSI scores, a qualitative look at the data shows that the results are comparable. Another key improvement to the MS network is the handling of incoming streams. In FIM 3.0.0.2, any NHDPlus HR stream segment that crosses the HUC boundary is assumed to be a MS headwater stream. This resulted in unwanted encroaching tributaries being included in the MS stream network. The new feature branch resolves this issue by preprocessing the network and filtering out unwanted tributaries.

[BradfordBates-NOAA]

After this is merged, it seems like the changelog should show 3.0.1.0. @nickchadwick-noaa thoughts?

CORRECTED -- I recommended 3.0.2.0 before, by accident.

[nickchadwick-noaa]

I agree @BradfordBates-NOAA , 3.0.1.0 seems reasonable since it's a larger update.

[RyanSpies-NOAA]

Reviewed code modifications/additions and examined new outputs. I did not run the new code, but the newly created vector files appear as expected. I also created a slide deck summarizing my review. Approving for merge.

[BradfordBates-NOAA]

Friendly reminder to update CHANGELOG to read 3.0.1.0

[nickchadwick-noaa]

Re-approved after updating CHANGELOG.md