Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SDR and probably other routed models have int32 limitations #1431

Closed
phargogh opened this issue Oct 19, 2023 · 1 comment · Fixed by #1434 or #1521
Closed

SDR and probably other routed models have int32 limitations #1431

phargogh opened this issue Oct 19, 2023 · 1 comment · Fixed by #1434 or #1521
Assignees
@phargogh
Labels
bug Something isn't working

Comments

@phargogh
Copy link
Member

In natcap/pygeoprocessing#351, we fixed an issue in MFD routing where rasters that had over 2^32 pixels, there would be an indexing issue and a related nodata comparison issue.

These problems with indexing also apply to all of the InVEST models that use MFD, which include SDR, NDR and SWY. @jagoldstein encountered this issue in SDR specifically (see logfile below), but we should fix the issue for all of the routed models as well.

This is important for the GEF work.

InVEST-natcap.invest.sdr.sdr-log-2023-10-17--14_22_25.txt
@phargogh phargogh added bug Something isn't working in progress This issue is actively being worked on labels Oct 19, 2023
@phargogh phargogh self-assigned this Oct 19, 2023
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Adding a 2-phase docker build for testing.
576f4c4 
RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Dockerfile syntaxerror RE:natcap#1431
495ea32
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Installing apt packages with -y. RE:natcap#1431
8cac295
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Allowing invest repo and build ref to be provided as build args. RE:n…
b9d38db 
…atcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Trying to quote the strings instead to try to get default arg values RE:
c9d72de 
natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Build contexts are not shared between images. RE:natcap#1431
3195875
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Adding venv, making dirname depend on repo. RE:natcap#1431
44d5b1e
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Updating requirements and removing wheel after build. RE:natcap#1431
55cb3d3
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Adding a docker container workflow building job.
a90a795 
RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Creating the environment file. RE:natcap#1431
efa1df2
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Removing the branch restriction for building the container. RE:natcap…
6bfd67f 
…#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Sanitizing the github ref. RE:natcap#1431
bb64267
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Using sed for substring replacement instead of bash builtings. GHA do…
a3f951f 
…esn't like all the nested curly braces. RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Not removing the wheel, not writeable. RE:natcap#1431
77be2d4
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Adding requests to requirements. RE:natcap#1431
beea25d
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Adding requirements to the list of triggering files. RE:natcap#1431
781789d
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Adding note about the docker container to HISTORY. RE:natcap#1431
5f5817e
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Building latest pygeoprocessing and installing into container. RE:nat…
c181d28 
…cap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Passing invest repo and version to the build image. RE:natcap#1431
f6ea5ec
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Sediment deposition now uses more memory, but can handle larger indexes.
eab9919 
RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Noting change in HISTORY for SDR. RE:natcap#1431
2d6cc1e
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Improving the number of pixels that can be included in the analysis.
0d647f3 
This comes at a near-doubling of required stack memory (and some heap
memory), but it should be OK for most users.  RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Upping stack/heap memory usage for indexes in seasonal water yield's …
6acc0a7 
…cython core. RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 19, 2023
@phargogh
Using the latest pygeoprocessing commit on main.
db1a05e 
RE:natcap#1431
@phargogh phargogh mentioned this issue Oct 21, 2023
Merged
1 task
phargogh added a commit to phargogh/invest that referenced this issue Oct 24, 2023
@phargogh
Monitoring a few other paths for rebuilding docker image.
274f669 
RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 24, 2023
@phargogh
Replacing native pygeoprocessing builds with better yml generation.
e6f850c 
This way pip will perform scm-based package installation like normal
with whatever extra dependencies it needs.

RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 24, 2023
@phargogh
Clearing pip cache during the build.
39ebcb3 
RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 24, 2023
@phargogh
Adding a compiler when we are building from source.
f7119e8 
RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 24, 2023
@phargogh
Specifying the base prefix. RE:natcap#1431
4800f4a
phargogh added a commit to phargogh/invest that referenced this issue Oct 24, 2023
@phargogh
Attempting to address a build warning about setuptools_scm/setuptools…
8bfd8d0 
… version incompatibility that I encountered in github actions. RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 24, 2023
@phargogh
Restoring pygeoprocessing 2.4.0 requirement.
8301c75 
RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 24, 2023
@phargogh
Explicitly prohibiting uploading to the registry when part of a PR, j…
76c84ba 
…ust in case. RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 24, 2023
@phargogh
Removing the pip constraint on pygeoprocessing. RE:natcap#1431
ab8400f
phargogh added a commit to phargogh/invest that referenced this issue Oct 24, 2023
@phargogh
Adding a compiler if on linux or it's needed.
89d840a 
A compiler will be needed to build from source.

RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Oct 25, 2023
@phargogh
Fixing history message heading. RE:natcap#1431
d9aeced
phargogh added a commit to phargogh/invest that referenced this issue Oct 25, 2023
@phargogh
Removing failure-case handling for git lfs install. RE:natcap#1431
cd492aa
@davemfish davemfish removed the in progress This issue is actively being worked on label Nov 20, 2023
phargogh added a commit to phargogh/invest that referenced this issue Jan 30, 2024
@phargogh
Logging some more debugging info, changing a couple types.
2cc5d1e 
RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Feb 1, 2024
@phargogh
Using the watersheds mask in align_and_resize_raster_stack.
e6388ed 
RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Feb 1, 2024
@phargogh
Adding some additional debugging information.
41ab9ca 
RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Feb 1, 2024
@phargogh
Adding some more debug logging.
2d4bd54 
RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Feb 1, 2024
@phargogh
Fixing datatype of processing stack.
dcf09ea 
We are overflowing the int32 space, so the flat_index of a pixel in the
processing queue is overflowing int32 (the type of the stack).

RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Feb 1, 2024
@phargogh
Correcting paths passed to alignment.
d6a36b8 
RE:natcap#1431
phargogh added a commit to phargogh/invest that referenced this issue Feb 2, 2024
@phargogh
Refactoring summary stats vector to minimize rasterizations.
3397273 
RE:natcap#1431
@phargogh
Copy link
Member Author

phargogh commented Feb 6, 2024

This came back up with a Colombia SDR run once again.

@phargogh phargogh reopened this Feb 6, 2024
phargogh added a commit to phargogh/invest that referenced this issue Feb 6, 2024
@phargogh
Removing my debugging code. RE:natcap#1431
e4721ab
phargogh added a commit to phargogh/invest that referenced this issue Feb 6, 2024
@phargogh
Restoring pygeoprocessing requirement. RE:natcap#1431
e50260b
phargogh added a commit to phargogh/invest that referenced this issue Feb 6, 2024
@phargogh
Removing the watersheds mask path so taskgraph doesn't fail on us. RE:n…
bccab6c 
…atcap#1431
@phargogh phargogh mentioned this issue Feb 6, 2024
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@phargogh phargogh

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
2 participants
@davemfish @phargogh