Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cogify): Update cogify to support the topo raster processes. BM-1116 #3388

Merged
merged 24 commits into from
Feb 27, 2025

Conversation

Wentao-Kuang
Copy link
Contributor

@Wentao-Kuang Wentao-Kuang commented Jan 8, 2025

Motivation

As a Basemaps user, I want to consume the NZTopo 50 & 250 Maps as a tile service.

Modifications

This work comprises a series of additions and modifications to the basemaps/cogify package.

These changes allow us to generate STAC files for an NZTopo Map Series imagery collection. They also allow us to generate standardised and cloud-optimised GeoTIFFs from the map sheet files of such collections. We have adjusted the basemaps/cogify package to handle the following collections stored in the AWS S3 TopoReleaseArchive directory:

s3://topographic-upload/TopoReleaseArchive/NZTopo50_GeoTif_Gridless/
s3://topographic-upload/TopoReleaseArchive/NZTopo250_GeoTif_Gridless/

There is also a new Argo workflow in development that depends on this work as part of its process flow. The workflow's purpose is to automate the standardisation of an NZTopo Map Series imagery collection.

Generating STAC files

This works defines a new process for generating a nested directory structure of STAC files for an NZTopo Map Series imagery collection. The process operates as follows:

  1. Loads the collection of GeoTiff images into memory
  2. Extracts the map code, version, and metadata (e.g. EPSG) from each image
  3. Identifies the latest version of each map sheet by code
  4. Groups the images by EPSG
  5. Generates a StacItem file for each image, and a StacCollection file for each EPSG grouping
  6. Structures the groups of StacItem and StacCollection files into a directory tree
  7. Saves the files as they are structured to a target location

Outputs

The command groups the images by EPSG and then structures the generated StacItem and StacCollection files, as illustrated:

Schema Example

The command then saves the generated tree of folders and files into the target location directory.

Processing Map Sheet files

This work extends the create CLI command of the basemaps/cogify package. This extension allows the command to recognise and process the map sheet files of an NZTopo Map Series imagery collection by way of particular properties added to the accompanying STAC files.

@Wentao-Kuang Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Jan 8, 2025
@Wentao-Kuang Wentao-Kuang marked this pull request as ready for review January 12, 2025 21:01
@Wentao-Kuang Wentao-Kuang requested a review from a team as a code owner January 12, 2025 21:01
@Wentao-Kuang Wentao-Kuang changed the title feat(cogify): Update the cogify create cog to support topo raster. feat(cogify): Update the cogify create cog to support topo raster. BM-1116 Jan 12, 2025
@Wentao-Kuang Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Jan 13, 2025
@Wentao-Kuang Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Jan 27, 2025
@tawera-manaena tawera-manaena force-pushed the feat/topo-raster-cog-creation branch from 24087c3 to 61f2b1a Compare February 23, 2025 21:28
@tawera-manaena
Copy link
Contributor

tawera-manaena commented Feb 23, 2025

I've re-based this branch to origin/master to bring in the tile matrix additions. Once I resolve Chatham Islands tile matrix workarounds, we can run an Argo workflow to make sure all of our Topo Raster processes work as expected.

@tawera-manaena tawera-manaena marked this pull request as draft February 23, 2025 22:32
@tawera-manaena
Copy link
Contributor

tawera-manaena commented Feb 23, 2025

Some functions have been refactored several times and therefore have outdated docs. I am correcting such instances now. Edit: I've also refined many of the Topo STAC creation functions as they had remnants of past refactorings that needed to be re-optimised.

@tawera-manaena tawera-manaena marked this pull request as ready for review February 24, 2025 02:35
@tawera-manaena tawera-manaena added container build pull request container for testing and removed container build pull request container for testing labels Feb 24, 2025
@blacha blacha enabled auto-merge February 27, 2025 02:39
@blacha blacha added this pull request to the merge queue Feb 27, 2025
Merged via the queue into master with commit 4366df6 Feb 27, 2025
17 checks passed
@blacha blacha deleted the feat/topo-raster-cog-creation branch February 27, 2025 02:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
container build pull request container for testing
Development

Successfully merging this pull request may close these issues.

3 participants