Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MOBT-797: Coordinate retention in percentile generation #2087

Merged
merged 2 commits into from
Jan 29, 2025

Conversation

bayliffe
Copy link
Contributor

This PR enables the retention of scalar coordinates that are generated when collapsing coordinates to generate percentiles. It also ensures that if these retained coordinates are time related (time, forecast_period, forecast_reference_time), they are modified to ensure that the point is aligned to the upper bound, adhering with IMPROVER's metadata standards.

The purpose of this change is to enable the production of typical wind speed diagnostics with our existing tooling. The suite chain will become:

flowchart TD

A(wind speed in 3-hours) --> B(merge CLI: construct 24-hour cube with time coordinate) --> C(generate_percentiles CLI: collapse time and realization coordinates to generate percentiles)
Loading

This change ensures that the time period covered by the diagnostic is available on the final diagnostic. Note that we will recycle the midnight validity time for adjacent days, e.g.

Day 1: 20250115T0000Z, 03, 06, 09, 12, 15, 18, 21, 20250116T0000Z
Day 2: 20250116T0000Z, 03, 06, 09, 12, 15, 18, 21, 20250117T0000Z

The wind speed diagnostics do no cover a period, so we use as many samples as we can for a given day. This means that, in the example shown here the 20250116T0000Z forecast is included in the forecast for day 1 and day 2. The generate_percentiles CLI will create bounds from the time points (as there are no bounds here), which results in bounds that span the whole 24-hours on the output cube.

Testing:

  • Ran tests and they passed OK
  • Added new tests for the new feature(s)

…ntile generation from coordinate collapse. This change also ensures that any scalar time coordinates that are created in this way adhere to the improver metadata standards.
Copy link

codecov bot commented Jan 27, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.40%. Comparing base (84a8944) to head (97ac22b).
Report is 64 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2087      +/-   ##
==========================================
+ Coverage   98.39%   98.40%   +0.01%     
==========================================
  Files         124      134      +10     
  Lines       12212    13254    +1042     
==========================================
+ Hits        12016    13043    +1027     
- Misses        196      211      +15     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Kat-90 Kat-90 self-assigned this Jan 27, 2025
Copy link
Contributor

@Kat-90 Kat-90 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this with a few coordinates to collapse and retain. It works as intended but I found that the bounds/points it puts on non-time related coordinates aren't always as useful. That's nothing to do with this PR though so happy to approve.

@Kat-90 Kat-90 removed their assignment Jan 27, 2025
@brhooper brhooper self-assigned this Jan 28, 2025
Copy link
Contributor

@brhooper brhooper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bayliffe, I've added a question about whether the CLI change requires an acceptance test to go along with it.

@brhooper brhooper assigned bayliffe and unassigned brhooper Jan 28, 2025
@brhooper brhooper merged commit 347a8a4 into metoppv:master Jan 29, 2025
10 checks passed
mo-philrelton pushed a commit to mo-philrelton/improver that referenced this pull request Jan 29, 2025
* Enable retention of scalar coordinates that are produced during percentile generation from coordinate collapse. This change also ensures that any scalar time coordinates that are created in this way adhere to the improver metadata standards.

* style fixes.
cpelley pushed a commit that referenced this pull request Jan 30, 2025
* MOBT-797: Coordinate retention in percentile generation (#2087)

* Enable retention of scalar coordinates that are produced during percentile generation from coordinate collapse. This change also ensures that any scalar time coordinates that are created in this way adhere to the improver metadata standards.

* style fixes.

* Setting percentiles with as_iterable

---------

Co-authored-by: bayliffe <benjamin.ayliffe@metoffice.gov.uk>
cpelley pushed a commit that referenced this pull request Jan 30, 2025
* MOBT-797: Coordinate retention in percentile generation (#2087)

* Enable retention of scalar coordinates that are produced during percentile generation from coordinate collapse. This change also ensures that any scalar time coordinates that are created in this way adhere to the improver metadata standards.

* style fixes.

* Setting percentiles with as_iterable

---------

Co-authored-by: bayliffe <benjamin.ayliffe@metoffice.gov.uk>
MoseleyS added a commit that referenced this pull request Feb 3, 2025
…ad_time using it (#2088)

* radius_by_lead_time improved input handling

* Make is_iterable changes applicable to percentiles option (#2090)

* MOBT-797: Coordinate retention in percentile generation (#2087)

* Enable retention of scalar coordinates that are produced during percentile generation from coordinate collapse. This change also ensures that any scalar time coordinates that are created in this way adhere to the improver metadata standards.

* style fixes.

* Setting percentiles with as_iterable

---------

Co-authored-by: bayliffe <benjamin.ayliffe@metoffice.gov.uk>

* TEDST: is_iterable

* Update improver/nbhood/__init__.py

Co-authored-by: Stephen Moseley <stephen.moseley@metoffice.gov.uk>

* Update __init__.py

* Update nbhood.py

---------

Co-authored-by: Phil <phil.relton@metoffice.gov.uk>
Co-authored-by: bayliffe <benjamin.ayliffe@metoffice.gov.uk>
Co-authored-by: Stephen Moseley <stephen.moseley@metoffice.gov.uk>
@bayliffe bayliffe added this to the IM2025.1/PS47 milestone Mar 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants