Cds 265 url area selector too small selection #194
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JamesVarndell
approved these changes
Aug 27, 2024
ecmwf-cobarzan
pushed a commit
that referenced
this pull request
Sep 11, 2024
* handle too small area selection
ecmwf-cobarzan
added a commit
that referenced
this pull request
Oct 3, 2024
* dev/cams * Steps towards a first rate limiting solution * Adapt hardcoded mapping * Small bug fix (for testing code) * Call the secondary adaptors (via a cdsapi request) * Send all params to secondary adaptors * Call retrieve_* from CAMSEuropeAirQualityForecastsAdaptorFor*Data * Add mock create_result_file in retrieve_xxx * Log how many req_groups * Create random file names in retrieve_xxx (check if these are not already generated) * Create random file names in new_retrieve_subrequest (check if these are not already generated) * Log how many req_groups[0]['retrieved_files'] * Add process_grib_files * MockResultFile to str * MochResultFile to string (at the right time, for now) * Covertions (netCDF, zip) * Revert likely previously commented create_result_file * Debug info * Debug format * Debug format * Add create_temp_file to context * Rate limiting: OK * See URLs for local fields * Test with hardcoded object store URL * Rewrite temporary_location for object store * Upload (for temporary files for now) * Upload test (for both permanent and temporary) * Fix model in the hardcoded mapping * Read context.request['mapping'] from the config * Debug * Remove hardcoded mapping and use the one in the configuration * Use an adaptor specific mapping function to apply the direct mapping from the configuration * Debug * Send the mapping from the adaptor to new_cams_regional_fc (as it is consummed by a config.pop in a superclass) * Deal with no result cases under new_retrieve_subrequest * Return None from CAMSEuropeAirQualityForecastsAdaptorFor*Data:retrieve if no data retrieved * Intersect constraints * Add _in_adaptor_no_cache to the list of recognised keys * Pop _in_adaptor_no_cache after the cache is avoided (to not propagate it to sub-requests) * Remove _in_adaptor_no_cache from the recognised keys list (likely not needed anymore) * Debug * Pop _in_adaptor_no_cache from the requests resulted after intersection * All components working (polishing to do) * Switch to using the standard apply_mapping (facilitated by mapping.remap and mapping.rename being populated as expected) * Add tabulate as a dependency * Load CDSAPI client credentials from environment * Remove unused imports, move imports to methods * Move tabulate from the list of mandatory dependencies to the complete version * Assume that the dataset specific bucket exists * Support netcdf_zip in cams_regional_fc (done by Luke) * Solve small bug in reassign_missing_to_archive * Bug fix in cams_regional_fc * Roocs adaptor: set timeout to a generous 30 seconds * Roocs adaptor: set timeout back to 3 seconds * Roocs adaptor: read the URL download timeout from the dataset configuration * Log request IDs of sub-requests done via the CDSAPI * Cds 265 url area selector too small selection (#194) * handle too small area selection * fix (#197) * mapped_requests * Remove temporary function * URL adaptor * adaptors refactor * Request type * Request type * mapped_requests * mapped_requests * debug * recall normalise_request * pre mapping mods * handle formats * handle embargo exceptions * handle download_formats * preserve _pre_retrieve so that adaptors can be updated gradually * intersect_constraints_bool * remove cyclical calling of intersect_constraints * cads-obs to use normalise_requests * cads-obs to use normalise_requests * some tidying * TODO update * set_download_format, with nice error * set_download_format, with nice error * download format, remove bug and add assertion check * CamsSolarRad adaptor to use normalise_request * type checking * QA + typing * URL adaptor/tools: add support for "fail_on_timeout_for_any_part" * Short revert * Revert * schema is now called at start of normalise request to ensure that we duplicate all the testing and modifications * Small typo fix (cads_adaptors.tools.download_tools) * CAMS solar radiation adaptor: adjust to request["format"] now being a list * Revert mapped request to scalar values (a specific requirement for this adaptor, at least for now) * Linting * Download sub-request result in a loop (for the cases when it fails to download completely the first time) * URL tools: set default for fail_on_timeout_for_any_part to True * clean update * typo * Refactor CAMSEuropeAirQualityForecastsAdaptor:retrieve * Refactor cams_regional_fc.py * Change destionation bucket to "cci2-cams-regional-fc-test" (a test one) * Refactor cacher.py * Refactor cads_adaptors/adaptors/__init__.py * Refactor cacher.py * Don't send the mapping to cams_regional_fc anymore * Remove debug code * Remove debug code * Send parent requests ID and user ID to the sub-request (and log them) * Restructure the directory structure the adaptor relies on * Switch to using the operational S3 bucket * Revert accidental cherry pick (wrt to InvalidRequest in legacy_intersect_constraints) * Linting * More linting * Type-checks * Avoid tabulate dependency (with dirty_manual_tabulate) * Debug * Debug * Debug * Debug --------- Co-authored-by: Luke Jones <luke.jones@ecmwf.int> Co-authored-by: Eddy Comyn-Platt <53045993+EddyCMWF@users.noreply.github.com> Co-authored-by: EddyCMWF <Edward.Comyn-Platt@ecmwf.int>
ecmwf-cobarzan
added a commit
that referenced
this pull request
Oct 25, 2024
* dev/cams * Steps towards a first rate limiting solution * Adapt hardcoded mapping * Small bug fix (for testing code) * Call the secondary adaptors (via a cdsapi request) * Send all params to secondary adaptors * Call retrieve_* from CAMSEuropeAirQualityForecastsAdaptorFor*Data * Add mock create_result_file in retrieve_xxx * Log how many req_groups * Create random file names in retrieve_xxx (check if these are not already generated) * Create random file names in new_retrieve_subrequest (check if these are not already generated) * Log how many req_groups[0]['retrieved_files'] * Add process_grib_files * MockResultFile to str * MochResultFile to string (at the right time, for now) * Covertions (netCDF, zip) * Revert likely previously commented create_result_file * Debug info * Debug format * Debug format * Add create_temp_file to context * Rate limiting: OK * See URLs for local fields * Test with hardcoded object store URL * Rewrite temporary_location for object store * Upload (for temporary files for now) * Upload test (for both permanent and temporary) * Fix model in the hardcoded mapping * Read context.request['mapping'] from the config * Debug * Remove hardcoded mapping and use the one in the configuration * Use an adaptor specific mapping function to apply the direct mapping from the configuration * Debug * Send the mapping from the adaptor to new_cams_regional_fc (as it is consummed by a config.pop in a superclass) * Deal with no result cases under new_retrieve_subrequest * Return None from CAMSEuropeAirQualityForecastsAdaptorFor*Data:retrieve if no data retrieved * Intersect constraints * Add _in_adaptor_no_cache to the list of recognised keys * Pop _in_adaptor_no_cache after the cache is avoided (to not propagate it to sub-requests) * Remove _in_adaptor_no_cache from the recognised keys list (likely not needed anymore) * Debug * Pop _in_adaptor_no_cache from the requests resulted after intersection * All components working (polishing to do) * Switch to using the standard apply_mapping (facilitated by mapping.remap and mapping.rename being populated as expected) * Add tabulate as a dependency * Load CDSAPI client credentials from environment * Remove unused imports, move imports to methods * Move tabulate from the list of mandatory dependencies to the complete version * Assume that the dataset specific bucket exists * Support netcdf_zip in cams_regional_fc (done by Luke) * Solve small bug in reassign_missing_to_archive * Bug fix in cams_regional_fc * Roocs adaptor: set timeout to a generous 30 seconds * Roocs adaptor: set timeout back to 3 seconds * Roocs adaptor: read the URL download timeout from the dataset configuration * Log request IDs of sub-requests done via the CDSAPI * Cds 265 url area selector too small selection (#194) * handle too small area selection * fix (#197) * mapped_requests * Remove temporary function * URL adaptor * adaptors refactor * Request type * Request type * mapped_requests * mapped_requests * debug * recall normalise_request * pre mapping mods * handle formats * handle embargo exceptions * handle download_formats * preserve _pre_retrieve so that adaptors can be updated gradually * intersect_constraints_bool * remove cyclical calling of intersect_constraints * cads-obs to use normalise_requests * cads-obs to use normalise_requests * some tidying * TODO update * set_download_format, with nice error * set_download_format, with nice error * download format, remove bug and add assertion check * CamsSolarRad adaptor to use normalise_request * type checking * QA + typing * URL adaptor/tools: add support for "fail_on_timeout_for_any_part" * Short revert * Revert * schema is now called at start of normalise request to ensure that we duplicate all the testing and modifications * Small typo fix (cads_adaptors.tools.download_tools) * CAMS solar radiation adaptor: adjust to request["format"] now being a list * Revert mapped request to scalar values (a specific requirement for this adaptor, at least for now) * Linting * Download sub-request result in a loop (for the cases when it fails to download completely the first time) * URL tools: set default for fail_on_timeout_for_any_part to True * clean update * typo * Multi-threaded caching in CAMS regional adaptor * Multi-threaded caching in CAMS regional adaptor * Multi-threaded caching in CAMS regional adaptor * Auto-linting of cams_regional_fc/cacher.py * Auto-linting of cams_regional_fc/mem_safe_queue.py * Harmonise cads_adaptors/__init__.py, cads_adaptors/adaptors/__init__.py with main * Delete cams_regional_fc/remote_copier.py * Bug fix in cams_regional_fc/cacher.py * Small qa --------- Co-authored-by: Luke Jones <luke.jones@ecmwf.int> Co-authored-by: Eddy Comyn-Platt <53045993+EddyCMWF@users.noreply.github.com> Co-authored-by: EddyCMWF <Edward.Comyn-Platt@ecmwf.int>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.