BUFR sequence for DAYCLI #51
Comments
|
In July 2018, the World Meteorological Organization, launched a one-year trial phase for the monthly reporting of daily climate data as per BUFR template 3 07 074 (WMO 20180730), named DAYCLI message. This message includes 6 variables: maximum temperature, minimum temperature, mean temperature, precipitation, depth of new snowfall and depth of total snow on the ground (See File WMO 20180730). An assessment report of the trial phase was published in May 2020 (See File WMO 20200918 ). It recommends enhancing the DAILY message, both in term of climatological practices (metadata) and BUFR structure. It suggest also a way forward on monitoring the exchange of DAYCLI messages, and the promotion of WMO Member participation on this project. EXTENSION OF THE TRIAL PHASE OF THE INTERNATIONAL EXCHANGE OF DAILY CLIMATE DATA WITH THE AIM OF RECOMMENDING ITS OPERATIONAL IMPLEMENTATION IN 2021 The Expert Team on Data Requirements for Climate Services (ET-DRC) from the WMO Services Commission has been proposed to specify all information needed for the complete data understanding, which means the necessary metadata. At first analysis the BUFR template should precise:
A start on the DAYCLI requirement is available here: |
|
Stations that send the CLIMAT message (normals values with monthly parameters and daily extremes, average and cumulative values) are forseen to send also the DAYCLI message. The reference guidelines for the CLIMAT message are : |
|
Those who are sharing their development :
|
|
Aside: libecbufr is now migrated to GitHub: https://github.com/ECCC-MSC/libecbufr |
|
The following Members submitted interest and/or written intents to the Secretariat to participate in the trial :
|
|
Let's present the Expert Team on Data Requirements for Climate Services
Core members: |
Thanks ! |
|
New version on issues and requirements : Draft on actors and tasks : |
|
I present here the summary of the meeting we had yesterday (Sergio and Denis) I believe that the information from this meeting is sufficient to make an initial outline of a new BUFR sequence that will be developed based on the existing DAYCLI sequence (3 07 074). I intend to make an initial document to put on GitHub. These are the points covered: 1 -The time slots to measure daily extreme, cumulative and average of variable as Max temperature, Min temperature, Precipitation and Average temperature differ from one NMHS to another, and also are not the same in one NMHS - Météo-France
India Luxembourg Brazil
In the case of the current template (3-07-074). The hour and time period are the same for maximum, minimum and mean temperature, which is not what is seen in a lot of NMHSs This is the part of 3-07-074 relative to the temperature One possible solution is put Day Hour and Time period or displacement inside of the loop. 1-05-003 - Replicate 5 descriptor 3X But there are other points to also be consider **2 ** There are many practical differences in calculating the average temperature. Therefore, it will probably be necessary to add a descriptor before the average temperature to identify the type of calculation used. 3 In the current template, the height out sensor (0-07-030) represent one value for all sensor (temperature and precipitation), but the height could be different in each case and not make sense in case of precipitation. At same time, this definition is already in OSCAR and not necessary in the template. So the most simple solution in this case would be to remove 0-07-030 4 Quality code for each variable- The idea is to create flag_table (10 bits probably is enough) to represent quality information of the data. Bit No. **5 ** There are many definition of Depth of fresh snow, Total snow depth. Example: Could be measurement each day at special time, or accumulation from one time to another time.
(Surely the second definition but need to be validated by experts) 6 Adding the quality of the measurement at the station point. There are quality classification for temperature and precipitation related to location and environment in CIMO Guide (WMO No.8) Part I, Chapter I, Annex 1B.: (See attached files) Those 2 classifications are not yet part of the WIGOS Metedata, but was discussed in previous TT-WMD meetings.
|
|
Hi Denis, Your suggestion for greater detail such as those listed below are good ones, but may have the potential to make taking observations difficult for observers. Given how difficult it can be to make good snowfall measurements given variations within a single observing site and effects of blowing snow, and ability of the observer to follow procedures, I know with some of our volunteer networks we are happy when we get well taken total snow and new snow measurements. Regarding thoughts to add a quality indicator. I am guessing you are referring to the ability to indicate if the observation is preliminary (no or limited QC applied), final (fully QCd)? I believe something like that could be useful. But like most organizations we run all the data we collect through our own QC processes. Which we will do regardless of any quality indicators provided. |
|
About the snow, I was wondering about the definitions. It was not in my intention to add more variables about snow. I suppose that total snow at a time of observation and new snow that fell in the previous 24 hours is sufficient for the Daycli message. Nevertheless we need to precise the time of observation, and the period of time for Fresh snow. For the quality, yes I was suggesting to add a quality code for each value in order to kown if the value is a raw one, or not. If the value has been checked or not, which kind of controls have been performed on the value and the result of the controls. But this international quality code as to be defined. As you said, we (WMO community) have so many different practices about quality code ! As regard current projects on interoperability and the opencdms project I think we really need the design of a "global" quality code that could allow NMHSs to keep their own QC and be translated into the Global-QC when exchanging data worldwide. |
|
Minimum modification for the DAYCLI message sequence (0 07 074)
Recommended modifications for Climatology purpose DAYCLI message with the CLIMAT message are non-real time data exchange, more particularly suitable for climatological studies. Those messages in non-real time give time to NMHSs to include their added values : the data quality. The DAYCLI message is a good opportunity to start sharing data with better quality information. On the other hand the Flagging System to use has not received (so far) the approval from the WMO community.
There is an intention to add indication of MQC in WMDS as well, and we have already tasked SC-MINT/Expert Team on Surface and Sub-surface Measurement (ET-SSM) to draft a proposal on how both classification schemes could be indicated properly not only in WMDS, but also in OSCAR. Of course it will be done in collaboration with SC-ON/ET-WIGOS Tools. (Krunoslav PREMEC)
Remarks |
|
Comments of Dr. Jörg Klausen on Data Quality Codes:
|
|
I agree with this point. There is a lot of redundancy in tables 0 33 BUFR and it is difficult to decide what to use. Creating a new table is not a good solution. It would further increase the redundancies in the tables. |
|
I think all quality information is relevant to the climato community. It is therefore necessary to list all the details relating to this aspect before thinking about a way to optimize their representation in the BUFR message. |
|
For information, here below, explanation on the 2 classifications, the siting classification and the measurement quality classification: |
|
Just one detail about adding the day, hour and 'time period or displacement' for each variable. For example
In these cases we can fixed day = 01 and use:
Tmax preceded by hour = 00UTC and displacement = +24h to indicate the time frame from 01/01/2020 00UTC to 01/02/2020 00 UTC |
|
Flagging systems from : MICROSTEP (Slovakian company) |
|
Dear @DenisStuber and @sergioh-pessoal |
For sure, we will work on it... |
|
Proposition from Sergio y Denis will include:
Need to Experts validation from :
|
|
Here is a draft proposal for the new DAYCLI message. 1- Add a new entry in the significance qualifier class of table B ( 0-08-094) and a new code table to describe the method used to calculate the daily average temperature.
Discussion: There are many formulas in use to estimate the average daily temperature. However, in general, these formulas are weighted average calculations of temperatures measured throughout the day, where the weights for each temperature are adjusted according to the time zone of each location. In this context, we see two possibilities for the code table for methods in use
Tm = (Tn+ Tx)/2 The code table bellow was based on option 1, but it cam be rewrite based in the option 2. There are 15 position to include methods in this code table. In the case of option 2, it may be better to create a code table with more positions. CODE TABLE 0-08-094
Note : The letters "a", "b", "c", "d" and "e" generically represent the weight associated with the respective temperature T. The sub-index of T: "1", "2", "3", "x ”And“ n represent the values measured at different times or maximum (x) or minimum (n) values. 2. Quality flag the for each climatological value (Temperatures, Precipitation and Snow). System Flagging in preparation from SERCOM 3 – Siting classification and Measurement Quality Classification Discussion: Proposal:
Note: It is necessary to verify if another descriptor will be necessary to snow. 4 – End measurement day Discussion:
Proposal:
Add 1 entry in code table 0-31-021 ( Associated filed significance) Code figure 20: 1 bit indicator for the end measurement day set 0: Current day (From previous day to current day); This 1 bit indicator would be add in each descriptors bellow 0-26-001 Principal time of daily reading in UTC of maximum temperature 5 – The new DAYCLI BUFR message Update descriptor 3-07-074 (Update the sequence or create a new descriptor in the next version of BUFR table D)
|
|
Here a graphic to illustrate the different climatological day definitions according parameters e.g. Luxembourg, France and Brazil |
|
summary of climatological needs : |
* issue 51 update * xml,txt files * issue 51 comment: See note 1 Note (1) : The letters "a", "b", "c", "d" and "e" generically represent the weight associated with the respective temperature T. The sub-index of T: "1", "2", "3", "x" and "n" represent the values measured at different times or maximum (x) or minimum (n) values. * xml,txt files * update issue 51 see note 1 (see Note 1) Note (1) : The letters "a", "b", "c", "d" and "e" generically represent the weight associated with the respective temperature T. The sub-index of T: "1", "2", "3", "x" and "n" represent the values measured at different times or maximum (x) or minimum (n) values. * xml,txt files * update issue 51 * update issue 51 * update issue 51 3. Siting classification and Measurement Quality Classification * update issue 51 008095 and 008096 * update issue 51 change from 36 - 254 to 36-254 * updated issue 51 element names for 013012 and 013013 * Update BUFRCREX_CodeFlag_en_31.csv * Update BUFRCREX_CodeFlag_en_31.csv * Update BUFRCREX_CodeFlag_en_31.csv * xml,txt files * updated issue 51 Create a new descriptor 3-07-075 in the next version of BUFR table D. * update issue 51 add "" * xml,txt files * update issue 51 (0 31 021) * Update BUFRCREX_CodeFlag_en_31.csv * xml,txt files * modified tables based on issue51 updates * xml,txt files * issue51 cleanup * xml,txt files * Update BUFR_TableD_en_03.csv remove note ID #51 * xml,txt files * Issue #51 (editorial edit) * xml,txt files * Updated dockerfile and readme, debugging issues due to docker caching data. * "Revert accidently commit from nested repository" This reverts commit 3b0a1ee. * #51 move sequence to correct csv * xml,txt files * Update BUFR_TableD_en_07.csv remove extra line * xml,txt files * #51 change sequence name * xml,txt files * issue #51, remove edits to Table B 13 * xml,txt files * issue #51 remove name change * xml,txt files * Update BUFRCREX_CodeFlag_en_08.csv update note reference * xml,txt files * Update BUFR_TableD_en_07.csv editorial * xml,txt files Co-authored-by: xchen <58480226+chenxiaoxia2019@users.noreply.github.com> Co-authored-by: Enrico Fucile <efucile@wmo.int> Co-authored-by: david-i-berry <dyb@noc.ac.uk>
|
"Here is a tar file containing DAYCLI test data for 10 stations in the US. This is for Nov 2021. If you have any trouble with these or have any questions please let me know." Jay Lawrimore, Meteorologist, Dataset Section, Climatic Science and Service Division NOAA's National Centers for Environmental Information (NCEI) |
|
issue will be closed after WMO adoption |
|
Anna, would it be possible not to close too soon the issue ? Or to have another place to fallow the DAYCLI message implementation? |
|
@DenisStuber -- I don't think there is an option to modify this sequence after WMO adoption. This validation should have been done before fast-track submission. @jitsukoh -- do you have a perspective here? |
|
@DenisStuber , @amilan17, @jitsukoh Just clarifying. If I understand correctly, template is validated. However, there is a intetion that the countries check that the BUFR is created as they intended and send their comments to TT-TDCF and to ET-DRC. Another topic is the preparation Document / Training material for the DAYCLI message. I believe we will need another channel (git) to continue this other work fase At moment I'm submitting the last BUFR encodings from US by e-mail to Denis. |
|
Sorry, I was not clear enough. My intention is not to change the validation process. But to have a place/area where to continue to get in touch with NMHSs in the implementation of the DAYCLI. Like the files from the US and other coming ones. As Sergio said, may be with another channel ? We may need to talk about that ... Thanks a lot Sergio for all the files that I received. |
|
Within the WIS team we will be restarting the monitoring of the DAYCLI messages and I will be talking with Peer this Friday to cover the cross cutting issues. I'll raise the issue of where the appropriate channel for discussions might be. |
|
Attached are the test files from the Japan Meteorological Agency (JMA). @sergio could you please code and decode them to check if everything is OK? I received also the files from Argentina and Libya but some modifications are needed from their side.DailyCLIMAT_Sample_JMA.zip |
|
2 modifications are needed : |
|
Here is the latest version of the template for the ASCII version of the DAYCLI message (20220308_fic_07630.docx) with the latest version on explanation on the codes (20220224_DAYCLI_USE_CASE.docx). Both are subject to modifications |
|
https://github.com/wmo-im/CCT/wiki/Teleconference-16.3.2022 Yann, the national focal point for France, will submit official request for this change, hopefully before March 22 when the review period ends |
Summary and purpose
Background, history and reasoning for the reporting of daily climate observations
The development of the principal measure of the state of the climate - the global temperature record - has extensively depended on monthly CLIMAT data provided by National Meteorological and Hydrological Services (NMHSs). Over the last 20 years, there has been a growing demand for indices and measures of the climate that also consider extremes (Jones et al., 2012). For many extreme measures, monthly data are insufficient and there is a need for operationally exchanged daily climate data. This need is not just for timeliness, but principally for data that is compatible with long historical daily series developed and made available by NMHSs. Attempts have been made to use SYNOP data for this purpose (e.g. by the European Climate Assessment and Dataset but there are serious issues of incompatibility of SYNOP data with traditional methods of climate measurement within NMHSs (see van den Besselaar et al., 2012). Daily summaries in SYNOP messages are based on measurements that occur between synoptic reporting times and often over a period of less than 24 hours. For instance, in Europe, minimum temperatures are recorded usually over the 18 to 06 UTC 12-hour period and maximum temperatures during the 06 to 18 UTC 12-hour period. Measured in this way, the true daily minimum and maximum temperatures may not be reported because they may have occurred outside those particular 12-hour periods. As a result, SYNOP reports have been shown to significantly underestimate extremes: minimum temperatures measured in this way may be higher than the true daily minimum temperature, and maximum temperatures reported may be lower than the true daily maximum temperature reported as 24-hour climate observation. Similar problems occur for precipitation. In other regions of the world, SYNOP reporting practices can differ but problems remain. The Commission for Basic Systems (CBS) Open Programme Area Group on Integrated Observing Systems (OPAG-IOS), Implementation/Coordination Team on Integrated Observing Systems (ICT-IOS), recommended in 2012 that daily climate observations be included in monthly CLIMAT reports as a means of addressing the gap in the quality of daily climate observations. The U.S. National Oceanic and Atmospheric Administration (NOAA) National Centers for Environmental Information (NCEI), in cooperation with WMO Inter-programme Expert Team on Data Representation Maintenance and Monitoring (IPET-DRMM) and NOAA National Centers for Environmental Prediction (NCEP), developed a BUFR template for transmission of daily climate observations in BUFR format. This template was approved by CBS for implementation in May 2015. It was subsequently tested in the United States, with the cooperation of the UK Met Office. A one-year trial phase for the monthly reporting of daily climate observations was accepted by delegates to the seventeenth session of the Commission for Climatology in April 2018 (see Recommendation 5 (CCl-17)).
Reporting daily climate observations: Technical solution
NOAA/NCEI, in cooperation with IPET-DRMM (taken over by the Inter-programme Expert Team on Codes Maintenance (IPET-CM) in 2016) and NOAA/NCEP, developed a BUFR template, 3 07 074 - Supplemental daily temperature and precipitation values, for daily climate observations in BUFR format, for monthly reporting. Please note that this does not replace the existing CLIMAT BUFR templates but offers complementary reporting of daily observations once per month.
BUFR template 3 07 074 enables NMHSs to provide 31 daily observations consistent with national climate databases for the following elements:
• Time of observation for temperature
• Daily maximum temperature
• Daily minimum temperature
• Daily mean temperature (if it differs from (Tmax+Tmin)/2)
• Time of observation for precipitation
• Total daily precipitation
• Depth of new snowfall
• Depth of total snow on the ground
Each of these observations should be recorded at the observing time consistent with the climate reporting practices of the NMHS and should reflect conditions over the previous 24-hour period. The climate convention varies from country to country; each country should retain its traditional observing practice in reporting daily climate summaries. For example, while in the U.S. the reporting time is local midnight, in Australia it is 9 a.m. local, and in Canada it is 06 UTC. These observations can be efficiently provided via daily CLIMAT reports or other methods specifically designed for climate purposes.
Consultations and Reviewers
References
Jones, P.D., Lister, D.H., Osborn, T.J., Harpham, C., Salmon, M., Morice, C.P., 2012: Hemispheric and large-scale land-surface air temperature variations: An extensive revision and an update to 2010. Journal of Geophysical Research, 117, D05127, doi:10.1029/2011JD017139.
Van den Besselaar, E.J.M., Klein Tank, A.M.G, van der Schrier, G. and Jones, P.D., 2012: Synoptic messages to extend climate data records. Journal of Geophysical Research, 117, D07101, doi:10.1029/2011JD1688.
Detailed proposal
(Final proposal, updated by @jitsukoh December 20.)
1. Add a new entry in the significance qualifier class of table B (0-08-094) and a new code table to describe the method used to calculate the daily average temperature.
CODE TABLE 0-08-094
0-08-094 Method used to calculate the average daily temperature
Note (1) : The letters "a", "b", "c", "d" and "e" generically represent the weight associated with the respective temperature T. The sub-index of T: "1", "2", "3", "x" and "n" represent the values measured at different times or maximum (x) or minimum (n) values.
2. Quality flag for each climatological value (Temperatures, Precipitation and Snow).
Add a new entry (5) of 8-bit indicator of quality control in the Associated field significance (0 31 021).
3. Siting classification and Measurement Quality Classification
Discussion:
It is necessary to add the Siting Classification (SC) and also Measurement Quality Classification (MQC). However, the MQC must be accompanied by the siting classification.
Proposal:
Add 2 entries in table B: One for Temperature and another for Precipitation. Both with 8-bit code tables, where, the first character represents the Siting Classification from "1" to "5", as defined by defined ISO/WMO standard 119289:2014(E) (see the Guide to Instruments and Methods of Observation (WMO-No. 8) edition 2014 Part I, Chapter I, Annex 1B for details), and the second character represent the Measurement Quality Classification from “A” to “D” defined by the Guide to Instruments and Methods of Observation (WMO-No. 8) edition 2020.
CODE TABLE for 0-08-095 and 0-08-096
4. Element names for 0 13 012 (depth of fresh snow) and 0 13 013 (total snow depth)
This change will not be implemented.
Update the element name for 0 13 012 (depth of fresh snow) and 0 13 013 (total snow depth) to add terminology used inWIGOS Metadata.
* CSV file: BUFRCREX_TableB_en_13.csv0 13 012Depth of fresh snowDepth of fresh snow (Depth of snowfall)0 13 013Total snow depthTotal snow depth (Snow depth) (see Note 2)5. The new DAYCLI BUFR message
Create a new descriptor 3-07-075 in the next version of BUFR table D.
(Depth of snowfall)(Snow depth)The text was updated successfully, but these errors were encountered: