Quality codes and QARTOD flags

[MathewBiddle]

There was a robust discussion and consensus on how to codify these. I need to digest and see what should be changed.

https://github.com/MathewBiddle/ATN-archive/blob/b738cbcf3836f50a3b518790662e376b7e3ad0d1/data/deployment_km_greyseal_v1.ncdump#L58-L67

https://github.com/MathewBiddle/ATN-archive/blob/b738cbcf3836f50a3b518790662e376b7e3ad0d1/data/deployment_km_greyseal_v1.ncdump#L185-L212

I think we need to bring in the the variable attribute ancillary_variables into some of these. See CF-conventions on Ancillary Data.

[MathewBiddle]

QARTOD variables

Here are my thoughts on what the standard_name for the qartod flag variables should be:

qartod_rollup_flag:standard_name = "aggregate_quality_flag" ;

qartod_speed_flag:standard_name = "gross_range_test_quality_flag" ;

qartod_location_flag:standard_name = "location_test_quality_flag" ;

qartod_time_flag:standard_name = "gross_range_test_quality_flag" ;

It's also recommended to have a references attribute for each of these variables.

This should be a URL to a resource that describes the test configuration, parameters used, etc, if such a resource is available. The global references attribute can also be used to describe QC methods in general.

location_class

The location class is currently defined as: location_class:standard_name = "quality_code" ;

However, quality_code does not exist in the CF standard name list. I recommend adjusting to status_flag as it's definition seems more applicable here:

A variable with the standard name of status_flag contains an indication of quality or other status of another data variable. This may include the status of the instrument producing the data as well as data quality information. The linkage between the data variable and the variable with the standard_name of status_flag is achieved using the ancillary_variables attribute. A variable which contains purely quality information may use the standard name of quality_flag to provide an assessed quality of the corresponding data.

It could also be quality_flag which is defined as:

A variable with the standard name of quality_flag contains an indication of assessed quality information of another data variable

Ancillary variables:

From CF:

When one data variable provides metadata about the individual values of another data variable it may be desirable to express this association by providing a link between the variables. For example, instrument data may have associated measures of uncertainty. The attribute ancillary_variables is used to express these types of relationships.

time should have the following attribute:
time:ancillary_variables = "qartod_time_flag qartod_rollup_flag" ;

lat, lon, and location_class should have the following attribute:
lat:ancillary_variables = "qartod_location_flag qartod_rollup_flag" ;
lon:ancillary_variables = "qartod_location_flag qartod_rollup_flag" ;
location_class:ancillary_variables = "qartod_location_flag qartod_rollup_flag" ;

I'm not sure what the qartod_speed_flag would be an ancillary variable to. Maybe time, lat, and lon?

I'm also wondering about the following variables and if they should be listed as ancillary_variables to lat and lon:

• error_radius
• semi_major_axis
• semi_minor_axis
• ellipse_orientation
• offset
• offset_orientation

[MathewBiddle]

From @iamchrisser:

• This is done. I think we talked about this last time.

Standard name additions

https://github.com/MathewBiddle/ATN-archive/blob/64bde94f37b4251696704ded97ecd80b766d7da4/templates/atn_trajectory_template_ncdump.txt#L65

https://github.com/MathewBiddle/ATN-archive/blob/64bde94f37b4251696704ded97ecd80b766d7da4/templates/atn_trajectory_template_ncdump.txt#L187

https://github.com/MathewBiddle/ATN-archive/blob/64bde94f37b4251696704ded97ecd80b766d7da4/templates/atn_trajectory_template_ncdump.txt#L194

https://github.com/MathewBiddle/ATN-archive/blob/64bde94f37b4251696704ded97ecd80b766d7da4/templates/atn_trajectory_template_ncdump.txt#L201

https://github.com/MathewBiddle/ATN-archive/blob/64bde94f37b4251696704ded97ecd80b766d7da4/templates/atn_trajectory_template_ncdump.txt#L208

Ancillary Variables:

https://github.com/MathewBiddle/ATN-archive/blob/64bde94f37b4251696704ded97ecd80b766d7da4/templates/atn_trajectory_template_ncdump.txt#L20

https://github.com/MathewBiddle/ATN-archive/blob/64bde94f37b4251696704ded97ecd80b766d7da4/templates/atn_trajectory_template_ncdump.txt#L46

https://github.com/MathewBiddle/ATN-archive/blob/64bde94f37b4251696704ded97ecd80b766d7da4/templates/atn_trajectory_template_ncdump.txt#L61

Questions

1. What about location_class:ancillary_variables? Does that not need to be included?

[iamchrisser]

Yes? So we want location_class:ancillary_variables = "lat lon". Anything missing?

[MathewBiddle]

I thought I commented on this already, but maybe I didn't hit the comment button. Sorry if these notes are somewhere else.

I think that's the last of the ancillary_variables attributes to add.

Within the qartod variables, we should have a references attribute which points to the appropriate QARTOD manual (or check if we can do that). url's are sufficient for that attribute.

[iamchrisser]

I think the most recent update addresses all of these.

[MathewBiddle]

Nice! Thanks for putting the references in there!

ioos-atn-data/data/atn_trajectory_template_ncdump_20211006.txt

Lines 186 to 213 in 2dcafb8

	int64 qartod_rollup_flag(obs) ;
	qartod_rollup_flag:_FillValue = 0LL ;
	qartod_rollup_flag:flag_meanings = "PASS NOT_EVALUATED SUSPECT FAIL MISSING" ;
	qartod_rollup_flag:flag_values = "1, 2, 3, 4, 9" ;
	qartod_rollup_flag:standard_name = "aggregate_quality_flag" ;
	qartod_rollup_flag:references = "" ;
	qartod_rollup_flag:long_name = "Aggregate QC value" ;
	int64 qartod_speed_flag(obs) ;
	qartod_speed_flag:_FillValue = 0LL ;
	qartod_speed_flag:flag_meanings = "PASS NOT_EVALUATED SUSPECT FAIL MISSING" ;
	qartod_speed_flag:flag_values = "1, 2, 3, 4, 9" ;
	qartod_speed_flag:standard_name = "gross_range_test_quality_flag" ;
	qartod_speed_flag:references = "https://cdn.ioos.noaa.gov/media/2020/03/QARTOD_TS_Manual_Update2_200324_final.pdf" ;
	qartod_speed_flag:long_name = "Speed QC test- gross range test" ;
	int64 qartod_location_flag(obs) ;
	qartod_location_flag:_FillValue = 0LL ;
	qartod_location_flag:flag_meanings = "PASS NOT_EVALUATED SUSPECT FAIL MISSING" ;
	qartod_location_flag:flag_values = "1, 2, 3, 4, 9" ;
	qartod_location_flag:standard_name = "location_test_quality_flag" ;
	qartod_location_flag:references = "https://cdn.ioos.noaa.gov/media/2020/03/QARTOD_TS_Manual_Update2_200324_final.pdf" ;
	qartod_location_flag:long_name = "Location QC test - Location test" ;
	int64 qartod_time_flag(obs) ;
	qartod_time_flag:_FillValue = 0LL ;
	qartod_time_flag:flag_meanings = "PASS NOT_EVALUATED SUSPECT FAIL MISSING" ;
	qartod_time_flag:flag_values = "1, 2, 3, 4, 9" ;
	qartod_time_flag:standard_name = "gross_range_test_quality_flag" ;
	qartod_time_flag:references = "https://cdn.ioos.noaa.gov/media/2020/03/QARTOD_TS_Manual_Update2_200324_final.pdf" ;
	qartod_time_flag:long_name = "Time QC test- gross range test" ;