NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTE: The cf-satellite mailing list is no longer active. The list archives are made available for historical reasons.
Jim, > I'm confused about a few things. If you don't mind spending a bit of > time clarifying, I'd appreciate it. Oops, sorry, it looks like I'm the one who is confused! > Here are my confused questions. > > * Isn't the _FillValue attribute the one that maps to '_'? (See the > discussion of _FillValue and missing_value here > <http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/Attribute-Conv > entions.html>.) Yes, you're right of course. Sorry for mixing up "_FillValue" and "missing_value". The latter was at one time deprecated, but now that deprecation has been removed from the latest CF conventions standard. > * Is the flag_values attribute considered applicable in the case of > floating point variables? That was my understanding. CF 1.5 says: ... The flag_values attribute is the same type as the variable to which it is attached, and contains a list of the possible flag values. I can't see anywhere that restricts flag_values to integers. Of course the "flag_masks" attribute wouldn't have much meaning for floating-point values, but I wasn't suggesting using "flag_masks". > * In what sense is this suggestion making a change to the > interpretation of the missing_value attribute? If "this suggestion" means extending missing_value to permit multiple values, then the only change it makes is changing a one-to-one mapping into a one-to-many mapping. Since my example of breaking ncdump/ncgen backward compatibility doesn't apply to the "missing_value" attribute, that may be harmless, unless there is other existing software that depends on "missing_value" having a single value. So I withdraw my confused objection to using "missing_value". However, I still think "flag_values" would also work for this purpose, especially if there were other reasons for special values than that they indicated a special type of missing value. Also "missing_values" might be more descriptive than "missing_value" for a multiple-value attribute, following the example of "flag_values", but I'm also in favor of keeping the additions of new CF attributes to a minimum, which argues for just using "missing_value". --Russ _____________________________________________________________________ Russ Rew UCAR Unidata Program russ@xxxxxxxxxxxxxxxx http://www.unidata.ucar.edu > Grace and peace, > > Jim > > On 7/15/2011 3:09 PM, Russ Rew wrote: > > Hi Jim, > > > >> In the ESIP Federation CF-Satellite session yesterday (July 14, 2011), I > >> suggested that it would be good to have a standard scheme for handling > >> multiple marker values in satellite data that indicate missing data. > >> There is confusion in the current documentation (between netCDF and CF) > >> about the proper use of the missing_value attribute, in particular about > >> whether or not it is acceptable for it to be a vector attribute. I > >> think there is good reason to allow it to be a vector attribute, and > >> that allowing the attribute to be a vector quantity is useful beyond the > >> satellite / remote sensing arena. The problem that arises with having a > >> vector missing_value attribute is the assignment of meanings to the > >> different values. > >> > >> One way to deal with this is to have a missing_value_meanings > >> attribute. The missing_value_meanings attribute would be formed in the > >> same way as the flag_meanings attribute. There would be one string of > >> non-whitespace characters per element of the missing_value attribute, > >> separated by spaces. Each string would be a human-readable phrase > >> (delineating the words of the phrase using underscores or camel-casing) > >> that explained the meaning associated with the corresponding > >> missing_value element. Here's an example fragment of cdl. > >> > >> float foo(points); > >> > >> foo:missing_value = -999.1, -999.2; > >> foo:missing_value_meanings = "InputsOutOfRange > >> SolutionDidNotConverge"; > > I agree that it would be very useful to have an attribute for multiple > > special values for satellite data as well as other kinds of data. IIRC, > > you gave an example of 10 different kinds of missing values for an NPP > > sensor. > > > > I'm a little concerned about breaking backward compatibility by > > extending the "missing_value" attribute for this purpose, because there > > may be software that depends on a one-to-one mapping between a > > missing_value for a variable and some alternate representation for it, > > such as the '_' notation ncdump already uses. The fact that the ncdump > > and ncgen utilities are inverses of each other, converting netCDF data > > to CDL and back to netCDF without changing the data, requires a single > > value for the missing value to avoid loss of information when converting > > from '_' back to the corresponding value. > > > > I suggest just using the existing CF flag conventions for this purpose > > instead, using the variable attributes "flag_values", and > > "flag_meanings" for any number of special values for a variable. > > > > If one of those values is a missing value, the associated flag_meaning > > could just be "missing_value", but flag_meanings can also specify an > > arbitrary number of other nuanced kinds of missing values or various > > other kinds of special values. > > > > http://cfconventions.org/documents/cf-conventions/1.5/cf-conventions.htm > l#flags > > > > --Russ > > _____________________________________________________________________ > > > > Russ Rew UCAR Unidata Program > > russ@xxxxxxxxxxxxxxxx http://www.unidata.ucar.edu > > -- > Jim Biard > > Government Contractor, STG Inc. > Remote Sensing and Applications Division (RSAD) > National Climatic Data Center > 151 Patton Ave. > Asheville, NC 28801-5001 > > jim.biard@xxxxxxxx > 828-271-4900 > > > --Boundary_(ID_xMNVEAaJBvpOih/JfomxKw) > Content-type: text/html; charset=ISO-8859-1 > Content-transfer-encoding: 7BIT > > <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> > <html> > <head> > <meta content="text/html; charset=ISO-8859-1" > http-equiv="Content-Type"> > </head> > <body text="#000000" bgcolor="#ffffff"> > Russ,<br> > <br> > I'm confused about a few things. If you don't mind spending a bit > of time clarifying, I'd appreciate it.<br> > <br> > Here are my confused questions.<br> > <ul> > <li>Isn't the _FillValue attribute the one that maps to '_'? (See > the discussion of _FillValue and missing_value <a > href="http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/Attribute-Conve > ntions.html">here</a>.)<br> > </li> > <li>Is the flag_values attribute considered applicable in the case > of floating point variables?</li> > <li>In what sense is this suggestion making a change to the > interpretation of the missing_value attribute?</li> > </ul> > Grace and peace,<br> > <br> > Jim<br> > <br> > On 7/15/2011 3:09 PM, Russ Rew wrote: > <blockquote cite="mid:10685.1310756965@xxxxxxxxxxxxxxxx" type="cite"> > <pre wrap="">Hi Jim, > > </pre> > <blockquote type="cite"> > <pre wrap="">In the ESIP Federation CF-Satellite session yesterday (J > uly 14, 2011), I > suggested that it would be good to have a standard scheme for handling > multiple marker values in satellite data that indicate missing data. > There is confusion in the current documentation (between netCDF and CF) > about the proper use of the missing_value attribute, in particular about > whether or not it is acceptable for it to be a vector attribute. I > think there is good reason to allow it to be a vector attribute, and > that allowing the attribute to be a vector quantity is useful beyond the > satellite / remote sensing arena. The problem that arises with having a > vector missing_value attribute is the assignment of meanings to the > different values. > > One way to deal with this is to have a missing_value_meanings > attribute. The missing_value_meanings attribute would be formed in the > same way as the flag_meanings attribute. There would be one string of > non-whitespace characters per element of the missing_value attribute, > separated by spaces. Each string would be a human-readable phrase > (delineating the words of the phrase using underscores or camel-casing) > that explained the meaning associated with the corresponding > missing_value element. Here's an example fragment of cdl. > > float foo(points); > > foo:missing_value = -999.1, -999.2; > foo:missing_value_meanings = "InputsOutOfRange > SolutionDidNotConverge"; > </pre> > </blockquote> > <pre wrap=""> > I agree that it would be very useful to have an attribute for multiple > special values for satellite data as well as other kinds of data. IIRC, > you gave an example of 10 different kinds of missing values for an NPP > sensor. > > I'm a little concerned about breaking backward compatibility by > extending the "missing_value" attribute for this purpose, because there > may be software that depends on a one-to-one mapping between a > missing_value for a variable and some alternate representation for it, > such as the '_' notation ncdump already uses. The fact that the ncdump > and ncgen utilities are inverses of each other, converting netCDF data > to CDL and back to netCDF without changing the data, requires a single > value for the missing value to avoid loss of information when converting > from '_' back to the corresponding value. > > I suggest just using the existing CF flag conventions for this purpose > instead, using the variable attributes "flag_values", and > "flag_meanings" for any number of special values for a variable. > > If one of those values is a missing value, the associated flag_meaning > could just be "missing_value", but flag_meanings can also specify an > arbitrary number of other nuanced kinds of missing values or various > other kinds of special values. > > <a class="moz-txt-link-freetext" href="http://cfconventions.org/documents/c > f-conventions/1.5/cf-conventions.html#flags">http://cfconventions.org/documen > ts/cf-conventions/1.5/cf-conventions.html#flags</a> > > --Russ > _____________________________________________________________________ > > Russ Rew UCAR Unidata Program > <a class="moz-txt-link-abbreviated" href="mailto:russ@xxxxxxxxxxxxxxxx";>russ@ > unidata.ucar.edu</a> <a class="moz-txt-link-freetext" hre > f="http://www.unidata.ucar.edu";>http://www.unidata.ucar.edu</a> > </pre> > </blockquote> > <br> > <pre class="moz-signature" cols="72">-- > Jim Biard > > Government Contractor, STG Inc. > Remote Sensing and Applications Division (RSAD) > National Climatic Data Center > 151 Patton Ave. > Asheville, NC 28801-5001 > > <a class="moz-txt-link-abbreviated" href="mailto:jim.biard@xxxxxxxx";>jim.biar > d@xxxxxxxx</a> > 828-271-4900 > </pre> > </body> > </html> > > --Boundary_(ID_xMNVEAaJBvpOih/JfomxKw)--
cf-satellite archives: