NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
> Thanks for the response, John. Perhaps you or Russ could talk to the
> NCL people and convince them to use missing_value instead of _FillValue
> as their preferred attribute for missing data since NCL can read
> multiple formats (including GrADS binary) and might run into the same
> issues.
>
> Don
>
> On 7/29/11 2:55 PM, Unidata netCDF Java Support wrote:
> > Hi Don:
> >
> > I think Russ and I have talked about this and decided that _FillValue shoul
> d match the bit pattern exactly (for portability?). valid_range, valid_min, v
> alid_max, and missing value are intended to match data values and so should b
> e allowed some slop. Im cc'ing Russ in case he remembers different.
The way ncdump handles it (which seems to differ from the documented
requirement for exact bit-for-bit equality), is to treat any floating
point value that's within one machine epsilon of the _FillValue (that is
one bit in the least significant bit) as equal to _FillValue, and
display it as "_". The code for this is in ncdump/dumplib.c, in the
ncfloat_val_equals() and corresponding double precision functions:
/*
* Return ( *(float* )v1p == *(float* )v2p);
* except use floating epsilon to compare very close vals as equal
* and handle IEEE NaNs and infinities.
*/
boolean
ncfloat_val_equals(const nctype_t *this,
const void *v1p, const void *v2p) {
float v1 = *(float* )v1p;
float v2 = *(float* )v2p;
if((v1 > 0.0f) != (v2 > 0.0f)) /* avoid overflow */
return false;
if(isfinite(v1) && isfinite(v2))
return (absval(v1 - v2) <= absval(float_eps * v2)) ;
if(isnan(v1) && isnan(v2))
return true;
if(isinf(v1) && isinf(v2))
return true;
return false;
}
It looks like I should fix the documentation to match the behavior. I
think it was necessary to add the 1-bit fuzz to make the ncdump tests
pass on every tested platform, but it may have been an artifact of bad
compiler code that's fixed now. I may try taking it out to see if all
the ncdump and ncgen tests pass without it.
> >
> > My own opinion is that the trouble stems from overloading _FillValue with m
> ultiple meanings, plus the lack of clarity to tell people what they should do
> . We should probably make another pass at
> >
> > http://www.unidata.ucar.edu/software/netcdf/docs/BestPractices.html
Yup.
--Russ
> > John
> >
> >> John-
> >>
> >> In the GrADS IOSP, I added a _FillValue attribute for the GrADS Data
> >> Descriptor File "undef" value:
> >>
> >> http://www.iges.org/grads/gadoc/descriptorfile.html#UNDEF
> >>
> >> I used _FillValue instead of missing_value because it seemed more
> >> accepted in the CF conventions until the recent change to undeprecate
> >> missing_value. Also, NCL prefers _FillValue over missing_value and I
> >> was working with NCL at the time I wrote the IOSP.
> >>
> >> I came across a file today where the undef was set to 9.99E8, but the
> >> "missing" values in the file ranged from 9.9899994E8 to 9.99E8 (We're
> >> looking into why it is inconsistent - the file is written by GrADS).
> >> Talking to one of the developers, they always use 32-bit values (floats)
> >> for the data when working with GrADS and I'm using float for the
> >> variable type.
> >>
> >> The logic in EnhanceScaleMissingImpl is different for _FillValue vs.
> >> missing_value. isMissingValue uses ucar.nc2.util.Misc.closeEnough
> >> whereas isFillValue uses straight equality.
> >>
> >> I can change the IOSP to have both _FillValue and missing_value
> >> attributes, or change _FillValue to missing_value and I get the results
> >> I expect. However, it seems like there may be some slop even in fill
> >> values, so perhaps isFillValue should use Misc.closeEnough instead of
> >> straight equality which also solves this problem. That seems more
> >> prudent for the typically large values that are used as _FillValues.
> >>
> >> Thoughts?
> >>
> >> Don
> >> --
> >> Don Murray
> >> NOAA/ESRL/PSD and CIRES
> >> 303-497-3596
> >> http://www.esrl.noaa.gov/psd/people/don.murray/
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: KJR-472449
> > Department: Support netCDF Java
> > Priority: Critical
> > Status: Open
> >
>
> --
> Don Murray
> NOAA/ESRL/PSD and CIRES
> 303-497-3596
> http://www.esrl.noaa.gov/psd/people/don.murray/