NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Interesting -- I've lost track a bit, but I thought that one of my colleagues had gotten a similar error with NCO. I think what may be going on here is that NCO made a copy by creating a new file, and copying everything over -- I haven't tried that with Python (or any way) yet, but maybe it doesn't check the _FillValue attribute if you set it after creating the variable? -CHB On Wed, Oct 2, 2024 at 4:46 PM Dave Allured - NOAA Affiliate via netcdfgroup <netcdfgroup@xxxxxxxxxxxxxxxx> wrote: > Chris, thanks for filing the issue with netcdf-C. > > Now this just got weirder. I wondered if NCO Utilities might be able to > copy that file, and somehow sanitize the offending _FillValue attributes. > Either fail like nccopy did, or fix the attribute data type, or something . > "ncks infile.nc outfile.nc." Well it replicated the input file with no > complaints. However, IT PRESERVED THE _FILLVALUE TYPE MISMATCHES. This > with modern NCO and library versions: > > ncks -r: ncks version 5.2.1 > ncdump -hs: :_NCProperties = "version=2,netcdf=4.9.2,hdf5=1.14.3" > ; > :_SuperblockVersion = 2 ; > > I thought this was no longer possible with the current netcdf-C. This > blows away my assertion that an older netcdf-C version was responsible. > How did NCO do that? Don't know. I am giving up for today. Perhaps an > NCO or netcdf developer can shed more light. > > If this is pursued any deeper, someone will need to isolate netcdf-C > library behavior from surrounding software, including the netcdf-python > library, and/or NCO code. > > > On Wed, Oct 2, 2024 at 5:14 PM Chris Barker via netcdfgroup < > netcdfgroup@xxxxxxxxxxxxxxxx> wrote: > >> OK -- issue posted: >> >> https://github.com/Unidata/netcdf-c/issues/3038 >> >> I only did the C lib, 'cause it's showing up in C and python -- is >> Fortran a different code base? or does it use the same C lib under hood? >> >> As for how high priority it is -- that's not up to me -- not a high >> enough priority for me to write a PR :-) >> >> Thanks, >> >> -CHB >> >> >> On Wed, Oct 2, 2024 at 4:02 PM Roy Mendelssohn - NOAA Federal via >> netcdfgroup <netcdfgroup@xxxxxxxxxxxxxxxx> wrote: >> >>> Actually now that I think of it, since you can read the file a better >>> solution is to just create a new file with the proper attributes and then >>> item by item copy the values ( i just checked that at least in Python you >>> can read the values and they appear to be properly masked). A pain, but >>> once written should work for any similar fils from cmem. >>> >>> -Roy >>> >>> > On Oct 2, 2024, at 3:52 PM, Roy Mendelssohn - NOAA Federal < >>> roy.mendelssohn@xxxxxxxx> wrote: >>> > >>> > To edit right now I think a hex editor will work. Even in a good text >>> editor FillValue shows up, it is just not clear how to reset it without >>> seeing the hex.. But a hex editor should be able to get at it. >>> > >>> > -Roy >>> > >>> > >>> >> On Oct 2, 2024, at 3:31 PM, Dave Allured - NOAA Affiliate via >>> netcdfgroup <netcdfgroup@xxxxxxxxxxxxxxxx> wrote: >>> >> >>> >> Yes in my opinion, this is worth issues in both the netcdf-c and >>> netcdf-python libraries, but at low priority. I think this is a relatively >>> obscure case. The hope is that if fixed in the libraries, it will avoid >>> more debugging efforts by future users. >>> >> >>> >> >>> >> On Wed, Oct 2, 2024 at 3:10 PM Chris Barker <chris.barker@xxxxxxxx> >>> wrote: >>> >> On Wed, Oct 2, 2024 at 1:44 PM Dave Allured - NOAA Affiliate via >>> netcdfgroup <netcdfgroup@xxxxxxxxxxxxxxxx> wrote: >>> >> This file was probably written by a pre-2013 version of the netcdf-C >>> library, >>> >> >>> >> Ahh, that may explain it. >>> >> >>> >> Modern libraries choke on the type mismatch, as seen here. >>> >> ... >>> >> However IMO, libraries should also support problem-free writes to >>> such existing files. >>> >> >>> >> Is this worth an issue on the netCDF C lib? >>> >> >>> >> -CHB >>> >> >>> >> >>> >> It may be possible to "fix" files like this, by replacing or deleting >>> the offending _FillValue attributes with lower level or capable software, >>> such as the HDF5 library. >>> >> >>> >> >>> >> On Wed, Oct 2, 2024 at 1:58 PM Chris Barker via netcdfgroup < >>> netcdfgroup@xxxxxxxxxxxxxxxx> wrote: >>> >> I was wondering what the NaNf meant :-) >>> >> >>> >> But yeah, that's probably it -- now to track down whoever wrote this >>> file -- >>> >> But I do wonder what software allowed this? >>> >> >>> >> -CHB >>> >> >>> >> >>> >> On Wed, Oct 2, 2024 at 12:53 PM Benjamin Root <ben.v.root@xxxxxxxxx> >>> wrote: >>> >> I think the issue is that the _FillValue of "NaN" is probably a >>> float64 while the variable's dtype is float32, which is probably what it >>> means by a type mismatch. When I have a NaN FillValue for a float32 >>> variable, it prints "NaNf" >>> >> >>> >> ``` >>> >> float sum_tp_mm(time, latitude, longitude) ; >>> >> sum_tp_mm:_FillValue = NaNf ; >>> >> ``` >>> >> >>> >> On Wed, Oct 2, 2024 at 12:05 PM Chris Barker via netcdfgroup < >>> netcdfgroup@xxxxxxxxxxxxxxxx> wrote: >>> >> Thanks Roy, at least it's not just us, and that error did give me a >>> hint, look at this from ncdump-ing the header: >>> >> >>> >> float uo(time, depth, latitude, longitude) ; >>> >> string uo:units = "m s-1" ; >>> >> uo:_FillValue = NaN ; >>> >> string uo:standard_name = "eastward_sea_water_velocity" ; >>> >> string uo:long_name = "Eastward velocity" ; >>> >> >>> >> There's no type for the uo:_FillValue attribute. >>> >> >>> >> How in the world did that happen? >>> >> >>> >> At least now I have a specific thing to point the file creators to. >>> >> >>> >> -CHB >>> >> >>> >> >>> >> On Wed, Oct 2, 2024 at 10:58 AM Roy Mendelssohn - NOAA Federal < >>> roy.mendelssohn@xxxxxxxx> wrote: >>> >> I can reproduce the error. I tried to copy the file using mccoy, >>> got the following: >>> >> >>> >> nccopy cmems_mod_nws_phy_anfc_0.027deg-3D_P1D-m_1727700325546.nc >>> test.nc >>> >> NetCDF: Not a valid data type or _FillValue type mismatch >>> >> Location: file ?; fcn ? line 1408 >>> >> >>> >> hope this helps. >>> >> >>> >> -Roy >>> >> >>> >> >>> >>> On Oct 2, 2024, at 10:46 AM, Chris Barker via netcdfgroup < >>> netcdfgroup@xxxxxxxxxxxxxxxx> wrote: >>> >>> >>> >>> I have a user unable to use a netcdf file with our code. >>> >>> >>> >>> It turns out that the file works just fine when read, but if you try >>> to open it for writing, it chokes: >>> >>> >>> >>> (this is the netCDF4 Python lib, version 1.6.5, which seems to be >>> the latest release) >>> >>> >>> >>> I get: >>> >>> In [3]: netCDF4.Dataset(' >>> cmems_mod_nws_phy_anfc_0.027deg-3D_P1D-m_1727700325546.nc', 'r+') >>> >>> >>> --------------------------------------------------------------------------- >>> >>> OSError Traceback (most recent >>> call last) >>> >>> Cell In[3], line 1 >>> >>> ----> 1 netCDF4.Dataset(' >>> cmems_mod_nws_phy_anfc_0.027deg-3D_P1D-m_1727700325546.nc', 'r+') >>> >>> >>> >>> File src/netCDF4/_netCDF4.pyx:2469, in >>> netCDF4._netCDF4.Dataset.__init__() >>> >>> >>> >>> File src/netCDF4/_netCDF4.pyx:2028, in >>> netCDF4._netCDF4._ensure_nc_success() >>> >>> >>> >>> OSError: [Errno -103] NetCDF: Can't write file: ' >>> cmems_mod_nws_phy_anfc_0.027deg-3D_P1D-m_1727700325546.nc' >>> >>> >>> >>> Any idea how I can get more info as to what's wrong with it? >>> >>> >>> >>> Again, we have had no problems reading the file. >>> >>> >>> >>> Thanks, >>> >>> - CHB >>> >>> >>> >>> File is 1.2MB, (not huge by today's standards?) so I've attached it >>> here if anyone wants to take a look. >>> >>> -- >>> >>> Christopher Barker, Ph.D. >>> >>> Oceanographer >>> >>> >>> >>> Emergency Response Division >>> >>> NOAA/NOS/OR&R (206) 526-6959 voice >>> >>> 7600 Sand Point Way NE (206) 526-6329 fax >>> >>> Seattle, WA 98115 (206) 526-6317 main reception >>> >>> >>> >>> Chris.Barker@xxxxxxxx >>> >> >> _______________________________________________ > NOTE: All exchanges posted to Unidata maintained email lists are > recorded in the Unidata inquiry tracking system and made publicly > available through the web. Users who post to any of the lists we > maintain are reminded to remove any personal information that they > do not want to be made public. > > > netcdfgroup mailing list > netcdfgroup@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > https://www.unidata.ucar.edu/mailing_lists/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@xxxxxxxx
netcdfgroup archives: