NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Wei and Barbara, I've attached a simplified demonstration for the HDF5 issue, wrrds.c, that writes a 75 MB file of floats, then reads them back, one value at a time. It finishes fine when linked to HDF5-1.8.12, but gets an HDF5 error when linked with HDF5-1.8.13. In either case, it takes about 5 minutes to run on my OSX laptop: $ uname -a Darwin milton.local 13.4.0 Darwin Kernel Version 13.4.0: Sun Aug 17 19:50:11 PDT 2014; root:xnu-2422.115.4~1/RELEASE_X86_64 x86_64 $ NCDIR=~/installs/nc432-1812 $ cc -o wrrds-1812 wrrds.c -I$NCDIR/include -L$NCDIR/lib -lnetcdf $ ./wrrds-1812 $ ./wrrds-1812 /tmp/wrrds.nc created /tmp/wrrds.nc written /tmp/wrrds.nc: reading values one-at-a-time successfully read file /tmp/wrrds.nc $ NCDIR=~/installs/nc432-1813 $ cc -o wrrds-1813 wrrds.c -I$NCDIR/include -L$NCDIR/lib -lnetcdf $ ./wrrds-1813 $ ./wrrds-1813 /tmp/wrrds.nc created /tmp/wrrds.nc written /tmp/wrrds.nc: reading values one-at-a-time error: NetCDF: HDF error reading var[0][16][130][1161] The fact that it fails the same way on yellowstone and a Mac laptop means its likely platform independent. --Russ > Here is NCAR's Yellowstone system info: > > uname -a > Linux geyser05 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 > x86_64 x86_64 x86_64 GNU/Linux > > Barbara, and Russ, > > Attached is a C code which I can reproduce this problem on NCAR's > Yellowstone > by read the NCL generated NetCDF file. > > Thank you all for check on this issue. > > Wei > > > > Wei > > > ================================================ > 1850 Table Mesa Dr. > Boulder, CO 80307 > Phone: 303-497-8924 > > On Fri, Sep 26, 2014 at 11:06 AM, <address@hidden> wrote: > > > Hi Wei, > > > > I probably should have figured that out! > > Joe Lee will look at this and we'll see what the next step is. > > I'll bring it up in the HDF5 meeting on Monday, as well. > > > > -Barbara > > > > > > On Fri, 26 Sep 2014, Wei Huang wrote: > > > > Barbara, > >> > >> I am a NCL developer in the NCL team. > >> As I mentioned in my last email, it seems that the problem > >> shows up when we tried to read netCDF4classic file quite some > >> million times. And the same script which worked with HDF5-1.8.12. > >> This read end-up only read each point at time with the way > >> the user coded in NCL. We know it is not an efficient way to read > >> data, but our code should not fail, and so HDF5 should not fail either. > >> > >> If you need, I can FTP the data file to you, and you can try to read this > >> data point by point with HDF5, > >> which may reproduce the problem. > >> If Russ wants to investigate more, I can FTP the data as well. > >> > >> Thanks for looking at this issue. > >> > >> Wei > >> > >> ================================================ > >> 1850 Table Mesa Dr. > >> Boulder, CO 80307 > >> Phone: 303-497-8924 > >> > >> On Fri, Sep 26, 2014 at 7:57 AM, <address@hidden> wrote: > >> > >> Hi Wei, > >>> > >>> I'm not sure we'll be able to determine the issue with your NCL > >>> scripts. However, I did find someone here who has used NCL before. > >>> I will forward your scripts to him, in case he can help. > >>> > >>> He suggested you contact the ncl-talk mailing list, as they may be > >>> familiar with this issue. It looks like you have to subscribe to > >>> ncl-talk. See: > >>> > >>> https://www.ncl.ucar.edu/Support/email_lists.shtml > >>> > >>> In the mean-time, we will see if we can determine anything from the > >>> scripts. > >>> > >>> -Barbara > >>> address@hidden > >>> > >>> > >>> > >>> I have been able to reproduce this issue with a simple NCL scripts. > >>> > >>>> Attached are two NCL scripts, where wrt.ncl created a data file. > >>>> Then read.ncl read a sub-set of the data in a loop. > >>>> > >>>> The read script can demonstrate that HDF5 error start to appear > >>>> loop index 414. (where the first 414 read just normal). > >>>> > >>>> I understand that it seems not fair to send you NCL scripts, > >>>> but that is how we got the problem, and little bit more frustrated > >>>> is that it seems the same code worked fine on my Mac, > >>>> but failed on NCAR's yellowstone. > >>>> > >>>> I hope I can give you more info, but that is what I got so far. > >>>> > >>>> Thanks, > >>>> > >>>> Wei > >>>> > >>>> > >>>> ================================================ > >>>> 1850 Table Mesa Dr. > >>>> Boulder, CO 80307 > >>>> Phone: 303-497-8924 > >>>> > >>>> On Thu, Sep 25, 2014 at 8:16 AM, <address@hidden> wrote: > >>>> > >>>> > >>>> Hi Wei, > >>>>> > >>>>> Thanks for the update. > >>>>> I'll wait to hear from you. > >>>>> > >>>>> -Barbara > >>>>> address@hidden > >>>>> > >>>>> > >>>>> On Wed, 24 Sep 2014, Wei Huang wrote: > >>>>> > >>>>> Barbara, > >>>>> > >>>>> > >>>>>> I am able to repeat this issue with a very short NCL script. > >>>>>> We'll do more debug and will let you know if we still have problem. > >>>>>> > >>>>>> Thanks, > >>>>>> > >>>>>> Wei > >>>>>> > >>>>>> ================================================ > >>>>>> 1850 Table Mesa Dr. > >>>>>> Boulder, CO 80307 > >>>>>> Phone: 303-497-8924 > >>>>>> > >>>>>> On Wed, Sep 24, 2014 at 3:01 PM, Wei Huang <address@hidden> wrote: > >>>>>> > >>>>>> Barbara, > >>>>>> > >>>>>> > >>>>>>> The problem came from a user running NCL. > >>>>>>> We got the data (two netcdf files, total of 35G), and could reproduce > >>>>>>> the > >>>>>>> issue when we compile NCL with HDF5-1.8.13, and NetCDF-4.3.2. > >>>>>>> But there is no problem when we use HDF5-1.8.12, and NetCDF-4.3.2. > >>>>>>> Then we try to reproduce the problem without read those two large > >>>>>>> files, > >>>>>>> but it seems NCL works with HDF5-1.8.13 and NetCDF-4.3.2 with > >>>>>>> simple code. > >>>>>>> > >>>>>>> This kind of put us in a place of no where, which we do not know > >>>>>>> where > >>>>>>> and how to debug this problem. That is why I send the question to > >>>>>>> netcdf-support. > >>>>>>> I do not know if want to send you the data and NCL script to debug > >>>>>>> this issue. If you feel the data is too much, and the problem is from > >>>>>>> NCL, > >>>>>>> could you please check with your developers to see if they have any > >>>>>>> idea? > >>>>>>> > >>>>>>> Thanks, > >>>>>>> > >>>>>>> Wei > >>>>>>> > >>>>>>> ================================================ > >>>>>>> 1850 Table Mesa Dr. > >>>>>>> Boulder, CO 80307 > >>>>>>> Phone: 303-497-8924 > >>>>>>> > >>>>>>> On Wed, Sep 24, 2014 at 12:39 PM, <address@hidden> wrote: > >>>>>>> > >>>>>>> Hi Wei, > >>>>>>> > >>>>>>> > >>>>>>>> Yes, please do send us a way to reproduce the issue. (I don't know > >>>>>>>> what > >>>>>>>> the problem is off-hand.) > >>>>>>>> > >>>>>>>> Thanks! > >>>>>>>> -Barbara > >>>>>>>> > >>>>>>>> ====================== > >>>>>>>> Barbara Jones > >>>>>>>> The HDF Group Helpdesk > >>>>>>>> address@hidden > >>>>>>>> ====================== > >>>>>>>> > >>>>>>>> > >>>>>>>> Hi Wei, > >>>>>>>> > >>>>>>>> > >>>>>>>> I will forward this question to address@hidden, because the > >>>>>>>>> symptoms > >>>>>>>>> are > >>>>>>>>> something that doesn't work with HDF5-1.8.13 but did work with > >>>>>>>>> HDF5-1.8.12. > >>>>>>>>> > >>>>>>>>> They may ask you for a way to duplicate the problem, unless they > >>>>>>>>> can > >>>>>>>>> tell what > >>>>>>>>> the problem is from the HDF5 error messages you have supplied. > >>>>>>>>> > >>>>>>>>> --Russ > >>>>>>>>> > >>>>>>>>> We try to create a netcdf file with write mode: netcdf4classic. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> It works with a test program, but failed with a real case, with > >>>>>>>>>> the > >>>>>>>>>> error > >>>>>>>>>> message below. > >>>>>>>>>> > >>>>>>>>>> We use HDF5-1.8.13 with netcdf4.3.2. > >>>>>>>>>> > >>>>>>>>>> If compiled with HDF5-1.8.12, and netcdf4.3.2, the real case > >>>>>>>>>> worked. > >>>>>>>>>> > >>>>>>>>>> We have checked the real case code few times, but did not find > >>>>>>>>>> anything > >>>>>>>>>> wrong. > >>>>>>>>>> > >>>>>>>>>> Thanks for any help on this issue. > >>>>>>>>>> > >>>>>>>>>> Wei > >>>>>>>>>> > >>>>>>>>>> -------------------------------- > >>>>>>>>>> > >>>>>>>>>> HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0: > >>>>>>>>>> #000: H5P.c line 303 in H5Pcreate(): unable to create property > >>>>>>>>>> list > >>>>>>>>>> major: Property lists > >>>>>>>>>> minor: Unable to create file > >>>>>>>>>> #001: H5Pint.c line 1747 in H5P_create_id(): unable to atomize > >>>>>>>>>> property > >>>>>>>>>> list > >>>>>>>>>> major: Property lists > >>>>>>>>>> minor: Unable to register new atom > >>>>>>>>>> #002: H5I.c line 895 in H5I_register(): can't insert ID node into > >>>>>>>>>> skip > >>>>>>>>>> list > >>>>>>>>>> major: Object atom > >>>>>>>>>> minor: Unable to insert object > >>>>>>>>>> #003: H5SL.c line 995 in H5SL_insert(): can't create new skip list > >>>>>>>>>> node > >>>>>>>>>> major: Skip Lists > >>>>>>>>>> minor: Unable to insert object > >>>>>>>>>> #004: H5SL.c line 687 in H5SL_insert_common(): can't insert > >>>>>>>>>> duplicate > >>>>>>>>>> key > >>>>>>>>>> major: Skip Lists > >>>>>>>>>> minor: Unable to insert object > >>>>>>>>>> > >>>>>>>>>> ================================================ > >>>>>>>>>> 1850 Table Mesa Dr. > >>>>>>>>>> Boulder, CO 80307 > >>>>>>>>>> Phone: 303-497-8924 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Russ Rew UCAR Unidata > >>>>>>>>>> Program > >>>>>>>>>> > >>>>>>>>>> address@hidden > >>>>>>>>> http://www.unidata.ucar.edu > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Ticket Details > >>>>>>>>> =================== > >>>>>>>>> Ticket ID: MRA-585540 > >>>>>>>>> Department: Support netCDF > >>>>>>>>> Priority: Normal > >>>>>>>>> Status: Closed > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>> > >>>>> > >>>>> > >>>>> > >>>> -- > >>> > >>> > >> > > -- > > > > Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu Ticket Details =================== Ticket ID: MRA-585540 Department: Support netCDF Priority: Normal Status: Closed
Attachment:
wrrds.c
Description: Binary data