NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hello, 2015-03-10 2:57 GMT+03:00 John Caron <caron@xxxxxxxx>: > Hi Antonio: > > Some answers embedded below: > > On Mon, Mar 9, 2015 at 1:43 PM, Antonio Rodriges <antonio.rrz@xxxxxxxxx> > wrote: >> >> Hello, >> >> Thank you for the 4.5 release. >> >> However, there are troubles migrating from 4.3 to 4.5 (I am trying to >> migrate from 4.3 to a higher version for quite long now). >> >> I have used a pre-release snapshot kindly provided at the 17th of >> January but I suppose the following issues may stlll hold for the >> final release. They are mainly concerned with IOSP grib2 backward >> compatibility. >> >> 1) Creation of index files >> version 4.3 does not create in the same directory where original grib2 >> files are located indexes for each date I access. But 4.5 does so for >> each date: >> 2015-02-18/06:09:50.118/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp] >> INFO GribCollectionBuilder write >> >> D:/RS_DATA/worker/symlinks/CFSR/windspeed/wnd10m.gdas.200508.grb2-20050818-180000.ncx2 >> ok=true >> 2015-02-18/06:09:50.120/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp] >> DEBUG createIndex for >> >> D:\RS_DATA\worker\symlinks\CFSR\windspeed\wnd10m.gdas.200508.grb2-20050815-000000.ncx2 >> 2015-02-18/06:09:50.120/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp] >> DEBUG write RecordMaps: bytes = 218 record = 14 bytesPerRecord=15 >> 2015-02-18/06:09:50.121/GMT-00:00 [ucar.nc2.grib.collection.Grib2Iosp] >> DEBUG write GribCollectionIndex= 551 bytes >> >> At the end of traversing the dates from a given interval one arrives >> at a point where the directory has files that are not anticipated to >> emerge (for one previously used 4.3). >> >> Even if index creation time is negligible and it significantly >> accelerates further operation and etc. etc. it is quite logical to >> keep this feature off since it was off or not present in 4.3. > > > If you want to put index files in a sperate directory from the data files, > use following in threddsConfig.xml : > > <GribIndex> > <alwaysUse>true</alwaysUse> > <dir>/tomcat_home/content/thredds/cache/grib/</dir> > </GribIndex> > I do not use THREDDS, I use NetCDF-Java library > see > > http://www.unidata.ucar.edu/software/thredds/v4.5/tds/reference/ThreddsConfigXMLFile.html#GribIndexWriting > > I dont think this has changed since 4.3 (?) And I have 4.3 and tried 4.5 so I can't update > >> >> You may argue that it is not so critical but it depends. The following >> issue is more severe. >> >> 2) Consider grib2 files of CFSR reanalysis, for example: >> http://nomads.ncdc.noaa.gov/data/cfsr/198209/wnd10m.gdas.198209.grb2 >> >> If I would like to read "u-component_of_wind_height_above_ground" in >> version 4.3 it is OK since it finds only a single variable with this >> name: >> >> float u-component_of_wind_height_above_ground(time=745, >> height_above_ground=1, lat=576, lon=1152); >> :long_name = "u-component of wind @ Specified height level above >> ground"; >> :units = "m/s"; >> :missing_value = NaNf; // float >> :abbreviation = "UGRD"; >> :Grib_Variable_Id = "VAR_0-2-2_L103"; >> :Grib2_Parameter = 0, 2, 2; // int >> :Grib2_Parameter_Discipline = "Meteorological products"; >> :Grib2_Parameter_Category = "Momentum"; >> :Grib2_Parameter_Name = "u-component of wind"; >> :Grib2_Level_Type = 103; // int >> :Grib2_Generating_Process_Type = "Forecast"; >> >> However, if I switch to 4.5 the program crashes since 4.5 finds 2 >> variables of the same name and findVariable returns null >> >> float u-component_of_wind_height_above_ground(time=745, >> height_above_ground=1, lat=576, lon=1152); >> :long_name = "u-component of wind @ Specified height level above >> ground"; >> :units = "m/s"; >> :missing_value = NaNf; // float >> :abbreviation = "UGRD"; >> :coordinates = "time height_above_ground "; >> :Grib_Variable_Id = "VAR_0-2-2_L103"; >> :Grib2_Parameter = 0, 2, 2; // int >> :Grib2_Parameter_Discipline = "Meteorological products"; >> :Grib2_Parameter_Category = "Momentum"; >> :Grib2_Parameter_Name = "u-component of wind"; >> :Grib2_Level_Type = "Specified height level above ground"; >> :Grib2_Generating_Process_Type = "Forecast"; > > >> >> float u-component_of_wind_height_above_ground(reftime=124, time=7, >> height_above_ground=1, lat=576, lon=1152); >> :long_name = "u-component of wind @ Specified height level above >> ground"; >> :units = "m/s"; >> :missing_value = NaNf; // float >> :abbreviation = "UGRD"; >> :coordinates = "reftime time height_above_ground "; >> :Grib_Variable_Id = "VAR_0-2-2_L103"; >> :Grib2_Parameter = 0, 2, 2; // int >> :Grib2_Parameter_Discipline = "Meteorological products"; >> :Grib2_Parameter_Category = "Momentum"; >> :Grib2_Parameter_Name = "u-component of wind"; >> :Grib2_Level_Type = "Specified height level above ground"; >> :Grib2_Generating_Process_Type = "Forecast"; >> >> Again, it is all about the backward compatibility. > > > CFSR encoding is notoriously messed up. Its likely there is some glitch like > table version that differs between two records. If you can send me a minimum > set of files that reproduces the problem, i will see if theres a workaround. > > Also, you could consider moving to version 4.6, which will once again write > new CDM index files for GRIB. The advantage is that it scales to large > collections. The disadvantage is that it is currently alpha. But if you are > interested, I can point you to the current snapshot. I have included the link in my previous e-mail, here it is again: http://nomads.ncdc.noaa.gov/data/cfsr/198209/wnd10m.gdas.198209.grb2 and you can find there a lot of other files for other dates > >> >> >> 3) Would it be helpful to maintain a collective repository of files >> from diverse domains and different formats that are read by >> NetCDF-Java users and test all subsequent releases against that >> repository? >> >> It could reveal differences/changes and document them at the early >> stage and avoid things like described (also recall HDF issues like >> http://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg12946.html >> ) > > > yes, that would be great. we have a set of files that we automatically read > in our unit tests, which would have caught this problem. It can be hard to > whittle the files down to a manageable size (a few Mbytes would be nice). Maybe there should be tests embedded in the library itself so everyone could run against their own collection and report the problems?
netcdf-java archives: