NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
The netCDF Operators NCO version 4.4.6 are ready. http://nco.sf.net (Homepage) http://dust.ess.uci.edu/nco (Homepage "mirror") This is a bugfix, stability, and performance release. It fixes an ncra bug that affects packed records with missing values, and it fixes an ncwa bug that affects minimization and maximization operations on packed variables. Affected users would likely have noticed being bit by these bugs since the answers generated were, well, noticeable. For these reasons alone, we recommend that users of packed datasets upgrade to 4.4.6. We rewrote ncrename to enhance its group-oriented features. We also implemented a performance optimization that dramatically improves access times to strided hyperslabs in netCDF4 datasets. Minor improvements include new ncks printing and hyperslab options, support for HDF4 chunking/compression, and ncap2 argument passing. Work on NCO 4.4.7 is underway, still focused on stability and speed. This includes more netCDF4 mop-up and new chunking features. Enjoy, Charlie NEW FEATURES (full details always in ChangeLog): A. ncrename expanded functionality on group-oriented files: ncrename now supports group attributes and variable attributes with the same syntax, provides distinct options to change only global metatdata or metadata in every group, and works with multiple instances of the same object, e.g., multiple instances of the same variable name in different groups. ncrename -a g1@old_att,new_att in.nc # Rename in g1 only ncrename -a group@old_att,new_att in.nc # Rename in all groups ncrename -a global@old_att,new_att in.nc # Rename in root only http://nco.sf.net/nco.html#ncrename B. Optimization of strided hyperslab access to netCDF4 data: Multiple users complain that access to strided hyperslabs of netCDF4 datasets is orders of magnitude slower than expected. This occurs with other languages (like NCL) and appears to be caused by lower-level libraries. Until the problem is solved at the root, we have developed and implemented a transparent workaround that fixes the issue for the common case where strides are taken in only one dimension, e.g., -d time,0,,12. Strided hyperslabs like ncks -O -d time,0,,12 ~/in.nc4 ~/foo.nc now complete in nearly the same amount of time as non-strided. This workaround works transparently with any version of netCDF. http://nco.sf.net#bug_nc4_srd C. ncap2 command-line and script arguments can now both be specified AND executed. Command-line args are executed before script: ncap2 -v -s "lon=${longitude}" -S in_4.nc out.nc This is useful for passing shell variables into NCO scripts. Pass the shell variables with -s and the -S scripts will use them. Thanks to Henry Butowsky for this. http://nco.sf.net/nco.html#ncap2 D. ncks -V = --var_val is new switch to print only variable values. Compare and contrast -Q, -V, and --no_nm_prn: % ncks -v three_dmn_rec_var -C -H in.nc time[0]=1 lat[0]=-90 lon[0]=0 three_dmn_rec_var[0]=1 % ncks --no_nm_prn -v three_dmn_rec_var -C -H in.nc 1 -90 0 1 % ncks -Q -v three_dmn_rec_var -C -H in.nc three_dmn_rec_var[0]=1 % ncks -Q --no_nm_prn -v three_dmn_rec_var -C -H in.nc 1 % ncks -V -v three_dmn_rec_var -C -H in.nc 1 http://nco.sf.net/nco.html#var_val E. ncks --cdl --hdn now prints extended file format (i.e., the format of the file or server supplying the data)as _SOURCE_FORMAT: % ncks --hdn --cdl ~/nco/data/cmip5.nc ... :_Format = "netCDF-4" ; :_SOURCE_FORMAT = "NC_FORMAT_HDF5" ; ... http://nco.sf.net/nco.html#hdn F. Hyperslabs specifications now accept -0 (negative zero) to indicate the last element of a dimension. This is consistent with existing (Python) syntax where negative integers indicate the number of elements from the last (-1 is penultimate, -0 is ultimate): ncks -d lon,1,-0 in.nc out.nc # Second through last longitude http://nco.sf.net/nco.html#hyp G. NCO now supports chunking and deflation features of HDF4 datasets when compiled with netCDF library 4.3.3-rc1 or later. Earlier versions of netCDF do not support these features for HDF4. Thanks to Unidata for continuing to improve HDF4 interoperability. http://nco.sf.net/nco.html#hdf4 BUG FIXES: A. Fix treatment of packed variables with missing values in ncra. Versions 4.3.7-4.4.5 of ncra could produce (wildly) inaccurate statistics when promoting (e.g., to double- from single-precision) variables that are packed and that have missing values. The two workarounds are to remove the scaling or to perform the statistics in single precision with the --flt option. The solution is to upgrade to NCO 4.4.6. Thanks to user John for reporting this. http://nco.sf.net#bug_ncra_dbl_pck_mss B. Fix promotion of packed variables subject to minimization and maximization operations in ncwa. Versions 4.3.7-4.4.5 of ncwa incorrectly handled packed variables during these operations. The two workarounds are to remove the scaling or to perform the statistics in single precision with the --flt option. The solution is to upgrade to NCO 4.4.6. Thanks to user Frederic Laliberte for reporting this. http://nco.sf.net#bug_ncwa_pck_min_max C. ncrename now fails (as intended) when required objects not present. Since version 4.3.x, ncrename incorrectly ignored missing objects. Now missing objects are ignored only when prefixed with a period. A missing object (variable, attribute, dimension, group) not prefixed with period now causes an error as intended. Thanks to user John for reporting this. http://nco.sf.net/nco.html#ncrename D. NCO now checks correct function to enable DAP with configure. NCO has been checking the wrong function to see if netCDF was enabled with DAP support since about NCO 4.3.1 (May 2013). Thanks to Parker Norton and Filipe Fernandes for reporting this. E. Fix --file_format switch as synonym for --fl_fmt Thanks to Dave Allured for reporting this. KNOWN PROBLEMS DUE TO NCO: This section of ANNOUNCE reports and reminds users of the existence and severity of known, not yet fixed, problems. These problems occur with NCO 4.4.6 built/tested with netCDF 4.3.3-rc2 (20141001) on top of HDF5 hdf5-1.8.13 with: cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or- cd ~/nco/bld;make dir;make allinone # Old Makefile mechanism A. NOT YET FIXED (NCO problem)Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments
Demonstration:ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
ncks -m -C -v att_var ~/foo.nc 20130724: Verified problem still exists TODO nco1102 Cause: NCO parsing of ncatted arguments is not sophisticated enough to handle arrays of NC_STRINGS with embedded delimiters. B. NOT YET FIXED (NCO problem?)ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions
Demonstration: ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc 20140826: Verified problem still exists 20140619: Problem reported by rmla Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD? Workaround: Convert to fixed dimensions then hyperslab KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS: A. NOT YET FIXED (netCDF4 or HDF5 problem?) Specifying strided hyperslab on large netCDF4 datasets leads to slowdown or failure with recent netCDF versions. Demonstration with NCO <= 4.4.5: time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc Demonstration with NCL: time ncl < ~/nco/data/ncl.ncl 20140718: Problem reported by Parker Norton 20140826: Verified problem still exists 20140930: Finish NCO workaround for problem Cause: Slow algorithm in nc_var_gets()? Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets()) Workaround #2: Convert file to netCDF3 first, then use stride B. NOT YET FIXED (would require DAP protocol change?) Unable to retrieve contents of variables including period '.' in name Periods are legal characters in netCDF variable names. Metadata are returned successfully, data are not. DAP non-transparency: Works locally, fails through DAP server. Demonstration:ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable
20130724: Verified problem still exists. Stopped testing because inclusion of var_nm.dot broke all test scripts.NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.
Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47 C. NOT YET FIXED (would require DAP protocol change) Correctly read scalar characters over DAP. DAP non-transparency: Works locally, fails through DAP server. Problem, IMHO, is with DAP definition/protocol Demonstration:ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists Bug report not filed Cause: DAP translates scalar characters into 64-element (this dimension is user-configurable, but still...), NUL-terminated strings so MD5 agreement fails "Sticky" reminders: A. Pre-built Debian Sid & Ubuntu packages: http://nco.sf.net#debian B. Pre-built Fedora and CentOS RPMs: http://nco.sf.net#rpm C. Pre-built Mac binaries: http://nco.sf.net#mac D. Pre-built Windows (native) and Cygwin binaries: http://nco.sf.net#windows F. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., HDF4: AMSR MERRA MODIS ... HDF5: GLAS ICESat Mabel SBUV ... HDF-EOS5: AURA HIRDLS OMI ... -- Charlie Zender, Earth System Sci. & Computer Sci. University of California, Irvine 949-891-2429 )'(
netcdfgroup archives: