NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
The netCDF Operators NCO version 4.3.5 are ready. http://nco.sf.net (Homepage) http://dust.ess.uci.edu/nco (Homepage "mirror") The current release brings group support to ncra, ncea, and ncrcat. Now ten operators fully support group hierarchies: ncbo, ncea, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat, ncrename, and ncwa. The operators that do no yet fully support group hierarchies are ncatted and ncap2. Rounding errors can accumulate to worrisome levels during arithmetic performed on large (>~10,000) arrays of single precision floats. Now NCO has the --dbl switch that forces operators to coerce single-precision numbers to double-precision for arithemetic. It is expensive in terms of time and RAM, and often not warranted. Use this switch to guarantee preserving maximum precision. Or if you want your NCO answers on single-precision fields to always agree exactly with tools that always promote (NCL, Matlab, IDL). The manual now contains a detailed discussion of the issues. Unfortunately progress sometimes comes at a price in bugs :) 4.3.5 fixes a nasty ncwa bug present in 4.3.3 and 4.3.4. Nasty means that it could produce wrong answers that look right. Fortunately, it could only occur when simultaneously averaging and hyperslabbing, e.g., ncwa -a lat,lon -d lat,0.,90. in.nc out.nc. Such users should immediately upgrade to 4.3.5 and re-check results. Full details below. Work on NCO 4.3.6 is underway and includes improved netCDF4 support for more NCO operators (ncatted) and improved support for HDF4 files. Enjoy, Charlie "New stuff" in 4.3.5 summary (full details always in ChangeLog): NEW FEATURES: A. ncra, ncea, and ncrcat now supports groups, Group Path Editing (GPE) and list unions, the same options already supported by ncbo, ncecat, ncflint, ncks, ncpdq, and ncwa. Features are controlled by -g, -G and --union, respectively. ncra -g g1 -v v1 -G dude -O -p ~/nco/data in_grp.nc ~/foo.nc ncea -g g1 -v v1 -G dude -O -p ~/nco/data in_grp.nc ~/foo.nc ncrcat -g g1 -v v1 -G dude -O -p ~/nco/data in_grp.nc ~/foo.nc http://nco.sf.net/nco.html#ncea http://nco.sf.net/nco.html#ncra http://nco.sf.net/nco.html#ncrcat http://nco.sf.net/nco.html#grp http://nco.sf.net/nco.html#gpe http://nco.sf.net/nco.html#union B. Forced conversion of float to double: NCO has always converted integers and single precision reals (i.e., floats) to double precision reals (i.e., doubles) prior to arithmetic when it would preserve precision of the result (the results are converted back to the original type after arithmetic). Until now, NCO never implicitly converted floats to doubles unless doubles were involved in the computation, because, well, GIGO ("garbage in, garbage out") and wisdom from Numerical Recipes. However, such promotion is often warranted on arithmetic involving large (> ~10,000) numbers of single-precision inputs. Now NCO allows a new arithmetic convention to implement this. The convention is manually invoked with --dbl: ncea --dbl in*.nc out.nc ncra --dbl in*.nc out.nc ncwa --dbl in.nc out.nc The current default behavior is explicitly selected with --flt. Depending on user feedback, --dbl may become the default in future versions of NCO. Send me your opinion! NB: Invoking arithmetic promotion of NC_FLOAT to NC_DOUBLE increases RAM memory usage by a factor of two, and slows down arithmetic, though perhaps negligibly compared to I/O time. The manual contains a detailed discussion of the issues: http://nco.sf.net/nco.html#dbl C. CDL-mode prints more verbose information as comments (after '// ') as the debugging level increases (i.e., ncks --cdl -D 1). Currently this includes a file summary, the ncgen command required to convert back to netCDF, and full path names for all groups. ncks --cdl -D 1 ~/nco/data/in_grp.nc # Adds file summary ncks --cdl -D 3 ~/nco/data/in_grp.nc # Adds full groupnames http://nco.sf.net/nco.html#cdl D. GPE operations allow netCDF3 output from netCDF4 input when groups are not required in the output file. This allows GPE flattening to directly create netCDF3 classic files for backward compatibility. For example, this extracts group "cesm" from a netCDF4 file and stores it at the root level of a netCDF3 output file: ncks -O -G : -g cesm -3 ~/nco/data/cmip5.nc ~/cesm.nc http://nco.sf.net/nco.html#gpe http://nco.sf.net/nco.html#xmp_flt BUG FIXES: A. ncwa hyperslabbing while averaging bug: Versions 4.3.3-4.3.4 of ncwa could return incorrect answers when user-specified hyperslabs were simultaneously extracted. In such cases, hyperslab limits were not consistently applied. This could produce incorrect answers that look correct. This bug only affected hyperslabbed statistics (those produced by simultaneously invoking -a and -d switches); "global averages" were unaffected. We urge all ncwa users to upgrade to NCO 4.3.5. http://nco.sf.net#bug_ncwa_lmt B. Group operators now correctly match variable names that are proper subsets of ancestor group names. C. Fixed ncpdq dimension reversal in group hierarchies. KNOWN ISSUES NOT YET FIXED: This section of ANNOUNCE reports and reminds users of the existence and severity of known, not yet fixed, problems. These problems occur with NCO 4.3.5 built/tested with netCDF 4.3.1-rc3 snapshot 20130827 on top of HDF5 hdf5-1.8.9 with these methods: cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or- cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism A. NOT YET FIXED Correctly read netCDF4 input over DAP, write netCDF4 output, then read resulting file. Replacing netCDF4 with netCDF3 in either location of preceding sentence leads to success. DAP non-transparency: Works locally, fails through DAP server. Unclear whether resulting file is "legal" because of dimension ID ordering assumptions. Demonstration: ncks -4 -O -v three_dmn_rec_var http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in_4.nc ~/foo.nc ncks ~/foo.nc # breaks with "NetCDF: Invalid dimension ID or name" 20130724: Unable to verify since in_4.nc no longer accessible on Unidata DAP server Bug report filed: netCDF #QUN-641037: dimension ID ordering assumptions B. NOT YET FIXED netCDF4 library fails when renaming dimension and variable using that dimension, in either order. Works fine with netCDF3. Problem with netCDF4 library implementation. Demonstration: ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc # Breaks with "NetCDF: HDF error" ncks -m ~/foo.nc 20130724: Verified problem still exists Bug report filed: netCDF #YQN-334036: problem renaming dimension and coordinate in netCDF4 file C. NOT YET FIXED (requires change to DAP protocol) Unable to retrieve contents of variables including period '.' in name Periods are legal characters in netCDF variable names. Metadata are returned successfully, data are not. DAP non-transparency: Works locally, fails through DAP server. Demonstration: ncks -O -C -D 3 -v var_nm.dot -p http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc # Fails to find variable 20130724: Verified problem still exists. Stopped testing because inclusion of var_nm.dot broke all test scripts. NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string. Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47 D. NOT YET FIXED (requires change to DAP protocol) Correctly read scalar characters over DAP. DAP non-transparency: Works locally, fails through DAP server. Problem, IMHO, is with DAP definition/protocol Demonstration: ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc 20120801: Verified problem still exists Bug report not filed Cause: DAP translates scalar characters into 64-element (this dimension is user-configurable, but still...), NUL-terminated strings so MD5 agreement fails E. NOT YET FIXED (NCO problem) Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments Demonstration: ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc 20130724: Verified problem still exists TODO nco1102 Cause: NCO parsing of ncatted arguments is not yet sophisticated enough to handle arrays of NC_STRINGS with embedded delimiters. "Sticky" reminders: A. Pre-built, up-to-date Debian Sid & Ubuntu packages: http://nco.sf.net#debian B. Pre-built Fedora and CentOS RPMs: http://nco.sf.net#rpm C. Pre-built Windows (native) and Cygwin binaries: http://nco.sf.net#windows D. Pre-built AIX binaries: http://nco.sf.net#aix E. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)? SWAMP efficiently schedules/executes NCO scripts on remote servers: http://swamp.googlecode.com SWAMP can work command-line operator analysis scripts besides NCO. If you must transfer lots of data from a server to your client before you analyze it, then SWAMP will likely speed things up. F. NCO support for netCDF4 features is tracked at http://nco.sf.net/nco.html#nco4 NCO supports netCDF4 atomic data types, compression, chunking, and groups. G. Reminder that ncks, ncecat, ncbo, ncflint, and ncpdq work on many common HDF5 datasets, e.g., NASA AURA HIRDLS HDF-EOS5 NASA ICESat GLAS HDF5 NASA SBUV HDF5... -- Charlie Zender, Earth System Sci. & Computer Sci. University of California, Irvine 949-891-2429 )'(
netcdfgroup archives: