NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
The netCDF operators NCO version 4.2.6 are ready. http://nco.sf.net (Homepage) http://dust.ess.uci.edu/nco (Homepage "mirror") This release continues to improve support of netCDF4 files in ncks and ncecat, and add a sprinkle of new and convenient features to the rest of NCO. ncks and ncecat now handle corner cases of netCDF4 groups, dimensions, and coordinates. We perverted our own test file with tortuous situations that all work, such as dimensions, coordinates, and variables all being in separate groups. ncks and ncecat now work well on many common HDF5 datasets, e.g., NASA AURA HIRDLS HDF-EOS5 NASA ICESat GLAS HDF5 NASA SBUV HDF5... Work on NCO 4.2.7 is underway and focuses on netCDF4 support for ncrename and ncbo operators, and on better support for Windows builds (enabling DAP, UDUnits). Enjoy, Charlie and Pedro "New stuff" in 4.2.6 summary (full details always in ChangeLog): NEW FEATURES: A. Regular expressions (rx's) now work on full paths, not just stubs. Previously, rx's only worked on last components of full names. Now rx's like ncks -v /cesm/temperature.+ can succeed. http://nco.sf.net/nco.html#rx B. ncks now counts and then ignores non-atomic type netCDF4 variables. Previously, ncks would die if it encountered the new netCDF4 non-atomic types: vlen, opaque, enum, and compound. Now ncks prints the variable name and then ignores it and processes all the atomic-type (float, double, int, string...) variables as usual. C. ncks prints fulls paths of non-local dimensions and coordinates. D. ncrename allows "mandatory attributes" (i.e., those not prefixed with a period which would make them "optional attributes") to be present in groups/global instead of in variables. E. ncflint accepts a new switch, --fix_rec_crd, that prevents record coordinates from being multiplied/interpolated. By default, ncflint multplies/interpolates record coordinates (like time) but not other coordinates (like, typically, latitude). This is because ncflint is most often used to time-interpolate files (so you want the time to change, but not the spatial coordinates). However, ncflint is also used quite often to "scale" files by multiplying the variables by a constant. Now one can easily multiply all variables (except coordinates) by a constant, and if --fix_rec_crd is specified, the record coordinate will not change. BUG FIXES: A. ncks correctly interprets scopes in corner cases including: 1. Dimension defined in ancestor of variable's group 2. Coordinate defined in relative (sibling, ancestor, descendent) of variable's group 3. Single dimension has multiple associated coordinates Handling these cases correctly is a major milestone for NCO. B. ncks prints accurate netCDF4 metadata summaries (with -M) and breaks dimensions into fixed vs. record, and variables into atomic-types vs. non-atomic. Formerly, the number of netCDF4 dimensions was ill-defined. C. ncks 4.2.4-4.2.5 failed when attempting to copy certain netCDF3 files (known as MM3 files because they contain multiple record variables that require a special copying algorithm) into netCDF4 files when group-path-editing (GPE) was invoked. This has been fixed so commands like this work again: ncks -G g1 -4 in.nc out.nc KNOWN BUGS NOT YET FIXED: This section of the ANNOUNCE file reports and reminds users of the existence and severity of known, not yet fixed, problems. These problems occur with NCO 4.2.6 built/tested with netCDF 4.3.0-rc3 released 20130311 on top of HDF5 hdf5-1.8.9 with these methods: cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or- cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism A. NOT YET FIXED Correctly read netCDF4 input over DAP, write netCDF4 output, then read resulting file. Replacing netCDF4 with netCDF3 in either location of preceding sentence leads to success. DAP non-transparency: Works locally, fails through DAP server. Unclear whether resulting file is "legal" because of dimension ID ordering assumptions Demonstration: ncks -4 -O -v three_dmn_rec_var http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in_4.nc ~/foo.nc ncks ~/foo.nc # breaks with "NetCDF: Invalid dimension ID or name" 20120731: Unable to verify since in_4.nc no longer accessible Bug report filed: netCDF #QUN-641037: dimension ID ordering assumptions B. NOT YET FIXED netCDF4 library fails when renaming dimension and variable using that dimension, in either order. Works fine with netCDF3. Problem with netCDF4 library implementation. Demonstration: ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc # Breaks with "NetCDF: HDF error" ncks -m ~/foo.nc 20130319: Verified problem still exists Bug report filed: netCDF #YQN-334036: problem renaming dimension and coordinate in netCDF4 file C. NOT YET FIXED Unable to retrieve contents of variables including period '.' in name Metadata is returned successfully, data is not. DAP non-transparency: Works locally, fails through DAP server. Demonstration: ncks -O -C -D 3 -v var_nm.dot -p http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc # Fails to find variable 20120731: Verified problem still exists. Stopped testing because inclusion of var_nm.dot broke all test scripts. NB: Problem hard to fix since DAP interprets '.' as structure delimiter in query string of HTTP requests. Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47 D. NOT YET FIXED Correctly read scalar characters over. DAP non-transparency: Works locally, fails through DAP server. Problem, IMHO, is with DAP definition/protocol Demonstration: ncks -O -D 1 -H -C -m --md5 -v md5_a -p http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc 20120801: Verified problem still exists Bug report not filed Cause: DAP translates scalar characters into 64-element, NUL-terminated strings so MD5 agreement fails E. NOT YET FIXED netCDF4 library can create dimensions with duplicate IDs if dimension with same name defined in a group and its ancestor group. Problem with HDF5 or with netCDF4 library implementation? Demonstration: ncks -O -v two_dmn_rec_var ~/nco/data/in_grp.nc ~/foo.nc ncks -m ~/foo.nc 20130319: Verified problem still exists Bug report filed 20120312: netCDF #SHH-257980: Re: [netcdfgroup] Dimensions IDs F. NOT YET FIXED ncdump is unable to dump the NCO test file in_grp.nc Unclear whether problem with ncdump, HDF5, or with netCDF4 library implementation Demonstration: ncdump ~/nco/data/in_grp.nc # breaks with "NetCDF: Invalid argument Location: file dumplib.c; line 970" 20130319: Verified Bug report not yet filed "Sticky" reminders: A. Pre-built, up-to-date Debian Sid & Ubuntu packages: http://nco.sf.net#debian B. Pre-built Fedora and CentOS RPMs: http://nco.sf.net#rpm C. Pre-built Windows (native) and Cygwin binaries: http://nco.sf.net#windows D. Pre-built AIX binaries: http://nco.sf.net#aix E. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)? SWAMP efficiently schedules/executes NCO scripts on remote servers: http://swamp.googlecode.com SWAMP can work command-line operator analysis scripts besides NCO. If you must transfer lots of data from a server to your client before you analyze it, then SWAMP will likely speed things up. F. NCO support for netCDF4 features is tracked at http://nco.sf.net/nco.html#nco4 NCO supports netCDF4 atomic data types, compression, chunking, and groups. -- Charlie Zender, Earth System Sci. & Computer Sci. University of California, Irvine 949-891-2429 )'(
netcdfgroup archives: