NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
The netCDF Operators NCO version 4.4.4 are ready. http://nco.sf.net (Homepage) http://dust.ess.uci.edu/nco (Homepage "mirror") This release is mainly a bugfix and stability release. A few new features improve some corner-cases: It includes workarounds that allow NCO to successfully work with some names that violate netCDF but not HDF rules. Users can now retain "orphaned" dimensions after variable subsetting. nces now works with hyperslabs that resolve to the same size. Group broadcasting by ncbo is more intelligent when differencing ensemble means (created by nces) from full datasets. A potentially serious bug in ncwa --mask_condition has been fixed. Users of ncwa --mask_condition are encouraged to upgrade. Work on NCO 4.4.5 is underway, still focused on stability and speed. This includes more netCDF4 mop-up, improved group broadcasting algorithms, and, possibly, improved HDF4 support, and new chunking features. Enjoy, Charlie NEW FEATURES (full details always in ChangeLog): A. NCO now contains automatic workarounds to handle element names that contain characters that are legal in HDF though are illegal in netCDF. For example, slashes and leading special characters are are legal in HDF and illegal in netCDF element (i.e., group, variable, dimension, and attribute) names. NCO converts these forbidden characters to underscores, and retains the original names of variables in automatically produced attributes named hdf_name. Two real-world examples: NCO translates the NASA CERES dimension "(FOV) Footprints" to "_FOV_ Footprints", and "Cloud & Aerosol, Cloud Only, Clear Sky w/Aerosol, and Clear Sky" (yes, that's really a dimension name) to "Cloud & Aerosol, Cloud Only, Clear Sky w_Aerosol, and Clear Sky". ncks CERES_MODIS.hdf CERES_MODIS.nc http://nco.sf.net/nco.html#hdf4 B. ncks --rad to Retain all dimensions. When invoked with --rad, ncks copies each dimension in the input file to the output file, regardless of whether the dimension is utilized by any variables. Normally ncks discards "orphan dimensions", i.e., dimensions not referenced by any variables. This switch allows users to keep non-referenced dimensions. When invoked in printing mode, this causes orphaned dimensions to be printed (they are not printed by default). ncks -O -C -v lat,lon ~/nco/data/in.nc ~/foo.nc ncks --rad -O -x -v lat ~/foo.nc ~/foo2.nc ncks --rad --cdl ~/foo2.nc http://nco.sf.net/nco.html#rad C. nces now computes statistics on ensemble members whose specified hyperslabs resolve to the same size. nces has always worked well with identically shaped ensemble members, and on ensemble members whose record-dimension hyperslabs resolve to the same size. Now nces works well with ensemble members whose specified hyperslabs along any dimension resolve to the same size. The use-case that demanded this was a researcher who wished to compare a certain lat/lon region (stored in some regional files) with the same region stored in other global files (with the same grid). The regional files (e.g., Greenland only) are much smaller than global files so nces's new capability allows users to "get by" with regional files rather than always storing global files.nces -O --nsm_grp -d lat,0 -d lon,2 -d time,0 -p ~/nco/data grp1.nc grp2.nc ~/foo.nc
http://nco.sf.net/nco.html#nces BUG FIXES: A. Fixed ncwa incorrect interpretation of some --mask_conditions. Lexer/Parser could return incorrect mask values for negative numbers. Thanks to Keith Lindsay for report, and to Henry Butowsky for fix. Prior to this fix, the ncwa lexer would drop the negative sign, if any, from the comparators appearing in the mask condition, e.g., ncwa --mask_condition "lat < -20" was parsed as "lat < 20" not "lat < -20". Hence, users of ncwa --mask_condition (or -B) should upgrade. NB: The -m -M -T form of ncwa masking is/was not buggy. http://nco.sf.net#bug_ncwa_mask_condition B. Restore former tolerance of -x for non-existent elements. NCO normally requires that variables and groups specified with -v and -g must match the input file contents. The exception is when the -x flag tells NCO to exclude these elements. This exception was broken since NCO 4.2.x, and is now working again. Now excluding non-existent variables should one again work, e.g., ncks -O -x -v no_such_variable ~/nco/data/in.nc ~/foo.nc Thanks to Keith Oleson for this bug report. http://nco.sf.net/nco.html#x C. Fix bug that caused CDL and XML printing to truncate trailing zeros of floating point exponents. Before this bugfix, ncks --cdl would print 1.100e-10 as 1.100e-1 instead of 1.1e-10. All CDL and XML printing prior to 4.4.4 mis-prints floating point numbers with exponents that end in zero (i.e., +/- 10, 20, 30 ... 200, 210...). Users of CDL and XML mode are urged to upgrade. This bug changed affected values by some multiple of 10 orders of magnitude. Hence it is likely that users affected by this bug would notice it when carefully performing any statistics, yet possible they would not for fields with large dynamic ranges. Thanks to Stubaan for report. D. Fix regression tests not to cause false negatives due to netCDF 4.3.2 record dimension chunk size changes E. ncdismember better handles whitespace in group names F. Fix numerous memory leaks and some corner-case segfaults KNOWN PROBLEMS: This section of ANNOUNCE reports and reminds users of the existence and severity of known, not yet fixed, problems. These problems occur with NCO 4.4.4 built/tested with netCDF 4.3.3-rc1 on top of HDF5 hdf5-1.8.12 with these methods: cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or- cd ~/nco/bld;make dir;make allinone # Old Makefile mechanism A. NOT YET FIXED (NCO problem)Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments
Demonstration:ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
ncks -m -C -v att_var ~/foo.nc 20130724: Verified problem still exists TODO nco1102 Cause: NCO parsing of ncatted arguments is not sophisticated enough to handle arrays of NC_STRINGS with embedded delimiters. B. NOT YET FIXED (netCDF library problem) Probe hidden attributes (chunking, compression) of HDF4 files Demonstration: ncdump -h -s ~/nco/data/hdf.hdf # (dies) ncks -m ~/nco/data/hdf.hdf # (works by avoiding fatal calls) 20131230: Verified problem still exists Cause: some libnetCDF library functions fail on HDF4 file inquiries.Bug report filed: netCDF #HZY-708311 ncdump/netCDF4 segfaults probing HDF4 file
Tracking tickets NCF-272, NCF-273 C. NOT YET FIXED (would require DAP protocol change?) Unable to retrieve contents of variables including period '.' in name Periods are legal characters in netCDF variable names. Metadata are returned successfully, data are not. DAP non-transparency: Works locally, fails through DAP server. Demonstration:ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable
20130724: Verified problem still exists. Stopped testing because inclusion of var_nm.dot broke all test scripts.NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.
Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47 D. NOT YET FIXED (would require DAP protocol change) Correctly read scalar characters over DAP. DAP non-transparency: Works locally, fails through DAP server. Problem, IMHO, is with DAP definition/protocol Demonstration:ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists Bug report not filed Cause: DAP translates scalar characters into 64-element (this dimension is user-configurable, but still...), NUL-terminated strings so MD5 agreement fails "Sticky" reminders: A. Pre-built Debian Sid & Ubuntu packages: http://nco.sf.net#debian B. Pre-built Fedora and CentOS RPMs: http://nco.sf.net#rpm C. Pre-built Mac binaries: http://nco.sf.net#mac D. Pre-built Windows (native) and Cygwin binaries: http://nco.sf.net#windows F. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., HDF4: AMSR MERRA MODIS ... HDF5: GLAS ICESat Mabel SBUV ... HDF-EOS5: AURA HIRDLS OMI ... -- Charlie Zender, Earth System Sci. & Computer Sci. University of California, Irvine 949-891-2429 )'(
netcdfgroup
archives: