NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
The netCDF Operators NCO version 4.6.2 are ready. http://nco.sf.net (Homepage, Mailing lists) http://github.com/nco (Source Code, Releases, Developers) What's new? 4.6.2 is mainly a stability release to polish existing features and to add minor new ones. The exception is the new JSON backend. NCO now prints CDL, "Traditional NCO", XML, and JSON. Babel-icious eh? We built the JSON backend to help a project (DOE Terraref). Our first choice was to adopt an off-the-shelf netCDF->JSON tool. However, no existing solution worked for us. JSON is a loose syntax, and we made necessary design choices that suited our application, and left some choices for later. Are there syntactical variants you want us to add? Some users of netCDF version 4.4.1 cannot build NCO from scratch because a bug in the nc-config command kills NCO's 'configure;make'. Unidata will ship a corrected nc-config in netCDF 4.4.2. The 4.4.2-development branch already contains the necessary fix. See (C) in KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS below. Work on NCO 4.6.3 has commenced. Planned improvements include more flexibility in handling extensive variables during regridding, CMake support, and brackets for multi-dimensional array values in JSON. Enjoy, Charlie NEW FEATURES (full details always in ChangeLog): A. JSON backend Similar the CDL and XML backends, ncks now supports JSON. To obtain JSON, use --jsn or --json instead of --cdl/--xml. ncks --json -v one ~/nco/data/in.nc { "variables": { "one": { "type": "float", "attributes": { "long_name": "one" }, "data": 1.0 } } } More sample output at: http://dust.ess.uci.edu/tmp/in.json and other *.json files. Other related options (-v, -g, -m, -M) have their usual meanings. The --jsn_att_fmt switch accesses formats of selectable verbosity. Thanks to Henry Butowsky for implementing the JSON backend, and to Pedro Vicente and Chris Barker for helpful discussions. http://nco.sf.net/nco.html#json B. Multi-arguments refer to the format of command-line key-value pairs that NCO uses to support four different features that can have too many parameters to set via traditional command line arguments. The four features are regridding (rgr), global-attribute-adding (gaa), precision-preserving compression (ppc), and ENVI image processing for the DOE Terraref project (trr). Formerly, users were required to supply one value per command line option for these features, e.g., "--rgr lat_nbr=64 --rgr lon_nbr=128 --rgr lat_grd=cap ..." The new multi-argument feature allows users to aggregate multiple key-value pairs per command line argument, e.g., "--rgr lat_nbr=64#lon_nbr=128#lat_grd=cap# ..." Now all key-value pairs related to a single feature (like regridding, compression) can be provided as one arument! The string that delimits each pair is user-configurable and defaults to a single hash "#". Change it with, e.g., --dlm=":". Ensure your delimiters are not special shell characters, and are escaped (with backslashes) if present in key or value strings. Thanks to Jerome Mao for implementing multi-arguments. http://nco.sf.net/nco.html#mta C. ncap2 now searches all paths in the NCO_PATH environment variable for any #include'd files that it cannot find in ${CWD}. D. All builds tested on LLVM 8.0.0 and GCC 6.2.1 toolchains. BUG FIXES: A. ncclimo and ncremap could fail if ${BASH_SOURCE[0]} is unavailable. This could occur, e.g., with the dash shell and/or due to software modules that modify users' environments. This portability issue has been resolved. B. ncatted failed to implement NaN on some MinGW systems with, e.g., ncatted -O -a _FillValue,fll_val,m,f,nan ~/nco/data/in.nc ~/foo.nc We think this has been fixed. The workaround is to upgrade or use ncatted on *NIX or MacOS. Thanks to James Adams for reporting this. C. ncks CDL now prints correct tokens for uint, int64, uint64 types. These are "ul", "ll", "ull", respectively. Previously NCO used different tokens because ncgen <= 4.3.0 choked on "ll" formats. So this was an intentional NCO "feature", not a bug. Use ncgen >= 4.3.1 to parse these CDL files. KNOWN PROBLEMS DUE TO NCO: This section of ANNOUNCE reports and reminds users of the existence and severity of known, not yet fixed, problems. These problems occur with NCO 4.6.2 built/tested under MacOS 10.12.1 with netCDF 4.4.1 on HDF5 1.8.16 and with Linux with netCDF 4.4.2-development (20161116) on HDF5 1.8.16. A. NOT YET FIXED (NCO problem)Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments
Demonstration:ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
ncks -m -C -v att_var ~/foo.nc 20130724: Verified problem still exists TODO nco1102 Cause: NCO parsing of ncatted arguments is not sophisticated enough to handle arrays of NC_STRINGS with embedded delimiters. B. NOT YET FIXED (NCO problem?)ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions
Demonstration: ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc 20140826: Verified problem still exists 20140619: Problem reported by rmla Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD? Workaround: Convert to fixed dimensions then hyperslab KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS: A. NOT YET FIXED (netCDF4 or HDF5 problem?) Specifying strided hyperslab on large netCDF4 datasets leads to slowdown or failure with recent netCDF versions. Demonstration with NCO <= 4.4.5: time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc Demonstration with NCL: time ncl < ~/nco/data/ncl.ncl 20140718: Problem reported by Parker Norton 20140826: Verified problem still exists 20140930: Finish NCO workaround for problem Cause: Slow algorithm in nc_var_gets()? Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets()) Workaround #2: Convert file to netCDF3 first, then use stride B. NOT YET FIXED (netCDF4 library bug)Simultaneously renaming multiple dimensions in netCDF4 file can corrupt output
Demonstration:ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but file is unreadable
ncks -v one ~/foo.nc20150922: Confirmed problem reported by Isabelle Dast, reported to Unidata
20150924: Unidata confirmed problem 20160212: Verified problem still exists in netCDF library 20160512: Ditto 20161028: Verified problem still exists with netCDF 4.4.1 Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm More details: http://nco.sf.net/nco.html#ncrename_crd C. NOT YET FIXED (netCDF4 library bug) in main netCDF release FIXED in netCDF Development branch as of 20161116nc-config/nf-config produce erroneous switches that cause NCO builds to fail
This problem affects netCDF 4.4.1 on all operating systems. Some pre-compiled netCDF packages may have patched the problem. Hence it does affect my MacPorts install of netCDF 4.4.1. Demonstration: % nc-config --cflags # Produces extraneous text that confuses make Using nf-config: /usr/local/bin/nf-config -I/usr/local/include -I/usr/local/include -I/usr/include/hdf If your nc-config output contains the "Using ..." line, you are affected by this issue. 20161029: Reported problem to Unidata20161101: Unidata confirmed reproducibility, attributed to netCDF 4.4.1 changes
20161116: Unidata patch is in tree for netCDF 4.4.2 release D. NOT YET FIXED (would require DAP protocol change?) Unable to retrieve contents of variables including period '.' in name Periods are legal characters in netCDF variable names. Metadata are returned successfully, data are not. DAP non-transparency: Works locally, fails through DAP server. Demonstration:ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable
20130724: Verified problem still exists. Stopped testing because inclusion of var_nm.dot broke all test scripts.NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.
Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47 E. NOT YET FIXED (would require DAP protocol change) Correctly read scalar characters over DAP. DAP non-transparency: Works locally, fails through DAP server. Problem, IMHO, is with DAP definition/protocol Demonstration:ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists Bug report not filed Cause: DAP translates scalar characters into 64-element (this dimension is user-configurable, but still...), NUL-terminated strings so MD5 agreement fails "Sticky" reminders: A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., HDF4: AMSR MERRA MODIS ... HDF5: GLAS ICESat Mabel SBUV ... HDF-EOS5: AURA HIRDLS OMI ... B. Pre-built executables for many OS's at: http://nco.sf.net#bnr -- Charlie Zender, Earth System Sci. & Computer Sci. University of California, Irvine 949-891-2429 )'(
netcdfgroup archives: