NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
The netCDF Operators NCO version 4.5.0 are ready. http://nco.sf.net (Homepage) http://dust.ess.uci.edu/nco (Homepage "mirror") This is the first NCO release that supports regridding datasets, currently limited to global datasets. These features build on infrastructure pioneered by Phil Jones (SCRIP), and the multi-agency Earth System Modeling Framework (ESMF). Work on NCO 4.5.1 has commenced and will support more regridding, additional ncra flexibility with weights, better threading, and easier builds on supercomputer systems that use modules. Enjoy, Charlie NEW FEATURES (full details always in ChangeLog): A. Regrid global datasets with ESMF or SCRIP remap files. Given a map-file that contains source-to-destination grid weights, ncks will regrid and input file to the destination grid. Acceptable file formats for map-files are described here: http://www.earthsystemcog.org/projects/regridweightgen http://oceans11.lanl.gov/svn/SCRIP/trunk/SCRIP ncks supports 1D->1D, 1D->2D, 2D->1D, and 2D->2D regridding for any unstructured 1D grid and any rectangular 2D grid. It has been tested converting among and between Gaussian, equiangular, FV, and unstructured cubed-sphere grids. Support for irregular 2D and regional grids (e.g., swath-like data) is planned. ncks automatically annotates the output with relevant metadata such as coordinate bounds, axes, and vertices (à la CF). When used with an identity remapping files, this feature can signficantly improve the metadata and usability of the original dataset. # Regrid entire file, same output format as input: ncks --map=map.nc in.nc out.nc # Deflated netCDF4 output, threading, selected variables: ncks -4 -L 1 -t 8 -v FS.?,T --map=map.nc in.nc out.nc http://nco.sf.net/nco.html#regrid B. ncdismember is a one-stop shop for flattening and CF-checking. When specifically requested, or when it cannot find a local cfchecker command, ncdismember will now upload each file (or, in the case of netCDF4 hierarchical files, each group) to the NERC CF compliance-checker, print the report summary, and save the full HTML reports for viewing after Orange is the New Black is over. Previously, ncdismember required a local cfchecker command to check files, though few users have managed to install a cfchecker. This need is obviated by automatic uploads to the NERC web service. To conserve bandwidth, ncdismember first "smallifies" the data file by hyperslabbing the dataset into a single element of latitude, longitude, and time---only then does it upload. Thanks to Phil Rasch for this idea and sample code. Using no checker, default (Decker) checker, NERC checker: ncdismember ~/nco/data/mdl_1.nc /tmp ncdismember ~/nco/data/mdl_1.nc /tmp decker ncdismember ~/nco/data/mdl_1.nc /tmp nerc http://nco.sf.net/nco.html#ncdismember C. NCO introduces a convention to retain the history-attribute contents of all files that were appended to a file. This convention stores those contents in the new global attribute called history_of_appended_files. This complements the history-attribute to provide a more complete provenance. The optional -h switch turns-off this behavior. http://nco.sf.net/nco.html#history D. When copying the variables date_written or time_written to the output file, NCO now updates their contents the GMT time reported by the system-supplied gmtime(). http://nco.sf.net/nco.html#cnv_acme BUG FIXES: A. Fix ncatted so it preserves global attributes (including history) from the output file when in append (-A) mode. KNOWN PROBLEMS DUE TO NCO: This section of ANNOUNCE reports and reminds users of the existence and severity of known, not yet fixed, problems. These problems occur with NCO 4.5.0 built/tested with netCDF 4.4.0-rc1 (20150610) on top of HDF5 hdf5-1.8.13 with: cd ~/nco;./configure # Configure mechanism -or- cd ~/nco/bld;make dir;make allinone # Old Makefile mechanism A. NOT YET FIXED (NCO problem)Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments
Demonstration:ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
ncks -m -C -v att_var ~/foo.nc 20130724: Verified problem still exists TODO nco1102 Cause: NCO parsing of ncatted arguments is not sophisticated enough to handle arrays of NC_STRINGS with embedded delimiters. B. NOT YET FIXED (NCO problem?)ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions
Demonstration: ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc 20140826: Verified problem still exists 20140619: Problem reported by rmla Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD? Workaround: Convert to fixed dimensions then hyperslab KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS: A. NOT YET FIXED (netCDF4 or HDF5 problem?) Specifying strided hyperslab on large netCDF4 datasets leads to slowdown or failure with recent netCDF versions. Demonstration with NCO <= 4.4.5: time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc Demonstration with NCL: time ncl < ~/nco/data/ncl.ncl 20140718: Problem reported by Parker Norton 20140826: Verified problem still exists 20140930: Finish NCO workaround for problem Cause: Slow algorithm in nc_var_gets()? Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets()) Workaround #2: Convert file to netCDF3 first, then use stride B. NOT YET FIXED (would require DAP protocol change?) Unable to retrieve contents of variables including period '.' in name Periods are legal characters in netCDF variable names. Metadata are returned successfully, data are not. DAP non-transparency: Works locally, fails through DAP server. Demonstration:ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable
20130724: Verified problem still exists. Stopped testing because inclusion of var_nm.dot broke all test scripts.NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.
Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47 C. NOT YET FIXED (would require DAP protocol change) Correctly read scalar characters over DAP. DAP non-transparency: Works locally, fails through DAP server. Problem, IMHO, is with DAP definition/protocol Demonstration:ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists Bug report not filed Cause: DAP translates scalar characters into 64-element (this dimension is user-configurable, but still...), NUL-terminated strings so MD5 agreement fails "Sticky" reminders: A. Pre-built Debian Sid & Ubuntu packages: http://nco.sf.net#debian B. Pre-built Fedora and CentOS RPMs: http://nco.sf.net#rpm C. Pre-built Mac binaries: http://nco.sf.net#mac D. Pre-built Windows (native) and Cygwin binaries: http://nco.sf.net#windows E. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., HDF4: AMSR MERRA MODIS ... HDF5: GLAS ICESat Mabel SBUV ... HDF-EOS5: AURA HIRDLS OMI ... -- Charlie Zender, Earth System Sci. & Computer Sci. University of California, Irvine 949-891-2429 )'(
netcdfgroup archives: