NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

[netcdfgroup] netCDF Operators NCO version 4.5.5 are ready

The netCDF Operators NCO version 4.5.5 are ready.

http://nco.sf.net (Homepage, Mailing lists)
http://github.com/nco (Source Code, Releases, Developers)

What's new?
NCO fully supports CDF5-format datasets, ncremap continues to
accrue useful features, and some corner case bugs were fixed.

Work on NCO 4.5.6 has commenced and will better support eliciting
latitude/longitude coordinates using the CF "coordinates" convention
and regridding variables whose horizontal dimensions are not the
most-rapidly-varying.

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. NCO supports the CDF5 file type first introduced by pnetCDF and now
   supported in Unidata netCDF as of version 4.4.0.
   CDF5 uses 64-bit offsets for a large address space (like the netCDF
   64BIT type) AND arrays may have more than 2^32 elements.
   Use '-5', '--5', '--fl_fmt=[cdf5|64bit_data|pnetcdf]'.
   Conversion to and between all netCDF filetypes is supported:
   ncks -5 in_3.nc out_5.nc
   ncks -5 in_7.nc out_5.nc
   ncks -7 in_5.nc out_7.nc
   http://nco.sf.net/nco.html#cdf5

B. ncremap: -P option triggers filetype-specific workflow procedures
   (such as automatic permutation). Currently filetype-specific
   handling is defined for AIRS, HIRDLS, MLS, and MPAS files:
   ncremap -P airs   -i airs.nc -m map.nc -o out.nc
   ncremap -P hirdls -i airs.nc -m map.nc -o out.nc
   ncremap -P mls    -i airs.nc -m map.nc -o out.nc
   ncremap -P mpas   -i mpas.nc -m map.nc -o out.nc
   One aspect of this handling is to rearrange dimension ordering
   from the way NASA/MPAS distributes generates these datasets so
   that horizontal spatial dimensions (lat, lon) come last.
   This is friendlier for regridding applications.

C. ncremap: -m map_fl supplies names to generated mapfiles and ncremap
   annotates map-files it creates with full history. This helps retain
   provenance in generated mapfiles. And ncremap has a friendlier
   mode, "map-only" mode, to generate mapfiles for later use.
   To use this mode, call ncremap without any input files to remap.
   ncremap will then generate the mapfile then exit.
   ncremap -i in.nc -d dst.nc -m map.nc -o out.nc # Named mapfile
   ncremap -s grd_in.nc -g grd_out.nc -m map.nc # Map-only mode
   http://nco.sf.net/nco.html#ncremap

D. ncremap -j job_nbr option for MPI parallelism
   http://nco.sf.net/nco.html#ncremap

E. ncra/nces/ncwa: New operation tabs()=total_absolute_value.
   The the -y tabs option is like -y ttl except the sum is over
   absolute values. tabs(), mibs(), mebs(), and mabs() are analogous
   to ttl(), min(), mean(), and max().
   nces -y tabs in.nc in.nc out.nc
   ncra -y tabs in.nc in.nc out.nc
   ncwa -y tabs in.nc out.nc
   http://nco.sf.net/nco.html#op_typ

F. Improve support for Cygwin builds

BUG FIXES:

A. ncremap: Inferred curvilinear grids always return points now in
   counterclockwise order. Version 4.5.4 sometimes did not.
   This caused bad things to happen in the weight-generators.

B. ncremap: Inferring grids from curvilinear coordinates no longer
   confused by "branch cuts" (i.e., the date-line). ncremap version
   4.5.4 would sometimes infer concave gridcells that then confused
   mapping software. Thanks to Feng Ding for reporting the problem.

C. ncpdq: Fix problem that could cause '--gaa' option to fail in
   earlier versions.

D. nces: Copy attributes of coordinate variables in nces --nsm_grp.
   Previous versions omitted the attributes of coordinates (not
   regular variables) in the ensemble average group. Thanks to Hannah
   Nissan for reporting and Pedro Vicente for fixing this.

E. ncra/nces: Correctly normalize "mebs" arithmetic. Under certain
   conditions previous versions did not correctly normalize sums
   computed with the mebs (mean absolute value) operator. Problem
   reported by Will Koeppen.

F. ncap2: Fix incorrect handling of negative dimension indices.
   Problem reported by Maksym Petrenko.

G. ncatted/ncrename: Fully support --glb_att_add=--gaa.
   This corrects an earlier oversight.
   http://nco.sf.net/nco.html#gaa

KNOWN PROBLEMS DUE TO NCO:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems.
   These problems occur with NCO 4.5.5 built/tested under
   MacOS with netCDF 4.3.3.1 on HDF5 1.8.16 and with
   Linux with netCDF 4.4.1-development (20160212) on HDF5 1.8.13.

A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments 

   Demonstration:
ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc 
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions 

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride

B. NOT YET FIXED (netCDF4 library bug)
Simultaneously renaming multiple dimensions in netCDF4 file can corrupt output 

   Demonstration:
ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but file is unreadable 
   ncks -v one ~/foo.nc
20150922: Confirmed problem reported by Isabelle Dast, reported to Unidata 
   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable 

   20130724: Verified problem still exists.
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string. 

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc 

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: