[netcdfgroup] netCDF Operators NCO version 4.6.6 are ready

• To: NCO Announce Mail List <nco-announce@xxxxxxxxxxxxxxxxxxxxx>, netCDF Mail List <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: [netcdfgroup] netCDF Operators NCO version 4.6.6 are ready
• From: Charlie Zender <zender@xxxxxxx>
• Date: Fri, 21 Apr 2017 14:41:16 -0700

The netCDF Operators NCO version 4.6.6 are ready.

http://nco.sf.net (Homepage, Mailing lists)
http://github.com/nco (Source Code, Releases, Developers)

What's new?
4.6.6 contains a potpurri of new features:
Output 2D rectangular grids formatted as UGRID meshes.
Calendar dates print for bounds and climatology variables.
Revamped HPSS/hsi support. The splitter produces cleaner output.
The regridder groks more files and masks, and annotates output with
more helpful metadata. CDL mode handles compound record variables.
JSON output has changed slightly to re-use NcML keywords.
ncclimo and ncremap have new extract/exclude options.

EVERYONE WHO USES NCKS TO PRINT DATA PLEASE READ THIS:

We would like to change the default behavior of ncks printing.
For 20+ years ncks has, by default, dumped the text representation
of a file in (what we call) "traditional" mode. ncdump produces a
format called CDL that is more useful than traditional mode for most
NCO users. ncks has produced CDL since only 2013 and so CDL is not the
default. ncdump has always printed CDL very cleanly and clearly there
was little point in making CDL the default for ncks.
Now ncks CDL mode rivals ncdump, and is superior in some ways.
In particular, ncks now prints times as human-readable calendar dates,
the last ncdump feature I used that ncks did not have.
We propose to change the default printed output of ncks to CDL.
Then one would type simply "ncks" instead of "ncks --cdl".
A new "--trd" option would still print traditional output.
Let us know if this bothers or pleases you prior to 4.6.7.
You have been warned :)

Work on NCO 4.6.7 has commenced. Planned improvements include
support for conda installs on MS Windows, more ncclimo and ncremap
features.

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. ncremap and ncclimo support new extract/exclude switches:
   --no_cll_msr: Exclude cell_measures variables
   --no_frm_trm: Exclude formula_terms variables
   --no_ntv_tms: Exclude native-grid timeseries (splitter only)
   --no_stg_grd: Exclude staggered grid coordinates
   By default, these variables are always extracted.
   Use the above switches to exclude them instead, e.g.,
   ncclimo --no_cll_msr --no_frm_trm --no_ntv_tms --no_stg_grd ...
   http://nco.sf.net/nco.html#no_cll_msr
   http://nco.sf.net/nco.html#no_frm_trm
   http://nco.sf.net/nco.html#no_ntv_tms
   http://nco.sf.net/nco.html#no_stg_grd

B. ncclimo's splitter mode no longer requires variable lists.
   Previously the splitter required a '-v var_lst' option to know
   which variables to split into timeseries. Now when the -v option
   is omitted the splitter creates timeseries of all variables of
   rank R >= 2. Typically this includes all spatial variables.
   Variables known to be extensive (like grid properties) are
   automatically excluded from this list. However, we expect there
   will be some cases where the splitter creates timeseries from
   R >= 2 variables that are not desirable. We are happy to encode
   manual exceptions as we receive notice of them.
   ncclimo -s 1979 -e 1981 -o $drc_out in*.nc
   http://nco.sf.net/nco.html#ncclimo

C. The regridder now produces UGRID output for 2D rectangular grids.
   Invoke ncremap in grid inferral mode with "--rgr ugrid=foo.nc"
   and voilà, an annoted UGRID file is produce along with the
   default SCRIP gridfile. The NCO UGRID is more complete than that
   produced by ESMF_Scrip2Unstruct, but ESMF can create UGRID for much
   more comlex grids. The NCO regridder can currently only generate
   UGRID for rectangular 2D input grids, and there are no plans to
   extend this to curvilinear or unstructured input grids.
   ncks --rgr nfr=Y --rgr grid=scrip.nc --rgr ugrid=ugrid.nc in.nc out.nc
   http://nco.sf.net/nco.html#rgr

D. ncks now prints the units, if any, in comments after CDL values
   when debugging is enabled with dbg_lvl >=1:
   zender@foo: ncks --cdl -D 1 in.nc
   lat = -90, 90 ; // units="degrees_north"
   lon = 0, 90, 180, 270 ; // units="degrees_east"
   tpt = 273.1, 273.2, ... 274 ; // units="kelvin"
   http://nco.sf.net/nco.html#cdl

E. ncap2 now implements the CF cell_methods convention.
   ncra, nces, and ncwa have long implemented this, and now the
   relevant ncap2 methods (avg, min, max, etc.) record the operation
   type in the cell_methods attribute.
   ncap2 -s 'avg=one_dmn_rec_var.avg()' ~/nco/data/in.nc ~/foo.nc
   http://nco.sf.net/nco.html#cll_mth

F. Since 4.6.5 ncks has printed the human-legible calendar string
   corresponding to values with UDUnits date units (time since
   basetime, e.g., "days since 2000-01-01") and CF calendar attribute,
   if any. Now ncks also translates and prints time values associated
   with CF climatology and bounds variables as human legible dates.
   As before, enact this with the --calendar option when printing.
   Using dbg_lvl >= 1 in CDL mode prints both the value and calendar
   string (one in comments):
   zender@aerosol:~$ ncks -D 1 --cdl --cal -v tm_365 ~/nco/data/in.nc
   tm_365 = "2013-03-01"; // double value: 59
   zender@aerosol:~$ ncks -D 1 --cdl -v tm_365 ~/nco/data/in.nc
   tm_365 = 59; // calendar format: "2013-03-01"
   This option is similar to the ncdump -t option.
   Does this work how you'd like? Let us know.
   http://nco.sf.net/nco.html#cal

G. ncks now prints "(no units)" instead of "(null)" as the units
   string for variables with no units. And in CDL mode ncks, when
   invoked with debugging (e.g., -D 1) prints the units string in the
   comment section after the values of a variable.

BUG FIXES:

A. NCO now exits with a clear message instead of core dump on missing
   files on HPSS systems with the hsi command.

B. ncremap will no longer define mask variables in inferred grid
   files unless it finds a mask variable in the source file.
   Previously ncremap could define a mask variable in the inferred
   grid file and never fill-it with data.

C. ncremap now invokes the neareststod algorithm when requested.
   Previously a typo caused ncremap to invoke nearestdtos when
   neareststod was specified (doh!). Solution is to upgrade.

D. ncremap now converts non-default masks correctly in infer mode.
   Previously a "mask" variable in an input file whose grid structure
   was being inferred could be incorrectly handled for some numeric
   types. This likely only affected masks with missing values.
   Sources of files possibly affected include AMSR and CICE.

E. ncks now prints braces required to disambiguate compound
   record dimensions in CDL mode. Such braces have been required by
   ncgen-like parsers since ~2013 in order to unambiguously determine
   the record dimension sizes in variables with multiple record
   dimensions. This fix only affects CDL printing of netCDF4 files
   that contain compound record variables.

KNOWN PROBLEMS DUE TO NCO:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems.
   These problems occur with NCO 4.6.6 built/tested under
   MacOS 10.12.3 with netCDF 4.4.1 on HDF5 1.8.16 and with
   Linux with netCDF 4.4.2-development (20161116) on HDF5 1.8.16.

A. NOT YET FIXED (NCO problem)

   Demonstration:

   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride

B. NOT YET FIXED (netCDF4 library bug)

   Demonstration:

   ncks -v one ~/foo.nc

   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library
   20160512: Ditto
   20161028: Verified problem still exists with netCDF 4.4.1
   20170323: Verified problem still exists with netCDF 4.4.2-development

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (netCDF4 library bug)

   Demonstration:

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
   More details: http://nco.sf.net/nco.html#ncrename_crd

   This problem affects netCDF 4.4.1 on all operating systems.
   Some pre-compiled netCDF packages may have patched the problem.
   Hence it does not affect my MacPorts install of netCDF 4.4.1.

   Demonstration:
   % nc-config --cflags # Produces extraneous text that confuses make
   Using nf-config: /usr/local/bin/nf-config
   -I/usr/local/include -I/usr/local/include -I/usr/include/hdf

   If your nc-config output contains the "Using ..." line, you are
   affected by this issue.

   20161029: Reported problem to Unidata

   20161116: Unidata patch is in tree for netCDF 4.4.2 release
   20161123: Fixed in maintenance release netCDF 4.4.1.1

E. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:

   20130724: Verified problem still exists.
   Stopped testing because inclusion of var_nm.dot broke all test scripts.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

F. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr

--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(