netCDF Utilities

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: netCDF Utilities
• From: hld@xxxxxxxxxxxxxxxxx (DAVIES Harvey)
• Date: Mon, 10 Feb 1992 21:50:52 +1100

My previous posing asked:

>Are there any utilities available besides ncgen & ncdump?

Steve Emmerson <steve@xxxxxxxxxxxxxxxx> replied as follows:

>The Unidata Program Center is implementing a suite of netCDF programs.
>A draft functional and syntactical specification is available via
>anonymous FTP:

>        host:   unidata.ucar.edu [128.117.140.3]
>        files:  pub/sdm/ncprogs.ps
>                pub/sdm/ncprogs.txt

>The first file contains a PostScript document; the second contains an
>ASCII equivalent.

>Comments are welcome and should be sent to me <steve@xxxxxxxxxxxxxxxx>
>-- unless, of course, you feel that the topic has sufficient general
>appeal to warrant being posted to the mailing-list (to which I also
>subscribe).

I was very interested in this draft document & would like to start some
discussion within netcdfgroup by sharing my comments.

My first reaction is that the proposals seem to assume a quite different
way of using netcdf from that I have adopted so far in my rather limited
experience. I use ncgen to create files with everything except the main
data arrays. Then I use my program nc_put_var to put the data into the file.
It would greatly simplify the design of utilties if they did not have to
create new variables & their related attributes & coordinates.
I assume that the structure of a file does not change, only its data. I would
also assume that one would process only one variable at a time. I would hope
that all utilities would:
- do unit conversion using units attribute values of input & output files
- do type conversion, allowing different types in input & output files
- utilize add_offset & scale_factor when reading & writing

So far nc_put_var is the only utility I have developed. It reads data from
standard input & puts it into an existing variable in a netCDF file. Usage:
Usage:
  nc_put_var [-hps] [-i INDEX] [-m MISVAL] [-u UNIT] [-v VAR] FILE
    -h: Do not write more history
    -p: Persevere after errors
    -s: Silent mode: Suppress warning messages
    INDEX: index vector of 1st data value (default: 0,0,...)
    MISVAL: input missing value
    UNIT: input unit (default: unit in file)
    VAR: name of variable (default: final variable)
    FILE: name of netCDF file to be updated

I am not happy with the facility for specifying arrays in either the current
version of nc_put_var or the NCAR plans.  Examples where something more
powerful that the hyperslab is needed are:
- month=12,1,2 (summer for us down under, winter for the rest of you!)
- the ability to select any longitude interval & duplicate one or more 
  longitudes at the ends

I propose a more sophisticated array syntax based partly on APL notation.

An 'array' is specified by:
- netCDf filename
- variable name
- a list of vector subscripts specifying what rows, columns, etc. are included

The suggested syntax is:
filename;varname[list]
filename;varname (for whole array)

list has the form:
cv=vector;cv=vector;cv=vector;...
where cv is name of coordinate variable

vector is a list of triplets separated by commas

triplet can be any of:
- subscript
- :
- subscript:
- :subscript
- subscript:subscript
- :subscript:stride
- subscript::stride
- subscript:subscript:stride
- *
Note that a triplet defines an arithmetic progression as in F90
* means append along unlimited dimension i.e. effectively n+1,n+2,...,infinity
where n is current size of unlimited dimension (* can only be used for output)

subscript can be any of:
- integer
- ~number
- <number
- >number
- ~number unit
- <number unit
- >number unit
The 1st form directly specifies a dimension index 1,2,3,... (or maybe 0,1,2,...)
The prefix ~ < or > means that the index is found via the coordinate variable
> means least greater than or equal to
< means greatest less than or equal to

Examples:
Assume file 'abc.cdf' has a variable 'rain' with dimensions lat, long, month &
year.

abc.cdf;rain
- whole variable

abc.cdf;rain[lat=:8;long=~-180:>-90 degrees_east:2;month=12,1:2]
first eight latitudes (indices 1 to 8)
every 2nd longitude from closest to -180 degrees to first >= -90 degrees
months 12 1 & 2
all years

I hope these ideas are of interest & not too 'way out'.

Harvey Davies                   CSIRO Division of Atmospheric Research,
Email: hld@xxxxxxxxxxxx         Private Bag No. 1, Mordialloc,
Phone: (03)5867574              Victoria 3195,  Australia