Re: Old Problem - Sparse Data

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: Re: Old Problem - Sparse Data
• From: Carlie Coats <xcc@xxxxxxxxxxxx>
• Date: Tue, 20 May 1997 06:53:27 -0400

[Jeremy Beal wrote that he has large quantities of both spatially and
temporally irregular/sparse data that he needs to store and retrieve 
efficiently in a platform independent manner, and wonders how best to 
do this.]

One additional question comes to mind immediately:  

    Do you want fast selective random access?

Be sure of your answer:  in many cases it can make an astounding
difference to the style of work you do.  Fast selective random access
makes an enormous difference for analysis and visualization. (I've also
seen too many met and met-related) models built around sequential
files that have become vast conspiracies to manipulate a complex
shared state centered around the positions of a multiplicity of
sequential file pointers.)

If you don't need/want fast selective random access, then the
XDR'ed binary file is an acceptable solution.  Otherwise, for sparse
data you need files with built-in indexing.  HDF VSets are a partial 
solution to this, provided you don't have very many time steps:  they 
have a doubly-linked list of index blocks interspersed with data blocks.
Be aware, though, that the overhead of sequential access to those index 
blocks can kill you if you do have lots of time steps.  If you have a 
year's worth of hourly met observations stored this way and you want
to look at the 0Z Dec 1 observations, be prepared to sit for five or
ten minutes while your disk drive grinds through the 8000 or so index
blocks for Jan 1-Dec 1 before it can even begin to think about data.

Something else worth checking is PDB, which is part of Livermore's 
Portable Application Code Toolkit; see 

    http://www.llnl.gov/def_sci/pact/hact_homepage.html

It seems to be a lower-level interface than netCDF, but does have support
for building efficient index structures.

fwiw

                                                 xcc@xxxxxxxxxxxxxxxx
Carlie J. Coats, Jr.                                   coats@xxxxxxxx
MCNC Environmental Programs                      phone: (919)248-9241
North Carolina Supercomputing Center               fax: (919)248-9245
3021 Cornwallis Road                                  P. O. Box 12889
Research Triangle Park, N. C.  27709-2889                         USA
"My opinions are my own, and I've got *lots* of them!"