Re: NetCDF for underway oceanographic data storage (Lindsay)

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: Re: NetCDF for underway oceanographic data storage (Lindsay)
• From: Tim Holt <holtt@xxxxxxxxxxxx>
• Date: Tue, 15 Sep 1992 09:24:02 PDT

Lindsay Pender and other netCDF Oceanographers...

I was really happy with the replies I got from my recent post.  Lindsay
Pender (pender@xxxxxxxxxxx) brought up a good point when, talking about
using netCDF for underway data, saying...

>... however came up with some conceptual difficulties when I was
>looking at ways to implement it. It may be that your data is different, but in
>our case we have data coming from many different sources, each with different
>sampling rates. Some of our instruments are sampled at 2.5kHz, while others are
>as slow as once a minute. For an underway data storage system using netCDF how
>do you store such data with only one 'unlimited' dimension? 

OSU currently can manage it's data by logging 1 minute averages for
all instruments.  No one yet has asked for finer resolution from our
common use equipment.  CTD, ADCP, and other such higher resolution
systems are managed and logged by their own software and are currently
independent of the new netCDF system.  Soon though, I will need to merge
in some finer res. data (5 second GPS and ADCP).  Here is my scheme, and
I'm real curious what kinds of alternatives others can suggest.

I'll see if I can describe my idea with a CDL file.  It may not be the best 
way, but I guess it will work...



<<< BEGIN multi_res.cdl >>>

netcdf multires {

dimensions:
        min_max_mean = 3;   // store 3 numbers:  min, max, mean
        ten_hz = 600;       // number of 10.0 hZ samples in 1 minute
        five_hz = 300;      // number of 5.0 hZ samples in 1 minute
        twopoint5_hz = 150; // number of 2.5 hZ samples in 1 minute
        one_hz = 60;        // number of 1.0 hZ samples in 1 minute
        five_second = 20;   // number of 0.05 hZ samples in 1 minute
    time = unlimited;   // the "time" dimension

variables:
        long   time(time);      // seconds since some fixed point in time
        float  gps_lat(time);   // gps latitude in sample period
    float  gps_lon(time);   // gps longitude in sample period
        short  n_sats(time);    // number of satellites used in fix
        float  raw_gps_lat(time, five_second);  // raw gps latitude
        float  raw_gps_lon(time, five_second);  // raw gps longitude
        float  sea_temp(time, min_max_mean);  // sea surface temperature
        float  towed_cdt_temp (time, ten_hz); // raw CTD temperature
        float  towed_ctd_cond (time, ten_hz); // raw CTD conductivity
}

<<< END multi_res.cdl >>>

The idea is to pick the least common denominator (1 minute data) and
pack anything that's a finer resolution into a new dimension. 

I did try this scheme for a towed vehicle logging/display system, but I
found the netCDF time overhead (on a PC) was too high for me to log real
time, raw 24 hZ CTD data.  Too many variables to log -- more than the
simple example above.  I still used the same idea, but went to a simpler
ASCII file for quick I/O.

Comments???

Tim Holt, OSU Oceanography
holtt@xxxxxxxxxxxx