Re: [thredds] Sparse Grids

• To: Kevin L Manross <kevin.manross@xxxxxxxx>
• Subject: Re: [thredds] Sparse Grids
• From: John Caron <caron@xxxxxxxxxxxxxxxx>
• Date: Fri, 19 Feb 2010 08:29:06 -0700

Kevin L Manross wrote:

Greetings!

For our CONUS grids, we make use of "sparse grids" in our netCDF files. From <http://cimms.ou.edu/~lakshman/WDSS2/dataformat.shtml> (about halfway down the page), our/the netCDF sparse grid format can be described as:

If your data are sparse, you can save a whole bunch of space by saving only the non-missing data in a simple run-length encoded format. The global attributes and the other variables are the same as the regular formats. However, the dimensions are provided by pixel_x, pixel_y for the first two dimensions and run_length for the number of values in a run (default=1 if run_length is omitted). The original dimensions are also required even if no variables actually use them. The default background value is MISSING_DATA unless a different value is provided (see the SparseRadialSet example below).

Example of a sparse lat-lon-grid:

dimensions:
        Lat = 650 ;
        Lon = 700 ;
        pixel = 23541 ;
variables:
        float Reflectivity_0C(pixel) ;
                Reflectivity_0C:Units = "dBZ" ;
        short pixel_x(pixel) ;
        short pixel_y(pixel) ;
        int pixel_count(pixel) ;

// global attributes:
                :TypeName = "Reflectivity_0C" ;
                :DataType = "SparseLatLonGrid" ;
                :Latitude = 37. ;
                :Longitude = -100. ;
                :Height = 0. ;
                :Time = 990376569 ;
                :FractionalTime = 0.584999999999127 ;
                :attributes = " ColorMap Unit" ;
                :ColorMap-unit = "dimensionless" ;
                :ColorMap-value = "Reflectivity" ;
                :Unit-unit = "dimensionless" ;
                :Unit-value = "dBZ" ;
                :LatGridSpacing = 0.01 ;
                :LonGridSpacing = 0.01 ;
                :MissingData = -99900.f ;
                :RangeFolded = -99901.f ;

I haven't dug far enough yet to see much on netCDF sparse grids. Considering that we were able to reduce a CONUS grid from 71 Mb to 4 Mb, we would like to continue to make use of sparse grids if we can.

Can anyone offer some guidance whether we can use THREDDS with sparse grid data? If we can, do I attempt to specify my coordinate transformations in the header, or is there more manipulation that I need to do on the data?

Thanks!!

-kevin.

TDS does not support sparse arrays. Have you considered using netcdf-4? I would 
expect the compression facilities there would give you similar size reduction, 
and it is transparent to all applications, including TDS. You should be able to 
relink your application with it, and make some minor modifications to your 
nc_create() call, and see what you get.

John

• References:
  • [thredds] Sparse Grids
    • From: Kevin L Manross