NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Packing and Grib to netCDF conversion

We have thought a lot about packing in netCDF and considered various schemes on how to do it. Variable length compression schemes would probably interfere with efficient subsetting, but it seems to me that fixed bit size packing should be doable. We have this on our wish list, i mean to-do list, for the next round of netCDF development.
since COARDS is lat/lon only, you cant do arbitrary GRIB conversion. CF seems like a good candidate, however. Has anyone investigated the feasibility of this?
gribtonc uses NUWG, and has various limitations. We are interested in possibly upgrading gribtonc. If anyone can make use of more flexible GRIB to netCDF conversion, I'd like to hear about your "use-case".
I think in order to correctly group the GRIB records into 3 or 4 dimensional netCDF variables, you will need (for the general case) some sort of configuration info for the converter, although I suppose a "common case" could be assumed. Anyone have any thoughts on that? 


Timothy Hume wrote:


Hi,

This discussion reminded me of how GRIB packs data. Ideally, it would be
nice for NetCDF to be able to handle data with an arbitrary number of
bits. Many meteorological data can be packed into only 9 or 10 bits (often
less), so packing them into 16 bit short integers is "wasteful". Aside
from that many satellite data are "naturally" 10 bit, and increasing them
to 16 bits can cause the file size to increase by tens of megabytes per
image.

By the way, does anyone know of software that can convert GRIB data to
COARDS or CF conventions? gribtonc converts GRIB to NUWG conventions?

Tim Hume

By the way, does anyone know of a GRIB to COARDS or CF

On Wed, 2 Oct 2002, Mark A Ohrenschall wrote:


Hello,

In the case of a packed variable (in which scale_factor and add_offset
are used) both the COARDS and CF conventions indicate that missing_value
and _FillValue should be likewise packed:

COARDS: "In cases where the data variable is packed via the scale_value
attribute this implies that the missing_value flag is likewise packed."
CF: "The missing values of a variable with scale_factor and/or
add_offset attributes (see section 8.1) are interpreted relative to the
variable's external values, i.e., the values stored in the netCDF file."

I'm assuming that for the sake of consistency, this means that all
statistical variable attributes should be packed as well, e.g.,
valid_range and actual_range, as well as mean and standard_deviation. Is
this true?

So for example, if I have real world data values for temperature between
-1.6 and 31.4 and I'm applying a scale_factor of 0.1 then I would say
the valid_range is -16, 314 and the mean is 116 (not 11.6)?

Thanks,

Mark
  • 2002 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: