storage of bitmasks

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: storage of bitmasks
• From: whfs@xxxxxxxxxxxxxx (Walter H. F. Smith)
• Date: Thu, 9 Jan 1992 09:52:46 PST

Dear NetCDF gurus:

        I seek your wisdom concerning the storage of bit streams in NetCDF.
We have an application where we encode bits indicating T/F to the question,
"Does this NetCDF variable have data in this bin", where a bin is a 1 degree
by 1 degree "square" of latitude and longitude on the earth.  There are 360
x 180 = 64800 such squares, so we want to store 64800 bits.
        If we were implementing this so that it worked only on our own 
machine, we might encode the bit stream so it could be decoded something
like this:
        unsigned long int       flag[2025];     /* 2025 = 64800/32  */
        .....
        for (bin = 0; bin < 64800; bin++) {
                word = bin/32;
                bit = bin%32;
                this_bin_has_data = ((flag[word] >> bit) & 1));
        }
or something; whatever works.
We are writing the database in NetCDF and we want to include these bit streams
in the NetCDF files.  The question is how to design the storage and encode / 
decode operations so that they can work on any machine.  Note that in C (which
is the language we will use for this) we can do bitwise logical operations
and bitshifts only on integer types.  Note also that we require that +0 and
-0 are distinguishable.

If, for example, we used the (signed) NC_LONG type for this, we would be
able to store numbers between -2147483648 and +2147483647.  This is fine;
we can think of it as the range 0 thru pow(2,31), plus a sign bit, and the
convention that -0 is -2147483648.  But now if we make the assignment of
this NC_LONG to an int, or an unsigned int, will the results be the same
on all machines?  Remember that the location of the bits, including the sign
bit, is going to be interpreted as indicative of a geographic location, so
we have to be able to encode and decode this consistently across machines.
If we store the stream as (signed) NC_BYTE and make an assignment of these
data to unsigned char, will the result be the same on all machines?  In
particular, will some platforms interpret -0 as a error condition?

Thank you for your advice.

walter smith
whfs@xxxxxxxxxxxxxx