RE: what's the best method to compute weight value for arbitrary data format?

• To: "Fei Liu" <Fei.Liu@xxxxxxxx>, <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: RE: what's the best method to compute weight value for arbitrary data format?
• From: "Hanson, Kurt" <khanson@xxxxxxx>
• Date: Tue, 21 Mar 2006 14:24:12 -0500

Hi Fei and group --

In my experience the biggest bottleneck in situations like these is usually 
disk IO. So if we can tolerate having vectors of weights -- w(i) for all i, 
w(j) for all j, etc -- in memory, then we could use a recursive algorithm.

Signature something like:

void compute_weights(
    std::vector<int> cursor,
    const std::vector<const NcVar *> & inputWeights,
    std::vector<std::vector<float> *> inputWeightData,
    std::vector<std::vector<float> *> & outputWeightData);

The idea is that initially, the cursor specifies "invalid" (-1) indices for 
each of the inputWeights. The function has logic:
* when the number of specified dims is < (N-1), give the left-most unspecified 
index a value of 0. Read the inputWeights corresponding to this new index into 
memory (inputWeightData).
* else only the last dim is unspecified.
    * Read the inputWeights for it into memory (inputWeightData).
    * Compute and store the output weights for each index of the last dimension.
    * Find the "next" suite of dimensions to the left of the last. (Increment 
the 2nd to last, or if its at the end, then increment the 3rd to last and set 
the 2nd to last to 0, etc.)
    * Call the function recursively.

...a rough sketch.

If one can't tolerate a full set of input weight vectors in memory, then 
presumably this algorithm can be generalized such that for the M (0<M<N) 
left-most dimensions, the full weight vectors should not be read in, and 
instead, results are computed one index at a time.

Thoughts?

Kurt Hanson

-----Original Message-----
From: owner-netcdfgroup@xxxxxxxxxxxxxxxx
[mailto:owner-netcdfgroup@xxxxxxxxxxxxxxxx]On Behalf Of Fei Liu
Sent: Tuesday, March 21, 2006 11:24 AM
To: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: what's the best method to compute weight value for arbitrary
data format?



Hi group, sometimes there is a need to compute a weight array when 
averaging netCDF result. For example, for data(z,y,x)

weight(i,j,k) = wx(i)*wy(j)*wz(k)

but for data(z,x), it's
weight(i,k) = wx(i)*wz(k)

This variable weight has to be constructed during run time without any 
assumption of the underlying data organization (data(t,x), data(t,z) 
etc.). I have an algorithm to compute this variable dynamicsally but it's 
not very efficient. What suggestions do you have?

-- 
Fei Liu, PhD.                    phone: 609-4526551
RSIS/GFDL/NOAA                   fax:   609-9875063
P.O. Box 308, 201 Forrestal Rd.  http://www.gfdl.noaa.gov/~fil
Princeton, NJ 08542-0308         email: Fei.Liu@xxxxxxxx