VisAD memory and speed performace with harge data set

• To: visad-list@xxxxxxxxxxxxx
• Subject: VisAD memory and speed performace with harge data set
• From: m huang <qomo@xxxxxxxxx>
• Date: Fri, 13 Dec 2002 17:50:51 -0800 (PST)

I am trying to find the optimum ways to use VisAD in two scenarios,
both of which are related to the change in existing arrays that
feed data into VisAD.

The first scenario is after I have fed VisAD with some data stored
in Java primitive arrays, and have made the data to be rendered on
a display, the data in these array changes. So I need to update the
display to replect the data change. I find that if I re-run the
flatField.setSamples() with the data arrays the corresponding changes
in the display will be made.
I would like to know if this is supposed to be the way to change the
display?

The second scenario is that I have fixed sized huge arrays. To
implify
the case suppose the data arrays are 1D double type for x and y,
represented by (x -> y) or (index->(x,y)) If I want to look at part
of
the data set, I can set the range in RangeMap of x (or set range of
index in ( index->(x,y)). My problem is that the part of x and y
arrays not being looked at are still processed by VisAD (e.g. when
looking for Max and Min when the data is set) and result extra memory
allocation inside VisAD. 

In a special case, if I only want to look at the first n points of
(x->y) where n is smaller than the length of x and y arrays, I
thought
I could use this to get a subset of x into x_set:

...
x_set=new Gridded1DDoubleSet(type, new double[][]{x_array}, n);
...

But I get an exception saying that `` visad.SetException:
visad.SetException: Gridded1DDoubleSet.init_doubles: samples
[0] length [x array length] doesn't match expected length [n]''.
What is the point of n if n is not allowed to be smaller than
the length of x ? and even if I do set "copy" to true, whereas
a new array is supposed be created and values copied from x, this
exception still happens (from reading source code). Can anyone
explain what is the idea behind the last parameter in constructor
Gridded1DDoubleSet(MathType type, float[][] samples, int lengthX)  ?

So for the second scenario, isn't there a generic way to tell 
VisAD only to look part of the data and not to process or
allocate memory for the part that is not being looked at?

Now for both scenarios, I find that the setSamples() in FlatField
actually ignores the "copy" argument. Why is the copy argument
"meaningless" (per JavaDoc) in FlatField? If really meaningless, why
is the argument there? Since FlatFlied is such a fundamental
data structure in VisAD, does this mean if anything in the orginal
primitive data array changes, be it data value  or the
subset of interest, VisAD MUST throw away the old FlatField and
whatever internal data structures created to plot the old data,
and make a new ones? I also notice that assignment loops are used
to copy arrays in FladField.java, even when copying the same type
of array. Why isn't System.arraycopy() used ?

If the original data structure is simple enough, is it possible 
to only pass the reference of the original data all the way through
until perhaps the screen coordinates are needed to be calculated?

To estimate memory use, in the above (x->y) case, how many
times more momory does VisAD  use in order to plot it? Are
there internal buffers that I haven't noticed in this post?

Unless I missed something big in VisAD (which happens often :-) 
the performance hit can be a show stopper in semi-realtime
situation such as plotting time series where there are new
data points available and appended to existing data set
at even modest rate.

Thanks for any insight.

__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

• Follow-Ups:
  • Re: VisAD memory and speed performace with harge data set
    • From: Bill Hibbard