Re: VisAD memory and speed performace with large data set / etc

• To: Bill Hibbard <billh@xxxxxxxxxxxxx>
• Subject: Re: VisAD memory and speed performace with large data set / etc
• From: m huang <qomo@xxxxxxxxx>
• Date: Sat, 14 Dec 2002 13:28:08 -0800 (PST)

Hi Bill,
Thanks for the reply.

--- Bill Hibbard <billh@xxxxxxxxxxxxx> wrote:
> On Fri, 13 Dec 2002, m huang wrote:
> 
> > So for the second scenario, isn't there a generic way to tell
> > VisAD only to look part of the data and not to process or
> > allocate memory for the part that is not being looked at?
> 
> You are free to create a custom DataRenderer (see tutorial)
> that handles your special case more efficiently.

OK. I have been chipping away the DataRenderer tutorial.
I thought I might need a custom dataRenderer but it
is not clear to me what the limits are for the existing ones.
I will be back with more specific questions.

> 
> > Now for both scenarios, I find that the setSamples() in FlatField
> > actually ignores the "copy" argument. Why is the copy argument
> > "meaningless" (per JavaDoc) in FlatField? If really meaningless,
> why
> > is the argument there?
> 
> This is not true. If you pass a double[][] array to a
> FlatField with the default FloatSet range Set, indicating
> internal storage as a float[][] array, then it must copy to
> convert the doubles to floats. But if copy = false and it
> can avoid copying, it will.
> 

Thanks for the clarification. I saw this in FlatField.java:
...
   * @param  copy    flag to copy values - meaningless for FlatField
   * ...
   */
  public void setSample(int index, Data range, boolean copy)
...

I wondered if the VisAD developers meant this to be copy 
was "meaningless" everywhere in FlatField. JavaDoc of
setSamples(double[][] range, boolean copy) does say "copy array
if copy flag is true", but after seeing the "meaningless" clause,
I thought I should get an confirmation from you guys that "do
not copy array if copy flag is not true"! Sorry to be so fussy
but I think it's prudent to regard the documentation as the 
only "legal" contract between the API developer and the API users.

> > I also notice that assignment loops are used
> > to copy arrays in FladField.java, even when copying the same type
> > of array. Why isn't System.arraycopy() used ?
> 
> Not true.

You are right. The places I saw assignment loops all turn out
to be between different types.

You pointed out that VisAD tries not to duplicate data as
much as possible. That is very encouraging to know.

> The
> other thing to keep in mind is that current performance
> problems will be long forgotten by 2007. Designs aimed for
> 5 years in the future should focus on adequate generality
> and flexibility to adapt to new needs, rather than current
> performance problems.
> 

Although the launch Herschel is scheduled in 2007, the software
development is driven by milestones. The hardware are being
fabricated and tested now. The IA software framework is intend
to support them. So we actually need something working soon
(read yesterday :).

I totally agree with you that flexibility is important.
But it can be hard to see for people who haven't experienced
what you must have when developing VisAD.
 
I attended Herschel IA working group meeting last
week. We need to decide on the software framework early next
year. That is the next milestone, after which the framework of
IA is largely frozen. Because the people who evaluated VisAD
found VisAD was difficult to understand and more so even to make
a prototype, a Herschel IA dataSet frame work is proposed without
structural consideration to use VisAD, yet it has abstract data model
that supports array of tuples, unit conversion, error propagation,
can be plotted in "layers" etc etc...  lots of what VisAD excels
in doing. This makes me wonder how many more projects out there
are trying to reinvent VisAD because they couldn't understand
what's already in VisAD. 

Because no one in the meeting really understandd VisAD much,
with my best effort I could only persuade my colleagues to send
me some key requirement for Herschel IA and I will tell them,
as best as I can, if/how these requirement can be met by using
VisAD, in the next month.  Then based on that we will re-evaluate
our situation. If anyone thinks I could better RTFM befor asking
some of my questions in the list, I appologize. I have a
short time line to "sell" VisAD.
I also persuaded some of my colleagues to take a closer look at
VisAD at the same time, and encourage them to ask questions here.



__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

• Follow-Ups:
  • Re: VisAD memory and speed performace with large data set / etc
    • From: Bill Hibbard

• References:
  • Re: VisAD memory and speed performace with harge data set
    • From: Bill Hibbard