Re: midas

• To: HOWARD ARA T <Ara.Howard@xxxxxxxxxxxx>
• Subject: Re: midas
• From: John Caron <caron@xxxxxxxxxxxxxxxx>
• Date: Mon, 16 Oct 2000 14:38:53 -0600

_From caron@xxxxxxxxxxxxxxxx Mon Oct 16 14:38:05 2000
Received: from unidata.ucar.edu (charlie.unidata.ucar.edu [128.117.140.86])
        by unidata.ucar.edu (UCAR/Unidata) with ESMTP id e9GKc2420664;
        Mon, 16 Oct 2000 14:38:03 -0600 (MDT)
Organization: UCAR/Unidata
Keywords: 200010162038.e9GKc2420664
X-Mailer: Mozilla 4.73 [en] (Windows NT 5.0; U)
X-Accept-Language: en
MIME-Version: 1.0
References: <200010161625.e9GGPP403204@xxxxxxxxxxxxxxxx>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Hi Howard:

First, you might check that you have the latest version, as I released
some fixes last week:

  http://www.unidata.ucar.edu/staff/caron/ma2/ma2.htm

There's one important change which would only affect you if you use
Variable.read(). For consistency, read() no longer does rank reduction
automatically. If you want rank reduction, do Variable.read().reduce().


> 
> > we've been going over the new cdf lib and are really liking it.  i've a
> > routine that pulls an image out from a netcdf file and converts in to an
> > image.  this is a really expensive operation that i'd like to speed up,
> > but need to overcome two 'Array' problems first...
> >
> > 1)    i'd like do directly pull out a primitive [] from an Array vs.
> >         iterating though it and copying values.  as a 'c' programmer i'm
> >       thinking i'd like a reference to the first element that i could
> >       simply assign to a pointer eg.
> >
> >       int *my_primitive_array = &(Array.getIterator().getIntNext());
> >
> >         however, there doesn't seem to be an easy way to do this.  i'm
> >       going to disect the Array class and make public the inner data
> >       so i can access it in this fashion, unless you have a slicker
> >       method.

You can get access to the primitive array by casting to AbstractArray,
then using 

  public java.lang.Object copyTo1DJavaArray();   // Copy this array to a
1D Java primitive array of type getElementClass(), with the physical
order of the result the same as logical order.

Note this makes a copy. If you want to get the array without a copy, you
can use getStorage(). This is package private, so you may need to make
it public. This of course breaks encapsulation, which is a Bad Thing.
However, Im *very* intererested in performance consequences of these
designs, so I would like to work with you to understand your specific
problem and where the actual time is taken. Are you using a profiler?

You may be interested in section 3 of this paper that compares the
reletive costs of Array vs primitive[] access:

  http://www.unidata.ucar.edu/staff/caron/ma2/JavaGrande2000.htm

It turns out if you use concrete classes (eg Double.3D) you can iterate
through an Array about as fast as with double[], using the Hotspot JVM. 

> 
> > 2)    reshape seems to work find if the shape array is of the same rank
> >         as the orignal, but not if it's reshape'd into a 1-D array
> >
> >         what i am trying to do is stretch a 2-D array into a 1-D array
> >       for use with MemoryImageSource - which uses 1-D arrays.
> >
> >         eg.
> >                   a = a.reduce();  //remove any size '1' dimensions
> >
> >                   int [] shape = a.getShape();
> >                   int [] len = new int[1];
> >                   len[0] = 1;
> >
> >                 for(int i=0; i<shape.length; i++)
> >                     len[0]*=shape[i];

  you can call getSize() for this.

> >
> >                   a = a.reshape(len);
> >
> >           len contains the correct value... but crashes with this error:
> >
> >         my output....
> >         len[0] = 1077297      //this is correct 897*1201 sat image
> >
> >         Exception in thread "main"
> >         java.lang.IllegalArgumentException
> >         at ucar.ma2.MAMath.copyByte(MAMath.java:233)
> >         at ucar.ma2.MAMath.copy(MAMath.java:127)
> >         at ucar.ma2.ArrayAbstract.reshape(ArrayAbstract.java:278)
> >         at reshape.main(reshape.java:43)
> >
> > i am probably doing something dumb here... and i do have other code that
> > works, i'm just trying to  streamline it as much as possible, and would
> > like to aviod iterating thru the 'Array' at all.

Actually, I'm the one who (was) doing something stupid. This should be
fixed in the version I mentioned at the top. please let me know if it
still gives problems.

However, since you just want a 1D array, you probably dont need to use
reshape() at all, just the copyTo1DJavaArray() as mentioned above.

There are some good reasons to keep the iterators, and I've been
investigating ways to make them fast. I'd be happy to work with you to
see how to make your code as fast as possible within HotSpot.

> 
> I think this problem is caused by trying to overwrite `a' with
> `a.reshape(len)', which is an Array of a different shape hence of a
> different type than `a'.  You will have to store `a.reshape(len)' in
> a different Array that's declared to have the desired shape.

  a = a.reshape(len) should be ok, since a is only an object reference,
its just getting replaced with a new reference.


> 
> > as i said before, we are really impressed thus far with your code.  we
> > also checked out the GDV and though it was pretty cool.  similar to what
> > we'll be doing, but we'll be using sat images instead of filled contours
> > for background.  the applet has some MAJOR speed issues to deal with!!
> > making images out of netcdf is only one of them.
> 

Thanks for your encouragement. You are one of the first users of the new
API and your feedback is useful.

Basically, java 1.3 which includes HotSpot is a must. applets are
difficult because of browser environments. We are investigating WebStart
as a replacement for applets.