Re: Problems with precision & ScalarMap

• To: Timo Thomas <tthomas@xxxxxxxxxxxxxxxxxxxxxxxx>, visad-list@xxxxxxxxxxxxx
• Subject: Re: Problems with precision & ScalarMap
• From: Don Murray <dmurray@xxxxxxxxxxxxxxxx>
• Date: Thu, 24 May 2001 12:42:00 -0600

Hi Timo-

An update on the precision problem.  I put a fix into AxisScale that
helps in calculating the axis tick marks which is where all the time was
being spent.  The problem was that I was using Math.round() on the
values and since your values were so large, I was getting some bogus
values.  I've put a corrected version of AxisScale.java at:

ftp://ftp.unidata.ucar.edu/pub/dmurray/AxisScale.java

and the next time the VisAD JAR files are updated, this change will be
included.  

However, after including my previous suggestion for setting the
DefaultSet to be a DoubleSet and using the AxisScale fix, I can get a
plot up quickly, but there is no data displayed.  I suspect there is
still a problem in precision deep in the bowels of VisAD (or else my
solution doesn't work), but Bill Hibbard or someone at SSEC will have to
look into that.  Bill is travelling this week so it may be next week
before you hear more on this.

Thanks for bringing this to our attention.

Don



*************************************************************
Don Murray                               UCAR Unidata Program
dmurray@xxxxxxxxxxxxxxxx                        P.O. Box 3000
(303) 497-8628                              Boulder, CO 80307
http://www.unidata.ucar.edu/staff/donm
*************************************************************

• References:
  • Problems with precision & ScalarMap
    • From: Timo Thomas
  • Re: Problems with precision & ScalarMap
    • From: Don Murray