Re: Newbie question about VisAD and large datasets

• To: "Cunningham, Daniel" <cunninghamd@xxxxxxxx>
• Subject: Re: Newbie question about VisAD and large datasets
• From: Curtis Rueden <curtis@xxxxxxxxxxxxx>
• Date: Fri, 16 Apr 2004 14:00:12 -0500

Hi Daniel,

This is a significant issue for any visualization program that needs to display 
large datasets. Depending on the structure of your data, there are a variety of 
ways you could address it.

For example, my VisBio application deals with large collections of images. In 
order for the software to obtain smooth browsing speed, and to be scalable to 
data of extremely large size, it creates a collection of thumbnails, displaying 
the currently visible image(s) in low resolution during quick browsing. When 
idle, it goes to disk, loads the full resolution data, and displays it. When 
the user starts browsing again, any full resolution data is dumped from memory.

If your data can be expressed as FlatFields, you could check out 
visad.data.FileFlatField, which conserves memory by performing disk caching for 
large FlatFields.

In order to obtain only part of a data object from a large data file, some file 
adapters implement visad.data.FormBlockReader, which breaks up data files into 
logical blocks. The most common examples of FormBlockReader are in adapters 
like visad.data.tiff.TiffForm, where each TIFF file can contain multiple pages 
(i.e., images), with each block constituting one image.

Also, Bill has written a package called visad.cluster that allows for 
distributed data objects across a cluster of machines. Maybe Bill could comment 
on that in more detail if you're interested.

Of course, the dataset you described is ~100MB in size. Since VisAD does things 
in either 32-bit (float) or 64-bit (double) precision, this becomes ~200MB in 
practice. Factoring in display transform logic, user interface, etc., your 
program might require several hundred megabytes of memory to run. But machines 
today can easily have 1GB or 2GB of memory, so your 100MB dataset shouldn't be 
a problem. But since Java has a practical limit of ~1.5GB of usable memory on 
most architectures, a 500-1000MB dataset is probably too large to completely 
visualize, and you'd have to start using one of the techniques I mentioned 
above.

In any case, visualizing that much data at once may not be useful, since it's 
unlikely the screen resolution is good enough to actually display it all. Even 
if it were, the human eye and brain probably can't interpret such a multitude 
of data at such fine detail all at once. It would likely work just as well to 
"play tricks," displaying data at low detail until the user focuses on a 
specific area.

Cheers,
Curtis

At 10:33 AM 4/16/2004, Cunningham, Daniel wrote:
>Hello everyone, 
>
>I am considering using VisAD to develop some neuroimaging applications.  I was 
>curious as to how well VisAD deals with large datasets.  For example, a 16 bit 
>dataset with dimensions 64 x 64 x 24 x 576 is what I consider to be large.
>
>If anybody has experience using data of this size or larger, please comment. 
>
>Thanks, 
>Daniel Cunningham 

• References:
  • Newbie question about VisAD and large datasets
    • From: Cunningham, Daniel