NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #YRD-168110]: Multiple Section reads appear to be skipping data? (java)

Hi Clifford,

> So lets say I have NetCDF file that I am trying to read with a Section that
> looks like this:
> 
> 0:0, 0:0, 0:719, 0:1439
> 
> When I read this entire section at once, I get back 1036800 elements
> clocking in at 4147200 bytes. So I says to myself, I says 'what if I split
> the read up into multiple sections, and process the partial Array results
> one at a time, so that I'm not holding a reference to ~4mb of memory".
> 
> So I split the read up such that it looks like the following (this is the
> actual toString() result of each Section immediately before the read takes
> place)
> 
> 0:0, 0:0, 0:359, 0:719
> 
> 0:0, 0:0, 360:719, 720:1439
> 
> BUT, the results from each of these arrays combined totals only
> 518400 elements weighing in at 2073600 bytes - I'm basically missing half
> the data here!

It appears that you have only requested half the data.  To get all the data,
you either need the four sections specified by

 0:0, 0:0, 0:359, 0:719
 0:0, 0:0, 0:359, 720:1439
 0:0, 0:0, 360:719, 0:719
 0:0, 0:0, 360:719, 720:1439

or you need to make each of the two sections request half the data:

 0:0, 0:0, 0:359, 0:1439
 0:0, 0:0, 360:719, 0:1439

--Russ

> Whats more, If I make the Sections smaller and increase the number of
> variable reads in an attempt to reduce memory usage even further, I lose
> even more data!
> 
> What am I doing wrong here? If Sections are not appropriate for 'chunking'
> data reads, is there a mechanism in the library that is more suited to this
> task?
> --
> Clifford M. Harms
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: YRD-168110
Department: Support netCDF
Priority: Normal
Status: Closed