NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: [netcdfgroup] Type conversion when writing netCDF

My nc_put_vara_double() call writes the entire array at once; in this case, it sounds like the library does more or less the same thing as my algorithm outlined in the original message.
Breaking up the write into pieces would probably be a performance hit, and would make my code a bit more complicated, since deciding if splitting is necessary and how many splits to make is highly system dependent.
I guess I will have to live with being a memory hog -- if I document this feature, then it becomes the user's problem, not mine. But it seems to me that it is still better to let the library do the type conversion. Will the library return a meaningful error if it is unable to allocate the required memory? Have I overlooked any other costs/ benefits? 

--Jennifer



On Nov 29, 2010, at 10:51 AM, Rob Ross wrote:
So you will want to split up the writes into multiple writes to keep memory allocation down. Perhaps 1GB at a time, given what you describe below. Still a lot easier than doing the conversion by hand! 

Rob

On Nov 29, 2010, at 9:45 AM, Ed Hartnett wrote:


Jennifer Adams <jma@xxxxxxxxxxxxx> writes:
That's great! Does the library use a chunk of memory to do this? If my array of doubles is very large, say 20Gb, and I'm running on a system with 
24GB of memory, will it work?
--Jennifer
On Nov 29, 2010, at 9:32 AM, Jim Edwards wrote:



Memory will be allocated, but only enough to convert data for each
nc_put_vara call. So if you are writing 1 MB of values at a time, it
will allocated 1 MB at a time to convert, and free that 1 MB after the 
write operation is complete.
So it doesn't matter how large your variables are, only how much you try 
and write at one time.

Thanks,

Ed
--
Ed Hartnett  -- ed@xxxxxxxxxxxxxxxx

_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/




--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma@xxxxxxxxxxxxx
  • 2010 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: