RE: Apparent filesize limit?

• To: "Dave Allured" <dave.allured@xxxxxxxx>
• Subject: RE: Apparent filesize limit?
• From: "Thaden, John J" <ThadenJohnJ@xxxxxxxx>
• Date: Sun, 10 Sep 2006 22:21:45 -0500

Dave, 
   Thanks for your reply. I also heard as much from Pavel Michna, author
of a NetCDF interface with the R language, whom I cc'd on my original
email. I'm working now on a way to distribute the data into ~50 files in
such a way that, for my required analysis, I should only need to access
them one at a time.  
- John Thaden

-----Original Message-----
From: Dave Allured [mailto:dave.allured@xxxxxxxx] 
Sent: Sunday, September 10, 2006 2:55 PM
To: Thaden, John J
Cc: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: Re: Apparent filesize limit?

John,

My guess is that you have run into one of the Netcdf 3 limitations. 
  There is a way to have *one* single variable larger than 2^31 
bytes, but you must do several things properly.  Please read the 
details here and check that you are following all of the rules:

http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/NetCDF-Classic-F
ormat-Limitations.html

For data in excess of one gig, I recommend that the data remain in 
multiple files, and that some simple scheme be used to read in the 
file set at the start of application programs.

One such scheme which is common, sometimes called "templating", uses 
a consistent part of the filename to span multiple files.  E.g. 
data.1990.nc, data.1991.nc, etc.  So a one line "open" statement 
becomes a loop that's a few lines long, and filenames are 
constructed on the fly.  HTH.

--Dave A.
CIRES Climate Diagnostics Center (CDC)
NOAA/ESRL, Physical Sciences Division (PSD)

Thaden, John J wrote:
> Hello, I've created a NetCDF convention to amass into one 
> outputfile results contained in many (e.g. 50) NetCDF files
> (of the AnDI for spectrometry type).  Data in the new 
> convention include two 3-D arrays, with their unlimited 
> dimension being the number of input files. But these 
> conventions/types may be unimportant to my question.
> 
> After some coding false starts, I can now iterative through
> a list of input files and load data into the output file
> without obvious memory problems. But all stops at iteration  
> 38, when the output file just exceeds 2 gigabytes, 
> specifically, when it is 2,097,144 kilobytes. 
> 
> My question:  Have I run up against a NetCDF file size 
> Limitation?  Or is this some glitch in R or in Windows?
> 
> Thanks,
> -John
> 
> P.S.  I'm working in the R (v. 2.3.1) statistical programming 
> environment. I use Pavel Michna's RNetCDF (v.1.2-1) package 
> within R to access netcdf (v. 3) library functions.  This is 
> all happening on a Windows XP laptop with 512 M RAM. The command
> in my iteration that fails is "var.put.nc".  The error is 
> generated from a line "stop(nc$errmsg, call. = FALSE). The error
> message is "Invalid Argument".  -JT
> 
> John Thaden PhD
> University of Arkansas for Medical Sciences
> Little Rock AR, USA
> 
> Confidentiality Notice: This e-mail message, including any
attachments, is for the sole use of the intended recipient(s) and may
contain confidential and privileged information.  Any unauthorized
review, use, disclosure or distribution is prohibited.  If you are not
the intended recipient, please contact the sender by reply e-mail and
destroy all copies of the original message.
> 
> 
>
=======================================================================
======
> To unsubscribe netcdfgroup, visit:
> http://www.unidata.ucar.edu/mailing-list-delete-form.html
>
=======================================================================
======
> 


Confidentiality Notice: This e-mail message, including any attachments, is for 
the sole use of the intended recipient(s) and may contain confidential and 
privileged information.  Any unauthorized review, use, disclosure or 
distribution is prohibited.  If you are not the intended recipient, please 
contact the sender by reply e-mail and destroy all copies of the original 
message.


==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================