NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: [netcdfgroup] Error with large filesizes

On Oct 27, 2014, at 8:46 PM,

However, you don't want chunks to be very small or very large just for the
> sake of efficiency.  The ideal size and shape for a chunk is dictated by
> what access patterns you anticipate to be common, to make each access one
> chunk.
>

Indeed, but I suspect that you'd hardly notice the difference between large
and really large chunk sizes. For instance a chunk larger that your disk
cache would probably buy you nothing. What's a typical disk cache these
days? I have no idea.

But I'll guess you won't see a noticeable drop in performance for a
multi-gigabyte read if you use a 10MB or so chunk size.

If you do any testing, please post your results here.

-Chris



> --Russ
>
> On Mon, Oct 27, 2014 at 12:13 AM, Samrat Rao <samrat.rao@xxxxxxxxx> wrote:
>
>>
>> Hi,
>>
>> I am using netCDF-fortran-4.1.1 with netCDF-4.3.2. I get an error 'Bad
>> chunk sizes' when the 3D data size is more than a certain number.
>>
>> I am unable to understand the cause of this error.
>>
>> My OS is CentOS 6.6. The errors are the same even if is use netCDF (v
>> 4.1.1) supplied by the CentOS community.
>>
>> i am attaching a minimal example. Please let me know if there are any
>> errors from my side.
>>
>> Thanks,
>> Samrat.
>>
>> --
>>
>> Samrat Rao
>> Research Associate
>> Engineering Mechanics Unit
>> Jawaharlal Centre for Advanced Scientific Research
>> Bangalore - 560064, India
>>
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe,  visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>
>


-- 

Samrat Rao
Research Associate
Engineering Mechanics Unit
Jawaharlal Centre for Advanced Scientific Research
Bangalore - 560064, India

_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit:
http://www.unidata.ucar.edu/mailing_lists/
  • 2014 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: