NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: [netcdfgroup] nc_open takes a long time to open a big file

I have tried it again with netcdf-4.1.2-beta1; same results.
I also tried calling nc_set_chunk_cache before nc_open, with args size=409600 (and also 4096000), nelems=51203, and preemption=0.75. That didn't seem to change anything for nc_open. Always 20 seconds for a file the OS hasn't touched in a while. 

--Jennifer

p.s. from config.log:
configure:4219: checking whether a default chunk size in bytes was specified 
configure:4229: result: 4194304
configure:4238: checking whether a maximum per-variable cache size for HDF5 was specified 
configure:4248: result: 67108864
configure:4257: checking whether a number of chunks for the default per-variable cache was specified 
configure:4267: result: 10
configure:4276: checking whether a default file cache size for HDF5 was specified 
configure:4286: result: 4194304
configure:4295: checking whether a default file cache maximum number of elements for HDF5 was specified 
configure:4305: result: 1009
configure:4314: checking whether a default cache preemption for HDF5 was specified 
configure:4324: result: 0.75



On Sep 9, 2010, at 8:06 PM, Denis Nadeau wrote:


Hi Jennifer,
Did you call nc_set_chunk_cache? If not, do you know what the default cache size was set up to? (Look in your config.log created after you called configure) 
I must admit that 20 seconds is quite long.

Denis
From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx [mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx ] On Behalf Of Jennifer Adams 
Sent: Thursday, September 09, 2010 7:34 PM
To: netCDF Mail List
Subject: [netcdfgroup] nc_open takes a long time to open a big file

Dear Experts,
I'm using netcdf-4.1.1-rc1 and hdf5-1.8.4-patch1 on a 64-bit linux server running CentOS-5.5. 
I have a netcdf-4 file that is 18404502496 bytes large.

My file's dimensions look like this:
        lon = 320 ;
        lat = 160 ;
        lev = 11 ;
        time = 1581 ;

It has 7 variables that look like this:
   float temp(time, lev, lat, lon) ;
                temp:_Storage = "chunked" ;
                temp:_ChunkSizes = 1, 1, 160, 320 ;
                temp:_DeflateLevel = 1 ;
                temp:_Shuffle = "true" ;

and 1 variable that looks like this:
    float sfp(time, lat, lon) ;
                sfp:_Storage = "chunked" ;
                sfp:_ChunkSizes = 1, 160, 320 ;
                sfp:_DeflateLevel = 1 ;
                sfp:_Shuffle = "true" ;
Is it normal for nc_open to take 20 seconds to open this file before returning control to my C program? 

--Jennifer


--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma@xxxxxxxxxxxxx





--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma@xxxxxxxxxxxxx
  • 2010 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: