NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi Jennifer, > I have tried it again with netcdf-4.1.2-beta1; same results. > > I also tried calling nc_set_chunk_cache before nc_open, with args > size=409600 (and also 4096000), nelems=51203, and preemption=0.75. > That didn't seem to change anything for nc_open. Always 20 seconds for > a file the OS hasn't touched in a while. This looks like a bug. We're occupied in project reviews this morning, but will dig into this further this afternoon. Thanks for reporting the problem! --Russ > p.s. from config.log: > configure:4219: checking whether a default chunk size in bytes was > specified > configure:4229: result: 4194304 > configure:4238: checking whether a maximum per-variable cache size for > HDF5 was specified > configure:4248: result: 67108864 > configure:4257: checking whether a number of chunks for the default > per-variable cache was specified > configure:4267: result: 10 > configure:4276: checking whether a default file cache size for HDF5 > was specified > configure:4286: result: 4194304 > configure:4295: checking whether a default file cache maximum number > of elements for HDF5 was specified > configure:4305: result: 1009 > configure:4314: checking whether a default cache preemption for HDF5 > was specified > configure:4324: result: 0.75 > > > > On Sep 9, 2010, at 8:06 PM, Denis Nadeau wrote: > > > Hi Jennifer, > > > > Did you call nc_set_chunk_cache? If not, do you know what the > > default cache size was set up to? (Look in your config.log created > > after you called configure) > > I must admit that 20 seconds is quite long. > > > > Denis > > > > > > From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx [mailto:netcdfgroup-bounces@unid > ata.ucar.edu > > ] On Behalf Of Jennifer Adams > > Sent: Thursday, September 09, 2010 7:34 PM > > To: netCDF Mail List > > Subject: [netcdfgroup] nc_open takes a long time to open a big file > > > > Dear Experts, > > I'm using netcdf-4.1.1-rc1 and hdf5-1.8.4-patch1 on a 64-bit linux > > server running CentOS-5.5. > > I have a netcdf-4 file that is 18404502496 bytes large. > > > My file's dimensions look like this: > > lon = 320 ; > > lat = 160 ; > > lev = 11 ; > > time = 1581 ; > > > > It has 7 variables that look like this: > > float temp(time, lev, lat, lon) ; > > temp:_Storage = "chunked" ; > > temp:_ChunkSizes = 1, 1, 160, 320 ; > > temp:_DeflateLevel = 1 ; > > temp:_Shuffle = "true" ; > > > > and 1 variable that looks like this: > > float sfp(time, lat, lon) ; > > sfp:_Storage = "chunked" ; > > sfp:_ChunkSizes = 1, 160, 320 ; > > sfp:_DeflateLevel = 1 ; > > sfp:_Shuffle = "true" ; > > > > Is it normal for nc_open to take 20 seconds to open this file before > > returning control to my C program? > > > > --Jennifer > > > > > > -- > > Jennifer M. Adams > > IGES/COLA > > 4041 Powder Mill Road, Suite 302 > > Calverton, MD 20705 > > jma@xxxxxxxxxxxxx > > > > > > > > -- > Jennifer M. Adams > IGES/COLA > 4041 Powder Mill Road, Suite 302 > Calverton, MD 20705 > jma@xxxxxxxxxxxxx > > > > > --Apple-Mail-5-829452394 > Content-Type: text/html; > charset=US-ASCII > Content-Transfer-Encoding: quoted-printable > > <html><body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = > -webkit-line-break: after-white-space; ">I have tried it again with = > netcdf-4.1.2-beta1; same results. <div><br></div><div>I also tried = > calling nc_set_chunk_cache before nc_open, with args size=3D409600 (and = > also 4096000), nelems=3D51203, and preemption=3D0.75. That didn't seem = > to change anything for nc_open. Always 20 seconds for a file the OS = > hasn't touched in a = > while. <div><br></div><div>--Jennifer</div><div><br></div><div>p.s. = > from config.log:</div><div><div><div><div><font class=3D"Apple-style-span"= > face=3D"Courier" size=3D"3"><span class=3D"Apple-style-span" = > style=3D"font-size: 12px;">configure:4219: checking whether a default = > chunk size in bytes was specified</span></font></div><div><font = > class=3D"Apple-style-span" face=3D"Courier" size=3D"3"><span = > class=3D"Apple-style-span" style=3D"font-size: 12px;">configure:4229: = > result: 4194304</span></font></div><div><font class=3D"Apple-style-span" = > face=3D"Courier" size=3D"3"><span class=3D"Apple-style-span" = > style=3D"font-size: 12px;">configure:4238: checking whether a maximum = > per-variable cache size for HDF5 was = > specified</span></font></div><div><font class=3D"Apple-style-span" = > face=3D"Courier" size=3D"3"><span class=3D"Apple-style-span" = > style=3D"font-size: 12px;">configure:4248: result: = > 67108864</span></font></div><div><font class=3D"Apple-style-span" = > face=3D"Courier" size=3D"3"><span class=3D"Apple-style-span" = > style=3D"font-size: 12px;">configure:4257: checking whether a number of = > chunks for the default per-variable cache was = > specified</span></font></div><div><font class=3D"Apple-style-span" = > face=3D"Courier" size=3D"3"><span class=3D"Apple-style-span" = > style=3D"font-size: 12px;">configure:4267: result: = > 10</span></font></div><div><font class=3D"Apple-style-span" = > face=3D"Courier" size=3D"3"><span class=3D"Apple-style-span" = > style=3D"font-size: 12px;">configure:4276: checking whether a default = > file cache size for HDF5 was specified</span></font></div><div><font = > class=3D"Apple-style-span" face=3D"Courier" size=3D"3"><span = > class=3D"Apple-style-span" style=3D"font-size: 12px;">configure:4286: = > result: 4194304</span></font></div><div><font class=3D"Apple-style-span" = > face=3D"Courier" size=3D"3"><span class=3D"Apple-style-span" = > style=3D"font-size: 12px;">configure:4295: checking whether a default = > file cache maximum number of elements for HDF5 was = > specified</span></font></div><div><font class=3D"Apple-style-span" = > face=3D"Courier" size=3D"3"><span class=3D"Apple-style-span" = > style=3D"font-size: 12px;">configure:4305: result: = > 1009</span></font></div><div><font class=3D"Apple-style-span" = > face=3D"Courier" size=3D"3"><span class=3D"Apple-style-span" = > style=3D"font-size: 12px;">configure:4314: checking whether a default = > cache preemption for HDF5 was specified</span></font></div><div><font = > class=3D"Apple-style-span" face=3D"Courier" size=3D"3"><span = > class=3D"Apple-style-span" style=3D"font-size: 12px;">configure:4324: = > result: = > 0.75</span></font></div><div><br></div></div></div></div><div><br></div><d= > iv><br></div><div><div><div><div><div>On Sep 9, 2010, at 8:06 PM, Denis = > Nadeau wrote:</div><br class=3D"Apple-interchange-newline"><blockquote = > type=3D"cite"><div lang=3D"EN-US" link=3D"blue" vlink=3D"purple" = > style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = > -webkit-line-break: after-white-space; "><div class=3D"WordSection1" = > style=3D"page: WordSection1; "><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; "><span style=3D"font-size: = > 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Hi = > Jennifer,<o:p></o:p></span></div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; "><span style=3D"font-size: = > 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); = > "><o:p> </o:p></span></div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; "><span style=3D"font-size: = > 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Did = > you call nc_set_chunk_cache? If not, do you know what the default = > cache size was set up to? (Look in your config.log created after = > you called configure)<o:p></o:p></span></div><div style=3D"margin-top: = > 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; = > font-size: 12pt; font-family: 'Times New Roman', serif; "><span = > style=3D"font-size: 11pt; font-family: Calibri, sans-serif; color: = > rgb(31, 73, 125); ">I must admit that 20 seconds is quite = > long.<o:p></o:p></span></div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; "><span style=3D"font-size: = > 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); = > "><o:p> </o:p></span></div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; "><span style=3D"font-size: = > 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); = > ">Denis<o:p></o:p></span></div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; "><span style=3D"font-size: = > 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); = > "><o:p> </o:p></span></div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; "><span style=3D"font-size: = > 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); = > "><o:p> </o:p></span></div><div><div style=3D"border-right-style: = > none; border-bottom-style: none; border-left-style: none; border-width: = > initial; border-color: initial; border-top-style: solid; = > border-top-color: rgb(181, 196, 223); border-top-width: 1pt; = > padding-top: 3pt; padding-right: 0in; padding-bottom: 0in; padding-left: = > 0in; "><div style=3D"margin-top: 0in; margin-right: 0in; margin-bottom: = > 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New = > Roman', serif; "><b><span style=3D"font-size: 10pt; font-family: Tahoma, = > sans-serif; ">From:</span></b><span style=3D"font-size: 10pt; = > font-family: Tahoma, sans-serif; "><span = > class=3D"Apple-converted-space"> </span><a = > href=3D"mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx";>netcdfgroup-bounces@u= > nidata.ucar.edu</a> [<a = > href=3D"mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx"; style=3D"color: = > blue; text-decoration: underline; = > ">mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx</a>]<span = > class=3D"Apple-converted-space"> </span><b>On Behalf Of<span = > class=3D"Apple-converted-space"> </span></b>Jennifer = > Adams<br><b>Sent:</b><span = > class=3D"Apple-converted-space"> </span>Thursday, September 09, = > 2010 7:34 PM<br><b>To:</b><span = > class=3D"Apple-converted-space"> </span>netCDF Mail = > List<br><b>Subject:</b><span = > class=3D"Apple-converted-space"> </span>[netcdfgroup] nc_open takes = > a long time to open a big file<o:p></o:p></span></div></div></div><div = > style=3D"margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; = > margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', = > serif; "><o:p> </o:p></div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; ">Dear = > Experts,<o:p></o:p></div><div><div><div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; ">I'm using = > netcdf-4.1.1-rc1 and hdf5-1.8.4-patch1 on a 64-bit linux server = > running CentOS-5.5.<o:p></o:p></div></div><div><div style=3D"margin-top: = > 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; = > font-size: 12pt; font-family: 'Times New Roman', serif; ">I have a = > netcdf-4 file that is 18404502496 bytes = > large. <o:p></o:p></div></div><div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; = > "><o:p> </o:p></div></div><div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; ">My file's dimensions look = > like this:<o:p></o:p></div></div><div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; "> = > lon =3D 320 ;<o:p></o:p></div></div><div><div = > style=3D"margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; = > margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', = > serif; "> lat =3D 160 = > ;<o:p></o:p></div></div><div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; "> = > lev =3D 11 ;<o:p></o:p></div></div><div><div = > style=3D"margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; = > margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', = > serif; "> time =3D 1581 = > ;<o:p></o:p></div></div><div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; = > "><o:p> </o:p></div></div><div><div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; ">It has 7 variables that = > look like this:<o:p></o:p></div></div><div><div style=3D"margin-top: = > 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; = > font-size: 12pt; font-family: 'Times New Roman', serif; "> = > float temp(time, lev, lat, lon) ;<o:p></o:p></div></div><div><div = > style=3D"margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; = > margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', = > serif; "> = > temp:_Storage =3D "chunked" ;<o:p></o:p></div></div><div><div = > style=3D"margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; = > margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', = > serif; "> = > temp:_ChunkSizes =3D 1, 1, 160, 320 = > ;<o:p></o:p></div></div><div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; "> = > temp:_DeflateLevel =3D 1 = > ;<o:p></o:p></div></div><div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; "> = > temp:_Shuffle =3D "true" = > ;<o:p></o:p></div></div></div><div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; = > "><o:p> </o:p></div></div><div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; ">and 1 variable that looks = > like this: <o:p></o:p></div></div><div><div><div style=3D"margin-top:= > 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; = > font-size: 12pt; font-family: 'Times New Roman', serif; "> = > float sfp(time, lat, lon) ;<o:p></o:p></div></div><div><div = > style=3D"margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; = > margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', = > serif; "> = > sfp:_Storage =3D "chunked" ;<o:p></o:p></div></div><div><div = > style=3D"margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; = > margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', = > serif; "> = > sfp:_ChunkSizes =3D 1, 160, 320 ;<o:p></o:p></div></div><div><div = > style=3D"margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; = > margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', = > serif; "> = > sfp:_DeflateLevel =3D 1 ;<o:p></o:p></div></div><div><div = > style=3D"margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; = > margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', = > serif; "> = > sfp:_Shuffle =3D "true" ;<o:p></o:p></div></div></div><div><div = > style=3D"margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; = > margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', = > serif; "><o:p> </o:p></div></div><div><div style=3D"margin-top: = > 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; = > font-size: 12pt; font-family: 'Times New Roman', serif; ">Is it normal = > for nc_open to take 20 seconds to open this file before returning = > control to my C program? <o:p></o:p></div></div><div><div = > style=3D"margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; = > margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', = > serif; "><o:p> </o:p></div></div><div><div style=3D"margin-top: = > 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; = > font-size: 12pt; font-family: 'Times New Roman', serif; = > ">--Jennifer<o:p></o:p></div></div><div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; = > "><o:p> </o:p></div></div><div><div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; = > "><o:p> </o:p></div><div><div><div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; "><span style=3D"font-size: = > 9pt; ">--<o:p></o:p></span></div></div><div><div style=3D"margin-top: = > 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; = > font-size: 12pt; font-family: 'Times New Roman', serif; "><span = > style=3D"font-size: 9pt; ">Jennifer M. = > Adams<o:p></o:p></span></div></div><div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; "><span style=3D"font-size: = > 9pt; ">IGES/COLA<o:p></o:p></span></div></div><div><div = > style=3D"margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; = > margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', = > serif; "><span style=3D"font-size: 9pt; ">4041 Powder Mill Road, Suite = > 302<o:p></o:p></span></div></div><div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; "><span style=3D"font-size: = > 9pt; ">Calverton, MD 20705<o:p></o:p></span></div></div><div><div = > style=3D"margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; = > margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', = > serif; "><span style=3D"font-size: 9pt; "><a = > href=3D"mailto:jma@xxxxxxxxxxxxx"; style=3D"color: blue; text-decoration: = > underline; = > ">jma@xxxxxxxxxxxxx</a><o:p></o:p></span></div></div><div><div = > style=3D"margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; = > margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', = > serif; "><span style=3D"font-size: 9pt; = > "><o:p> </o:p></span></div></div><div style=3D"margin-top: 0in; = > margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: = > 12pt; font-family: 'Times New Roman', serif; "><span style=3D"font-size: = > 9pt; "><o:p> </o:p></span></div></div></div><div style=3D"margin-top:= > 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; = > font-size: 12pt; font-family: 'Times New Roman', serif; = > "><o:p> </o:p></div></div></div></div></div></div></div></blockquote>= > </div><br><div apple-content-edited=3D"true"> <div style=3D"word-wrap: = > break-word; -webkit-nbsp-mode: space; -webkit-line-break: = > after-white-space; font-size: 12px; "><div>--</div><div>Jennifer M. = > Adams</div><div>IGES/COLA</div><div>4041 Powder Mill Road, Suite = > 302</div><div>Calverton, MD 20705</div><div><a = > href=3D"mailto:jma@xxxxxxxxxxxxx";>jma@xxxxxxxxxxxxx</a></div><div><br = > class=3D"khtml-block-placeholder"></div><br = > class=3D"Apple-interchange-newline"></div> = > </div><br></div></div></div></div></body></html>= > > --Apple-Mail-5-829452394-- > > > --===============0634312372== > Content-Type: text/plain; charset="us-ascii" > MIME-Version: 1.0 > Content-Transfer-Encoding: 7bit > Content-Disposition: inline > > _______________________________________________ > netcdfgroup mailing list > netcdfgroup@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/m > ailing_lists/ > --===============0634312372==--
netcdfgroup archives: