NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
There is an option to nccopy, -u option, that copies a file while converting all unlimited dimensions to fixed size. As an experiment, you might nccopy -u your file and then rerun you performance tests to see if they improve. =Dennis Heimbigner Unidata On 6/2/2016 2:35 PM, Burlen Loring wrote:
That sounds like it, ncdump on 1 file shows "time = UNLIMITED ; // (8 currently)" it's kind of unexpected that these 8 values not be in a contiguous array! Oh well. Thanks for clarifying. This is simulation output, so our options may be limited. I will be sure to mention this to the scientists. Hopefully they can write them as a fixed dimension. On 06/02/2016 01:24 PM, Bowman, Kenneth P wrote:Hi Burlen, If time is your unlimited (record) dimension, then the time values are scattered through the 433 MB file. That is true for any variables that have a time dimension. To read the time variable, the netCDF library has to jump through the file and collect the values. The longitude variable is contiguous in memory and can be read quickly. If you know the number of time steps in the file before you write the file, you can change the unlimited time dimension to a fixed dimension. Then something dimensioned by (only) time will be contiguous in memory. Or you can rewrite the files with fixed dimensions. That read performance penalty is one of the tradeoffs of having the flexibility of an unlimited dimension. Good luck! Ken Date: Thu, 2 Jun 2016 12:41:53 -0700 From: Burlen Loring <bloring@xxxxxxx <mailto:bloring@xxxxxxx>> To: Tom Fogal <tfogal@xxxxxxxxxxxx <mailto:tfogal@xxxxxxxxxxxx>>, <mailto:netcdfgroup@xxxxxxxxxxxxxxxx>netcdfgroup@xxxxxxxxxxxxxxxx Subject: Re: [netcdfgroup] abysmal performance Message-ID: <961631fd-2aad-d348-ce1d-8a70a9e67287@xxxxxxx <mailto:961631fd-2aad-d348-ce1d-8a70a9e67287@xxxxxxx>> Content-Type: text/plain; charset=windows-1252; format=flowed Hi Tom, That's not an option, and it has it's own issues. for example if file size exceeds the size of a tape drive we can't archive it. Beside it doesn't seem like a lustre metadata issue, open is relatively fast, like 0.096 sec. and wouldn't explain why reading the time dimension with only 8 values takes on the order of 1 sec while reading the lon dimension with 1152 values takes on the order of 1e-4 sec. ? Burlen ----------------------------------------------------------------------------- Dr. Kenneth P. Bowman 1014A Eller Building David Bullock Harris Professor of Geosciences 979-862-4060 Department of Atmospheric Sciences 979-862-4466 fax Texas A&M University 3150 TAMU College Station, TX 77843-3150 _http://atmo.tamu.edu/people/faculty/bowmankenneth.html________________________________________________ NOTE: All exchanges posted to Unidata maintained email lists are recorded in the Unidata inquiry tracking system and made publicly available through the web. Users who post to any of the lists we maintain are reminded to remove any personal information that they do not want to be made public. netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
netcdfgroup archives: