NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Is this a netcdf-3 (classic) file or netcdf-4? =Dennis Heimbigner Unidata On 6/2/2016 1:12 PM, Burlen Loring wrote:
I am working on a climate analysis app. it must scan the dataset, comprised of many netcdf files to determine the available time steps. I'm finding that during reading the time axis from the file the performance is really bad. In one example the dataset is 10k files, each file has 8 time values, each value is 8 bytes, so 64 bytes are read per file. the time taken to read these arrays is between 0.3 and 2.2 seconds which puts the measured performance between 213 bytes per second and 29 bytes per second! I've had dial modems faster than that! reading other much larger arrays is much much faster, eg reading lon array with 1152 values only takes 7.7e-5 sec. I don't get it. I'd like some advise on this. On my workstation the read times of time values on the same dataset ranges between 1.9e-5 and 4.7e-5 sec. Thus something seems very wrong with the performance on the cray. btw this is on the lustre scratch2 file system and files are 433 MB each and striped across 24 ost's with a stripe size of 1MB. Burlen here is the raw data gathered from edison $/usr/common/graphics/teca/builds/TECA/bin/bin/teca_metadata_probe --input_regex /scratch2/scratchdirs/prabhat/TCHero/data/cam5_1_amip_run2'\.cam2\.h2.2005-09-[0-9][0-9]-10800.nc$' scan_files = 1.1513 open_file = 0.0967366 x_axis_metadata = 7.4109e-05 y_axis_metadata = 7.732e-06 z_axis_metadata = 3.84e-07 t_axis_metadata = 5.418e-06 variables = 0.000824452 read_x_axis = 5.9976e-05 read_y_axis = 1.0444e-05 read_z_axis = 4.294e-06 read_t_axis = 0.368079 open_file = 0.122236 t_axis_metadata = 4.906e-05 read_t_axis = 0.431482 open_file = 0.0953903 t_axis_metadata = 3.4205e-05 read_t_axis = 0.3815 open_file = 0.0853393 t_axis_metadata = 2.9607e-05 read_t_axis = 0.396472 open_file = 0.0664037 t_axis_metadata = 2.8239e-05 read_t_axis = 0.351707 open_file = 0.748844 t_axis_metadata = 3.3047e-05 read_t_axis = 1.03634 open_file = 0.161732 t_axis_metadata = 2.6955e-05 read_t_axis = 0.377919 open_file = 0.0820469 t_axis_metadata = 3.1613e-05 read_t_axis = 0.363014 open_file = 0.0903407 t_axis_metadata = 3.1844e-05 read_t_axis = 0.370521 open_file = 0.092586 t_axis_metadata = 2.9547e-05 read_t_axis = 0.37146 open_file = 0.083997 t_axis_metadata = 4.0095e-05 read_t_axis = 0.396375 open_file = 0.0799897 t_axis_metadata = 2.9833e-05 read_t_axis = 0.386237 open_file = 0.105688 t_axis_metadata = 0.000124456 read_t_axis = 0.481453 open_file = 0.0816038 t_axis_metadata = 3.0969e-05 read_t_axis = 0.355051 open_file = 0.101204 t_axis_metadata = 2.5927e-05 read_t_axis = 0.408186 open_file = 0.108662 t_axis_metadata = 2.6313e-05 read_t_axis = 0.314209 open_file = 0.0916682 t_axis_metadata = 2.9697e-05 read_t_axis = 0.319686 open_file = 0.0812744 t_axis_metadata = 2.7813e-05 read_t_axis = 0.391357 open_file = 0.101898 t_axis_metadata = 2.8607e-05 read_t_axis = 0.305515 open_file = 0.0748274 t_axis_metadata = 1.7247e-05 read_t_axis = 0.69522 open_file = 0.100151 t_axis_metadata = 1.7887e-05 read_t_axis = 0.511695 open_file = 0.0700686 t_axis_metadata = 1.8225e-05 read_t_axis = 2.27155 open_file = 0.121193 t_axis_metadata = 1.9918e-05 read_t_axis = 0.497648 open_file = 0.0734922 t_axis_metadata = 2.1134e-05 read_t_axis = 1.70601 open_file = 0.312652 t_axis_metadata = 1.888e-05 read_t_axis = 0.468489 open_file = 0.0906366 t_axis_metadata = 1.9165e-05 read_t_axis = 0.348474 open_file = 0.0776435 t_axis_metadata = 2.3417e-05 read_t_axis = 0.387956 open_file = 0.157729 t_axis_metadata = 4.9765e-05 read_t_axis = 0.344205 open_file = 0.114566 t_axis_metadata = 2.8431e-05 read_t_axis = 0.374615 open_file = 0.0788776 t_axis_metadata = 2.7865e-05 read_t_axis = 0.381856 collect_t_axis = 19.47 total = 20.72 A total of 240 steps available in 30 files. Using the noleap calendar. Times are specified in units of days since 1979-01-01 00:00:00. The available times range from 2005-9-1 3:0:0 (9733.12) to 2005-10-1 0:0:0 (9763). _______________________________________________ NOTE: All exchanges posted to Unidata maintained email lists are recorded in the Unidata inquiry tracking system and made publicly available through the web. Users who post to any of the lists we maintain are reminded to remove any personal information that they do not want to be made public. netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
netcdfgroup archives: