NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi John: On Apr 25, 2011, at 12:42 PM, John Caron wrote: > On 4/25/2011 1:37 PM, Roy Mendelssohn wrote: >> yes, internal compression. All the files were made from netcdf3 files using >> NCO with the options: >> >> ncks -4 -L 1 >> >> The results so far show a decrease in file size from 40% of original to >> 1/100 th of the original file size. If the internally compressed data >> requests are cached differently than request to netcdf3 files, we want to >> take that into account when we do the tests, so that we do not just see the >> affect of differential cacheing. >> >> When we have done tests on just local files, the reads where about 8 times >> slower from a compressed file. But Rich Signell has found that the >> combination of serialization/bandwidth is the bottleneck, and you hardly >> notice the difference in a remote access situation. That is what we want to >> find out, because we run on very little money and with compression as >> mentioned above our RAIDS would go a lot farther, as long the hit to the >> access time is not too great. >> >> Thanks, >> >> -Roy > > in netcdf4/hdf5, compression is tied to the chunking. Each chunk is > individually compressed, and must be completely decompressed to retrieve even > one value from that chunk. So the trick is to make your chunks correspond to > your "common cases" of data access. If thats possible, you should find that > compressed access is faster than non-compressed access, because IO is > smaller. but it will be highly dependent on that. Hi John: But do you cache the entire chunk that you decompress, or do you toss it? So if I make a second request that has data in that chunk i it saved or is it reread from the file. -Thanks, -Roy ********************** "The contents of this message do not reflect any position of the U.S. Government or NOAA." ********************** Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center 1352 Lighthouse Avenue Pacific Grove, CA 93950-2097 e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address) voice: (831)-648-9029 fax: (831)-648-8440 www: http://www.pfeg.noaa.gov/ "Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected"
thredds archives: