NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
I looked at the GDAL driver for netcdf. Adding nc_set_chunk_cache(100000000, 10000, 0.75) in GDALRegister_netCDF() helps, but I'm not sure whether it's a good general solution. The code on lines 482-500 <https://github.com/OSGeo/gdal/blob/trunk/gdal/frmts/netcdf/netcdfdataset.cpp#L482> sets some chunking-related properties if NETCDF_HAS_NC4 is defined. My problem persists, though, whether or not this is defined (unless I add the caching call). Note that later on line 511 nBlockSize is set to 1 regardless of the define for bottom-up datasets (which mine is). However, for other datasets that I have nBlockSize is also set to 1, and they are fast. Does anything else in netcdfdataset.cpp ring any bells? I can bring this up with gdal folks, but wanted to check here first for a possible recommendation. On Thu, Dec 15, 2016 at 5:24 PM, dmh@xxxxxxxx <dmh@xxxxxxxx> wrote: > The interactions between two independent caches can cause > problems. I should look at the netcdf cache and see how it interacts > with the hdf5 cache. > =Dennis Heimbigner > Unidata > > On 12/15/2016 6:03 PM, Dave Allured - NOAA Affiliate wrote: > >> On Thu, Dec 15, 2016 at 4:46 PM, Chris Barker <chris.barker@xxxxxxxx >> <mailto:chris.barker@xxxxxxxx>> wrote: >> >> On Thu, Dec 15, 2016 at 1:00 PM, dmh@xxxxxxxx <mailto:dmh@xxxxxxxx> >> >> <dmh@xxxxxxxx <mailto:dmh@xxxxxxxx>> wrote: >> >> 1. Adding this feature to ncdump also requires adding >> it to the netcdf-c library API. But providing some means >> for client programs to pass thru parameter settings to the >> hdf5 lib >> seems like a good idea. >> >> >> absolutely! that would be very helpful. >> >> -CHB >> >> >> This may be premature. The netcdf API already has its own chunk cache >> with at least two functions to adjust tuning parameters. It seems to me >> that the netcdf facility would probably handle the current ncdump and >> gdal cases nicely, though I have not tested it. Please see this >> relevant documentation: >> >> http://www.unidata.ucar.edu/software/netcdf/docs/netcdf_perf >> _chunking.html >> >> Simon, you might want to ask your gdal maintainer to give this a try. >> If it works, it should be simple and robust. I would suggest increasing >> the per-variable chunk size to at least 5 qualityFlags.nc uncompressed >> chunks, and probably more. 5 is the number of chunks that span a single >> row for this particular file. This advice presumes that your typical >> read pattern is similar to ncdump, which I speculate is first across >> single whole rows, as I said earlier. >> >> columns = 4865 ; >> rows = 3682 ; >> uint quality_flags(rows, columns) ; >> quality_flags:_ChunkSizes = 891, 1177 ; >> >> 5 x 891 x 1177 x 4 bytes per uint uncompressed ~= 21 Mbytes >> >> Note this is likely to be a little larger than the default cache size in >> the current netcdf-C library, thus explaining some of the slow read >> behavior. >> >> You might also consider rechunking such data sets to smaller chunk >> size. Nccopy and ncks can do that. Rechunking may depend on your >> anticipated spatial read patterns, so give that a little thought. >> >> You might also consider reading the entire grid in a single get_vara >> call to the netcdf API. That is what my fast fortran test program did. >> A naive reader that, for example, loops over single rows may incur bad >> cache activity that could be avoided. >> >> --Dave >> >> >> _______________________________________________ >> NOTE: All exchanges posted to Unidata maintained email lists are >> recorded in the Unidata inquiry tracking system and made publicly >> available through the web. Users who post to any of the lists we >> maintain are reminded to remove any personal information that they >> do not want to be made public. >> >> >> netcdfgroup mailing list >> netcdfgroup@xxxxxxxxxxxxxxxx >> For list information or to unsubscribe, visit: >> http://www.unidata.ucar.edu/mailing_lists/ >> >> > _______________________________________________ > NOTE: All exchanges posted to Unidata maintained email lists are > recorded in the Unidata inquiry tracking system and made publicly > available through the web. Users who post to any of the lists we > maintain are reminded to remove any personal information that they > do not want to be made public. > > > netcdfgroup mailing list > netcdfgroup@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ >
netcdfgroup archives: