NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi Nico, > We are using NetCDF4 to write our data products. These data products > are variable in size, but typically: > > * Total file sizes ranging from 10GB to several hundreds of GB > * Tens to hundreds of groups, in 3-4 levels > * Hundreds to thousands of variables, ranging in size from several > kB to several GB > * Each variable having several attributes > > The problem we are facing now is the following: Due to chunking, each > variable is assigned a chunking cache. Currently we have found no way to > end the access to a NetCDF variable. Due to the large number of variables > this has the following effects: > > * The total chunking cache size is very large, causing large memory > footprint of our software. > * A lot of data will only be written to the disk when the file is closed > > It is possible to force writing data to disk with nc_sync, but this works > at a file level rather than variable level and can thus result in a huge > performance penalty, as this would also force (re-)writing chunks in other > datasets that are not yet complete. Moreover, it will not release the > chunking cache and does thus not affect the software's memory footprint. > > Closing and reopening the file would probably release all resources, > but this will cause a huge overhead and a lot of additional code to find > and reopen groups and variables again. > > Is it possible with NetCDF4 to explicitly end the access to a variable > (like it is for example possible in HDF5, using H5Dclose()), thus freeing > all resources associated with it and forcing all cached data for the > variable to be written to disk? No, unfortunately there is no such function in the netCDF-4, but you have made a convincing justification of the need for such a function. I've created an issue in our bug tracking system for the addition of such a function, so we don't lose track of it and so you can follow any progress on the issue or comment on it in the future: https://bugtracking.unidata.ucar.edu/browse/NCF-292 Thanks for reporting the problem. --Russ Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu Ticket Details =================== Ticket ID: MME-788623 Department: Support netCDF Priority: Normal Status: Closed