NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Dennis,Great! Glad to know you are working on it! NetCDF C API is the only one I care about.
Your scenario 1 is the one I'm using to hide latency of Lustre file system when we need to extract metadata from large dataset. We have some that contain 10k files. A pool of threads picks away at the list of NetCDF files, only a single thread accesses any given file. We hold a lock while we open and close the file, what happens in between open and close, the reading and parsing of the metadata, occurs freely. Only 1 thread is touching any given file. This is the scenario where I need thread safety baked into HDF5, or I end up intermittently crashing.
To get more info about the complication of configuring HDF5 thread safe with the HDF5 HL API try to configure HDF5 with both thread safety and the HL API. here is the output from hdf5 1.8.17
./configure --enable-threadsafe --enable-hlchecking for thread safe support... configure: error: The thread-safe library is incompatible with the high-level library. --disable-hl can be used to prevent building the high-level library (recommended). Alternatively, --enable-unsupported will allow building the high-level library, though this configuration is not supported by The HDF Group.
Building in the unsupported mode is what we are currently doing to work around the issue. I don't know what the plan for Hdf5 HL API and theads is, but perhaps NetCDF should make use of the "low level" HDF5 API and avoid the issue altogether?
Your scenario 2 is also of interest. in that scenario I currently protect all nc calls and this serializes the I/O but still lets our calculations run in parallel. The calcs take much longer than the I/O so even though I/O is serialized this still works out well.
Burlen On 07/18/2016 03:23 PM, dmh@xxxxxxxx wrote:
> Unfortunately HDF5's thread safe config is officially mutually > exclusive with the HDF5 HL API used by NetCDF. I was unaware of this. My understanding was that the thread-safe HDf5 operated by providing a global lock to serialize all accesses. Assuming I am correct, you seem to be saying that this lock is not used at the HL API level. To date WRT netcdf, there have been two notions of thread-safe: 1. Allow multiple threads to operate as long as they are operating on different files. 2. Allow multiple threads to operate on the same file. #1 is doable -- just time consuming to implement. In fact I have a netcdf-c branch that should allow this for netcdf 3 (classic) files. The approach is to isolate all mutable global state used by the library and surround operations on that state (both read and write) with a mutex lock. Since none of the state accesses are all that long, this should not affect performance very much. Note that an implicit assumption is that all c-library calls (esp. malloc) are or can be made thread-safe. This approach might also work for netcdf-4 files except that we are limited by what the HDF5 library does. If there API is globally serialized, then our locking regime will not help. There is no obvious reason AFIAK why the HDF5 library could not be modified to do a similar isolation of global state. Note this issue crops up for the pnetcdf library also. #2 is much harder and would require significant refactoring of any library that attempted it. The reason is that access to EVERY piece of state (global or not) must be made thread-safe. Finally, note that this issue is largely independent of parallel IO using e.g. MPIO. I look forward to further discussion of this issue; especially any complication I might be overlooking. =Dennis Heimbigner Unidata On 7/18/2016 12:24 PM, Burlen Loring wrote:Hi All, Just wanted to voice concern about the status of thread safety in NetCDF 4 HDF5. The locking strategy we've successfully used with NetCDF classic is not sufficient for NetCDF 4 with HDF5. In addition to our locking strategy HDF5 needs to be compiled with a thread safe option. Unfortunately HDF5's thread safe config is officially mutually exclusive with the HDF5 HL API used by NetCDF. When HDF5 is forced to compile with thread safety and HDF5 HL API, our threaded code runs without issue. It also performs well, which is important. My concern is the fact that wenow rely upon a build configuration that is officially unsupported by HDF5.Given the continual evolution to many core architectures, the horrendous latency on modern parallel file systems on super computing platforms, and that we have to deal with datasets structured such that latency is a major issue, threading is ever more critical. It's really important that we have a viable path to thread safety that is officially supported by HDF5 and performant. We don't want to be facing problems down the road due to use of the unsupported HDF5 config. Using the unsupported config creates a deployment issue as we'd like to rely on HDF5 installed at HPC centers or in official Linux distros, neither of whom will likely be compiling HDF5 in an unsupported configuration. I also believe that for the best performance locking is better done at the lowest level where it can be fine grained, hence locking all NetCDF I/O in our application is undesirable. I'm hoping that this conversation can be a data point that people are using threads to speed processing of large datasets on parallel file systems. It's important for us to have an officially supported thread safe option for NetCDF 4 HDF5 format. Burlen _______________________________________________ NOTE: All exchanges posted to Unidata maintained email lists are recorded in the Unidata inquiry tracking system and made publicly available through the web. Users who post to any of the lists we maintain are reminded to remove any personal information that they do not want to be made public. netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/_______________________________________________ NOTE: All exchanges posted to Unidata maintained email lists are recorded in the Unidata inquiry tracking system and made publicly available through the web. Users who post to any of the lists we maintain are reminded to remove any personal information that they do not want to be made public. netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxxFor list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
netcdfgroup archives: