NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi Nick, yes, they are all writing to the same file - we want to have one file at the end. I've been scanning through the source-code of netcdf3. I guess the problem of the partly written sections is caused by the translation of the nc_put_vara calls to internal pages, and the from the internal pages to disk. And eventually, the internal pages are not aligned with my nc_put_vara calls, so even when the region of nc_put_vara doesn't overlap between concurrent calls, the internal pages do? Is there a way to enforce proper alignment? I see nc__enddef has several align parameters. I'm aware that concurrent writes are not officially supported by the netcdf-library. But IT-infrastructure has changed a lot since the start of the netcdf-library and systems are nowadays highly parallelized, both on CPU and also in IO/filesystems. I'm trying to find a way to allow for simple parallelization. Having many output-files from a model is risky for data-consistency - so I would like to avoid it without sacrificing to much speed. Best regards, Heiko On 2015-09-21 15:18, Nick Papior wrote: > So, are they writing to the same files? > > I.e. job1 writes a(:,1) to test.nc <http://test.nc> and job2 writes > a(:,2) to test.nc <http://test.nc>? > Because that is not allowed. > > 2015-09-21 15:13 GMT+02:00 Heiko Klein <Heiko.Klein@xxxxxx > <mailto:Heiko.Klein@xxxxxx>>: > > Hi, > > I'm trying to convert about 90GB of NWP data 4 times daily from grib to > netcdf. The grib-files arrive as fast as the data can be downloaded from > the HPC machines. They come by 10 files/forecast timestep. > > Currently, I manage to convert 1 file/forecast timestep and I would like > to parallelize the conversion into independent jobs (i.e. neither MPI or > OpenMP), with a theoretical performance increase of 10. The underlying > IO system is fast enough to handle 10 jobs, and I have enough CPUs, but > the concurrently written netcdf-files show data which is only written > half to the disk, or mixed with other slices. > > What I do is create a _FILL_VALUE 'template' file, containing all > definitions before the NWP job runs. When a new set of files arrives, > the data is put to the respective data-slices which don't have any > overlap, there is never a redefine, only functions like: nc_put_vara_* > with different slices. > > Since the nc_put_vara_* calls are non-overlapping, I hoped that this > type of concurrent write would work - but it doesn't. Is my idea really > so bad to write data in parallel (e.g. there are internal buffers which > are rewritten)? Any ideas how to improve the conversion process? > > Best regards, > > Heiko > > _______________________________________________ > netcdfgroup mailing list > netcdfgroup@xxxxxxxxxxxxxxxx <mailto:netcdfgroup@xxxxxxxxxxxxxxxx> > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ > > > > > -- > Kind regards Nick -- Dr. Heiko Klein Norwegian Meteorological Institute Tel. + 47 22 96 32 58 P.O. Box 43 Blindern http://www.met.no 0313 Oslo NORWAY
netcdfgroup archives: