NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Your code fragment looks fine.
You can give PnetCDF a try to see if there is any difference.
In general, independent I/O mode performs worse than collective.
This is because the underlying file systems handle poorly for
noncontiguous file access resulted from independent mode.
If you can find a way to write a whole variable at a time,
using collective mode will significantly improve the performance.
Wei-keng
On Sep 22, 2015, at 9:43 AM, Heiko Klein wrote:
> Hi Wei-keng,
>
> thanks for the information. I got now the parallel version working with
> NC_INDEPENDENT and from 2 processors, I see some benefit, e.g. real-time
> reduction from 40s to 30s. But when I add more processors performance
> get worse.
>
> I'm using parallel hdf5 with an uncompressed netcdf4 file. Is
> NC_INDEPENDENT faster with netcdf3 pNetcdf?
>
>
> What I'm basically doing now is:
>
> for (size_t vi = 0; vi < vars.size(); ++vi) {
> if ((vi % mifi_mpi_size) != mifi_mpi_rank) {
> continue; // skipping
> }
> check(nc_var_par_access(ncId, varId[vi], NC_INDEPENDENT));
> //... read data for the variable from grib
> check(nc_put_vara_float(ncId, varId[vi], start, count, data));
> }
>
> Maybe there are better ways to do that?
>
>
> Best regards,
>
> Heiko
>
>
> On 2015-09-22 09:07, Wei-keng Liao wrote:
>> Hi, Heiko
>>
>> In that case, you can use independent mode.
>> I.e. nc_var_par_access(ncid, varid, NC_INDEPENDENT)
>>
>> It still allows you to write to a shared file from multiple
>> MPI processes independently, at different time.
>>
>> However, the performance will not be as good as the collective mode.
>>
>> Wei-keng
>>
>> On Sep 22, 2015, at 1:45 AM, Heiko Klein wrote:
>>
>>> Hei Wei-keng,
>>>
>>> thanks for your tip about using pnetcdf. I've worked with MPI, but only
>>> for modeling, i.e. when all processes do approximately the same thing at
>>> the same time.
>>>
>>> The problem here is that the 10 input-files don't appear on my machines
>>> at the same time. They are ensemble members and downloaded from
>>> different machines with different processors, so the first file might
>>> appear 30s before the last file (within a total time-step time of 2
>>> minutes). I would like to start as soon as the first file appears, but
>>> this sounds very difficult with MPI, isn't it? (I'm more familiar with
>>> OpenMP, and there exist task-based parallelization (what I would use
>>> here), and loop-base parallelization (which is more like MPI?))
>>>
>>> Best regards,
>>>
>>> Heiko
>>>
>>> On 2015-09-22 03:24, Wei-keng Liao wrote:
>>>> Hi, Heiko
>>>>
>>>> Parallel I/O to the classical netCDF format is supported by netCDF through
>>>> PnetCDF underneath.
>>>> It allows you to write concurrently to a single shared file from multiple
>>>> MPI processes.
>>>> Of course, you will have to build PnetCDF first and then build netCDF with
>>>> --enable-pnetcdf configure option.
>>>>
>>>> Your netCDF program does not need much changes to make use this feature.
>>>> All you have to
>>>> do is the followings.
>>>> 1. call nc_create_par() instead of nc_create()
>>>> 2. add NC_PNETCDF to the create mode argument of nc_create_par
>>>> 3. call nc_var_par_access(ncid, varid, NC_COLLECTIVE) after nc_enddef to
>>>> enable collective I/O mode
>>>>
>>>> There are a couple example codes available in this URL.
>>>> http://cucis.ece.northwestern.edu/projects/PnetCDF/#InteroperabilityWithNetCDF4
>>>>
>>>> There are instructions in each example file for building netCDF with
>>>> PnetCDF.
>>>> For downloading PnetCDF, please see
>>>> http://cucis.ece.northwestern.edu/projects/PnetCDF/download.html
>>>>
>>>> Wei-keng
>>>>
>>>> On Sep 21, 2015, at 9:14 AM, Heiko Klein wrote:
>>>>
>>>>> Hi Nick,
>>>>>
>>>>> yes, they are all writing to the same file - we want to have one file at
>>>>> the end.
>>>>>
>>>>> I've been scanning through the source-code of netcdf3. I guess the
>>>>> problem of the partly written sections is caused by the translation of
>>>>> the nc_put_vara calls to internal pages, and the from the internal pages
>>>>> to disk. And eventually, the internal pages are not aligned with my
>>>>> nc_put_vara calls, so even when the region of nc_put_vara doesn't
>>>>> overlap between concurrent calls, the internal pages do? Is there a way
>>>>> to enforce proper alignment? I see nc__enddef has several align
>>>>> parameters.
>>>>>
>>>>>
>>>>> I'm aware that concurrent writes are not officially supported by the
>>>>> netcdf-library. But IT-infrastructure has changed a lot since the start
>>>>> of the netcdf-library and systems are nowadays highly parallelized, both
>>>>> on CPU and also in IO/filesystems. I'm trying to find a way to allow for
>>>>> simple parallelization. Having many output-files from a model is risky
>>>>> for data-consistency - so I would like to avoid it without sacrificing
>>>>> to much speed.
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Heiko
>>>>>
>>>>>
>>>>> On 2015-09-21 15:18, Nick Papior wrote:
>>>>>> So, are they writing to the same files?
>>>>>>
>>>>>> I.e. job1 writes a(:,1) to test.nc <http://test.nc> and job2 writes
>>>>>> a(:,2) to test.nc <http://test.nc>?
>>>>>> Because that is not allowed.
>>>>>>
>>>>>> 2015-09-21 15:13 GMT+02:00 Heiko Klein <Heiko.Klein@xxxxxx
>>>>>> <mailto:Heiko.Klein@xxxxxx>>:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm trying to convert about 90GB of NWP data 4 times daily from grib to
>>>>>> netcdf. The grib-files arrive as fast as the data can be downloaded from
>>>>>> the HPC machines. They come by 10 files/forecast timestep.
>>>>>>
>>>>>> Currently, I manage to convert 1 file/forecast timestep and I would like
>>>>>> to parallelize the conversion into independent jobs (i.e. neither MPI or
>>>>>> OpenMP), with a theoretical performance increase of 10. The underlying
>>>>>> IO system is fast enough to handle 10 jobs, and I have enough CPUs, but
>>>>>> the concurrently written netcdf-files show data which is only written
>>>>>> half to the disk, or mixed with other slices.
>>>>>>
>>>>>> What I do is create a _FILL_VALUE 'template' file, containing all
>>>>>> definitions before the NWP job runs. When a new set of files arrives,
>>>>>> the data is put to the respective data-slices which don't have any
>>>>>> overlap, there is never a redefine, only functions like: nc_put_vara_*
>>>>>> with different slices.
>>>>>>
>>>>>> Since the nc_put_vara_* calls are non-overlapping, I hoped that this
>>>>>> type of concurrent write would work - but it doesn't. Is my idea really
>>>>>> so bad to write data in parallel (e.g. there are internal buffers which
>>>>>> are rewritten)? Any ideas how to improve the conversion process?
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Heiko
>>>>>>
>>>>>> _______________________________________________
>>>>>> netcdfgroup mailing list
>>>>>> netcdfgroup@xxxxxxxxxxxxxxxx <mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
>>>>>> For list information or to unsubscribe, visit:
>>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Kind regards Nick
>>>>>
>>>>> --
>>>>> Dr. Heiko Klein Norwegian Meteorological Institute
>>>>> Tel. + 47 22 96 32 58 P.O. Box 43 Blindern
>>>>> http://www.met.no 0313 Oslo NORWAY
>>>>>
>>>>> _______________________________________________
>>>>> netcdfgroup mailing list
>>>>> netcdfgroup@xxxxxxxxxxxxxxxx
>>>>> For list information or to unsubscribe, visit:
>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>
>>>
>>> --
>>> Dr. Heiko Klein Norwegian Meteorological Institute
>>> Tel. + 47 22 96 32 58 P.O. Box 43 Blindern
>>> http://www.met.no 0313 Oslo NORWAY
>>
>
> --
> Dr. Heiko Klein Norwegian Meteorological Institute
> Tel. + 47 22 96 32 58 P.O. Box 43 Blindern
> http://www.met.no 0313 Oslo NORWAY
netcdfgroup archives: