NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

[netcdfgroup] Advice on parallel netcdf

I was looking for general advice on using parallel netcdf/hdf5. I'm working on development an atmospheric model at NASA. The model of course is distributed with mpi (each process is essentially working one section of the grid describing the world) but much of the file IO is serial. The arrays to be written are gathered on the root process and the root process then does the reading/writing to the netcdf file. In an attempt to improve the overall IO performance I've been experimenting with parallel netcdf/hdf5, where the file for IO is opened for parallel access on all processes and each process read/writes the data for the piece of the world it is working in the netcdf file. Here is an outline of what I am doing in the code with a few actual code snippets: 

set some mpi info ...

call MPI_info_create(info,STATUS)
call MPI_Info_set(info,"romio_cb_read", "enable" ,STATUS)
call MPI_Info_set(info,"romio_cb_write", "enable" ,STATUS)
call ESMF_ConfigGetAttribute(CF, cb_buffer_size, Label='cb_buffer_size:', __RC__) 
call MPI_Info_set(info,"cb_buffer_size", "16777216" ,STATUS)
status = nf90_create("file_parallel.nc4",IOR(IOR(NF90_CLOBBER,NF90_HDF5),NF90_MPIIO),ncid,comm=comm,info=info) 

define dimensions ...
define vars ...
set access to collective for each variable ...

status = nf90_var_par_access(ncid,varid,NF90_COLLECTIVE)

determine start and cnt for process ...
read or write ...


Here are a few general observations.
- In general the IO does not scale with the number of processors and I'm seeing about the same write time for 1 or hundreds of mpi tasks.
- Gathering to root and having root write (and the converse for reading) was generally almost as fast or only marginally slower (2x) than using parallel IO regardless of mpi tasks.
- Setting the access of each variable to collective was crucial to write performance. If the access was set to independent the writing was horribly slow, 10 to 20 times longer than the gather to root/root write method.
- In general playing with the buffer size had no appreciable affect on the performance.
Does anyone have any tricks I haven't thought of or has seen the same thing with parallel IO performance? There really aren't that many things one can play with other than setting the MPI hints or changing the access type for variables (collective or independent). So far I have been using intel 11, intel mpi 3 on the gpfs file system but I plan to play with this on newer intel versions, different MPI stacks, and on lustre instead of gpfs. 

--
Ben Auer, PhD   SSAI, Scientific Programmer/Analyst
NASA GSFC,  Global Modeling and Assimilation Office
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD  20771
Phone: 301-286-9176               Fax: 301-614-6246