[netcdfgroup] how to run netcdf-c parallel I/O tests without using mpiexec on login nodes of a supercomputer

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: [netcdfgroup] how to run netcdf-c parallel I/O tests without using mpiexec on login nodes of a supercomputer
• From: Edward Hartnett - NOAA Affiliate <edward.hartnett@xxxxxxxx>
• Date: Wed, 7 Aug 2024 07:26:36 -0600

All,

NetCDF tests should always be run after every install, to make sure
netcdf-c is working correctly on that platform. Parallel I/O tests are of
particular importance on supercomputers, because the tests can reveal
system problems with MPI or the parallel file systems. It's much better to
find these problems when installing netCDF, than later when your weather
model doesn't work.

One challenge is that most supercomputers don't want us launching jobs with
mpiexec, and that's how netcdf-c launches its parallel tests. However, this
can be overridden at build time. I have just added this to the netCDF
documentation:

The parallel I/O tests will only run if the additional configure
> option is used: --enable-parallel-tests. Those tests run (by default)
> with mpiexec, on 4, 16, or 32 processors. If mpiexec cannot be used on
> your login-nodes, a different command can be used to launch the
> parallel I/O tests. Used the --with-mpiexec configure option to set a
> different parallel I/O job launcher:


> $ CC=mpicc CPPFLAGS=-I/usr/local/hdf5-1.14.3_mpich/include
LDFLAGS=-L/usr/local/hdf5-1.14.3_mpicj/lib ./configure
--prefix=/usr/local/netcdf-c-4.9.3_mpich --enable-parallel-tests
--with-mpiexec='srun -A acct_name -q queue_name'

So you can use the --with-mpiexec option to cause an srun command to be
used instead of mpiexec. Please let me know if this is not clear.

The CMake build did not have this capability, so I added the CMake option
NETCDF_MPIEXEC to handle this, which will (I suspect) be in the 4.9.3
release.

Supercomputer installers can run all the netcdf-c tests, including parallel
I/O tests, to be sure that netcdf-c has built correctly on that platform.

Keep on NetCDFing!
Ed


-- 
Edward Hartnett
CIRES/NOAA EMC