[netcdfgroup] parallel I/O testing with MVAPICH2

• To: netCDF Mail List <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: [netcdfgroup] parallel I/O testing with MVAPICH2
• From: Dan Kokron <daniel.kokron@xxxxxxxx>
• Date: Tue, 21 Sep 2010 16:07:01 -0400

I am comparing parallel I/O performance between NetCDF-4 and direct
MPI-IO.  I modified the ~netcdf/nc_test4/tst_nc4perf.c to add parameter
scans for direct MPI-IO calls.  It works fine on the discover platform
using Intel MPI, but fails when using MVAPICH2-1.5.1p1.

Any ideas?

I'm using ...
NetCDF-4.1.1          2010-04-01
configured with
--enable-hdf4 --enable-dap --disable-shared --enable-f90
--enable-fortran --enable-netcdf-4
--with-max-default-cache-size=268435456 --disable-cxx --with-udunits
--enable-hdf4

HDF5-1.8.4-patch1 released on Tue Feb 23 11:31:09 CST 2010
configured with
--disable-shared --disable-cxx --enable-hl --enable-fortran
--disable-sharedlib-rpath --enable-parallel CFLAGS=-fPIC

Here is a trace.
>mpirun_rsh -hostfile $PBS_NODEFILE -np 1 tst_nc4perf_mv2.5760x2881x72

HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) MPI-process 0:
  #000: H5Dio.c line 266 in H5Dwrite(): can't write data
    major: Dataset
    minor: Write failed
  #001: H5Dio.c line 578 in H5D_write(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5Dmpio.c line 552 in H5D_contig_collective_write(): couldn't finish 
shared collective MPI-IO
    major: Low-level I/O
    minor: Write failed
  #003: H5Dmpio.c line 1586 in H5D_inter_collective_io(): couldn't finish 
collective MPI-IO
    major: Low-level I/O
    minor: Can't get value
  #004: H5Dmpio.c line 1632 in H5D_final_collective_io(): optimized write failed
    major: Dataset
    minor: Write failed
  #005: H5Dmpio.c line 334 in H5D_mpio_select_write(): can't finish collective 
parallel write
    major: Low-level I/O
    minor: Write failed
  #006: H5Fio.c line 167 in H5F_block_write(): file write failed
    major: Low-level I/O
    minor: Write failed
  #007: H5FDint.c line 185 in H5FD_write(): driver write request failed
    major: Virtual File Layer
    minor: Write failed
  #008: H5FDmpio.c line 1726 in H5FD_mpio_write(): can't convert from size to 
size_i
    major: Internal error (too specific to document in detail)
    minor: Out of range


-- 
Dan Kokron
Global Modeling and Assimilation Office
NASA Goddard Space Flight Center
Greenbelt, MD 20771
Daniel.S.Kokron@xxxxxxxx
Phone: (301) 614-5192
Fax:   (301) 614-5304

• Follow-Ups:
  • Re: [netcdfgroup] parallel I/O testing with MVAPICH2
    • From: Rob Latham