Hi John,
You wrote:
> Greetings,
>
> I am sending this e-mail to the group of people who I think might
> be able to give me some insight into the current status of
> parallel netcdf (or parallel hdf/netcdf). I currently have a couple
> of months to investigate implementing some sort of parallel netcdf
> capability in our code. Note that this is not meant to be a research
> endeavor, but to actually introduce some new technology /
> improved capabilities.
>
> A couple of notes on the code itself. It is a parallel global
> Atmospheric Chemical Transport Model (ACTM) that uses MPI to
> communicate between the processors. It is written in F77/F90 and
> uses netCDF for much of its I/O. It is a mature code, highly portable,
> and has been run on a variety of MP platforms over the years. Of
> course like most models, it is continues to be "enhanced" regularly.
> We do parallel reads of large amounts of netCDF data, but currently all
> significant netCDF output is funneled through a single Master processor.
> This output can be many gigabytes as well.
>
> After looking into this for a short time and trying to figure out if
> there was a particular direction I should go in, the following choices /
> issues seemed to emerge:
>
> * Wait a couple of years for the newly funded proposal to merge netCDF
> with HDF to be completed? I wish I could, but we really would like
> something now, and I also have much of my time funded to do this
> between now and 9/30/03.
As the PI of this effort, I agree that you should try one of the other
parallel netCDF implementations, with your time constraints. From
what I've seen, the Northwestern Univ./ANL effort
<http://www-unix.mcs.anl.gov/parallel-netcdf/> has achieved some
impressive performance results and is closest to ready to try at this
point, although I haven't kept up with recent progress of the
NERSC/LBL effort.
> * Work with NERSC/LBL and try to use the parallel netCDF capability
> that they have developed. Not sure if this just runs on a single
> platform, if it has a Fortran interface, what state it is in, etc.?
>
> * Ditto for Northwestern Univ./ANL. It was actually a tutorial I took
> at SuperComputing from Rob Ross, et.al. that got me interested in this.
>
> * Somehow use HDF to provide the parallel netCDF capability. Convert
> the code to use HDF (probably a significant amount of work) and convert
> the netCDF input files to HDF? Or maybe HDF can read netCDF files? Would
> probably want to convert the HDF output files as well, but not sure. If
> this sort of format conversion is possible, how much time would it take?
> We have lots and lots of netCDF input files.
>
> * Other possibilities?
>
> Any guidance any of you can give me would be much appreciated.
>
> Regards,
> John
>
> --
> ===========================
> John R. Tannahill
> Lawrence Livermore Nat. Lab.
> P.O. Box 808, M/S L-103
> Livermore, CA 94551
> 925-423-3514
> Fax: 925-423-4908
> ===========================
--Russ
_____________________________________________________________________
Russ Rew UCAR Unidata Program
russ@xxxxxxxxxxxxxxxx http://my.unidata.ucar.edu
