NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTE: The netcdf-hdf mailing list is no longer active. The list archives are made available for historical reasons.
Hi John, You wrote: > Greetings, > > I am sending this e-mail to the group of people who I think might > be able to give me some insight into the current status of > parallel netcdf (or parallel hdf/netcdf). I currently have a couple > of months to investigate implementing some sort of parallel netcdf > capability in our code. Note that this is not meant to be a research > endeavor, but to actually introduce some new technology / > improved capabilities. > > A couple of notes on the code itself. It is a parallel global > Atmospheric Chemical Transport Model (ACTM) that uses MPI to > communicate between the processors. It is written in F77/F90 and > uses netCDF for much of its I/O. It is a mature code, highly portable, > and has been run on a variety of MP platforms over the years. Of > course like most models, it is continues to be "enhanced" regularly. > We do parallel reads of large amounts of netCDF data, but currently all > significant netCDF output is funneled through a single Master processor. > This output can be many gigabytes as well. > > After looking into this for a short time and trying to figure out if > there was a particular direction I should go in, the following choices / > issues seemed to emerge: > > * Wait a couple of years for the newly funded proposal to merge netCDF > with HDF to be completed? I wish I could, but we really would like > something now, and I also have much of my time funded to do this > between now and 9/30/03. As the PI of this effort, I agree that you should try one of the other parallel netCDF implementations, with your time constraints. From what I've seen, the Northwestern Univ./ANL effort <http://www-unix.mcs.anl.gov/parallel-netcdf/> has achieved some impressive performance results and is closest to ready to try at this point, although I haven't kept up with recent progress of the NERSC/LBL effort. > * Work with NERSC/LBL and try to use the parallel netCDF capability > that they have developed. Not sure if this just runs on a single > platform, if it has a Fortran interface, what state it is in, etc.? > > * Ditto for Northwestern Univ./ANL. It was actually a tutorial I took > at SuperComputing from Rob Ross, et.al. that got me interested in this. > > * Somehow use HDF to provide the parallel netCDF capability. Convert > the code to use HDF (probably a significant amount of work) and convert > the netCDF input files to HDF? Or maybe HDF can read netCDF files? Would > probably want to convert the HDF output files as well, but not sure. If > this sort of format conversion is possible, how much time would it take? > We have lots and lots of netCDF input files. > > * Other possibilities? > > Any guidance any of you can give me would be much appreciated. > > Regards, > John > > -- > =========================== > John R. Tannahill > Lawrence Livermore Nat. Lab. > P.O. Box 808, M/S L-103 > Livermore, CA 94551 > 925-423-3514 > Fax: 925-423-4908 > =========================== --Russ _____________________________________________________________________ Russ Rew UCAR Unidata Program russ@xxxxxxxxxxxxxxxx http://my.unidata.ucar.edu
netcdf-hdf archives: