NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Ed,
See my answer/comments to your email below.
Thanks,
Wei Huang
huangwei@xxxxxxxx
VETS/CISL
National Center for Atmospheric Research
P.O. Box 3000 (1850 Table Mesa Dr.)
Boulder, CO 80307-3000 USA
(303) 497-8924
On Sep 19, 2011, at 4:43 PM, Ed Hartnett wrote:
> Wei Huang <huangwei@xxxxxxxx> writes:
>
>> Hi, netcdfgroup,
>>
>> Currently, we are trying to use parallel-enabled NetCDF4. We started with
>> read/write a 5G file and some computation, we got the following timing (in
>> wall-clock) on a IBM power machine:
>> Number of Processors Total(seconds) read(seconds) Write(seconds)
>> Computation(seconds)
>> seq 89.137 28.206 48.327
>> 11.717
>> 1 178.953 44.837 121.17
>> 11.644
>> 2 167.25 46.571 113.343
>> 5.648
>> 4 168.138 44.043 118.968
>> 2.729
>> 8 137.74 25.161 108.986
>> 1.064
>> 16 113.354 16.359 93.253
>> 0.494
>> 32 439.481 122.201 311.215
>> 0.274
>> 64 831.896 277.363 588.653
>> 0.203
>>
>> First thing we can see is that when run parallel-enabled code at one
>> processor, the total
>> wall-clok time doubled.
>> Then we did not see the scaling when more processors added.
>>
>> Anyone wants to share their experience?
>>
>> Thanks,
>>
>> Wei Huang
>> huangwei@xxxxxxxx
>> VETS/CISL
>> National Center for Atmospheric Research
>> P.O. Box 3000 (1850 Table Mesa Dr.)
>> Boulder, CO 80307-3000 USA
>> (303) 497-8924
>>
>>
>
> Howdy Wei and all!
>
> Are you using the 4.1.2 release? Did you configure with
> --enable-parallel-tests, and did those tests pass?
I am using 4.1.3, and configured with "-enanle-parallel-tests", and those tests
passed.
>
> I would suggest building netCDF with --enable-parallel-tests and then
> running nc_test4/tst_nc4perf. This simple program, based on
> user-contributed test code, performs parallel I/O with a wide variety of
> options, and prints a table of results.
I have run nc_test4/tst_nc4perf for 1, 2, 4, and 8 processors, results attached.
To me, the performance decreases when processors increase.
Someone may have a better interpret.
I also run tst_parallel4, with result:
num_proc time(s) write_rate(B/s)
1 9.2015 1.16692e+08
2 12.4557 8.62048e+07
4 6.30644 1.70261e+08
8 5.53761 1.939e+08
16 2.25639 4.75866e+08
32 2.28383 4.7015e+08
64 2.19041 4.90202e+08
>
> This will tell you whether parallel I/O is working on your platform, and
> at least give some idea of reasonable settings.
>
> Parallel I/O is a very complex topic. However, if everything is working
> well, you should see I/O improvement which scales reasonably linearly,
> for less then about 8 processors (perhaps more, depending on your
> system, but not much more.) At this point, your parallel application is
> saturating your I/O subsystem, and further I/O performance is
> marginal.
>
> In general, HDF5 I/O will not be faster than netCDF-4 I/O. The netCDF-4
> layer is very light in this area, and simply calls the HDF5 that the
> user would call anyway.
>
> Key settings are:
>
> * MPI_IO vs. POSIX_IO (varies from platform to platform which is
> faster. See nc4perf results for your machine/compiler.)
Tested both, POSIX is better.
>
> * Chunking and caching play a big role, as always. Caching is
> turned off by default, otherwise netCDF caches on all the processors
> will consume too much memory. But you should set this to at least the
> size of one chunk. Note that this cache will happen on all processors
> involved.
>
We use chunking, can probably try caching.
> * Collective vs. independent access. Seems (to my naive view) like
> independent should usually be faster, but the opposite seems to be
> the case. This is because the I/O subsystems are good at grouping I/O
> requests into larger, more efficient units. Collective access gives
> the I/O layer the maximum chance to exercise its magic.
tried both, no significant difference.
>
> Best thing to do is get tst_nc4perf working on your platform, and then
> modify it to write data files that match yours (i.e. same size
> variables). The program will then tell you the best set of settings to
> use in your case.
We can modify this program to mimic our data size, but do not know if this
will help us.
>
> If the program shows that parallel I/O is not working, take a look at
> the netCDF test program h5_test/tst_h_par.c. This is a HDF5-only program
> (no netcdf code at all) that does parallel I/O. If this program does not
> show that parallel I/O is working, then your problem is not with the
> netCDF layer, but somewhere in HDF5 or even lower in the stack.
>
> Thanks!
>
> Ed
>
> --
> Ed Hartnett -- ed@xxxxxxxxxxxxxxxx
Attachment:
mpi_io_bluefire.perf
Description: Binary data
netcdfgroup archives: