Re: access under f90

• To: Jörg Kaduk <joerg@xxxxxxxxxxxxxxxxxxx>
• Subject: Re: access under f90
• From: Olaf Heudecker <oheudeck@xxxxxxxxxxxxxxxxxx>
• Date: Wed, 26 May 1999 09:34:37 +0200 (MET DST)

Jörg Kaduk wrote:
>Hello,
>this might be a stupid or inappropriate question - however:
>I wonder whether I could speed up netcdf I/O on a Cray J90
>and I'm thinking about buffers and their sizes, etc.
I'd be interested in your results.


Hello Mr. Kaduk,

I did some work on optimizing buffers with netCDF on the Cray-T3E
(which should be essentially the same as on the C90) and
would like to refer you to:

- Cray's: Application Programmers I/O-Guide
- the netCDF-PATCH for more versatile assign-Parameters:
http://www.unidata.ucar.edu/packages/netcdf/known_problems.html#cray-assigns



>I wondered about is:
>in f90 language, is netcdf doing direct or sequential access -
>i.e. what should the buffering favor? I think it must be
>direct. But just to be sure...

I never thought about the Fortran Access Type but you must consider
that direct output is (a) fortran specific and (b) operating system
specific (as far as I know) and therefore not well adapted to the idea 
of architecture independent data formats.

In my tests it proved important to adapt the buffers to another aspect 
of file access: random or sequential access to data in the file:

The standard netCDF-assigns on Crays (bufa:336:2) are very well
adapted to sequential access. For random access it is more efficient
to allocate more smaller buffers (eg. with cache:8:256 in the
environment variable NETCDF_FFIOSPEC).  For random access in large
chunks, a more useful setting might be cachea:256:8:2, the optional
last parameter 2 reads two blocks ahead or writes two blocks
behind... .

Also the use of striped file systems (netCDF-Patch with -p and -q
option of the ffio assign command) can yield substantial increases in
bandwidth.

There is some more information about buffers and tests in the Intranet 
of our institute, but I am afraid it is in german.

>In general, I'm reading and writing record orientated, i.e.
>all/some fields for each time and the for the next time...
Strange, that I would call that sequential.

Good luck
Olaf R. Heudecker

• References:
  • access under f90
    • From: Jörg Kaduk