Re: Asynchronous IO

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: Re: Asynchronous IO
• From: "Glenn P. Davis" <davis@xxxxxxxxxxxxxxxx>
• Date: Thu, 5 Sep 1996 15:16:36 -0600

On Sep 5,  2:54pm, Aubrey Holland wrote:
> Hello,
>    I was just wondering if asynchronous io was supported by netcdf.  My
> application is in need of a speed boost in the area of its NetCDf read.
>       Thanks
>
>       Aubrey Holland
>       holland@xxxxxxxxxxxxx
>
>-- End of excerpt from Aubrey Holland

By default, the io in netcdf is asynchronous.

The implementation in netcdf-2.4 (libsrc/xdrposix.c) uses
a single buffer of size BIOBUFSIZ. This macro is set in libsrc/local_nc.h
and defaults to 8192.

The scheme in netcdf-3 is similar (libsrc/posixio.c), except that the
size is more dynamic. The library attempts to query the system as to
to the preferred block size using "struct stat" member "st_blksize". (If this
isn't supported on a system it just uses 8192.)
The size of the buffer is initialized to this, and may grow in units of this
amount, so that a single request from the higher layers may all fit in the
buffer. To protect "small" systems, the size of a single request is limited
by a compile time macro NC_PG_CHUNK, defined in libsrc/nc.h.  This defaults
to 16384. This default is probably too small for real workstations. If you've
got lots of memory, you can bump it up accordingly.

In netcdf-3, you can turn off buffering (make all the writes synchronous)
by open'ing with the NC_SHARE flag.

Also, in netcdf-3, the io layer is much more isolated, making it possible to
implement your own in special circumstances. The io interface is defined by
libsrc/ncio.h. There are two implementations provided, libsrc/posixio.c
(usual unix system calls) and libsrc/ffio.c (uses CRAY ffio library calls).

Thank you for using netcdf.

-glenn

• References:
  • Asynchronous IO
    • From: Aubrey Holland