NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

Re: [netcdfgroup] PDL::NetCDF slow since netcdf-4.1.0

  • To: Heiko Klein <Heiko.Klein@xxxxxx>
  • Subject: Re: [netcdfgroup] PDL::NetCDF slow since netcdf-4.1.0
  • From: Doug Hunt <dhunt@xxxxxxxx>
  • Date: Tue, 5 Mar 2013 09:38:20 -0700 (MST)
Hi Heiko: I think these calls to nc_sync have been there a long time. I don't recall the original reason for them. Before netcdf version 4.1 was nc_sync just a no-op? If this is the case, then maybe we should put in an AUTOSYNC option with the default = 0 (do not sync).
If the netcdf group has ideas about the utility of nc_sync before netcdf version 4.1, then perhaps we should add the AUTOSYNC option with default = 1 (do sync).
Another alternative would be to remove all calls to nc_sync and then make available and advertise a sync method in PDL::NetCDF. 

NetCDF group:  Was nc_sync useful before netcdf version 4.1?

Thanks!

--Doug

dhunt@xxxxxxxx
Software Engineer
UCAR - COSMIC, Tel. (303) 497-2611

On Tue, 5 Mar 2013, Heiko Klein wrote:


Hi Doug,
we just upgraded to Ubuntu Precise (12.04) and have for the first time a netcdf version >= 4.1. With this upgrade PDL::NetCDF became awfully slow when writing data, in particularly when writing small amounts of data.
Reason for that is that netcdf now calls 'fsync' when nc_sync is called. Syncing the complete filesystem is very costy and I don't really understand why the netcdf-folks did that by default (it might make sense in some HPC filesystems - and fsync is not available from FORTRAN). It can be disabled in build-time, but who really does that - most people just don't use nc_sync, in particular since 'close' does this automatically.
But PDL::NetCDF calls nc_sync automatically after each put*. I would like to just remove the nc_sync calls from PDL::NetCDF, and let users call them manually if they really need syncronisation. If you oppose to that, I would like to put a flag to new: (AUTOSYNC => 0|1) (with default to 1). What do you think? 

Best regards,

Heiko

--
Dr. Heiko Klein                              Tel. + 47 22 96 32 58
Development Section / IT Department          Fax. + 47 22 69 63 55
Norwegian Meteorological Institute           http://www.met.no
P.O. Box 43 Blindern  0313 Oslo NORWAY