NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: [netcdfgroup] PDL::NetCDF slow since netcdf-4.1.0

  • To: Heiko Klein <Heiko.Klein@xxxxxx>
  • Subject: Re: [netcdfgroup] PDL::NetCDF slow since netcdf-4.1.0
  • From: Doug Hunt <dhunt@xxxxxxxx>
  • Date: Tue, 5 Mar 2013 09:38:20 -0700 (MST)
Hi Heiko: I think these calls to nc_sync have been there a long time. I don't recall the original reason for them. Before netcdf version 4.1 was nc_sync just a no-op? If this is the case, then maybe we should put in an AUTOSYNC option with the default = 0 (do not sync).
If the netcdf group has ideas about the utility of nc_sync before netcdf version 4.1, then perhaps we should add the AUTOSYNC option with default = 1 (do sync).
Another alternative would be to remove all calls to nc_sync and then make available and advertise a sync method in PDL::NetCDF. 

NetCDF group:  Was nc_sync useful before netcdf version 4.1?

Thanks!

--Doug

dhunt@xxxxxxxx
Software Engineer
UCAR - COSMIC, Tel. (303) 497-2611

On Tue, 5 Mar 2013, Heiko Klein wrote:


Hi Doug,
we just upgraded to Ubuntu Precise (12.04) and have for the first time a netcdf version >= 4.1. With this upgrade PDL::NetCDF became awfully slow when writing data, in particularly when writing small amounts of data.
Reason for that is that netcdf now calls 'fsync' when nc_sync is called. Syncing the complete filesystem is very costy and I don't really understand why the netcdf-folks did that by default (it might make sense in some HPC filesystems - and fsync is not available from FORTRAN). It can be disabled in build-time, but who really does that - most people just don't use nc_sync, in particular since 'close' does this automatically.
But PDL::NetCDF calls nc_sync automatically after each put*. I would like to just remove the nc_sync calls from PDL::NetCDF, and let users call them manually if they really need syncronisation. If you oppose to that, I would like to put a flag to new: (AUTOSYNC => 0|1) (with default to 1). What do you think? 

Best regards,

Heiko

--
Dr. Heiko Klein                              Tel. + 47 22 96 32 58
Development Section / IT Department          Fax. + 47 22 69 63 55
Norwegian Meteorological Institute           http://www.met.no
P.O. Box 43 Blindern  0313 Oslo NORWAY
  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: