Re: [netcdfgroup] NF90_SYNC question

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: Re: [netcdfgroup] NF90_SYNC question
• From: Karsten Bolding <karsten@xxxxxxxxxxxxxxxxxxxx>
• Date: Thu, 21 Feb 2013 08:02:02 +0000

Hello

I'm working on a 1D water column model using the F90 NetCDF interface.
Recently I've introduced a namelist option to specify when nf90_sync()
is called. We used to call nf90_sync() everytime data was saved.
Now we can specify every n'th  saving will also do syncing. Setting
sync_out=0 will leave everything to the operating system. From an annual
run with with 110 vertical levels the result is (data are saved once pr
day):

sync_out = 0
kb@orca /data/kb/gotm-cases/nns_annual $ time gotm
:
real    0m1.045s
user    0m0.972s
sys     0m0.028s

sync_out = 1
kb@orca /data/kb/gotm-cases/nns_annual $ time gotm
:
real    0m12.567s
user    0m1.288s
sys     0m0.052s

For a single run this might not be very important - but when doing
ensemble and/or parameter estimations with possible thousends of
simulations is does make a difference.

The only downside I see is if the model blows up the latest saved
results will not be on disk making debugging a bit harder.

So unless you you have an external program reading the .nc file my
suggestion is to leave it to the operating system.

Karsten


On Wed, Feb 20, 2013 at 11:00:59 -0600, Leon Maurer wrote:
> I've got a -- possibly ill posed -- question about the NF90_SYNC command:
> does calling it append new data to the already existing file (in the same
> way that a few more 'write(1,*)'s would) and modify a few existing things,
> or does it rewrite the file from scratch? Does it do any/much reading in
> the process?
> 
> I'm using a netCDF-4/HDF5 file with compression.
> 
> I'm using HTCondor <http://research.cs.wisc.edu/htcondor/>'s standard
> universe, which effectively captures all file input/output and redirects it
> over a network. Although my code only produces ~100MB of new data per day,
> I'm told that the resulting network usage is dangerously high. The program
> calls NF90_SYNC every 5 minutes, so I'm wondering if that could be
> the culprit. (i.e. if it's doing a lot of reading/writing in the process.)
> 
> (Also, I just want to confirm that the NF90_SHARE flag doesn't work for
> netCDF-4/HDF5 files. The NF90_SYNC
> page<http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-f90/NF90_005fSYNC.html>
> recommends using
> the flag, but doesn't mention netCDF-4/HDF5file incompatibility; mentioning
> the incompatibility is saved for the NF90_CREATE
> page<http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-f90/NF90_005fCREATE.html>
> .)
> 
> Thanks.
> -Leon

> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit: 
> http://www.unidata.ucar.edu/mailing_lists/ 


-- 
----------------------------------------------------------------------
Karsten Bolding                    Bolding & Burchard ApS
Strandgyden 25                     Phone: +45 64422058
DK-5466 Asperup                    Fax:   +45 64422068
Denmark                            Email: karsten@xxxxxxxxxxxxxxxxxxxx

http://www.findvej.dk/Strandgyden25,5466,11,3

"Anti-intellectualism has been a constant thread winding its way 
through our political and cultural life, nurtured by the false notion 
that democracy means that my ignorance is just as good as your 
knowledge." -- Isaac Asimov
----------------------------------------------------------------------

• References:
  • [netcdfgroup] NF90_SYNC question
    • From: Leon Maurer