Hi-
I'm kind of new to writing unlimited dimensions in netcdf and ran into
some odd behavior when attempting to write unlimited dimension with
netCDF4 and Fortran90. In short, I find two problems (for details of the
problems see below examples and code):
1) writing unlimited dimensions in netCDF4 significantly increases file
size over same data written with version 3.6. I realize I could turn on
deflate ... but why would netcdf4-unlimited be 10x bigger than the
ver.3.6 or ver.4??
2) writing large amounts of unlimited data seems impossible? Is there a
performance issue? I realize my code is likely not optimized. Maybe I'm
using the wrong netCDF flags? When attempting to write 1 million points
with an unlimited dimension, ver3 file (not shown) completes quickly,
ver.4 file never completed.
Thanks for any insight.
-Ken
p.s. I have Lahey-Fortran v6.20d and am using netcdf-4.0-snapshot2007052502
I've attached code used to produce 3 files: ver.3.6 unlimited-dimension,
ver.4 fixed-dimension data and ver.4 unlimited-dimension data
First, I run with NMAX=10,000
haze:~/for/tests $ time ./test_unlimited ; lss Test_*
real 0m3.040s
user 0m2.640s
sys 0m0.370s
48K Test_Limited_v4.nc 40K Test_Unlimited_v3.nc 420K
Test_Unlimited_v4.nc
Since it is storing 10K int*4's, the file size should be ~40K. But the
ver.4 unlimited-dimension file is 10x too big?
OK, so I then ran it with NMAX=100,000
haze:~/for/tests $ time ./test_unlimited ; lss Test_*
real 0m30.478s
user 0m27.220s
sys 0m3.240s
404K Test_Limited_v4.nc 396K Test_Unlimited_v3.nc 4.0M
Test_Unlimited_v4.nc
Same issue: ver.4 w/ unlimited is 10x too big? And now it completed in
10x3sec=30sec. not bad since number of writes is 10x before.
Now the kicker, NMAX=1,000,000 ... it never finishes on my system (after
20 minutes ... where it should conceivably finish after
3secx100=300sec=5 minutes). It completes the first 2 files, but it never
completes writing the Test_Unlimited_v4.nc file. Here's the output after
Cntrl-C'ing the job:
haze:~/for/tests $ time ./test_unlimited ; lss Test_*
real 20m22.557s
user 4m19.840s
sys 0m53.840s
3.9M Test_Limited_v4.nc 3.9M Test_Unlimited_v3.nc 40M
Test_Unlimited_v4.nc
It never completed the write to Test_Unlimited_v4.nc, but the file sizes
are roughly the right size. In fact, the memory goes through the roof
and begins page swapping. Why?
program unlimited
use netcdf
implicit none
include 'netcdf4.inc'
integer :: ncid, dim, vid, vid2, i
integer :: nmax=1000000
!NetCDF3 file
call check( nf90_create('Test_Unlimited_v3.nc',NF90_CLOBBER,ncid) )
call check( nf90_def_dim(ncid,"cell",NF90_UNLIMITED,dim) )
call check( nf90_def_var(ncid,"temp",NF90_INT,dim,vid) )
call check(nf90_enddef(ncid))
do i=1,nmax
call check( nf90_put_var(ncid,vid,i,start=(/ i /) ) )
end do
call check( nf90_close(ncid) )
call check( nf90_create('Test_Limited_v4.nc',NF_NETCDF4,ncid) )
call check( nf90_def_dim(ncid,"cell",nmax,dim) )
call check( nf90_def_var(ncid,"temp",NF90_INT,dim,vid) )
call check(nf90_enddef(ncid))
do i=1,nmax
call check( nf90_put_var(ncid,vid,i,start=(/i /) ) )
end do
call check( nf90_close(ncid) )
call check( nf90_create('Test_Unlimited_v4.nc',NF_NETCDF4,ncid) )
call check( nf90_def_dim(ncid,"cell",NF90_UNLIMITED,dim) )
call check( nf90_def_var(ncid,"temp",NF90_INT,dim,vid) )
call check(nf90_enddef(ncid))
do i=1,nmax
call check( nf90_put_var(ncid,vid,i,start=(/i /) ) )
end do
call check( nf90_close(ncid) )
contains
subroutine check(status)
integer, intent ( in) :: status
if(status /= nf90_noerr) then
print *, trim(nf90_strerror(status))
stop "Stopped"
end if
end subroutine check
end program unlimited
--
Ken Knapp
Remote Sensing and Applications Division
National Climatic Data Center
151 Patton Ave
Asheville, NC 28801
828-271-4339 (voice) 828-271-4328 (fax)
==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================
