NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi-I'm kind of new to writing unlimited dimensions in netcdf and ran into some odd behavior when attempting to write unlimited dimension with netCDF4 and Fortran90. In short, I find two problems (for details of the problems see below examples and code): 1) writing unlimited dimensions in netCDF4 significantly increases file size over same data written with version 3.6. I realize I could turn on deflate ... but why would netcdf4-unlimited be 10x bigger than the ver.3.6 or ver.4??
2) writing large amounts of unlimited data seems impossible? Is there a performance issue? I realize my code is likely not optimized. Maybe I'm using the wrong netCDF flags? When attempting to write 1 million points with an unlimited dimension, ver3 file (not shown) completes quickly, ver.4 file never completed.
Thanks for any insight. -Ken p.s. I have Lahey-Fortran v6.20d and am using netcdf-4.0-snapshot2007052502I've attached code used to produce 3 files: ver.3.6 unlimited-dimension, ver.4 fixed-dimension data and ver.4 unlimited-dimension data
First, I run with NMAX=10,000
Since it is storing 10K int*4's, the file size should be ~40K. But the ver.4 unlimited-dimension file is 10x too big?haze:~/for/tests $ time ./test_unlimited ; lss Test_* real 0m3.040s user 0m2.640s sys 0m0.370s48K Test_Limited_v4.nc 40K Test_Unlimited_v3.nc 420K Test_Unlimited_v4.nc
OK, so I then ran it with NMAX=100,000
Same issue: ver.4 w/ unlimited is 10x too big? And now it completed in 10x3sec=30sec. not bad since number of writes is 10x before.haze:~/for/tests $ time ./test_unlimited ; lss Test_* real 0m30.478s user 0m27.220s sys 0m3.240s404K Test_Limited_v4.nc 396K Test_Unlimited_v3.nc 4.0M Test_Unlimited_v4.nc
Now the kicker, NMAX=1,000,000 ... it never finishes on my system (after 20 minutes ... where it should conceivably finish after 3secx100=300sec=5 minutes). It completes the first 2 files, but it never completes writing the Test_Unlimited_v4.nc file. Here's the output after Cntrl-C'ing the job:
It never completed the write to Test_Unlimited_v4.nc, but the file sizes are roughly the right size. In fact, the memory goes through the roof and begins page swapping. Why?haze:~/for/tests $ time ./test_unlimited ; lss Test_* real 20m22.557s user 4m19.840s sys 0m53.840s3.9M Test_Limited_v4.nc 3.9M Test_Unlimited_v3.nc 40M Test_Unlimited_v4.nc
program unlimited
use netcdf
implicit none
include 'netcdf4.inc'
integer :: ncid, dim, vid, vid2, i
integer :: nmax=1000000
!NetCDF3 file
call check( nf90_create('Test_Unlimited_v3.nc',NF90_CLOBBER,ncid) )
call check( nf90_def_dim(ncid,"cell",NF90_UNLIMITED,dim) )
call check( nf90_def_var(ncid,"temp",NF90_INT,dim,vid) )
call check(nf90_enddef(ncid))
do i=1,nmax
call check( nf90_put_var(ncid,vid,i,start=(/ i /) ) )
end do
call check( nf90_close(ncid) )
call check( nf90_create('Test_Limited_v4.nc',NF_NETCDF4,ncid) )
call check( nf90_def_dim(ncid,"cell",nmax,dim) )
call check( nf90_def_var(ncid,"temp",NF90_INT,dim,vid) )
call check(nf90_enddef(ncid))
do i=1,nmax
call check( nf90_put_var(ncid,vid,i,start=(/i /) ) )
end do
call check( nf90_close(ncid) )
call check( nf90_create('Test_Unlimited_v4.nc',NF_NETCDF4,ncid) )
call check( nf90_def_dim(ncid,"cell",NF90_UNLIMITED,dim) )
call check( nf90_def_var(ncid,"temp",NF90_INT,dim,vid) )
call check(nf90_enddef(ncid))
do i=1,nmax
call check( nf90_put_var(ncid,vid,i,start=(/i /) ) )
end do
call check( nf90_close(ncid) )
contains
subroutine check(status)
integer, intent ( in) :: status
if(status /= nf90_noerr) then
print *, trim(nf90_strerror(status))
stop "Stopped"
end if
end subroutine check
end program unlimited
-- Ken Knapp Remote Sensing and Applications Division National Climatic Data Center 151 Patton Ave Asheville, NC 28801 828-271-4339 (voice) 828-271-4328 (fax) ============================================================================== To unsubscribe netcdfgroup, visit: http://www.unidata.ucar.edu/mailing-list-delete-form.html ==============================================================================
netcdfgroup archives: