Re: splitting netcdf file into multiple files by date

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: Re: splitting netcdf file into multiple files by date
• From: Selina Satterfield <satterfields2000@xxxxxxxxx>
• Date: Fri, 13 May 2005 11:42:23 -0700 (PDT)

Just found out that the method of using get_rec from
the original file and put_recs to the new file works.
I had a minor coding error causing the problem. Just
wanted to give you all an update in case this comes up
for some one else. Thank you for all of your
suggestions.
--- Selina Satterfield <satterfields2000@xxxxxxxxx>
wrote:

> I believe I found a way to split the file up into
> multiple files by data through code since I can't
> use
> a third party tool however the code seems to have an
> 80 file limit and I'm not sure why.
> 
> First I search for a date from a database within a
> SearchForTime function
> 
> //Get the coordinate variable for the unlimited
> dimension
>  
>
ncVarUnlimDim=ncFile->get_var(ncFile->rec_dim()->name());
>   
>   //Get the type of the unlimited dimension variable
>   ncTypeUnlimDim=ncVarUnlimDim->type();
>   
>   //Get number of values in unlimited dimension
>   lNumValsUnlimDim=ncVarUnlimDim->num_vals();
> //Scan for date based upon type
>   if(ncTypeUnlimDim==ncDouble)
>   {
>     //double date times may not match exactly
> created
> an epsilon to handle this
>     dHalfSec=dEpsilon/(double)timeUnitSec;
>    
> dRcdTime=_GetTimeAxisValDouble(strDateTime.c_str());
>     
>     //loop through unlimited dimension to find date
> time match or near match
>     for(iCount=0; iCount < lNumValsUnlimDim;
> iCount++)
>     {
>       if( fabs(dRcdTime -
> ncVarUnlimDim->as_double(iCount)) < dHalfSec)
>       {
>         break;
>       }
>     }
>   }
>   else if(ncTypeUnlimDim == ncFloat)
> 
> etc........
> 
> 
> 
> if there was a match
> 
>  //Matching time was found; set time index to this
> value
>     ncVarUnlimDim->set_cur((long)iCount);
>     timeIndex = iCount;
> 
> Then the AddTime function is called
> Which adds the date to the new file at location 0
> //Note all the dimensions, attitbutes, an variables
> were already created before this.
> 
> Then I have an AddData function which accepts the
> original file and the time index found in the
> Searchfortime function. However for some reason it
> dies after creating 80 files it is dying on the
> get_rec
> function. Do you all know any limitations on the
> get_rec function? Here is the code
> 
> for( int i=0 ; i< ncOrgFile->num_vars() ; i++ )
> {
>   NcVar * ncOrgVar = ncOrgFile->get_var(i);
>   NcVar * ncNewVar = ncFile->get_var(i);
>   
>   // Skip coordinate variables (i.e., variables with
> names
>         // that match dimension names)
>   if( strcmp(ncOrgVar->name(),
> ncOrgVar->get_dim(0)->name()) == 0)
>   {
>     continue;
>   }
>   NcValues* varOrgVals;
>  
>   varOrgVals = ncOrgVar->get_rec( orgTimeIndex );
>   ncOrgType = ncOrgVar->type();    
>   // Insert values at new location
>   NcBool rtn;
>   if( ncOrgType == ncByte )
>   {
>     ncbyte *copyVals = new
> ncbyte[varOrgVals->num()];
>     for(int k=0; k < varOrgVals->num() ;k++) 
>     { 
>       copyVals[k] = varOrgVals->as_ncbyte(k);
>     }
>     rtn = ncNewVar->put_rec( copyVals, timeIndex);
>     delete [] copyVals;
>   }
>   else if( ncOrgType ==  ncChar )
>   {
>     etc......
> }
> 
> --- t.hume@xxxxxxxxxx wrote:
> > Hi,
> > 
> > This is nasty, but it would be quick. Your C++
> > program could make a call
> > to the ncks utility (which is part of the NetCDF
> > operators). In C you
> > use the system function ... I guess it is similar
> or
> > the same in C++.
> > Compared to using the C++ interface, this will
> save
> > you many many lines
> > of code, at the expense of being a very inelegant
> > solution.
> > 
> > Tim Hume
> > 
> > 
> > On Mon, 11 Apr 2005 11:42:20 -0700 (PDT)
> > Selina Satterfield <satterfields2000@xxxxxxxxx>
> > wrote:
> > 
> > > I'm new to netCDF development and currently I
> have
> > a
> > > large number of netcdf files
> > > that contain multiple dimensions, variables ,
> and
> > data
> > > for a period of time.  Here is a cdl that I
> > generated
> > > with ncdump of one of the files
> > > 
> > > netcdf test {
> > > dimensions:
> > >   lat = 141 ;
> > >   lon = 360 ;
> > >   time = UNLIMITED ; // (31 currently)
> > > variables:
> > >   float lat(lat) ;
> > >           lat:long_name = "Latitude" ;
> > >           lat:units = "degrees_north" ;
> > >   float lon(lon) ;
> > >           lon:long_name = "Longitude" ;
> > >           lon:units = "degrees_east" ;
> > >   float time(time) ;
> > >           time:long_name = "Time" ;
> > >           time:units = "hours since 1997-01-01" ;
> > >   float age_rcnt_obs(time, lat, lon) ;
> > >           age_rcnt_obs:_FillValue = 999.f ;
> > >           age_rcnt_obs:long_name = "Age of Recent
> > Observation"
> > > ;
> > >           age_rcnt_obs:missing_value = 999.f ;
> > >           age_rcnt_obs:units = "Hours" ;
> > >           age_rcnt_obs:valid_max = 255.f ;
> > >           age_rcnt_obs:valid_min = 0.f ;
> > >   float anal_temp(time, lat, lon) ;
> > >           anal_temp:_FillValue = 9999.f ;
> > >           anal_temp:long_name = "Analysis Temperature" ;
> > >           anal_temp:missing_value = 999.9f ;
> > >           anal_temp:scale_factor = 0.1f ;
> > >           anal_temp:units = "Deg. C" ;
> > >           anal_temp:valid_max = 70.f ;
> > >           anal_temp:valid_min = -10.f ;
> > >   float avgrad(time, lat, lon) ;
> > >           etc.....;
> > > 
> > > // global attributes:
> > >           :title = "SST 100 KM FIELD" ;
> > > data:
> > > 
> > >  lat = -70, -69, -68, -67, -66, -65, -64, -63,
> > -62,
> > > -61, -60, -59, -58, -57, 
> > >     -56, -55, -54, -53, -52, -51, -50, -49, -48,
> > -47,
> > > -46, -45, -44, -43, 
> > >     etc..............;
> > > 
> > >  lon = -180, -179, -178, -177, -176, -175, -174,
> > -173,
> > > -172, -171, -170, 
> > >     -169, -168, -167, -166, -165, -164, -163,
> > -162,
> > > -161, -160, -159, -158, 
> > >     -157, -156, -155, -154, -153, -152, -151,
> > -150,
> > > -149, -148, -147, -146, 
> > >     -145, -144, -143, -142, -141, -140, -139,
> > -138,
> > > -137, -136, -135, -134, 
> > >     etc..............;
> > > 
> > >  time = 60600, 60624, 60648, 60672, 60696,
> 60720,
> > > 60744, 60768, 60792, 60816, 
> > >     60840, 60864, 60888, 60912, 60936, 60960,
> 
=== message truncated ==



                
__________________________________ 
Yahoo! Mail Mobile 
Take Yahoo! Mail with you! Check email on your mobile phone. 
http://mobile.yahoo.com/learn/mail