NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: [thredds] Best way to handle missing grids in an aggregation.

Hi Ken:
has anyone responded to this? My guess when you have multiple grids in the same file is that the "missing grid" has to be there in the sense of having all missing values at that time period. What we do is just have separate files for each grid, as you suggested at the end. Look at: 

http://oceanwatch.pfeg.noaa.gov/thredds/catalog.html

-Roy

On Oct 13, 2009, at 12:15 PM, Ken Tanaka wrote:


Hi,
I'm aggregating some gridded data with an ncml entry in our THREDDS catalog.xml file. The entry looks like this: 

<?xml version="1.0" encoding="UTF-8"?>
<catalog name="NGDC THREDDS Server Default Catalog"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0 " 
     xmlns:xlink="http://www.w3.org/1999/xlink";
     version="1.0.2">

<service name="aggServices" serviceType="compound" base="">
<service name="dapService2" serviceType="OpenDAP" base="/thredds/ dodsC/" /> 
 <service name="wcsService2" serviceType="WCS" base="/thredds/wcs/" />
 <service name="wms2" serviceType="WMS" base="/thredds/wms/" />
</service>

<dataset name="USTEC STEC Aggregation" ID="STEC-Agg"
        urlPath="stec-aggregation">
 <metadata inherited="true">
   <serviceName>aggServices</serviceName>
   <dataType>Grid</dataType>
 </metadata>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ ncml-2.2"> 
   <attribute name="Comment" value="Aggregation for USTEC STEC" />
<aggregation dimName="Time" type="joinExisting" recheckEvery="15 min"> 
     <scan location="/nfs/ustec_netcdf/stec/latest7days"
           suffix=".nc" />
   </aggregation>
 </netcdf>

</dataset>
</catalog>

Inside the netCDF files are grids from satellites:
time 1:  satellite_01, satellite_02, satellite_03
time 2:  satellite_01, satellite_02, satellite_03
time 3:  satellite_01, satellite_03
time 4:  satellite_03, satellite_02, satellite_01
and so on.
Sometimes data from a satellite is not present, as satellite_02 is missing at time 3 at in the example above. This appears as an all white image in the time sequence when viewing the satellite_02 result with IDV. What is the best way to handle this. Is there a missing grid flag, or should there be an empty image inserted, so that there is always a satellite_02 grid? We are also getting an error from IDV
java.io.IOException: Inconsistent array length read: 1165128303 != 1914731274 
at ucar.nc2.dt.grid.GeoGrid.readDataSlice(GeoGrid.java:616)
at ucar.nc2.dt.grid.GeoGrid.readDataSlice(GeoGrid.java:524)
at ucar.nc2.dt.grid.GeoGrid.readVolumeData(GeoGrid.java:445)
at ucar.visad.data.GeoGridFlatField.readData(GeoGridFlatField.java: 220) at ucar.visad.data.CachedFlatField.getMyValues(CachedFlatField.java: 479) at ucar.visad.data.CachedFlatField.getMyValues(CachedFlatField.java: 467) at ucar.visad.data.CachedFlatField.unpackFloats(CachedFlatField.java: 614) 
at visad.FlatField.getFloats(FlatField.java:1567)
at visad.ShadowFunctionOrSetType.doTransform (ShadowFunctionOrSetType.java:1404) at visad.java3d.ShadowFunctionOrSetTypeJ3D.doTransform (ShadowFunctionOrSetTypeJ3D.java:195) at visad.java3d.ShadowFunctionOrSetTypeJ3D.recurseRange (ShadowFunctionOrSetTypeJ3D.java:1209) at visad.java3d.ShadowFunctionOrSetTypeJ3D$1.run (ShadowFunctionOrSetTypeJ3D.java:181) 
at visad.util.ThreadManager$1.run(ThreadManager.java:292)
at visad.util.ThreadPool$ThreadMinnow.run(ThreadPool.java:98)

Do you think this error is due to the missing satellite_02 grids?
One more question: Also, as can be seen at time 4, the satellite grids are not always in the same order in the netCDF file. Is this a problem? I can have my colleague regenerate the data with grids in a consistent order if needed.
If I have answers to the above, I can probably figure out the following on my own. If all satellites must be present, maybe we need to start a new aggregation if a new satellite is added to the set or one is retired from service? Maybe we should be splitting out each satellite into their own netCDF files? 

Thanks,
Ken

_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/


**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA." 
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
  • 2009 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: