NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

[thredds] Best way to handle missing grids in an aggregation.

Hi,
I'm aggregating some gridded data with an ncml entry in our THREDDS catalog.xml file. The entry looks like this: 

<?xml version="1.0" encoding="UTF-8"?>
<catalog name="NGDC THREDDS Server Default Catalog"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"; 
       xmlns:xlink="http://www.w3.org/1999/xlink";
       version="1.0.2">

 <service name="aggServices" serviceType="compound" base="">
<service name="dapService2" serviceType="OpenDAP" base="/thredds/dodsC/" /> 
   <service name="wcsService2" serviceType="WCS" base="/thredds/wcs/" />
   <service name="wms2" serviceType="WMS" base="/thredds/wms/" />
 </service>

 <dataset name="USTEC STEC Aggregation" ID="STEC-Agg"
          urlPath="stec-aggregation">
   <metadata inherited="true">
     <serviceName>aggServices</serviceName>
     <dataType>Grid</dataType>
   </metadata>

   <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
     <attribute name="Comment" value="Aggregation for USTEC STEC" />
     <aggregation dimName="Time" type="joinExisting" recheckEvery="15 min">
       <scan location="/nfs/ustec_netcdf/stec/latest7days"
             suffix=".nc" />
     </aggregation>
   </netcdf>

 </dataset>
</catalog>

Inside the netCDF files are grids from satellites:
time 1:  satellite_01, satellite_02, satellite_03
time 2:  satellite_01, satellite_02, satellite_03
time 3:  satellite_01, satellite_03
time 4:  satellite_03, satellite_02, satellite_01
and so on.
Sometimes data from a satellite is not present, as satellite_02 is missing at time 3 at in the example above. This appears as an all white image in the time sequence when viewing the satellite_02 result with IDV. What is the best way to handle this. Is there a missing grid flag, or should there be an empty image inserted, so that there is always a satellite_02 grid? We are also getting an error from IDV
java.io.IOException: Inconsistent array length read: 1165128303 != 1914731274 
  at ucar.nc2.dt.grid.GeoGrid.readDataSlice(GeoGrid.java:616)
  at ucar.nc2.dt.grid.GeoGrid.readDataSlice(GeoGrid.java:524)
  at ucar.nc2.dt.grid.GeoGrid.readVolumeData(GeoGrid.java:445)
  at ucar.visad.data.GeoGridFlatField.readData(GeoGridFlatField.java:220)
  at ucar.visad.data.CachedFlatField.getMyValues(CachedFlatField.java:479)
  at ucar.visad.data.CachedFlatField.getMyValues(CachedFlatField.java:467)
at ucar.visad.data.CachedFlatField.unpackFloats(CachedFlatField.java:614) 
  at visad.FlatField.getFloats(FlatField.java:1567)
at visad.ShadowFunctionOrSetType.doTransform(ShadowFunctionOrSetType.java:1404) at visad.java3d.ShadowFunctionOrSetTypeJ3D.doTransform(ShadowFunctionOrSetTypeJ3D.java:195) at visad.java3d.ShadowFunctionOrSetTypeJ3D.recurseRange(ShadowFunctionOrSetTypeJ3D.java:1209) at visad.java3d.ShadowFunctionOrSetTypeJ3D$1.run(ShadowFunctionOrSetTypeJ3D.java:181) 
  at visad.util.ThreadManager$1.run(ThreadManager.java:292)
  at visad.util.ThreadPool$ThreadMinnow.run(ThreadPool.java:98)

Do you think this error is due to the missing satellite_02 grids?
One more question: Also, as can be seen at time 4, the satellite grids are not always in the same order in the netCDF file. Is this a problem? I can have my colleague regenerate the data with grids in a consistent order if needed.
If I have answers to the above, I can probably figure out the following on my own. If all satellites must be present, maybe we need to start a new aggregation if a new satellite is added to the set or one is retired from service? 
Maybe we should be splitting out each satellite into their own netCDF files?

Thanks,
Ken
  • 2009 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: