NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: [thredds] Time Partition Syntax
Here's an update on attempting a Time Partition for model analyses that are in GRIB1:
I have created a local data structure for testing where I can write to the file directories (until I can get permission to write to the file directories of our main archive). The structure is as follows: 

/glade/nwsc_scratch/manross/ds083.2/grib1/yyyy/yyyy.MM/fnl_yyyyMMdd_HH_00_c
When I found the message thread : http://www.unidata.ucar.edu/support/help/MailArchives/thredds/msg01820.html
I took the advice mentioned therein and grabbed the 4.3.17 version of the thredds war, and set my catalog.xml config as follows: 

<?xml version="1.0" encoding="UTF-8"?>
<catalog name="THREDDS Server Default Catalog : You must change this to fit your server!" 
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
        xmlns:xlink="http://www.w3.org/1999/xlink";>

  <service name="all" base="" serviceType="compound">
    <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
<service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" /> 
    <!--service name="wcs" serviceType="WCS" base="/thredds/wcs/" /-->
    <service name="wms" serviceType="WMS" base="/thredds/wms/" />
<service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/grid/" /> 
  </service>

  <!--datasetRoot path="FNL" location="content/ds083.2" /-->

  <featureCollection name="LOCAL_FNL"
                     path="LOCALFNL"
                     harvest="true"
                     featureType="GRIB">

    <metadata inherited="true">
      <serviceName>all</serviceName>
      <documentation type="summary">LOCAL FNLS FOR TESING
      </documentation>
      <dataFormat>GRIB-1</dataFormat>
    </metadata>

    <collection
spec="/glade/nwsc_scratch/manross/ds083.2/grib1/**/fnl_.*_c$"
        timePartition="1 day"
        dateFormatMark="#fnl_#yyyyMMdd_HH"
        />

    <update startup="true" rescan="0 0 0 * * ? *"/>
    <gribConfig datasetTypes="Best Files"/>

  </featureCollection>
</catalog>
Upon restarting tomcat, the directory was scanned and I was met with numerous "Too many open files" errors:
2013-04-29T13:12:17.795 -0600 [ 36341][ ] ERROR - fc.LOCAL_FNL - Grib1CollectionBuilder LOCAL_FNL-20130416 : reading/Creating gbx9 index for file /glade/nwsc_scratch/manross/ds083.2/grib1/2013/2013.04/fnl_20130416_18_00_c failed java.io.FileNotFoundException: /usr/share/tomcat6/content/thredds/cache/grib/glade-nwsc_scratch-manross-ds083.2-grib1-2013-2013.04-fnl_20130416_18_00_c.gbx9 (Too many open files) 
    at java.io.FileOutputStream.open(Native Method)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:209)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:160)
    at ucar.nc2.grib.grib1.Grib1Index.makeIndex(Grib1Index.java:199)
at ucar.nc2.grib.GribIndex.readOrCreateIndexFromSingleFile(GribIndex.java:143) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.makeAggregatedGroups(Grib1CollectionBuilder.java:469) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:434) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:167) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85) 
    at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177)
at ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:163) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73) 
    at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
at thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224) 
    at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
at thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:616)
    at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
at thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213) at thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392) at thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287) 
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) 2013-04-29T13:12:17.796 -0600 [ 36342][ ] ERROR - fc.LOCAL_FNL - Failed to open partition LOCAL_FNL-20130416 java.io.FileNotFoundException: /usr/share/tomcat6/content/thredds/cache/grib/glade-nwsc_scratch-manross-ds083.2-grib1-LOCAL_FNL-20130416.ncx (Too many open files) 
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118)
    at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:277)
    at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
at ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:569) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:435) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:167) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85) 
    at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177)
at ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:163) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73) 
    at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
at thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224) 
    at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
at thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:616)
    at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
at thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213) at thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392) at thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287) 
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) 2013-04-29T13:12:17.803 -0600 [ 36349][ ] ERROR - fc.LOCAL_FNL - Fail to create/update collection java.io.FileNotFoundException: /glade/nwsc_scratch/manross/ds083.2/grib1/LOCAL_FNL-20120605.ncx (Too many open files) 
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118)
    at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:277)
    at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
at ucar.nc2.grib.grib1.Grib1CollectionBuilder.readIndex(Grib1CollectionBuilder.java:196) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:163) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85) 
    at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177)
at ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.checkPartitions(Grib1TimePartitionBuilder.java:215) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:187) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73) 
    at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
at thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224) 
    at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
at thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:616)
    at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
at thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213) at thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392) at thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287) 
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
So I pared back the dataset from 3.5 years worth of 4 daily files to just the first month (still 4 files daily) of 2010-2013. This eliminated the "Too many open files" errors, and created the respective *.gbx9 and .ncx files, but threw a cast error:
2013-04-29T13:40:04.188 -0600 [ 108434][ ] INFO - fc.LOCAL_FNL - fnl_20130131_00_00_c: createIndex /glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_00_00_c.ncx 2013-04-29T13:40:04.351 -0600 [ 108597][ ] INFO - fc.LOCAL_FNL - fnl_20130131_06_00_c: createIndex /glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_06_00_c.ncx 2013-04-29T13:40:04.508 -0600 [ 108754][ ] INFO - fc.LOCAL_FNL - fnl_20130131_12_00_c: createIndex /glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_12_00_c.ncx 2013-04-29T13:40:04.649 -0600 [ 108895][ ] INFO - fc.LOCAL_FNL - fnl_20130131_18_00_c: createIndex /glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_18_00_c.ncx 2013-04-29T13:40:06.936 -0600 [ 111182][ ] ERROR - fc.LOCAL_FNL - Fail to create/update collection java.lang.ClassCastException: ucar.nc2.grib.GribCollection$VariableIndex cannot be cast to ucar.nc2.grib.TimePartition$VariableIndexPartitioned at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.writeGroupProto(Grib1TimePartitionBuilder.java:445) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.writeIndex(Grib1TimePartitionBuilder.java:413) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:197) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73) 
    at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
at thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224) 
    at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163)
at thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:616)
    at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177)
at thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213) at thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392) at thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287) 
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
And when trying to access my TDS from a browser, the catalog could not be made:
2013-04-29T13:40:33.330 -0600 [ 137576][ 7] INFO - threddsServlet - Remote host: 128.117.8.232 - Request: "GET /thredds/catalog/LOCALFNL/catalog.html HTTP/1.1" 2013-04-29T13:40:33.335 -0600 [ 137581][ 7] ERROR - fc.LOCAL_FNL - Error making catalog for LOCALFNL 
java.lang.NullPointerException
at thredds.catalog.InvDatasetFcGrib.makeCatalogTop(InvDatasetFcGrib.java:453) at thredds.catalog.InvDatasetFcGrib.makeCatalog(InvDatasetFcGrib.java:377) at thredds.servlet.DataRootHandler.makeDynamicCatalog(DataRootHandler.java:1467) at thredds.servlet.DataRootHandler.getCatalog(DataRootHandler.java:1430) at thredds.server.catalogservice.LocalCatalogServiceController.handleRequestInternal(LocalCatalogServiceController.java:191) at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153) at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936) 
    ...


For grins, I tried changing the partition to

        timePartition="directory"
        dateFormatMark="#grib1/#yyyy"

And got the same ClassCastException.
If I'm missing something obvious, please let me know. I understand that the GRIB feature collection is still in flux, so please consider this some testing info. I'm able to make a GRIB collection without the time partition, but for file browsing, that partition would be a feature we would really want to incorporate. 

Cheers!

-kevin.

--
Kevin Manross
NCAR/CISL/Data Support Section
Phone: (303)-497-1218
Email:manross@xxxxxxxx <mailto:manross@xxxxxxxx>
Web:http://rda.ucar.edu
  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: