NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi Dan,Thank you (and Tim Pugh) for the reminder to look at the JVM settings on the OS. Through John and Lansings's work, I have been able to get some of the time partitioning to work with my data set.
I am still getting the "too many files" error when trying to set up a daily partition, but no such errors with a monthly partition. I have yet to modify the OS JVM, and I'll try that today.
Much appreciated! -kevin. On 5/1/13 7:49 AM, Dan Swank - NOAA Affiliate wrote:
Just a quick plug... Been closely working with Jahn Caron (up until he left for EGU), and trying to get featureCollections working for all the major NCDC-NOMADS Archive datasets. Both Grib1 & 2 ~ With moderate success. Alot (but not all) of the issues I found were due to problems on our own file system. Not permissions, but rather zero-byte and/or corrupt/damaged GRIB records wrecking havoc with the GRIB scanner and cause it to fail to create the master time partition / index files. Caron added some modifications in the last few .wars to make the GRIB scanner more tolerant, but as you mentioned there are still lingering issues with it. The "too many open files" sounds like you may need to adjust an OS or JAVA env setting. Not an expert on this and perhaps someone else can chime in. In short, it may not necessarily be related to the featureCollectionScan. -Dan NCDC-NOMADS projectOn Mon, Apr 29, 2013 at 4:17 PM, Kevin Manross <manross@xxxxxxxx <mailto:manross@xxxxxxxx>> wrote:Here's an update on attempting a Time Partition for model analyses that are in GRIB1: I have created a local data structure for testing where I can write to the file directories (until I can get permission to write to the file directories of our main archive). The structure is as follows: /glade/nwsc_scratch/manross/ds083.2/grib1/yyyy/yyyy.MM/fnl_yyyyMMdd_HH_00_c When I found the message thread : http://www.unidata.ucar.edu/support/help/MailArchives/thredds/msg01820.html I took the advice mentioned therein and grabbed the 4.3.17 version of the thredds war, and set my catalog.xml config as follows: <?xml version="1.0" encoding="UTF-8"?> <catalog name="THREDDS Server Default Catalog : You must change this to fit your server!"xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";<http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0> xmlns:xlink="http://www.w3.org/1999/xlink"; <http://www.w3.org/1999/xlink>> <service name="all" base="" serviceType="compound"> <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" /> <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" /> <!--service name="wcs" serviceType="WCS" base="/thredds/wcs/" /--> <service name="wms" serviceType="WMS" base="/thredds/wms/" /> <service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/grid/" /> </service> <!--datasetRoot path="FNL" location="content/ds083.2" /--> <featureCollection name="LOCAL_FNL" path="LOCALFNL" harvest="true" featureType="GRIB"> <metadata inherited="true"> <serviceName>all</serviceName> <documentation type="summary">LOCAL FNLS FOR TESING </documentation> <dataFormat>GRIB-1</dataFormat> </metadata> <collection spec="/glade/nwsc_scratch/manross/ds083.2/grib1/**/fnl_.*_c$" timePartition="1 day" dateFormatMark="#fnl_#yyyyMMdd_HH" /> <update startup="true" rescan="0 0 0 * * ? *"/> <gribConfig datasetTypes="Best Files"/> </featureCollection> </catalog> Upon restarting tomcat, the directory was scanned and I was met with numerous "Too many open files" errors: 2013-04-29T13:12:17.795 -0600 [ 36341][ ] ERROR - fc.LOCAL_FNL - Grib1CollectionBuilder LOCAL_FNL-20130416 : reading/Creating gbx9 index for file /glade/nwsc_scratch/manross/ds083.2/grib1/2013/2013.04/fnl_20130416_18_00_c failed java.io.FileNotFoundException: /usr/share/tomcat6/content/thredds/cache/grib/glade-nwsc_scratch-manross-ds083.2-grib1-2013-2013.04-fnl_20130416_18_00_c.gbx9 (Too many open files) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:209) at java.io.FileOutputStream.<init>(FileOutputStream.java:160) at ucar.nc2.grib.grib1.Grib1Index.makeIndex(Grib1Index.java:199) at ucar.nc2.grib.GribIndex.readOrCreateIndexFromSingleFile(GribIndex.java:143) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.makeAggregatedGroups(Grib1CollectionBuilder.java:469) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:434) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:167) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85) at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177) at ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:163) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73) at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96) at thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224) at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163) at thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177) at thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213) at thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392) at thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287) at org.quartz.core.JobRunShell.run(JobRunShell.java:213) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) 2013-04-29T13:12:17.796 -0600 [ 36342][ ] ERROR - fc.LOCAL_FNL - Failed to open partition LOCAL_FNL-20130416 java.io.FileNotFoundException: /usr/share/tomcat6/content/thredds/cache/grib/glade-nwsc_scratch-manross-ds083.2-grib1-LOCAL_FNL-20130416.ncx (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118) at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:277) at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:259) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:569) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.createIndex(Grib1CollectionBuilder.java:435) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:167) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85) at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177) at ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:163) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73) at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96) at thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224) at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163) at thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177) at thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213) at thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392) at thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287) at org.quartz.core.JobRunShell.run(JobRunShell.java:213) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) 2013-04-29T13:12:17.803 -0600 [ 36349][ ] ERROR - fc.LOCAL_FNL - Fail to create/update collection java.io.FileNotFoundException: /glade/nwsc_scratch/manross/ds083.2/grib1/LOCAL_FNL-20120605.ncx (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118) at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:277) at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:259) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.readIndex(Grib1CollectionBuilder.java:196) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.readOrCreateIndex(Grib1CollectionBuilder.java:163) at ucar.nc2.grib.grib1.Grib1CollectionBuilder.factory(Grib1CollectionBuilder.java:85) at ucar.nc2.grib.GribCollection.factory(GribCollection.java:177) at ucar.nc2.grib.TimePartition$Partition.makeGribCollection(TimePartition.java:172) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.checkPartitions(Grib1TimePartitionBuilder.java:215) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:187) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73) at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96) at thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224) at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163) at thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177) at thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213) at thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392) at thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287) at org.quartz.core.JobRunShell.run(JobRunShell.java:213) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) So I pared back the dataset from 3.5 years worth of 4 daily files to just the first month (still 4 files daily) of 2010-2013. This eliminated the "Too many open files" errors, and created the respective *.gbx9 and .ncx files, but threw a cast error: 2013-04-29T13:40:04.188 -0600 [ 108434][ ] INFO - fc.LOCAL_FNL - fnl_20130131_00_00_c: createIndex /glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_00_00_c.ncx 2013-04-29T13:40:04.351 -0600 [ 108597][ ] INFO - fc.LOCAL_FNL - fnl_20130131_06_00_c: createIndex /glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_06_00_c.ncx 2013-04-29T13:40:04.508 -0600 [ 108754][ ] INFO - fc.LOCAL_FNL - fnl_20130131_12_00_c: createIndex /glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_12_00_c.ncx 2013-04-29T13:40:04.649 -0600 [ 108895][ ] INFO - fc.LOCAL_FNL - fnl_20130131_18_00_c: createIndex /glade/nwsc_scratch/manross/ds083.2/grib1.1/2013/2013.01/fnl_20130131_18_00_c.ncx 2013-04-29T13:40:06.936 -0600 [ 111182][ ] ERROR - fc.LOCAL_FNL - Fail to create/update collection java.lang.ClassCastException: ucar.nc2.grib.GribCollection$VariableIndex cannot be cast to ucar.nc2.grib.TimePartition$VariableIndexPartitioned at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.writeGroupProto(Grib1TimePartitionBuilder.java:445) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.writeIndex(Grib1TimePartitionBuilder.java:413) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.createPartitionedIndex(Grib1TimePartitionBuilder.java:197) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:128) at ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73) at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96) at thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:224) at thredds.catalog.InvDatasetFcGrib.update(InvDatasetFcGrib.java:163) at thredds.catalog.InvDatasetFeatureCollection.handleCollectionEvent(InvDatasetFeatureCollection.java:236) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at ucar.nc2.util.ListenerManager.sendEvent(ListenerManager.java:177) at thredds.inventory.CollectionManagerAbstract.sendEvent(CollectionManagerAbstract.java:213) at thredds.inventory.MFileCollectionManager.scan(MFileCollectionManager.java:392) at thredds.inventory.CollectionUpdater$UpdateCollectionJob.execute(CollectionUpdater.java:287) at org.quartz.core.JobRunShell.run(JobRunShell.java:213) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) And when trying to access my TDS from a browser, the catalog could not be made: 2013-04-29T13:40:33.330 -0600 [ 137576][ 7] INFO - threddsServlet - Remote host: 128.117.8.232 - Request: "GET /thredds/catalog/LOCALFNL/catalog.html HTTP/1.1" 2013-04-29T13:40:33.335 -0600 [ 137581][ 7] ERROR - fc.LOCAL_FNL - Error making catalog for LOCALFNL java.lang.NullPointerException at thredds.catalog.InvDatasetFcGrib.makeCatalogTop(InvDatasetFcGrib.java:453) at thredds.catalog.InvDatasetFcGrib.makeCatalog(InvDatasetFcGrib.java:377) at thredds.servlet.DataRootHandler.makeDynamicCatalog(DataRootHandler.java:1467) at thredds.servlet.DataRootHandler.getCatalog(DataRootHandler.java:1430) at thredds.server.catalogservice.LocalCatalogServiceController.handleRequestInternal(LocalCatalogServiceController.java:191) at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153) at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936) ... For grins, I tried changing the partition to timePartition="directory" dateFormatMark="#grib1/#yyyy" And got the same ClassCastException. If I'm missing something obvious, please let me know. I understand that the GRIB feature collection is still in flux, so please consider this some testing info. I'm able to make a GRIB collection without the time partition, but for file browsing, that partition would be a feature we would really want to incorporate. Cheers! -kevin.-- Kevin ManrossNCAR/CISL/Data Support Section Phone: (303)-497-1218 <tel:%28303%29-497-1218> Email:manross@xxxxxxxx <mailto:manross@xxxxxxxx> Web:http://rda.ucar.edu _______________________________________________ thredds mailing list thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx> For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/ -- ======================================= Dan Swank NCDC-NOMADS Project Team ERT/STG (NCDC Government Contractor) Data Access & Applications Branch Veach-Baley Federal Building 151 Patton Avenue Asheville, NC 28801-5001 Email: Dan.Swank@xxxxxxxx <mailto:Dan.Swank@xxxxxxxx> Phone: 828-271-4007 ============================================ Any opinions expressed in this message are mine personally and do not represent any position from STG, ERT, NCDC, or NOAA ============================================
-- Kevin Manross NCAR/CISL/Data Support Section Phone: (303)-497-1218 Email:manross@xxxxxxxx <mailto:manross@xxxxxxxx> Web:http://rda.ucar.edu
thredds archives: