NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: [thredds] Cannot get FeatureCollection Updated by tdm

  • To: "Sean Arms" <sarms@xxxxxxxx>
  • Subject: Re: [thredds] Cannot get FeatureCollection Updated by tdm
  • From: 唐健 <tangjian@xxxxxxxxxx>
  • Date: Tue, 19 Apr 2016 14:00:52 +0800 (GMT+08:00)
HI Sean,


Thanks a lot for help me! And bad news, it is still there.


Sorry about forgetting to mention before, I have already been using tdm 4.6.5 
the link just like you provided.


using:


/usr/java -Xmx4g -Dtds.content.root.path="/usr/local/tomcat/content" -jar 
tdmFat-4.6.5.jar -tds "http://localhost:8080/"; -cred tdm:secret -log DEBUG

I actually use "test" for rewrite and just change to "always" to see whether 
anything could happen.

the tdm log is:
2016-04-19T13:09:00.006 +0800 DEBUG - 2016-04-19T05:09:02016-04-19T13:05:51.447 
+0800 INFO  - FeatureCollection config=FeatureCollectionConfig name ='ncep' 
collectionName='ncep_all' type='GRIB2'
  spec='/home/jtang/Documents/THREDDS_NEW/**/.*\.grib2$'
  timePartition =directory
  updateConfig =UpdateConfig{userDefined=false, recheckAfter='null', 
rescan='null', triggerOk=true, updateType=never}
  tdmConfig =UpdateConfig{userDefined=true, recheckAfter='null', rescan='0 0/1 
* * * ? *', triggerOk=true, updateType=test}
  ProtoConfig{choice=Penultimate, change='null', param='null', 
outerNcml='null', cacheAll=true}
  hasInnerNcml =false
  GribConfig{datasets=[TwoD, Best, Files, Latest], useGenType=false, 
useTableVersion=false, intvMerge=true, useCenter=false, 
filesSortIncreasing=true}

2016-04-19T13:05:51.458 +0800 INFO  - scheduleJob startup scan force=test for 
'ncep_all' at 2016-04-19T13:05:54.453+0800
2016-04-19T13:05:51.467 +0800 INFO  - scheduleJob recurring scan for 'ncep_all' 
cronExpr=0 0/1 * * * ? *
2016-04-19T13:05:54.494 +0800 DEBUG - Grib2CollectionBuilder 
ncep_all-2016041500: makeGroups
2016-04-19T13:05:54.494 +0800 DEBUG -  
dcm=thredds.inventory.partition.DirectoryCollection@4c3c53c4
2016-04-19T13:05:54.635 +0800 DEBUG -   Index written: f003.grib2.gbx9 == 366 
records
2016-04-19T13:05:55.390 +0800 DEBUG -  Counter: nvars=135 records 366/366 
(1.000000) filtered=0 dups=0 (0.000000)

2016-04-19T13:05:55.415 +0800 DEBUG -  createIndex for 
/home/jtang/Documents/THREDDS_NEW/2016041500/ncep_all-2016041500.ncx3
2016-04-19T13:05:55.430 +0800 DEBUG -   write RecordMaps: bytes = 6322 record = 
366 bytesPerRecord=17
2016-04-19T13:05:55.486 +0800 DEBUG -   write GribCollectionIndex= 10150 bytes
2016-04-19T13:05:55.487 +0800 DEBUG - That took 994 msecs
2016-04-19T13:05:55.585 +0800 DEBUG -      Using canonical partition 
ncep_all-2016041500
2016-04-19T13:05:55.662 +0800 INFO  - updateGribCollection ncep_all changed 
true took 1188 msecs
2016-04-19T13:05:55.662 +0800 DEBUG - 2016-04-19T05:05:55.662Z ncep_all changed 
true
2016-04-19T13:06:01.650 +0800 INFO  - send trigger to 
http://localhost:8080/thredds/admin/collection/trigger?trigger=never&collection=ncep_all
 status = 200
2016-04-19T13:07:00.003 +0800 DEBUG - Grib2CollectionBuilder 
ncep_all-2016041500: makeGroups
2016-04-19T13:07:00.003 +0800 DEBUG -  
dcm=thredds.inventory.partition.DirectoryCollection@34fedf53
2016-04-19T13:07:00.037 +0800 DEBUG -   Index written: f006.grib2.gbx9 == 366 
records
2016-04-19T13:07:00.077 +0800 DEBUG -  Counter: nvars=135 records 732/732 
(1.000000) filtered=0 dups=0 (0.000000)

2016-04-19T13:07:00.077 +0800 DEBUG -  createIndex for 
/home/jtang/Documents/THREDDS_NEW/2016041500/ncep_all-2016041500.ncx3
2016-04-19T13:07:00.085 +0800 DEBUG -   write RecordMaps: bytes = 11061 record 
= 732 bytesPerRecord=15
2016-04-19T13:07:00.089 +0800 DEBUG -   write GribCollectionIndex= 10195 bytes
2016-04-19T13:07:00.089 +0800 DEBUG - That took 86 msecs
2016-04-19T13:07:00.093 +0800 DEBUG -      Using canonical partition 
ncep_all-2016041500
2016-04-19T13:07:00.116 +0800 INFO  - updateGribCollection ncep_all changed 
true took 114 msecs
2016-04-19T13:07:00.117 +0800 DEBUG - 2016-04-19T05:07:00.116Z ncep_all changed 
true
2016-04-19T13:07:00.133 +0800 INFO  - send trigger to 
http://localhost:8080/thredds/admin/collection/trigger?trigger=never&collection=ncep_all
 status = 200
2016-04-19T13:08:00.015 +0800 INFO  - updateGribCollection ncep_all changed 
false took 14 msecs
2016-04-19T13:08:00.015 +0800 DEBUG - 2016-04-19T05:08:00.015Z ncep_all changed 
false
2016-04-19T13:09:00.006 +0800 INFO  - updateGribCollection ncep_all changed 
false took 5 msecs
2016-04-19T13:09:00.006 +0800 DEBUG - 2016-04-19T05:09:00.006Z ncep_all changed 
false
2016-04-19T13:10:00.006 +0800 INFO  - updateGribCollection ncep_all changed 
false took 4 msecs
2016-04-19T13:10:00.006 +0800 DEBUG - 2016-04-19T05:10:00.006Z ncep_all changed 
false
2016-04-19T13:11:00.004 +0800 INFO  - updateGribCollection ncep_all changed 
false took 3 msecs
2016-04-19T13:11:00.005 +0800 DEBUG - 2016-04-19T05:11:00.005Z ncep_all changed 
false
2016-04-19T13:12:00.005 +0800 INFO  - updateGribCollection ncep_all changed 
false took 4 msecs
0.006Z ncep_all changed false
2016-04-19T13:10:00.006 +0800 INFO  - updateGribCollection ncep_all changed 
false took 4 msecs
2016-04-19T13:10:00.006 +0800 DEBUG - 2016-04-19T05:10:00.006Z ncep_all changed 
false
2016-04-19T13:11:00.004 +0800 INFO  - updateGribCollection ncep_all changed 
false took 3 msecs
2016-04-19T13:11:00.005 +0800 DEBUG - 2016-04-19T05:11:00.005Z ncep_all changed 
false
2016-04-19T13:12:00.005 +0800 INFO  - updateGribCollection ncep_all changed 
false took 4 msecs

As I described before,
in the http://localhost:8080/thredds/catalog/nwp/NCEP/catalog.html
ncep
|--Full collection
|--Best ncep Time Series
|--Latest Collection for ncep
|--ncep-2016041500

the first two (Full, Best) will be updated as I manually put more file in,
but not for the "Latest" and the "ncep-2016041500".

I know it is very weird and hope you could help get some clue.


Best wishes,
Jian




-----原始邮件-----
发件人:"Sean Arms" <sarms@xxxxxxxx>
发送时间:2016-04-18 23:59:37 (星期一)
收件人: "唐健" <tangjian@xxxxxxxxxx>
抄送: thredds <thredds@xxxxxxxxxxxxxxxx>
主题: Re: [thredds] Cannot get FeatureCollection Updated by tdm


Greetings Jian,


The configuration looks ok, but you could simplify it by removing the update 
element. The default for update (if the element is missing) is:


<update startup="nocheck" trigger="allow"/>


Also, I would set the TDM element's rewrite attribute to test, especially if 
the collection is really big - otherwise, you may run the risk of not being 
able to regenerate the collection index files within your 1 minute window. So, 
something like this:


<tdm rewrite="test" rescan="0 0/1 * * * ? *" trigger="allow"/>


Do you kno which version of the TDM you are using? If not, I would go ahead and 
download the latest TDM to be sure you are using the most up-to-date version.


http://artifacts.unidata.ucar.edu/content/repositories/unidata-releases/edu/ucar/tdmFat/4.6.5/tdmFat-4.6.5.jar


Let me know if that fixes the issue.


Cheers,


Sean


On Mon, Apr 18, 2016 at 8:17 AM, 唐健 <tangjian@xxxxxxxxxx> wrote:
HI everyone,
I am having hard time upgrading from 4.5.x to 4.6.5.
Everything works fine in 4.5.
The thing is when I am in 4.6.5, using FeatureCollection to get a directory of 
Grib2(orGRIB1) files, let's say NCEP_GFS fcst files, the directory is 
2016041600, and when the GRIB file comes in, the first time ncx3 and gbx9 was 
built, the TDM will send trigger to TDS, and it will be updated. But when more 
and more file comes in, it still will send trigger, but the TDS will not 
respond to it anymore.

First I thought it could be some problem due to windows or something. So I 
install openSuse on my laptop, and install both tomcat8 and Java(JDK)8. And I 
get SSL working as the same.

So I have the following XML:
Please help me check if I did something wrong.
in catalog.xml
<catalogRef xlink:title="Global Forecast System (GFS) 0p5deg" 
xlink:href="nwp_ncep_gfs_0p5deg.xml" name=""/>

and in nwp_ncep_gfs_0p5deg.xml

<?xml version="1.0" encoding="UTF-8"?>
<catalog name="NMC THREDDS Server Catalog" 
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"; 
xmlns:xlink="http://www.w3.org/1999/xlink";  version="1.0.2" >


<featureCollection name="ncep" featureType="GRIB2" path="nwp/NCEP" >
    <metadata inherited="true">
     <serviceName>GribServices</serviceName>
     <dataFormat>GRIB-2</dataFormat>
    </metadata>
    <collection spec="/home/jtang/Documents/THREDDS_NEW/**/.*\.grib2$"  
timePartition="directory" name="ncep_all"/>

    <update startup="test" trigger="allow"/>
    <tdm rewrite="always" rescan="0 0/1 * * * ? *" trigger="allow"/>

</featureCollection>

</catalog>

Since I use "GribServices" as the Service,  Full Collection (Reference / 
Forecast Time) Dataset, it will change when I manually put more file into the 
directory, but the 
"http://localhost:8080/thredds/catalog/nwp/NCEP/ncep_all-2016041500/catalog.html";
 it will not change.

If anyone have any clue, please feel free to jump in.

Thank you very much.

--
Dr. Jian TANG
Central Meteorological Office
National Meteorological Center of CMA

No. 46, Zhongguancun South Street, Haidian District, Beijing
P. R. China, 100081




_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/




--

Dr. Jian TANG
Central Meteorological Office
National Meteorological Center of CMA

No. 46, Zhongguancun South Street, Haidian District, Beijing
P. R. China, 100081




2016-04-19T12:32:08.484 +0800 INFO  - 
org.springframework.context.support.FileSystemXmlApplicationContext - 
Refreshing 
org.springframework.context.support.FileSystemXmlApplicationContext@148080bb: 
startup date [Tue Apr 19 12:32:08 CST 2016]; root of context hierarchy
2016-04-19T12:32:08.515 +0800 INFO  - 
org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML 
bean definitions from class path resource [resources/application-config.xml]
2016-04-19T12:32:08.730 +0800 INFO  - org.quartz.impl.StdSchedulerFactory - 
Using default implementation for ThreadExecutor
2016-04-19T12:32:08.732 +0800 INFO  - org.quartz.simpl.SimpleThreadPool - Job 
execution threads will use class loader of thread: main
2016-04-19T12:32:08.743 +0800 INFO  - org.quartz.core.SchedulerSignalerImpl - 
Initialized Scheduler Signaller of type: class 
org.quartz.core.SchedulerSignalerImpl
2016-04-19T12:32:08.744 +0800 INFO  - org.quartz.core.QuartzScheduler - Quartz 
Scheduler v.2.2.0 created.
2016-04-19T12:32:08.744 +0800 INFO  - org.quartz.simpl.RAMJobStore - 
RAMJobStore initialized.
2016-04-19T12:32:08.745 +0800 INFO  - org.quartz.core.QuartzScheduler - 
Scheduler meta-data: Quartz Scheduler (v2.2.0) 'DefaultQuartzScheduler' with 
instanceId 'NON_CLUSTERED'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
  Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support 
persistence. and is not clustered.

2016-04-19T12:32:08.745 +0800 INFO  - org.quartz.impl.StdSchedulerFactory - 
Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource 
file in Quartz package: 'quartz.properties'
2016-04-19T12:32:08.745 +0800 INFO  - org.quartz.impl.StdSchedulerFactory - 
Quartz scheduler version: 2.2.0
2016-04-19T12:32:08.745 +0800 INFO  - org.quartz.core.QuartzScheduler - 
Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
2016-04-19T12:32:08.892 +0800 INFO  - thredds.tdm.Tdm - TDS server added 
http://localhost:8080/
2016-04-19T12:32:08.894 +0800 INFO  - thredds.tdm.Tdm - ThreddsConfigReader 
reading xml file = /usr/local/tomcat/content/thredds/threddsConfig.xml
2016-04-19T12:32:08.924 +0800 INFO  - thredds.tdm.Tdm - TDM set 
DiskCache2{cachePathPolicy=NestedDirectory, alwaysUseCache=false, 
cachePathPolicyParam='null', 
root='/usr/local/tomcat/content/thredds/cache/grib/', scourEveryMinutes=-1, 
persistMinutes=-1, fail=false}
2016-04-19T12:32:08.957 +0800 INFO  - serverStartup - LoggerFactorySpecial add 
logger= fc.ncep_all file= /usr/local/tomcat/content/tdm/fc.ncep_all.log
2016-04-19T12:32:08.972 +0800 INFO  - 
org.springframework.context.support.FileSystemXmlApplicationContext - Closing 
org.springframework.context.support.FileSystemXmlApplicationContext@148080bb: 
startup date [Tue Apr 19 12:32:08 CST 2016]; root of context hierarchy
2016-04-19T12:32:12.108 +0800 INFO  - thredds.tdm.Tdm - 
GribCdmIndex.updateGribCollection ncep_all changed false
2016-04-19T12:33:00.557 +0800 INFO  - thredds.tdm.Tdm - 
GribCdmIndex.updateGribCollection ncep_all changed true
2016-04-19T13:05:50.549 +0800 INFO  - 
org.springframework.context.support.FileSystemXmlApplicationContext - 
Refreshing 
org.springframework.context.support.FileSystemXmlApplicationContext@148080bb: 
startup date [Tue Apr 19 13:05:50 CST 2016]; root of context hierarchy
2016-04-19T13:05:50.608 +0800 INFO  - 
org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML 
bean definitions from class path resource [resources/application-config.xml]
2016-04-19T13:05:51.028 +0800 INFO  - org.quartz.impl.StdSchedulerFactory - 
Using default implementation for ThreadExecutor
2016-04-19T13:05:51.032 +0800 INFO  - org.quartz.simpl.SimpleThreadPool - Job 
execution threads will use class loader of thread: main
2016-04-19T13:05:51.051 +0800 INFO  - org.quartz.core.SchedulerSignalerImpl - 
Initialized Scheduler Signaller of type: class 
org.quartz.core.SchedulerSignalerImpl
2016-04-19T13:05:51.053 +0800 INFO  - org.quartz.core.QuartzScheduler - Quartz 
Scheduler v.2.2.0 created.
2016-04-19T13:05:51.054 +0800 INFO  - org.quartz.simpl.RAMJobStore - 
RAMJobStore initialized.
2016-04-19T13:05:51.056 +0800 INFO  - org.quartz.core.QuartzScheduler - 
Scheduler meta-data: Quartz Scheduler (v2.2.0) 'DefaultQuartzScheduler' with 
instanceId 'NON_CLUSTERED'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
  Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support 
persistence. and is not clustered.

2016-04-19T13:05:51.056 +0800 INFO  - org.quartz.impl.StdSchedulerFactory - 
Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource 
file in Quartz package: 'quartz.properties'
2016-04-19T13:05:51.056 +0800 INFO  - org.quartz.impl.StdSchedulerFactory - 
Quartz scheduler version: 2.2.0
2016-04-19T13:05:51.056 +0800 INFO  - org.quartz.core.QuartzScheduler - 
Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
2016-04-19T13:05:51.354 +0800 INFO  - thredds.tdm.Tdm - TDS server added 
http://localhost:8080/
2016-04-19T13:05:51.357 +0800 INFO  - thredds.tdm.Tdm - ThreddsConfigReader 
reading xml file = /usr/local/tomcat/content/thredds/threddsConfig.xml
2016-04-19T13:05:51.406 +0800 INFO  - thredds.tdm.Tdm - TDM set 
DiskCache2{cachePathPolicy=NestedDirectory, alwaysUseCache=false, 
cachePathPolicyParam='null', 
root='/usr/local/tomcat/content/thredds/cache/grib/', scourEveryMinutes=-1, 
persistMinutes=-1, fail=false}
2016-04-19T13:05:51.445 +0800 INFO  - serverStartup - LoggerFactorySpecial add 
logger= fc.ncep_all file= /usr/local/tomcat/content/tdm/fc.ncep_all.log
2016-04-19T13:05:51.468 +0800 INFO  - 
org.springframework.context.support.FileSystemXmlApplicationContext - Closing 
org.springframework.context.support.FileSystemXmlApplicationContext@148080bb: 
startup date [Tue Apr 19 13:05:50 CST 2016]; root of context hierarchy
2016-04-19T13:05:55.662 +0800 INFO  - thredds.tdm.Tdm - 
GribCdmIndex.updateGribCollection ncep_all changed true
2016-04-19T13:07:00.116 +0800 INFO  - thredds.tdm.Tdm - 
GribCdmIndex.updateGribCollection ncep_all changed true
2016-04-19T13:08:00.015 +0800 INFO  - thredds.tdm.Tdm - 
GribCdmIndex.updateGribCollection ncep_all changed false
2016-04-19T13:09:00.006 +0800 INFO  - thredds.tdm.Tdm - 
GribCdmIndex.updateGribCollection ncep_all changed false
2016-04-19T13:10:00.006 +0800 INFO  - thredds.tdm.Tdm - 
GribCdmIndex.updateGribCollection ncep_all changed false
2016-04-19T13:11:00.005 +0800 INFO  - thredds.tdm.Tdm - 
GribCdmIndex.updateGribCollection ncep_all changed false
2016-04-19T13:12:00.006 +0800 INFO  - thredds.tdm.Tdm - 
GribCdmIndex.updateGribCollection ncep_all changed false
2016-04-19T13:13:00.004 +0800 INFO  - thredds.tdm.Tdm - 
GribCdmIndex.updateGribCollection ncep_all changed false
2016-04-19T13:14:00.004 +0800 INFO  - thredds.tdm.Tdm - 
GribCdmIndex.updateGribCollection ncep_all changed false
2016-04-19T13:15:00.005 +0800 INFO  - thredds.tdm.Tdm - 
GribCdmIndex.updateGribCollection ncep_all changed false
  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: