Re: [thredds] How to speed up gbx9 file generation

• To: "Sean Arms" <sarms@xxxxxxxx>
• Subject: Re: [thredds] How to speed up gbx9 file generation
• From: 唐健 <tangjian@xxxxxxxxxx>
• Date: Thu, 7 Jul 2016 08:49:29 +0800 (GMT+08:00)

Hi Sean,


Thanks for the quick reply.
I was trying to google yesterday and found it in application-config.xml also.
But cannot find the flag to use it beside from ant the source.


One more question, is it safe to use multi-thread? Because I find this 
https://github.com/Unidata/thredds/issues/496 .
I am using TDM 4.5 and TDS 4.5.5 right now.


Yesterday I also try to run multi TDM by using the flag like " -catalog 
nwp_cma_grapes_global.xml "
like this, acutually it works...


runTDM.bat:


start "" "C:/Program Files/Java/jdk1.7.0_55/bin/java" -Xmx1g 
-Dtds.content.root.path="C:\Program Files\Apache Software Foundation\Tomcat 
7.0\content" -jar tdm-4.5.jar -catalog nwp_cma_grapes_global.xml -tds 
"http://thredds.cma.gov.cn/";
start "" "C:/Program Files/Java/jdk1.7.0_55/bin/java" -Xmx1g 
-Dtds.content.root.path="C:\Program Files\Apache Software Foundation\Tomcat 
7.0\content" -jar tdm-4.5.jar -catalog nwp_cma_grapes_meso.xml -tds 
"http://thredds.cma.gov.cn/";
start "" "C:/Program Files/Java/jdk1.7.0_55/bin/java" -Xmx1g 
-Dtds.content.root.path="C:\Program Files\Apache Software Foundation\Tomcat 
7.0\content" -jar tdm-4.5.jar -catalog nwp_cma_grapes_ruc.xml -tds 
"http://thredds.cma.gov.cn/";
start "" "C:/Program Files/Java/jdk1.7.0_55/bin/java" -Xmx1g 
-Dtds.content.root.path="C:\Program Files\Apache Software Foundation\Tomcat 
7.0\content" -jar tdm-4.5.jar -catalog nwp_cma_t639.xml -tds 
"http://thredds.cma.gov.cn/";


I will try the multi-threads way this afternoon.
Thanks very much again!


Cheers,
Jian



-----原始邮件-----
发件人:"Sean Arms" <sarms@xxxxxxxx>
发送时间:2016-07-07 05:27:37 (星期四)
收件人: "唐健" <tangjian@xxxxxxxxxx>
抄送: thredds <thredds@xxxxxxxxxxxxxxxx>
主题: Re: [thredds] How to speed up gbx9 file generation


Greetings Jian,


The performance of the TDM will be limited to the performance of your I/O 
system. While running more than one TDM against a single TDS won't work due to 
the way the TDM finds collections that need scanned, you can run a single TDM 
in a multithreaded mode. In your TDM run script, you can set the number of 
threads to use by setting the flag


-nthreads=<number of threads>


By default, the TDM runs with one thread.


Cheers,


Sean




On Wed, Jul 6, 2016 at 12:40 AM, 唐健 <tangjian@xxxxxxxxxx> wrote:
HI everyone

We run our thredds server and TDM on a virtual workstation with 2.0GHz 
E78850*2, and 32GB memory, windows server 2008.
The script run TDM is like this
"C:/Program Files/Java/jdk1.7.0_55/bin/java" -Xmx10g 
-Dtds.content.root.path="C:\Program Files\Apache Software Foundation\Tomcat 
7.0\content" -jar tdm-4.5.jar -tds "http://thredds.cma.gov.cn/"; -cred ..
In operation it only take like 0.5GB memory.

My problem is almost all global model and meso-scale model output comes in at 
the most busy time around 3-4 pm local time.
and there are a lot of files. it takes quite long time for tdm to build indexes.
for example, when one model output is all there, tdm is busy building indexes 
for another model, and it cannot generate gbx9 and ncx2 file for this model.
Is there anything I can do about it?
Maybe run multi-TDM?
or put one TDM in charge of every single model output?

or can I use any other program to scan dirs and generate gbx9 for each file 
once it comes in ?

If anyone has any suggestion, please let me know.
Many thanks


Cheers
Jian


--
Dr. Jian TANG
Central Meteorological Office
National Meteorological Center of CMA



_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/




--

Dr. Jian TANG
Central Meteorological Office
National Meteorological Center of CMA

No. 46, Zhongguancun South Street, Haidian District, Beijing
P. R. China, 100081

• Follow-Ups:
  • Re: [thredds] How to speed up gbx9 file generation
    • From: Sean Arms

• References:
  • [thredds] How to speed up gbx9 file generation
    • From: 唐健
  • Re: [thredds] How to speed up gbx9 file generation
    • From: Sean Arms