Re: [thredds] TDS not refreshing catalog after TDM trigger

• To: Arnaud Dumont <dumont@xxxxxxxx>
• Subject: Re: [thredds] TDS not refreshing catalog after TDM trigger
• From: Sean Arms <sarms@xxxxxxxx>
• Date: Fri, 20 Mar 2020 09:56:18 -0600

Greetings Arnaud,

Is this TDS publicly accessible? I'm unable to access it from home,
but it could be on my end, so I just want to double check.

It looks like you are running the TDS out of a different context
(/gtgnTDS/ instead of /thredds/). Is that correct?

Sean

On Fri, Mar 20, 2020 at 8:23 AM Arnaud Dumont <dumont@xxxxxxxx> wrote:
>
> I'm struggling to get TDS to update its indices for a realtime GRIB dataset. 
> I'm running
>
>
>   version= 4.6.14 - 2019-07-23T11:04:31-0600
>
>   tdmFat-4.6.14.jar
>
>
> with the catalog.xml featureCollection set up as:
>
>   <featureCollection name="gtgn" featureType="GRIB2" path="gtgn">
>
>
>     <metadata inherited="true">
>
>       <serviceName>all</serviceName>
>
>       <dataType>Grid</dataType>
>
>     </metadata>
>
>
>
>     <collection name="gtgn"
>
> spec="/d1/data_archive/gtg_ncst_grib/**/gtgn.*\.grb2$"
>
> timePartition="file"
>
>                 dateFormatMark="gtg_ncst_grib/#yyyyMMdd'/gtgn.t'HHmm#" />
>
>
>
>     <update startup="never" trigger="allow"/>
>
>     <tdm rewrite="test" rescan="0 0/5 * * * ? *" trigger="allow" />
>
>
>   </featureCollection>
>
>
> When I run the tdm with the following args:
>
>   java -Xmx4g -Dtds.content.root.path=/d1/data_archive -jar 
> /d1/data_archive/thredds/tdmFat-4.6.14.jar  -cred "user:password" -log DEBUG 
> &> /d1/data_archive/tdm/runTdm.log
>
>
> I see that it correctly regenerates the indicies and triggers:
>
>
> 2020-03-20T13:19:48.644 +0000 DEBUG -  createIndex for 
> /d1/data_archive/gtg_ncst_grib/20200320/gtgn.t0730z.edr.f000.grb2.ncx3
>
> 2020-03-20T13:19:48.644 +0000 DEBUG -   write RecordMaps: bytes = 675 record 
> = 52 bytesPerRecord=12
>
> 2020-03-20T13:19:48.645 +0000 DEBUG -   write GribCollectionIndex= 700 bytes
>
> 2020-03-20T13:19:48.645 +0000 DEBUG - That took 152 msecs
>
> 2020-03-20T13:19:48.658 +0000 DEBUG -      Using canonical partition 
> gtgn.t1300z.edr.f000.grb2
>
> 2020-03-20T13:19:48.681 +0000 INFO  - RewriteFilePartition gtgn-20200320 took 
> 6648 msecs
>
> 2020-03-20T13:19:48.898 +0000 DEBUG -      Using canonical partition 
> gtgn-20200319
>
> 2020-03-20T13:19:48.988 +0000 INFO  - updateGribCollection gtgn changed true 
> took 20896 msecs
>
> 2020-03-20T13:19:48.989 +0000 DEBUG - 2020-03-20T13:19:48.988Z gtgn changed 
> true
>
> 2020-03-20T13:19:49.013 +0000 INFO  - send trigger to 
> http://localhost:8080/thredds/admin/collection/trigger?trigger=never&collection=gtgn
>  status = 200
>
> 2020-03-20T13:20:01.609 +0000 INFO  - updateGribCollection gtgn changed false 
> took 1269 msecs
>
> 2020-03-20T13:20:01.610 +0000 DEBUG - 2020-03-20T13:20:01.609Z gtgn changed 
> false
>
>
> However, the catalog (below) never updates:
>
>
>   http://poseidon.rap.ucar.edu:8080/gtgnTDS/catalog/gtgn/catalog.html
>
>
> even though the WMS getCapabilities (below) DOES update for the new files:
>
>
>   
> http://poseidon.rap.ucar.edu:8080/thredds/wms/gtgn/TP?service=WMS&version=1.3.0&request=GetCapabilities
>
>
> I'm relying on the xml catalog to determine run times for some other forecast 
> datasets, so having updated information is critical to my display application.
>
> ...and a seemingly unrelated oddity is that the getCapabilities' times are 
> all rounded to the hour, even though the files are 15-minutely. The HTML 
> catalog gets the times correct (when it's updating correctly, that is), but 
> the getCapabilities has each 15-minute file listed as being at the top of the 
> hour (dropping minutes field?). See the latest values in the "time" dimension 
> below:
>
> <Layer queryable="1">
> <Name>Eddy_dissipation_parameter_altitude_above_msl</Name>
> <Title>
> Eddy dissipation parameter @ Specific altitude above mean sea level
> </Title>
> <Abstract>
> Eddy dissipation parameter @ Specific altitude above mean sea level
> </Abstract>
> <EX_GeographicBoundingBox>
> <westBoundLongitude>-139.97003181300516</westBoundLongitude>
> <eastBoundLongitude>-57.26792343693751</eastBoundLongitude>
> <southBoundLatitude>16.20865459974452</southBoundLatitude>
> <northBoundLatitude>55.51688759604331</northBoundLatitude>
> </EX_GeographicBoundingBox>
> <BoundingBox CRS="CRS:84" minx="-139.97003181300516" 
> maxx="-57.26792343693751" miny="16.20865459974452" maxy="55.51688759604331"/>
> <Dimension name="elevation" units="m" default="0.0">
> 0.0,30.0,304.0,609.0,914.0,1219.0,1524.0,1828.0,2133.0,2438.0,2743.0,3048.0,3352.0,3657.0,3962.0,4267.0,4572.0,4876.0,5181.0,5486.0,5791.0,6096.0,6400.0,6705.0,7010.0,7315.0,7620.0,7924.0,8229.0,8534.0,8839.0,9144.0,9448.0,9753.0,10058.0,10363.0,10668.0,10972.0,11277.0,11582.0,11887.0,12192.0,12496.0,12801.0,13106.0,13411.0,13716.0,14020.0,14325.0,14630.0,14935.0,15240.0
> </Dimension>
> <Dimension name="time" units="ISO8601" multipleValues="true" current="true" 
> default="2020-03-20T13:00:00.000Z">
> 2019-11-13T19:00:00.000Z,2019-11-13T20:00:00.000Z,2019-11-13T21:00:00.000Z,2019-11-13T22:00:00.000Z,
> ...
> 2020-03-20T11:00:00.000Z,2020-03-20T11:00:00.000Z,2020-03-20T11:00:00.000Z,2020-03-20T11:00:00.000Z,2020-03-20T12:00:00.000Z,2020-03-20T12:00:00.000Z,2020-03-20T12:00:00.000Z,2020-03-20T12:00:00.000Z,2020-03-20T13:00:00.000Z,2020-03-20T13:00:00.000Z,2020-03-20T13:00:00.000Z,2020-03-20T13:00:00.000Z
>
> </Dimension>
>
>
> Any ideas on what I might be doing wrong or suggestions on how to work around 
> these issues?
>
> Thanks,
> Arnaud
> ---
> Arnaud Dumont
> Research Applications Laboratory (RAL)
> National Center for Atmospheric Research (NCAR)
> 3450 Mitchell Ln, Boulder CO, 80301
> dumont@xxxxxxxx
> +1(303)497-8434
>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit: 
> https://www.unidata.ucar.edu/mailing_lists/

• References:
  • [thredds] TDS not refreshing catalog after TDM trigger
    • From: Arnaud Dumont