Re: [thredds] Set harvest attribute using datasetScan

• To: Chiara Scaini <saetachiara@xxxxxxxxx>, thredds@xxxxxxxxxxxxxxxx
• Subject: Re: [thredds] Set harvest attribute using datasetScan
• From: Antonio S. Cofiño <cofinoa@xxxxxxxxx>
• Date: Wed, 4 Jul 2018 19:46:35 +0200

Hi Chiara,

I'm answering inline.



On 04/07/18 18:23, Chiara Scaini wrote:

Hi all, I'm setting up a geospatial data and metadata portal based on thredds catalog and the Geonetwork engine and web application. I am working on Linux CentOS and my applications are deployed with Tomcat8.

Which TDS version are you using?

I am populating a thredds catalog based on a filesystem containing meteorological data. Geonetwork then harvests the catalog and populates the application. However, and given that I'm updating the data on the web side, I would like to harvest only once the data.

I tried to set the 'harvest' attribute from the catalog, but without success. Here's an excerpt of my catalog.xml file:

  <datasetScan name="AUXILIARY" ID="testAUXILIARY"

               path="AUXILIARY" location="content/testdata/auxiliary-aux" harvest="true">

This harvest is correct.

    <metadata inherited="true">
      <serviceName>all</serviceName>
      <dataType>Grid</dataType>
      <dataFormatType>NetCDF</dataFormatType>
        <DatasetType harvest="true"></DatasetType>
        <harvest>true</harvest>

      <keyword>WRF outputs</keyword>

        <documentation type="summary">This is a summary for my test ARPA catalog for WRF runs. Runs are made at 12Z and 00Z, with analysis an         d forecasts every 6 hours out to 60 hours. Horizontal = 93 by 65 points, resolution 81.27 km, LambertConformal projection. Vertical = 1000 to

         100 hPa pressure levels.</documentation>
       <timeCoverage>
         <end>present</end>
         <duration>5 years</duration>
       </timeCoverage>
       <variables vocabulary="GRIB-1" />
       <variables vocabulary="">

         <variable name="Z_sfc" vocabulary_name="Geopotential H" units="gp m">Geopotential height, gpm</variable>

       </variables>
    </metadata>

    <filter>
      <include wildcard="*wrfout_*"/>
    </filter>

    <addDatasetSize/>

    <addTimeCoverage datasetNameMatchPattern="([0-9]{2})_([0-9]{4})-([0-9]{2})-([0-9]{2})_([0-9]{2}):([0-9]{2}):([0-9]{2})"

           startTimeSubstitutionPattern="$2-$3-$4T$5:00:00"
                  duration="6 hours" />

    <namer>

    <regExpOnName regExp="([0-9]{4})([0-9]{2})([0-9]{2})_([0-9]{2})" replaceString="WRF $1-$2-$3T$4:00:00" />     <regExpOnName regExp="([0-9]{2})_([0-9]{4})-([0-9]{2})-([0-9]{2})_([0-9]{2}):([0-9]{2}):([0-9]{2})" replaceString="WRF Domain-$1 $2-$3-$4T$5:00:00" />

    </namer>

  </datasetScan>

Even if I set the harvest="true" attribute, it is not inherited by the datasets and thus the harvester does not get the files. I can also ignore the 'harvest' attribute while harvesting, but my aim is to harvest only new files using an auxiliary catalog that contains symbolic links (and updating the Thredds path after harvesting).

Am I missing something? How would you sistematically add the harvest attribute to all inner datasets in a nested filesystem? Or, would it make sense to create two catalogs using the time filter options (ex. all up to yesterday in one catalog, and today's files in another)? Can you show me an example of usage of those filters in a datasetScan?

Many thanks,
Chiara

How this helps
Regards

Antonio

--
Antonio S. Cofiño
Dep. de Matemática Aplicada y
        Ciencias de la Computación
Universidad de Cantabria
http://www.meteo.unican.es

--
Chiara Scaini


_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/

• References:
  • [thredds] Set harvest attribute using datasetScan
    • From: Chiara Scaini