Re: [thredds] NcML aggregation strangely pulling files from multiple datasets

• To: John Caron <caron@xxxxxxxxxxxxxxxx>
• Subject: Re: [thredds] NcML aggregation strangely pulling files from multiple datasets
• From: John Maurer <jmaurer@xxxxxxxxxx>
• Date: Thu, 19 May 2011 14:37:02 -1000

Hi John:

On Thu, May 19, 2011 at 2:17 PM, John Caron <caron@xxxxxxxxxxxxxxxx> wrote:

>  Hi John:
>
>
> On 5/19/2011 3:24 PM, John Maurer wrote:
>
> Hi all,
> I'm seeing some weird behavior today with one of our joinExisting NcML
> aggregations for high-frequency radio (HFR). The OPeNDAP page lists
> variables that are from a totally different dataset (e.g. HFR does not
> measure water temperature):
> http://oos.soest.hawaii.edu/thredds/idd/hfr.html?dataset=kokagg. I looked
> at "content/thredds/cache/agg/" at the aggregation cache file for the
> dataset (hioos-hfr-kokagg) and it confirms that it's strangely pulling files
> from 3 unrelated datasets: a glider, an NCOM global FMRC, and the HFR data.
> Since nothing has changed recently in my config file for the dataset and I
> haven't recently upgraded TDS, this has me worried about a possible disk
> problem from recent power outages? The three datasets are not subdirectories
> of each other, so I don't see how TDS can be lumping them together. Any
> ideas what could be causing this?
>
>
> A brief look at what you put on ftp confirms it : its a mystery ;^(
>
> No chance you are playing games with symlinks is there?
>

Nope, no use of symlinks involved.


> In addition to the above, 5 of my FMRC's are suddenly inaccessible today as
> well and were recently working just fine. It would seem their file
> aggregations may have similarly gone haywire as well, but their cache files
> under "content/thredds/collection/" (right?) are binary (.jdb) so I can't
> confirm whether they're combining multiple datasets.
>
>
> If you have remote debugging on (and you should!) you can look at the files
> with:
>
> https://server/thredds/admin/debug?Collections/showFmrcCache<https://motherlode.ucar.edu:8443/thredds/admin/debug?Collections/showFmrcCache>
>


Yep, I later tried that as well. For all of the FMRC's that were failing,
the cache had not been updated for several days.


> The catalogs give an error in threddsServlet.log with the message:
>         at
> ucar.nc2.ft.fmrc.FmrcDataset.getNetcdfDataset2D(FmrcDataset.java:166)
>         at ucar.nc2.ft.fmrc.Fmrc.getDataset2D(Fmrc.java:183)
>
>
> can you send the file content/thredds/logs/featureCollectionScan.log ?
>

Sorry, I would send that log except that it seems it was overwritten when I
restarted TDS so I can't find the same time period and error messages
anymore. Here are the relevant snippets that I dug out earlier during my
investigation. TDS can't seem to find files anymore, although the
directories and filenames are all correct and haven't changed for quite some
time:

[2011-05-19T12:21:46.145-1000] ERROR ucar.nc2.ft.fmrc.FmrcDataset: Fmrc
collection is empty =fmrc:NCOM Hawaii Regional Ocean Model
[2011-05-19T12:21:46.145-1000] ERROR ucar.nc2.ft.fmrc.Fmrc:
/export/lawelawe1/model/ncom/
hawaii/**/ncom_relo_hawaii_u_#yyyyMMddHH#_t.*\.nc$: makeFmrcInv failed

[2011-05-19T12:10:22.497-1000] ERROR ucar.nc2.ft.fmrc.FmrcDataset: Fmrc
collection is empty =fmrc:ROMS Hawaii Regional Ocean Model
[2011-05-19T12:10:22.497-1000] ERROR ucar.nc2.ft.fmrc.Fmrc:
/export/lawelawe1/model/ocn/roms_forecast/hiig/hiig_forec_#yyyyMMdd#\.nc$:
makeFmrcInv failed

[2011-05-19T12:22:16.456-1000] ERROR ucar.nc2.ft.fmrc.FmrcDataset: Fmrc
collection is empty =fmrc:ROMS Oahu South Shore Regional Ocean Model
[2011-05-19T12:22:16.456-1000] ERROR ucar.nc2.ft.fmrc.Fmrc:
/export/lawelawe1/model/ocn/roms_forecast/hiomsg/hiomsg_forec_#yyyyMMdd#\.nc$:
makeFmrcInv failed

[2011-05-19T12:10:12.752-1000] ERROR ucar.nc2.ft.fmrc.FmrcDataset: Fmrc
collection is empty =fmrc:WRF Maui-Oahu Regional Atmospheric Model
[2011-05-19T12:10:12.753-1000] ERROR ucar.nc2.ft.fmrc.Fmrc:
/export/lawelawe1/model/atm/wrf_mo/wrf_mo_#yyyyMMdd#\.nc$: makeFmrcInv
failed

[2011-05-19T12:11:11.026-1000] ERROR ucar.nc2.ft.fmrc.FmrcDataset: Fmrc
collection is empty =fmrc:WaveWatch III Global Wave Model
[2011-05-19T12:11:11.027-1000] ERROR ucar.nc2.ft.fmrc.Fmrc:
/export/lawelawe1/model/ore/ww3/ww3pac_#yyyyMMdd#\.nc$: makeFmrcInv failed



> What's the best way for me to clear all TDS caches? Is it safe to remove
> "content/thredds/cache/" entirely?, and then restart Tomcat/TDS? Will it get
> re-built then automatically after I access each dataset?
>
>
> yes, its safe, as roy says, it just slows things down. stop tds, remove,
> restart.
>
> i just did a bug fix release 4.2.7 with slightly better logging in case you
> want to install it.
>
> if you see this again, zip up content/thredds/cache/agg/* and send it.
>

OK, will do.

Thanks!,
John Maurer

• Follow-Ups:
  • Re: [thredds] NcML aggregation strangely pulling files from multiple datasets
    • From: John Caron

• References:
  • [thredds] NcML aggregation strangely pulling files from multiple datasets
    • From: John Maurer
  • Re: [thredds] NcML aggregation strangely pulling files from multiple datasets
    • From: John Caron