Re: [thredds] joinExisting and FMRC aggregation performance

• To: "Signell, Richard" <rsignell@xxxxxxxx>
• Subject: Re: [thredds] joinExisting and FMRC aggregation performance
• From: John Caron <caron@xxxxxxxx>
• Date: Fri, 13 Mar 2015 20:22:04 -0600

Hi Rich:

NcML Aggregations should only be used for small collections of files ( a
few dozen?) , because they are created on-the-fly. FMRC does a better job
of caching information so things go quicker. It handles the case of a
single time dimension as a special case of a Forecast model collection.
However, they too are limited in how much they will scale up, (< 100 ?)

GRIB collection in v4.6.0 finally are almost ready for large scale
collections (> 10K files). I will be improving the FMRC to use some of the
techniques in GRIB collections, slated for version 4.6.1. Not yet sure how
far that will get in scaling up, but I think we can do much better than now.

So how many files and variables are in the HF Radar collection?

John

On Fri, Mar 13, 2015 at 2:50 PM, Signell, Richard <rsignell@xxxxxxxx> wrote:

> Thredds community,
>
> The largest archive of HF Radar ocean surface current data is being
> served by THREDDS at
> http://hfrnet.ucsd.edu/thredds/catalog.html, but the erratic
> performance of the joinExisting aggregations has made them difficult
> to use.  The folks at UCSD discovered that if they use FMRC
> aggregations they work much better than the joinExisting, as borne out
> by this Ipython Notebook, where we just request WMS services from the
> two aggregations every minute for one hour:
>
> http://nbviewer.ipython.org/gist/rsignell-usgs/139d5481d74a1181e576
>
> I don't understand this behavior.  The joinExisting was designed for
> this type of aggregation (simply joining netcdf files along the time
> dimension) and the FMRC was instead designed for files with
> overlapping forecast times.  But there is no arguing with the results
> of this test: FMRC is clearly working better.
>
> Anyone have insight into why we are getting these results?
>
> Are there settings that could be changed to improve the performance of
> the joinExisting aggregation?
>
> Thanks,
> Rich
>
> P.S. the existing aggregation catalog and threddsConfig.xml settings
> are shown at the end of the notebook
>
> --
> Dr. Richard P. Signell   (508) 457-2229
> USGS, 384 Woods Hole Rd.
> Woods Hole, MA 02543-1598
>
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
>

• Follow-Ups:
  • Re: [thredds] joinExisting and FMRC aggregation performance
    • From: Signell, Richard
  • Re: [thredds] joinExisting and FMRC aggregation performance
    • From: Roy Mendelssohn - NOAA Federal

• References:
  • [thredds] joinExisting and FMRC aggregation performance
    • From: Signell, Richard