NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi Roland: Before I respond. can you give a rough idea of your metric for "big" is (or how many dataset elements and catalogRefs). Thanks, -Roy On Jan 4, 2011, at 8:35 AM, Roland Schweitzer wrote: > Thanks John. Among the groups we collaborate with there are some folks that > are quite concerned about the scaling issue. Personally, my direct > experience at this point that indicates that the performance is just fine (at > least so far) even with our largest catalogs. > > What's the experience of the list? Are folks seeing unacceptable TDS > initialization because of time spend reading catalogs? The thread from John > Maurer about aggregation access issues notwithstanding. > > Roland > > On 01/03/2011 07:34 PM, John Caron wrote: >> On 1/3/2011 10:53 AM, Roland Schweitzer wrote: >>> Hi, >>> >>> We're starting to put together some "big" server-side configuration >>> catalogs (both with "lots" of dataset elements and "lots" of catalogRef >>> elements). We are wondering about the process TDS goes through to read the >>> catalog when is starts. What gets cached? Does it have a way to know a >>> referenced catalog is unchanged? When do referenced catalogs get scanned? >>> And so on. >>> >>> Is there some documentation or a flow chart on how TDS initializes itself? >>> >>> Thanks, >>> Roland >>> >>> _______________________________________________ >>> thredds mailing list >>> thredds@xxxxxxxxxxxxxxxx >>> For list information or to unsubscribe, visit: >>> http://www.unidata.ucar.edu/mailing_lists/ >> >> Hi Roland: >> >> The sad answer is theres not much documentation. Weve been on the verge of >> redoing the initialization sequence for a few years now, so weve been >> waiting so we can document the clean, cool refactor instead of the crufty, >> lame current one. >> >> Anyway, the TDS reads in all the config catalogs at startup. It caches all >> of them, and uses the "expires" attribute on the catalog to decide if/when >> it needs to reread a catalog. It needs to read all catalogs, including >> catalogRef, because it has to know what the possible dataset URLs are, and >> there is no contract that a client has to read a catalog first. >> >> Obviously this doesnt scale forever. Ethan can probably fill in some details. >> >> see: >> http://www.unidata.ucar.edu/projects/THREDDS/tech/catalog/v1.0.2/InvCatalogSpec.html#catalog >> >> >> John >> >> _______________________________________________ >> thredds mailing list >> thredds@xxxxxxxxxxxxxxxx >> For list information or to unsubscribe, visit: >> http://www.unidata.ucar.edu/mailing_lists/ > > _______________________________________________ > thredds mailing list > thredds@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ ********************** "The contents of this message do not reflect any position of the U.S. Government or NOAA." ********************** Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center 1352 Lighthouse Avenue Pacific Grove, CA 93950-2097 e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address) voice: (831)-648-9029 fax: (831)-648-8440 www: http://www.pfeg.noaa.gov/ "Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected"
thredds archives: