NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: [thredds] Read-Only GribIndex cache conflict

  • To: John Caron <caron@xxxxxxxx>
  • Subject: Re: [thredds] Read-Only GribIndex cache conflict
  • From: Dan Swank - NOAA Affiliate <dan.swank@xxxxxxxx>
  • Date: Mon, 10 Aug 2015 10:46:04 -0400
Correct,
I have a DatasetScan (that gives the R/O file system error) along
side featureCollections using the read-only pool of indecies - those work
OK.
The datasetScans want to put .ncx3's into the GribIndex configured path.

On Fri, Aug 7, 2015 at 3:45 PM, John Caron <caron@xxxxxxxx> wrote:

> You only want one TDM for each set of files that need indexing. Multiple
> TDMs would step on each other. So that part is right.
>
> Your problem is that you are serving the files out individually also.
> Presumably you want those individual indices to be made only on demand?
> Cached for a bit then discarded? Are these in a datasetScan?
>
> John
>
> On Fri, Aug 7, 2015 at 1:23 PM, Dan Swank - NOAA Affiliate <
> dan.swank@xxxxxxxx> wrote:
>
>> I have a 3 tier server environment.   And the production
>> will be load balenced.   So the sense is that we cannot
>> afford to have 4+ TDM servers all indexing 32+ Million file units
>> even semi-regularly.   Too much I/O.   So I am trying to make
>> a master TDM server that upkeeps the GribIndex, while the
>> TDS server(s) run from a read-only copy of that.
>> The TDM master will signal all TDS instances when it finishes a pass.
>>
>> Perhaps I am overlooking something obvious
>> about how to set this up better.
>>
>> On Thu, Aug 6, 2015 at 9:07 PM, John Caron <caron@xxxxxxxx> wrote:
>>
>>> Hi Dan:
>>>
>>> A a read-only GribIndex Cache doesnt make too much sense. Why read-only?
>>>
>>> John
>>>
>>> On Thu, Aug 6, 2015 at 8:47 AM, Dan Swank - NOAA Affiliate <
>>> dan.swank@xxxxxxxx> wrote:
>>>
>>>> (v.4.6.2)  For a few THREDDS instances I need to run
>>>> the server from a read-only GribIndex Cache that TDM may update
>>>> on other servers ~ and this file system is shared with various
>>>>  virtual machines for static use.
>>>>
>>>> I've run into an issue where the featureCollections will
>>>> work just fine under the above arrangement, but if attempting to use
>>>> Services [OpenDap/Ncss] on a directory listed file units - outside
>>>> the featureCollections, would attempt to create a .ncx3 into this
>>>>  GribIndex cache location - and fail.
>>>>
>>>> Is there currently a way a workaround?
>>>>
>>>> Being able to configure a fail-over location to wirte
>>>> on-demand non-FC, gribIndex files may be needed.
>>>>
>>>> ---
>>>> Dan Swank
>>>> NOMADS System Support, STG Inc. - Federal Government Contractor
>>>> NOAA's National Centers for Environmental Information (NCEI)
>>>> Data Stewardship Devision (DSD)
>>>> 151 Patton Ave
>>>> Asheville, NC 28801
>>>> dan.swank@xxxxxxxx
>>>> Phone: 828-271-4007
>>>> Customer Support:  828-271-4800 or http://www.ncdc.noaa.gov/contact
>>>>
>>>> The newly formed NCEI merges the National Climatic Data Center (NCDC),
>>>> the National Geophysical Data Center (NGDC), and the National Oceanographic
>>>> Data Center (NODC).
>>>>
>>>> Facebook: http://www.facebook.com/NOAANationalClimaticDataCenter
>>>> Twitter: @NOAANCDC @NOAAOceanData
>>>>
>>>> ======================= Disclaimer ===========
>>>> I am not a Federal Employee.
>>>> Any opinions expressed in this message are mine personally and
>>>> do not represent official positions from STG Inc, ERT, NCEI, or NOAA
>>>> ============================================
>>>>
>>>> _______________________________________________
>>>> thredds mailing list
>>>> thredds@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe,  visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>
>>>
>>>
>>
>>
>> --
>>
>> Dan Swank
>> NOMADS System Support, STG Inc. - Federal Government Contractor
>> NOAA's National Centers for Environmental Information (NCEI)
>> Data Stewardship Devision (DSD)
>> 151 Patton Ave
>> Asheville, NC 28801
>> dan.swank@xxxxxxxx
>> Phone: 828-271-4007
>> Customer Support:  828-271-4800 or http://www.ncdc.noaa.gov/contact
>>
>> The newly formed NCEI merges the National Climatic Data Center (NCDC),
>> the National Geophysical Data Center (NGDC), and the National Oceanographic
>> Data Center (NODC).
>>
>> Facebook: http://www.facebook.com/NOAANationalClimaticDataCenter
>> Twitter: @NOAANCDC @NOAAOceanData
>>
>> ======================= Disclaimer ===========
>> I am not a Federal Employee.
>> Any opinions expressed in this message are mine personally and
>> do not represent official positions from STG Inc, ERT, NCEI, or NOAA
>> ============================================
>>
>
>


-- 

Dan Swank
NOMADS System Support, STG Inc. - Federal Government Contractor
NOAA's National Centers for Environmental Information (NCEI)
Data Stewardship Devision (DSD)
151 Patton Ave
Asheville, NC 28801
dan.swank@xxxxxxxx
Phone: 828-271-4007
Customer Support:  828-271-4800 or http://www.ncdc.noaa.gov/contact

The newly formed NCEI merges the National Climatic Data Center (NCDC), the
National Geophysical Data Center (NGDC), and the National Oceanographic
Data Center (NODC).

Facebook: http://www.facebook.com/NOAANationalClimaticDataCenter Twitter:
@NOAANCDC @NOAAOceanData

======================= Disclaimer ===========
I am not a Federal Employee.
Any opinions expressed in this message are mine personally and
do not represent official positions from STG Inc, ERT, NCEI, or NOAA
============================================
  • 2015 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: