NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
I was just addressing the issue of the peculiar dds and trying to guess why Hydrax and TDS came up with very different structures -- renaming the dimensions should make the two servers serve a more similar structure. Updating is a different issue. Benno On Thu, Feb 23, 2012 at 4:23 PM, Claude DEROGNAT <cderognat@xxxxxxx> wrote: > Hi Benno,**** > > ** ** > > I just have to change the name of dimension to have a continuous update of > the targeted file, we mean ?**** > > Thank you for the correction in my interpretation of the CF convention.*** > * > > ** ** > > Regards, **** > > ** ** > > Ing. Claude DEROGNAT, PhD* > **ARIA** **TECHNOLOGIES *SA > 8-10, rue de la Ferme* - *92100 Boulogne-Billancourt * *- France > tél. / *phone:* 33 (0)1 46 08 68 60 > ligne directe / *direct line: *33 (0)1 46 08 68 65 > fax: 33 (0)1 41 41 93 17 > e-mail: *cderognat@xxxxxxx** *- http : www.aria.fr **** > > *P** **Avant d'imprimer ce courriel**, réfléchissez à l'impact sur > l'environnement, > et dans la mesure du possible éditez les documents en recto verso, merci!* > **** > > ** ** > > ** ** > > *De :* bennoblumenthal@xxxxxxxxx [mailto:bennoblumenthal@xxxxxxxxx] *De > la part de* Benno Blumenthal > *Envoyé :* jeudi 23 février 2012 22:12 > *À :* Claude DEROGNAT > *Cc :* Nathan Potter; support@xxxxxxxxxxx; Patrick West; Nathan Potter; > thredds@xxxxxxxxxxxxxxxx > *Objet :* Re: [thredds] OLFS and/or BES configuration to obtain a dynamic > refresh of existing files ...**** > > ** ** > > While I haven't seen the original file, my point is that you are violating > the CF convention (assuming I am correct in thinking that the TDS matches > the netcdf structure): variables that have the same name as a dimension > need to be one dimensional. > > Benno**** > > On Thu, Feb 23, 2012 at 3:57 PM, Claude DEROGNAT <cderognat@xxxxxxx> > wrote:**** > > Hi Benno,**** > > **** > > Thank you for your answer. The corresponding name between grid coordinates > and array dimension are driven by the NetCdf CF convention…**** > > What about my real time constrain, so the require continuous update of the > provides data ? How can I reach the property from my TDS catalog ?**** > > **** > > Regards,**** > > **** > > Claude**** > > **** > > Ing. Claude DEROGNAT, PhD* > **ARIA** **TECHNOLOGIES *SA > 8-10, rue de la Ferme* - *92100 Boulogne-Billancourt * *- France > tél. / *phone:* 33 (0)1 46 08 68 60 > ligne directe / *direct line: *33 (0)1 46 08 68 65 > fax: 33 (0)1 41 41 93 17 > e-mail: *cderognat@xxxxxxx** *- http : www.aria.fr **** > > *P** **Avant d'imprimer ce courriel**, réfléchissez à l'impact sur > l'environnement, > et dans la mesure du possible éditez les documents en recto verso, merci!* > **** > > **** > > **** > > *De :* bennoblumenthal@xxxxxxxxx [mailto:bennoblumenthal@xxxxxxxxx] *De > la part de* Benno Blumenthal > *Envoyé :* jeudi 23 février 2012 19:45 > *À :* Nathan Potter > *Cc :* Claude DEROGNAT; support@xxxxxxxxxxx Support; Patrick West; Nathan > Potter; thredds@xxxxxxxxxxxxxxxx > *Objet :* Re: [thredds] OLFS and/or BES configuration to obtain a dynamic > refresh of existing files ...**** > > **** > > As for the two different structures -- the TDS generated a pure flat > structure, Hyrax made Grid structures, from the netcdf API point of view > they are very similar, though there is a significant difference w.r.t. > height. > > I wonder if the height variable is the reason TDS did not try to generate > Grid structures -- It is a netcdf convention that the height dimension > should have a height variable that is also one-dimensional. In this case, > the height variable is four-dimensional. If the height dimension were > renamed slightly, e.g. heightdim, you might get very different (and more > consistent) results. > > Benno**** > > On Thu, Feb 23, 2012 at 1:09 PM, Nathan Potter <ndp@xxxxxxxxxxx> wrote:*** > * > > **** > > **** > > **** > > On Feb 23, 2012, at 4:24 AM, Claude DEROGNAT wrote:**** > > ** ** > > Hi, > > My IT said that the file I send you is an old one. My problem is probably > link to my THREEDS catalog. > The question is so probably to threeds mailing list. I send them my > question on Monday just after your message and I still have no response > back ... > So, I send you my catalog if you can have a look to it you may find why > there a mismatch between > > http://web.aria.fr:443/thredds/dodsC/CHIMERE@CAMAC@reference@p02/CF_CHIMERE_20120115_d03.nc.html > and > > http://web.aria.fr:443/LENVIS/CHIMERE/CAMAC/reference/p02/CF_CHIMERE_20120115_d03.nc.html > **** > > **** > > **** > > Claude, > > I look at the first URL, saw that it was a instance of THREDDS Data Server > (a fairly recent version 4.0.26) and I forwarded you message to their > support list. > > It was only later that I realized the second URL is what appears to be a > VERY old (circa version 1.5.0), locally compiled, version of our Hyrax > server. > > So you are running two entirely different implementations of an OPeNDAP > server. > > You can see this clearly when you look at one of the collections with a > browser: > > *TDS*: > http://web.aria.fr:443/thredds/dodsC/CHIMERE@CAMAC@reference@p02/ > > *Hyrax*: http://web.aria.fr:443/LENVIS/CHIMERE/CAMAC/reference/p02/*** > * > > **** > > why the second is continuously updated but doesn't allow the access to the > whole set of variables stored in NetCdf file. > Why the first one are not continuously updated even if the date in the > threeds catalogs presentation always mentioned the right modified time .. . > **** > > > So with regards to how often datasets are "updated" - this is a function > of how the servers are implemented. I don't know what is going on in the > TDS, but in the Hyrax server each new request gets the file information > regarding file size and last modified times from the files system on which > the dataset resides. Hyrax also doesn't cache responses, so it will > re-interrogate the file every time you ask for something in or about the > file.**** > > **** > > The TDS folks are better informed to comment on what's going on in the TDS > with regards to this.**** > > Now I think you are also saying that these two URLs:**** > > **** > > > http://web.aria.fr:443/thredds/dodsC/CHIMERE@CAMAC@reference@p02/CF_CHIMERE_20120115_d03.nc > **** > > > http://web.aria.fr:443/LENVIS/CHIMERE/CAMAC/reference/p02/CF_CHIMERE_20120115_d03.nc > **** > > **** > > Reference the exact same file on the exact same file system. Is that the > case?**** > > **** > > If so, the fact that the data structure representations generated by each > server for this file are so different is somewhat surprising:**** > > **** > > **** > > From the TDS:**** > > **** > > > http://web.aria.fr:443/thredds/dodsC/CHIMERE@CAMAC@reference@p02/CF_CHIMERE_20120115_d03.nc.dds > **** > > **** > > Dataset {**** > > String transverse_mercator;**** > > Float32 x[lon = 101];**** > > Float32 y[lat = 101];**** > > Float32 lat[time = 48][lat = 101][lon = 101];**** > > Float32 lon[time = 48][lat = 101][lon = 101];**** > > String projectionCoordSys;**** > > Float32 height[time = 48][height = 25][lat = 101][lon = 101];**** > > Float32 time[time = 48];**** > > Float32 Time[Time = 48];**** > > Float32 REL[time = 48][lat = 101][lon = 101];**** > > Float32 PHI[time = 48][height = 25][lat = 101][lon = 101];**** > > Float32 M001S001[time = 48][height = 25][lat = 101][lon = 101];**** > > } CHIMERE%40CAMAC%40reference%40p02/CF_CHIMERE_20120115_d03.nc;**** > > **** > > **** > > **** > > From (very old) Hyrax:**** > > **** > > > http://web.aria.fr:443/LENVIS/CHIMERE/CAMAC/reference/p02/CF_CHIMERE_20120115_d03.nc.dds > **** > > **** > > Dataset {**** > > String transverse_mercator;**** > > Grid {**** > > Array:**** > > Float32 x[lon = 101];**** > > Maps:**** > > Float32 lon[lon = 101];**** > > } x;**** > > Grid {**** > > Array:**** > > Float32 y[lat = 101];**** > > Maps:**** > > Float32 lat[lat = 101];**** > > } y;**** > > String projectionCoordSys;**** > > Grid {**** > > Array:**** > > Float32 REL[time = 48][lat = 101][lon = 101];**** > > Maps:**** > > Float32 time[time = 48];**** > > Float32 lat[lat = 101];**** > > Float32 lon[lon = 101];**** > > } REL;**** > > Grid {**** > > Array:**** > > Float32 PHI[time = 48][height = 25][lat = 101][lon = 101];**** > > Maps:**** > > Float32 time[time = 48];**** > > Float32 height[height = 25];**** > > Float32 lat[lat = 101];**** > > Float32 lon[lon = 101];**** > > } PHI;**** > > Grid {**** > > Array:**** > > Float32 M001S001[time = 48][height = 25][lat = 101][lon = 101];*** > * > > Maps:**** > > Float32 time[time = 48];**** > > Float32 height[height = 25];**** > > Float32 lat[lat = 101];**** > > Float32 lon[lon = 101];**** > > } M001S001;**** > > Float32 Time[Time = 48];**** > > } CF_CHIMERE_20120115_d03.nc;**** > > **** > > **** > > This may be expected behavior, or not. I am tempted to say that you should > upgrade both the TDS Hyrax to the latest releases before we dig to deeply... > **** > > **** > > **** > > Anyone else care to comment?**** > > **** > > **** > > **** > > Thanks,**** > > **** > > Nathan**** > > **** > > **** > > **** > > **** > > **** > > **** > > **** > > **** > > **** > > **** > > ** ** > > For instance, > > why the second is continuously updated but doesn't allow the access to the > whole set of variables stored in NetCdf file. > Why the first one are not continuously updated even if the date in the > threeds catalogs presentation always mentioned the right modified time .. . > > Regards > > Ing. Claude DEROGNAT, PhD > ARIA TECHNOLOGIES SA > 8-10, rue de la Ferme - 92100 Boulogne-Billancourt - France > tél. / phone: 33 (0)1 46 08 68 60 > ligne directe / direct line: 33 (0)1 46 08 68 65 > fax: 33 (0)1 41 41 93 17 > e-mail: cderognat@xxxxxxx - http : www.aria.fr **** > > P Avant d'imprimer ce courriel, réfléchissez à l'impact sur > l'environnement, et dans la mesure du possible éditez les documents en > recto verso, merci! > > > > -----Message d'origine----- > De : Nathan Potter [mailto:ndp@xxxxxxxxxxx] **** > > Envoyé : mardi 21 février 2012 17:02 > À : Claude DEROGNAT > Cc : Nathan Potter; Nathan Potter; support@xxxxxxxxxxx; Patrick West > Objet : Re: OLFS and/or BES configuration to obtain a dynamic refresh of > existing files ... > > > > > Claude, > > It would appear that you are using the THREDDS Data Server (TDS). I say > this because the configuration file that you attached is unique to the TDS > and is not a part of Hyrax. The TDS is a UNIDATA product and I think that > they would be better positioned to answer your questions regarding this, > you xan reach them here: thredds@xxxxxxxxxxxxxxxx > > If you are pretty sure that you are trying to use Hyrax then I think we > may have entered a twilight zone where parts of the two servers are being > mashed up into one... > > Can you confirm which server software you are attempting to configure and > use? > > > Thanks, > > > Nathan > > > > On Feb 21, 2012, at 5:03 AM, Claude DEROGNAT wrote: > > **** > > HI, **** > > **** > > Sorry for the delay of my answer but I had to deal with another huge > project.**** > > **** > > My system will be dedicated to real time plume modeling.**** > > The model runs every 30 minutes. At the beginning of each day, it creates > a result file then every 30 minutes the result file is overwritten with a > new one containing the additional time frame.**** > > **** > > I observed à strange behavior between Opendap and thredds.**** > > The file is continuously updated in the OpenDap server.**** > > The threeds catalog notices in the 'Dates' field that the file are updates > but the Access/OpenDap target file are not updated. The available time > doesn't be continuously updated. May I have to perform any modification in > the thredds configuration ? A attached you my threddsConfig.xml file...** > ** > > **** > > Regards **** > > **** > > Ing. Claude DEROGNAT, PhD**** > > ARIA TECHNOLOGIES SA**** > > 8-10, rue de la Ferme - 92100 Boulogne-Billancourt - France**** > > tél. / phone: 33 (0)1 46 08 68 60**** > > ligne directe / direct line: 33 (0)1 46 08 68 65**** > > fax: 33 (0)1 41 41 93 17**** > > e-mail: cderognat@xxxxxxx - http : www.aria.fr **** > > P Avant d'imprimer ce courriel, réfléchissez à l'impact sur > l'environnement, et dans la mesure du possible éditez les documents en > recto verso, merci!**** > > **** > > **** > > **** > > -----Message d'origine-----**** > > De : Nathan Potter [mailto:npotter@xxxxxxxxxxx] **** > > Envoyé : mardi 11 octobre 2011 19:31**** > > À : Claude DEROGNAT; support@xxxxxxxxxxx support**** > > Cc : Nathan Potter; Patrick West**** > > Objet : Re: OLFS and/or BES configuration to obtain a dynamic refresh of > existing files ...**** > > **** > > **** > > On Oct 11, 2011, at 9:22 AM, Claude DEROGNAT wrote:**** > > **** > > Dear all,**** > > **** > > With a correct title it should be better.**** > > **** > > Regards,**** > > ...**** > > **** > > Dear all,**** > > **** > > I have a question about OpenDap functionality.**** > > Is-it possible to configure OLFS and/or BES to obtain a dynamic **** > > refresh of existing files.**** > > **** > > If your model is writing to files that are within the scope of the > BES.Catalog.catalog.RootDirectory, then Hyrax will just pick up the changes > in the filesystem and refresh it's output. Little of that is cached, and if > it is, it's triggered to refresh by a change in a files last modified time. > **** > > **** > > If writing your model output into the**** > > BES.Catalog.catalog.RootDirectory doesn't work you might be able to use > disk mounts and soft links to achieve what you want.**** > > **** > > But I have a feeling James and I haven't answered quite the question you > were asking. The fact that James's earlier reply didn't help makes me think > that repeating his point in more detail probably didn't either.**** > > **** > > - Are you perhaps wanting to define aggregations for a periodically > changing file set? So that the aggregation definition always reflects the > current state of the model?**** > > **** > > - Are you talking about automatically updating thredds catalog metadata > for aggregations?**** > > **** > > - Updating NcML definitions of aggregations to reflect the new changing > model results?**** > > **** > > **** > > **** > > I performed daily forecast of 48h and used Opendap to provide model **** > > results to partners.**** > > Thank to assimilation each 30 minutes, I want to**** > > - add a new time frame on an existing file**** > > **** > > Your model can just write to the file within the scope of > BES.Catalog.catalog.RootDirectory. Hyrax will see that the last- modified > time of the file has changed.**** > > **** > > - replace (modified) some fields stored few hours ago.**** > > **** > > Again, I think this is just your model writing to files that are within > the scope of BES.Catalog.catalog.RootDirectory**** > > **** > > **** > > Is-is possible ?**** > > **** > > Regards,**** > > **** > > Ing. Claude DEROGNAT, PhD**** > > ARIA TECHNOLOGIES SA**** > > 8-10, rue de la Ferme - 92100 Boulogne-Billancourt - France**** > > tél. / phone: 33 (0)1 46 08 68 60**** > > ligne directe / direct line: 33 (0)1 46 08 68 65**** > > fax: 33 (0)1 41 41 93 17**** > > e-mail: cderognat@xxxxxxx - http : www.aria.fr**** > > P Avant d'imprimer ce courriel, réfléchissez à l'impact sur **** > > l'environnement, et dans la mesure du possible éditez les documents **** > > en recto verso, merci!**** > > **** > > **** > > **** > > -----Message d'origine-----**** > > De : Nathan Potter [mailto:ndp@xxxxxxxxxxx]**** > > Envoyé : vendredi 18 septembre 2009 18:14**** > > À : Patrick West**** > > Cc : Nathan Potter; Claude DEROGNAT; support@xxxxxxxxxxx**** > > Objet : [SPAM?] Re: [Support] Bad Request 400 after a complete re- **** > > installation ....**** > > **** > > **** > > **** > > Claude,**** > > **** > > The OLFS relies on Tomcat's built in authentication mechanism. REad**** > > this section:**** > > **** > > > http://docs.opendap.org/index.php/Hyrax_-_OLFS_Configuration#Authentication_.26_Authorization > **** > > **** > > Of our documents for more information on that, although you will need**** > > to read the Tomcat documentation to get it sorted out. You can connect**** > > the authentication service to an LDAP server (using the JNDIRealm) if**** > > you wish, or you could use the example "MemoryRealm" if all you want**** > > is something simple but not as secure as LDAP.**** > > **** > > **** > > Nathan**** > > **** > > **** > > **** > > On Sep 18, 2009, at 8:26 AM, Patrick West wrote:**** > > **** > > **** > > On Sep 18, 2009, at 5:38 AM, Claude DEROGNAT wrote:**** > > **** > > Patrick,**** > > **** > > Thank you very much for your help ... Thank also Nathan and all the**** > > opendap support !!!!**** > > I never success without your help !!!!**** > > **** > > **** > > Glad we could help. We are always working to make the installs**** > > easier, and have formed a working group around the issue.**** > > **** > > the configuration similar to http://opendap.rpi.edu is perfect for**** > > my application.**** > > **** > > A last question ... the last I promise ;-))).**** > > I didn't find simple way to introduce "authentication" to restrict**** > > access to the opendap (OLFS) server.**** > > You look very familiar with tomcat. Do you know simple way to only**** > > introduce one (or more) login/password in the tomcat server access.**** > > I have in mind something similar to ftp access : http://login:password@IP* > *** > > :port .**** > > **** > > I'm not sure about this last part, whether the OLFS supports this,**** > > can get the username and password and do anything with it. Nathan?**** > > **** > > Patrick**** > > **** > > **** > > Regards,**** > > **** > > Claude Derognat • Ingénieur R&D**** > > ARIA TECHNOLOGIES SA**** > > 8-10, rue de la Ferme - 92100 Boulogne-Billancourt - France**** > > tél. / phone: 33 (0)1 46 08 68 60**** > > ligne directe / direct line: 33 (0)1 46 08 68 65**** > > fax: 33 (0)1 41 41 93 17**** > > e-mail: cderognat@xxxxxxx - http : www.aria.fr**** > > P Avant d'imprimer ce courriel, réfléchissez à l'impact sur**** > > l'environnement, et dans la mesure du possible éditez les documents**** > > en recto verso, merci!**** > > **** > > **** > > -----Message d'origine-----**** > > De : Patrick West [mailto:pwest@xxxxxxxxxxx]**** > > Envoyé : jeudi 17 septembre 2009 19:38**** > > À : Nathan Potter**** > > Cc : Claude DEROGNAT; support@xxxxxxxxxxx**** > > Objet : Re: [Support] Bad Request 400 after a complete re-**** > > installation ....**** > > **** > > For one of our projects here, actually two of them, we have a **** > > default**** > > context. We set this in the server.xml file, before the closing Host**** > > tag:**** > > **** > > <Context docBase="opendap" path="" reloadable="true"**** > > source="org.eclipse.j st.j2ee.server:opendap"/>**** > > </Host>**** > > **** > > And, if you don't want to have to use :8080 when going to the **** > > server,**** > > and you don't already have an apache server running, you can use **** > > this**** > > in your server.xml file.**** > > **** > > <Connector port="80" address="128.113.28.182" protocol="HTTP/1.1"**** > > connectionTimeout="20000"**** > > useIPVHosts="true" redirectPort="443" />**** > > **** > > This configuration gives us http://opendap.rpi.edu, where we serve**** > > some sample data.**** > > **** > > Might need to change this in other locations as well (port 80 for **** > > the**** > > port tomcat listens on, and 443 for ssl connections.)**** > > **** > > But if you have apache running, then this last part won't work,**** > > unless**** > > you change the apache port to something other than 80. There is a **** > > way**** > > to redirect in httpd.conf, but I'm not sure how to do that.**** > > **** > > In our case, we have one machine with three different IP addresses,**** > > each one with either an apache server or tomcat server running, each**** > > listening on port 80 for that IP address, which is why we specify **** > > the**** > > address in the Connector.**** > > **** > > Patrick**** > > **** > > On Sep 17, 2009, at 12:58 PM, Nathan Potter wrote:**** > > **** > > **** > > On Sep 17, 2009, at 9:23 AM, Claude DEROGNAT wrote:**** > > **** > > Nathan,**** > > **** > > Thank you for your help.**** > > But your mail induces additional questions ...**** > > **** > > How do you configure Tomcat to use Hyrax (running in**** > > it's context of "opendap") as the "default" servlet ?**** > > **** > > At this point I would have to go read through the Tomcat**** > > documentation in order to answer your question.**** > > **** > > I think the answer is on this page:**** > > **** > > http://tomcat.apache.org/tomcat-6.0-doc/default-servlet.html**** > > **** > > **** > > But I haven't tried this and I have no idea if it's actually**** > > possible (or even advisable...)**** > > **** > > **** > > **** > > **** > > I tried to couple Tomcat with Apache:**** > > NC Services (link to BES) work properly but I lost the most part **** > > of**** > > the layout: (no logo, no blue horizontal bar, not link to**** > > documentation...).**** > > **** > > Did you read the section in the Hyrax documentation on integrating**** > > Hyrax with Apache??**** > > **** > > http://docs.opendap.org/index.php/Hyrax_-_Apache_Integration**** > > **** > > I know people have done this and they have been able to make all of**** > > it work.**** > > **** > > **** > > **** > > What's why, I tried to deploy OLFS on the "default" Tomcat URL ( > http://IP:8080**** > > ) and NAT the external IP adresse to the LAN IP(:8080) of my**** > > OpenDap sever. Any idea about that ?**** > > **** > > That's always worked for me.**** > > **** > > I just set up the firewall so that all requests sent to http://**** > > WAN_ADDRESS:80 are forwarded to http://NAT_ADDRESS:8080**** > > **** > > I hope that helps,**** > > **** > > **** > > Nathan**** > > **** > > ... > > [Message clipped] **** > > > > > -- > Dr. M. Benno Blumenthal benno@xxxxxxxxxxxxxxxx > International Research Institute for climate and society > The Earth Institute at Columbia University > Lamont Campus, Palisades NY 10964-8000 (845) 680-4450 **** > -- Dr. M. Benno Blumenthal benno@xxxxxxxxxxxxxxxx International Research Institute for climate and society The Earth Institute at Columbia University Lamont Campus, Palisades NY 10964-8000 (845) 680-4450
thredds
archives: