NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
TDS support : I have tried to get 3 seperate aggregations working during the few hours I've had to work on this during the past week. In our TDS test area, Please take a look at the following 3 entries: http://nomads.ncdc.noaa.gov:8085/thredds/testDatasets.html o Blended Ocean Winds Test Daily Aggregation o NARR-A Test Daily Aggregation o Test Narr hgtprs Subset The configuration file entries for these follow: ============================================== <!-- Test BOW Aggregation --> <dataset name="Blended Ocean Winds Test Daily Aggregation" ID="test/BOWdailyagg" urlPath="test/BOWdailyagg"> <serviceName>allTest</serviceName> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> <dimension name="time" length="6688" /> <variable name="time" type="int" shape="time"> <attribute name="long_name" value="Aggregated Time axis in days" /> <attribute name="units" value="days since 1987-07-09" /> <attribute name="_CoordinateAxisType" value="Time" /> <values start="0" incr="1" npts="6688" /> </variable> <aggregation dimName="time" type="joinNew"> <variableAgg name="wind" /> <scan dateFormatMark="#yyyyMMdd" location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1980s/" suffix=".nc" /> <scan dateFormatMark="#yyyyMMdd" location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1990s/" suffix=".nc" /> <scan dateFormatMark="#yyyyMMdd" location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/2000s/" suffix=".nc" /> </aggregation> </netcdf> </dataset> <!-- Test NARR Aggregation (local) --> <dataset name="NARR-A Test Daily Aggregation" ID="test/narragg" urlPath="test/narragg"> <access serviceName="allTest" urlPath="test/narragg" dataFormat="GRIB-1" /> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> <dimension name="time" length="8" /> <variable name="time" type="int" shape="time"> <attribute name="long_name" value="Aggregated Time axis in 3hr" /> <attribute name="units" value="hours since 2006-01-01 00:00" /> <attribute name="_CoordinateAxisType" value="Time" /> <values start="0" incr="3" npts="8" /> </variable> <aggregation dimName="time" type="joinNew"> <variableAgg/> <scan dateFormatMark="#narr-a_211_yyyyMMdd_hhmm_000" location="/nomads3_data/raid5a/data/merged_AWIP32/200601/20060101" suffix=".grb" /> </aggregation> </netcdf> </dataset> <!-- NARR hgtprs test agg (remote) --> <dataset name="Test Narr hgtprs Subset" ID="NARR/hgtprs" urlPath="NCEP_NARR_DAILY/narr-a_221_hgtprs.subset" serviceName="gds" > <catalogRef xlink:title="Test NARR Subset hgtprs" xlink:href="http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY/narr-a_221_hgtprs.subset.thredds"; /> <serviceName>gds</serviceName> <variable name="Geopotential Height" orgName="hgtprs"> <attribute name="long_name" value="Geopotential Height on Pressure Levels" /> <attribute name="units" value="gpm" /> </variable> </dataset> ============================================== Services defined as follows: ============================================== <service name="allTest" serviceType="Compound" base=""> <service name="ncdods" serviceType="OpenDAP" base="/thredds/dodsC/"/> <service name="HTTPServer" serviceType="HTTPServer" base="/thredds/fileServer/"/> <service name="WCS" serviceType="WCS" base="/thredds/wcs/"/> </service> <service name="gds" serviceType="DODS" base="http://nomads.ncdc.noaa.gov:9091/dods/"; /> ============================================== The Blended Ocean winds almost work, The OpENDAP interface comes up and allows the subsetting functions, but whenever I submit a request for anything that spans time steps it returns an error. The other two, I may not even have in the same ballpark. I thought it would be a good idea to just try a single day of NARR in an aggregation with a scan element, but even that ~ so far ~ gives nothing but a blank page for OpENDAP service. Roy and Jerome : I'd like to take up you offer to see your NcML configuration. So far I've had no luck with this on my own. Need some pointers on what I need in the configuration. Thanks in advance for any assistance. -Dan Roy Mendelssohn wrote the following on 6/15/2006 10:44 AM: > TDS in fact allows for just what you are asking ('If it is possible to > create hooks that allow TDS to read and import data/metadata from a > source OPeNDAP URL') and we have a test of this implemented (go to > http://oceanwatch.pfeg.noaa.gov:8081/thredds/catalog.html and look at > 'Test remote AUV data TDS'. These data are served by OPeNDAP at MBARI, > but we both aggregate and serve them through TDS at our sie. > > If you want info on how the THREDDS catalogs and NcML files are set up > to do this, email Jeorme King at jerome.king@xxxxxxxx. I would > apprecaiae being cc:'ed ; also. > > Hope this helps. > > -Roy > > > At 10:07 AM -0400 6/15/06, dan.swank wrote: > >> We have had the NARR on GDS for quite some time : >> http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY >> GDS works very well with GRIB data in general. >> Our particular goal is to get it on TDS due to its suporior metadata >> handling and WCS service. > >> If it is possible to create hooks that allow TDS to read and import > >> data/metadata from a source OPeNDAP URL, similar to what the Live Access >> Server does... that would be outstanding. May pose an easier task to >> implement as well. > >> -Dan > >> >> >> Rob Cermak wrote the following on 6/14/2006 6:50 PM: >> > Dan, >> > >> > The Grads GDS OPeNDAP server is suppose to work with GRIB files. The > >> > large unknown is if TDS can then aggregate the data across the Grads GDS >> > server. To help answer that is to see if TDS can aggregate across maybe >> > one of the NOMADs GFS servers? I'm pretty sure they are using the Grads >> > GDS server. >> > >> > http://www.iges.org/grads/gds/ >> > >> > Rob >> > >> > On Tue, June 13, 2006 2:28 pm, dan.swank wrote: >> > >> >>Ethan: >> >>Understood. consider this a wish-list then :) >> >>-Dan >> >> >> >>Ethan Davis wrote the following on 6/13/2006 6:22 PM: >> >> >> >>>Hi Dan, >> >>> >> >>>Evidently, aggregation of GRIB data may not be working. I thought since >> >>>GRIB files are read by the netCDF-java library it would just work but I >> >>>just heard that there may be some issues with aggregating GRIB >> involving >> >>>how the underlying code writes indexes into the GRIB files. We'll have >> >>>to wait till John is back from vacation for a more detailed >> >>>answer/explanation. >> >>> >> >>>Sorry for the mistake. >> >>> >> >>>There currently isn't a way to have a single scan element scan a set of >> >>>directories. >> >>> >> >>>I believe the underlying netCDF-java code that reads GRIB files should >> >>>result in fairly well-formed, CF (I think) compliant netCDF views >> of the >> >>>dataset. Not sure how this would be affected by aggregation. >> >>> >> >>>Ethan >> >>> >> >>>dan.swank wrote: >> >>> >> >>> >> >>>>Ethan: >> >>>> >> >>>>The NARR is a reanalysis, so it don't have forecast times. I would be >> >>>>a >> >>>>simple 03 hr chain (00 hr fct time) spanning 26 years. >> >>>> >> >>>>See an existing GDS subset aggregation: >> >>>>http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY/narr-a_221_tmpprs.subset.info >> >>>> >> >>>>This will give a sense for the nature of the beast. >> >>>> >> >>>>The directory structure is set up as such: >> >>>>http://nomads.ncdc.noaa.gov/data/narr/ >> >>>> >> >>>> >> >>>>Heres the TDS aggregation I set up while experimenting yesterday, on a >> >>>>non-related dataset: >> >>>> >> >>>> <dataset name="OceanWinds Test Daily Aggregation" >> >>>> ID="test/dailyagg" urlPath="test/agg"> >> >>>> <serviceName>allTest</serviceName> >> >>>> <netcdf >> >>>>xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> >> >>>> <aggregation dimName="time" type="joinNew"> >> >>>> <variableAgg name="wind" /> >> >>>> <scan dateFormatMark="#yyyyMMdd" >> >>>>location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1980s/" >> >>>>suffix=".nc" /> >> >>>> <scan dateFormatMark="#yyyyMMdd" >> >>>>location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1990s/" >> >>>>suffix=".nc" /> >> >>>> <scan dateFormatMark="#yyyyMMdd" >> >>>>location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/2000s/" >> >>>>suffix=".nc" /> >> >>>> </aggregation> >> >>>> <variable name="time" orgName="time"> >> >>>> <attribute name="long_name" value="Days"/> > >> >>>> <attribute name="units" value="days since 1987-07-09" /> >> >>>> </variable> >> >>>> </netcdf> >> >>>> </dataset> >> >>>> >> >>>> >> >>>>Would this automatically detect the source of data were GRIB rather >> >>>>than >> >>>>NetCDF? and it seems like you need to set the <scan> on each >> >>>>individual >> >>>>directory... Doing so the way NARR is set up would create one chunky >> >>>>configuration file. Is there anyway to have this scan a pattern >> >>>>(YYYYMM/YYYYMMDD) of directories? >> >>>> >> >>>>I understand GRIB requires a certain amount of "supplemented" metadata >> >>>>for complience. Where do you enter this? >> >>>> >> >>>>-Dan >> >>>> >> >>>> >> >>>>Ethan Davis wrote the following on 6/13/2006 1:21 PM: >> >>>> >> >>>> >> >>>> >> >>>>>Hi Dan, >> >>>>> >> >>>>>Aggregation should work the same for GRIB as for netCDF files. The >> >>>>>issue >> >>>>>would be how your GRIB files are structured and how you want to >> >>>>>aggregate them. Our GRIB files each contain one full model run (all >> >>>>>parameters, all forecast times). We haven't tried aggregating beyond >> >>>>>that. >> >>>>> >> >>>>>We have started tracking what is available for the NCEP models on our >> >>>>>server. This is from the TDS 3.8 announcement (with links updated): >> >>>>> >> >>>>> We also are now tracking detailed inventory of NCEP model output, >> >>>>>eg: >> >>>>> >> >>>>>http://motherlode.ucar.edu:8080/thredds/modelInventory/model/NCEP/NAM/CONUS_12km/ >> >>>>> >> >>>>> >> >>>>> These are all linked from the "collection dataset" pages; For >> >>>>> example from >> >>>>> >> >>>>>http://motherlode.ucar.edu:8080/thredds/catalog/model/NCEP/NAM/CONUS_12km/catalog.html >> >>>>> >> >>>>> >> >>>>> choose the top "CONUS_12_km" link, then choose "Available >> >>>>>Inventory" >> >>>>> Documentation. >> >>>>> >> >>>>>One idea for this work is to eventually provide access to alternate >> >>>>>datasets, for instance, a dataset that contains all the 3hr forecast >> >>>>>times from the different runs, or one that contained all the 12Z >> valid >> >>>>>times from the different runs. Tracking these detailed inventories is >> >>>>>just the first step but aggregation and alternate groupings of the >> >>>>>data >> >>>>>is pretty interesting to think about. >> >>>>> >> >>>>>How are your GRIB files structured and what kind of aggregation where >> >>>>>you thinking about? >> >>>>> >> >>>>>Ethan >> >>>>> >> >>>>>dan.swank wrote: >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>>>Hello, >> >>>>>> >> >>>>>>I've been tinkering with the TDS aggregation capabilities and they >> >>>>>>work >> >>>>>>quite well for NetCDF data, however, I can't seem to find >> anything in >> >>>>>>the docs regarding aggregating GRIB. >> >>>>>>We want to get The NARR dataset which we have here at NCDC-NOMADS on >> >>>>>>the >> >>>>>>TDS. It consists of hundreds of thousands of 50 Mb + GRIB files >> in a >> >>>>>>YYYYMM/YYYYMMDD tree. >> >>>>>> Just scouting for a quick answer here: >> >>>>>>Is aggregating the NARR GRIB currently feasable with the current >> >>>>>>release >> >>>>>>of TDS? If so, do any docs exist which could give me a starting >> >>>>>>point? >> >>>>>>Converting it to NetCDF will not be possible (volume). >> >>>>>> >> >>>>>> >> >>>>> >> >>>>> >> >>>> >> >>>> >> >>> >> >>-- >> >>Dan Swank <dan.swank@xxxxxxxx> >> >>NOMADS Project: Software & Data Management >> >>Contractor - STG, Incorporated >> >>Veach-Baley Federal Building >> >>151 Patton Avenue >> >>Asheville, NC 28801-5001 >> >>Phone: 828-271-4007 >> >> >> >>============================================================================== >> >>To unsubscribe thredds, visit: >> >>http://www.unidata.ucar.edu/mailing-list-delete-form.html >> >>============================================================================== >> >> >> > >> > >> > >> >> -- >> Dan Swank <dan.swank@xxxxxxxx> >> NOMADS Project: Software & Data Management >> Contractor - STG, Incorporated >> Veach-Baley Federal Building >> 151 Patton Avenue >> Asheville, NC 28801-5001 >> Phone: 828-271-4007 >> >> ============================================================================== >> To unsubscribe thredds, visit: >> http://www.unidata.ucar.edu/mailing-list-delete-form.html >> ============================================================================== > > > > -- > > ********************** > "The contents of this message do not reflect any position of the U.S. > Government or NOAA." > ********************** > Roy Mendelssohn > Supervisory Operations Research Analyst > NOAA/NMFS > Environmental Research Division > Southwest Fisheries Science Center > 1352 Lighthouse Avenue > Pacific Grove, CA 93950-2097 > > e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address) > voice: (831)-648-9029 > fax: (831)-648-8440 > www: http://www.pfeg.noaa.gov/ > > "Old age and treachery will overcome youth and skill." -- Dan Swank <dan.swank@xxxxxxxx> NOMADS Project: Software & Data Management Contractor - STG, Incorporated Veach-Baley Federal Building 151 Patton Avenue Asheville, NC 28801-5001 Phone: 828-271-4007 ============================================================================== To unsubscribe thredds, visit: http://www.unidata.ucar.edu/mailing-list-delete-form.html ==============================================================================
thredds archives: