NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: Grads GDS / Re: GRIB Aggregation

Sorry its taken so long to get back to you on this one.

One possibility:
 NcML aggregation requires homogenous files, GRIB files are notorious for not being 
homogenous, because they are collections of 2D GRIB records. We are working on a more 
powerfil kind of aggregation called "Forecast Model Run" aggregation, that 
should solve this problem.

anyway, do you have error messages in the log files?

dan.swank wrote:

TDS support :

I have tried to get 3 seperate aggregations working during the few hours
I've had to work on this during the past week.

In our TDS test area, Please take a look at the following 3 entries:
http://nomads.ncdc.noaa.gov:8085/thredds/testDatasets.html

  o  Blended Ocean Winds Test Daily Aggregation
  o  NARR-A Test Daily Aggregation
  o  Test Narr hgtprs Subset

The configuration file entries for these follow:

==============================================

        <!-- Test BOW Aggregation -->

  <dataset name="Blended Ocean Winds Test Daily Aggregation"
      ID="test/BOWdailyagg" urlPath="test/BOWdailyagg">
    <serviceName>allTest</serviceName>
    <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>

     <dimension name="time" length="6688" />

     <variable name="time" type="int" shape="time">
       <attribute name="long_name" value="Aggregated Time axis in days" />
       <attribute name="units" value="days since 1987-07-09" />
       <attribute name="_CoordinateAxisType" value="Time" />
       <values start="0" incr="1" npts="6688" />
     </variable>

     <aggregation dimName="time" type="joinNew">
       <variableAgg name="wind" />
       <scan dateFormatMark="#yyyyMMdd"
location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1980s/"
suffix=".nc" />
       <scan dateFormatMark="#yyyyMMdd"
location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1990s/"
suffix=".nc" />
       <scan dateFormatMark="#yyyyMMdd"
location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/2000s/"
suffix=".nc" />
     </aggregation>
    </netcdf>
  </dataset>

        <!-- Test NARR Aggregation (local) -->

  <dataset name="NARR-A Test Daily Aggregation"
      ID="test/narragg" urlPath="test/narragg">
    <access serviceName="allTest" urlPath="test/narragg"
dataFormat="GRIB-1" />
    <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>

     <dimension name="time" length="8" />
     <variable name="time" type="int" shape="time">
       <attribute name="long_name" value="Aggregated Time axis in 3hr" />
       <attribute name="units" value="hours since 2006-01-01 00:00" />
       <attribute name="_CoordinateAxisType" value="Time" />
       <values start="0" incr="3" npts="8" />
     </variable>

     <aggregation dimName="time" type="joinNew">
       <variableAgg/>
       <scan dateFormatMark="#narr-a_211_yyyyMMdd_hhmm_000"
location="/nomads3_data/raid5a/data/merged_AWIP32/200601/20060101"
suffix=".grb" />
     </aggregation>
    </netcdf>
  </dataset>

        <!-- NARR hgtprs test agg (remote) -->

  <dataset
        name="Test Narr hgtprs Subset" ID="NARR/hgtprs"
        urlPath="NCEP_NARR_DAILY/narr-a_221_hgtprs.subset"
        serviceName="gds" >
    <catalogRef xlink:title="Test NARR Subset hgtprs"

xlink:href="http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY/narr-a_221_hgtprs.subset.thredds";
/>
    <serviceName>gds</serviceName>

    <variable name="Geopotential Height" orgName="hgtprs">
       <attribute name="long_name" value="Geopotential Height on
Pressure Levels" />
       <attribute name="units" value="gpm" />
    </variable>
  </dataset>

==============================================
Services defined as follows:
==============================================
 <service name="allTest" serviceType="Compound" base="">
   <service name="ncdods" serviceType="OpenDAP" base="/thredds/dodsC/"/>
   <service name="HTTPServer" serviceType="HTTPServer"
base="/thredds/fileServer/"/>
   <service name="WCS" serviceType="WCS" base="/thredds/wcs/"/>
 </service>
  <service name="gds" serviceType="DODS"
base="http://nomads.ncdc.noaa.gov:9091/dods/"; />
==============================================

The Blended Ocean winds almost work,
The OpENDAP interface comes up and allows the subsetting functions, but
whenever I submit a request for anything that spans time steps it
returns an error.

The other two, I may not even have in the same ballpark.
I thought it would be a good idea to just try a single day of NARR in an
aggregation with a scan element, but even that ~ so far ~ gives nothing
but a blank page for OpENDAP service.

Roy and Jerome :
I'd like to take up you offer to see your NcML configuration.
So far I've had no luck with this on my own.

Need some pointers on what I need in the configuration.
Thanks in advance for any assistance.

-Dan



Roy Mendelssohn wrote the following on 6/15/2006 10:44 AM:


TDS in fact allows for just what you are asking ('If it is possible to
create hooks that allow TDS to read and import data/metadata from a
source OPeNDAP URL') and we have a test of this implemented  (go to
http://oceanwatch.pfeg.noaa.gov:8081/thredds/catalog.html and look at 'Test remote AUV data TDS'. These data are served by OPeNDAP at MBARI, 
but we both aggregate and serve them through TDS at our sie.

If you want info on how the THREDDS catalogs and NcML files are set up
to do this, email Jeorme King at jerome.king@xxxxxxxx.  I would
apprecaiae being cc:'ed ; also.

Hope this helps.

-Roy


At 10:07 AM -0400 6/15/06, dan.swank wrote:



We have had the NARR on GDS for quite some time :
http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY
GDS works very well with GRIB data in general.
Our particular goal is to get it on TDS due to its suporior metadata
handling and WCS service.



If it is possible to create hooks that allow TDS to read and import



data/metadata from a source OPeNDAP URL, similar to what the Live Access
Server does... that would be outstanding.  May pose an easier task to
implement as well.



-Dan




Rob Cermak wrote the following on 6/14/2006 6:50 PM:


Dan,

The Grads GDS OPeNDAP server is suppose to work with GRIB files.  The



large unknown is if TDS can then aggregate the data across the Grads GDS
server.  To help answer that is to see if TDS can aggregate across maybe
one of the NOMADs GFS servers?  I'm pretty sure they are using the Grads
GDS server.

http://www.iges.org/grads/gds/

Rob

On Tue, June 13, 2006 2:28 pm, dan.swank wrote:



Ethan:
Understood.  consider this a wish-list then :)
-Dan

Ethan Davis wrote the following on 6/13/2006 6:22 PM:



Hi Dan,

Evidently, aggregation of GRIB data may not be working. I thought since
GRIB files are read by the netCDF-java library it would just work but I
just heard that there may be some issues with aggregating GRIB


involving


how the underlying code writes indexes into the GRIB files. We'll have
to wait till John is back from vacation for a more detailed
answer/explanation.

Sorry for the mistake.

There currently isn't a way to have a single scan element scan a set of
directories.

I believe the underlying netCDF-java code that reads GRIB files should
result in fairly well-formed, CF (I think) compliant netCDF views


of the


dataset. Not sure how this would be affected by aggregation.

Ethan

dan.swank wrote:




Ethan:

The NARR is a reanalysis, so it don't have forecast times.  I would be
a
simple 03 hr chain (00 hr fct time) spanning 26 years.

See an existing GDS subset aggregation:
http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY/narr-a_221_tmpprs.subset.info

This will give a sense for the nature of the beast.

The directory structure is set up as such:
http://nomads.ncdc.noaa.gov/data/narr/


Heres the TDS aggregation I set up while experimenting yesterday, on a
non-related dataset:

<dataset name="OceanWinds Test Daily Aggregation"
    ID="test/dailyagg" urlPath="test/agg">
  <serviceName>allTest</serviceName>
  <netcdf
xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
   <aggregation dimName="time" type="joinNew">
     <variableAgg name="wind" />
     <scan dateFormatMark="#yyyyMMdd"
location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1980s/"
suffix=".nc" />
     <scan dateFormatMark="#yyyyMMdd"
location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1990s/"
suffix=".nc" />
     <scan dateFormatMark="#yyyyMMdd"
location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/2000s/"
suffix=".nc" />
   </aggregation>
   <variable name="time" orgName="time">
     <attribute name="long_name" value="Days"/>



     <attribute name="units" value="days since 1987-07-09" />
   </variable>
  </netcdf>
</dataset>


Would this automatically detect the source of data were GRIB rather
than
NetCDF?  and it seems like you need to set the <scan> on each
individual
directory... Doing so the way NARR is set up would create one chunky
configuration file.  Is there anyway to have this scan a pattern
(YYYYMM/YYYYMMDD) of directories?

I understand GRIB requires a certain amount of "supplemented" metadata
for complience.  Where do you enter this?

-Dan


Ethan Davis wrote the following on 6/13/2006 1:21 PM:





Hi Dan,

Aggregation should work the same for GRIB as for netCDF files. The
issue
would be how your GRIB files are structured and how you want to
aggregate them. Our GRIB files each contain one full model run (all
parameters, all forecast times). We haven't tried aggregating beyond
that.

We have started tracking what is available for the NCEP models on our
server. This is from the TDS 3.8 announcement (with links updated):

 We also are now tracking detailed inventory of NCEP model output,
eg:

http://motherlode.ucar.edu:8080/thredds/modelInventory/model/NCEP/NAM/CONUS_12km/


 These are all linked from the "collection dataset" pages; For
 example from

http://motherlode.ucar.edu:8080/thredds/catalog/model/NCEP/NAM/CONUS_12km/catalog.html


 choose the top "CONUS_12_km" link, then choose "Available
Inventory"
 Documentation.

One idea for this work is to eventually provide access to alternate
datasets, for instance, a dataset that contains all the 3hr forecast
times from the different runs, or one that contained all the 12Z


valid


times from the different runs. Tracking these detailed inventories is
just the first step but aggregation and alternate groupings of the
data
is pretty interesting to think about.

How are your GRIB files structured and what kind of aggregation where
you thinking about?

Ethan

dan.swank wrote:






Hello,

I've been tinkering with the TDS aggregation capabilities and they
work
quite well for NetCDF data, however, I can't seem to find


anything in


the docs regarding aggregating GRIB.
We want to get The NARR dataset which we have here at NCDC-NOMADS on
the
TDS.  It consists of hundreds of thousands of 50 Mb + GRIB files


in a


YYYYMM/YYYYMMDD tree.
Just scouting for a quick answer here:
Is aggregating the NARR GRIB currently feasable with the current
release
of TDS?  If so, do any docs exist which could give me a starting
point?
Converting it to NetCDF will not be possible (volume).








--
Dan Swank <dan.swank@xxxxxxxx>
NOMADS Project:  Software & Data Management
Contractor - STG, Incorporated
Veach-Baley Federal Building
151 Patton Avenue
Asheville, NC 28801-5001
Phone: 828-271-4007

==============================================================================
To unsubscribe thredds, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================






--
Dan Swank <dan.swank@xxxxxxxx>
NOMADS Project:  Software & Data Management
Contractor - STG, Incorporated
Veach-Baley Federal Building
151 Patton Avenue
Asheville, NC 28801-5001
Phone: 828-271-4007

==============================================================================
To unsubscribe thredds, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================




--

**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."





==============================================================================
To unsubscribe thredds, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================
  • 2006 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: