NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Dear all, coming back to my question on NCEP grib aggregation a while back: John Caron wrote: > What you are seeing is the limitations of aggregations. In this > case, there are 3 different time coordinates in the collection, but > NcML aggregation can only aggregate on one of them. You want to use > feature collections instead. Replace your entire <dataset> element > with something like: > > <featureCollection name="myCollectionName" featureType="GRIB" > path="grib/NCEP/GFS/etc"> > <collection > spec="/pub/data/nccf/com/gfs/prod/gfs.2013060600/gfs\.t00z\.pgrb2f..$" > dateFormatMark="#prod/gfs.#yyyyMMddHH" /> > </featureCollection> I've finally implemented the GRIB featurecollection, and it seems to be working. I can access my grib files (although indexing of the full ncep ensemble takes a while!) and the data comes out OK. (timeseries for a single location, for each ensemble member). I'm experiencing a problem though: data extraction is extremely slow. I'm comparing to my old situation where I listed all ensemble member grib files in a single ncml file. A data extraction for one location (all members, for a single 15 day forecast) took 5 minutes in thredds 4.1 with this system. In the new situation (thredds 4.3), I use a grib feature collection sorted by directory (forecast cycle). After half an hour, the extraction is still not done. In the featureCollectionScan.log I can see that thredds keeps scanning all folders: > [2013-08-29T12:40:28.786+0000] INFO > thredds.inventory.MFileCollectionManager: 2013082618 : was scanned > MCollection{name='2013082618', > dirName='/output/operational/atmosphere/ncep/gefs/1.0deg/2013082618', > wantSubdirs=true, ff=WildcardMatchOnPath{wildcard=null > regexp=gefs\..*\.f.*\.grib2$}} It does this for ALL forecast cycle folders (I have about 20), even though I am accessing only the 2013082900 directory. Could anyone give me tips on how to prevent thredds from continuously re-scanning the whole directory structure with grib files? Current setup: <featureCollection name="gefs_col" featureType="GRIB" path="ncep/gefs/1.0deg"> <!-- be specific here with the file selector, other grib2 files may be hanging around in the tree --> <collection spec="/output/operational/atmosphere/ncep/gefs/1.0deg/**/gefs\..*\.f.*\.grib2$" dateFormatMark="#0deg/#yyyyMMddHH" timePartition="directory" name="gefs_col_unique" /> <update startup="true" trigger="allow"/> </featureCollection> This organizes the data the way I want: I get a single url per cycle: .../thredds/dodsC/ncep/gefs/1.0deg/2013082900/best .../2013082800/best .../2013082700/best The data comes out the way I want, but as mentioned above it's _extremely_ slow, likely due to re-scanning of the disk structure. I don't really need automatic updating, a manual trigger when a new forecast is downloaded would be ok too. I would prefer thredds to scan and index the grib files only once upon a manual trigger. Any hints on how to improve this? Kind regards, Hein Zelle > Send thredds mailing list submissions to > thredds@xxxxxxxxxxxxxxxx > > To subscribe or unsubscribe via the World Wide Web, visit > http://mailman.unidata.ucar.edu/mailman/listinfo/thredds > or, via email, send a message with subject or body 'help' to > thredds-request@xxxxxxxxxxxxxxxx > > You can reach the person managing the list at > thredds-owner@xxxxxxxxxxxxxxxx > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of thredds digest..." > > > thredds mailing list > thredds@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ > > Today's Topics: > > 1. Re: aggregating GFS data, problem with accumulated (Hein Zelle) > 2. Re: aggregating GFS data, problem with accumulated (John Caron) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 6 Jun 2013 11:30:36 +0200 > From: Hein Zelle <hein.zelle@xxxxxxxxxxxxx> > To: thredds@xxxxxxxxxxxxxxxx > Subject: Re: [thredds] aggregating GFS data, problem with accumulated > Message-ID: <20130606093035.GA17727@xxxxxxxxxxxxxxxxxxxx> > Content-Type: text/plain; charset="us-ascii" > > Dear John, > > attached to this email is a complete ncml file that we place next to > the data files. The data files themselves are too big to upload, but > they are standard gfs grib2 files, you can find them at > > ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.2013060600 > > (that's for this morning, modify the date as needed) > The files are the grib2 files at 0.5 degree, e.g. gfs.t00z.pgrb2bf30 > (50 mb each). > > The previous snippet of ncml I sent should also work, you'll have to modify > the paths to the correct folder of course. A variable to check is for > example > > Total_precipitation_surface_3_Hour_Accumulation > > These should have multiple time steps, but I get only 1 time step (the > first, for the +03 forecast). The +00 analysis doesn't contain the > precipitation fields. Any variable with an accumulation or averaging > interval exhibits the problem. > > > Kind regards, > Hein Zelle > > -- Dr. Hein Zelle Senior consultant meteorology & oceanography BMT ARGOSS Tel: +31 (0)527-242299 Fax: +31 (0)527-242016 E-mail: hein.zelle@xxxxxxxxxxxxx Website: www.bmtargoss.com BMT ARGOSS b.v. Voorsterweg 28, 8316 PT Marknesse, the Netherlands Postal address: P.O. Box 61, 8325 ZH Vollenhove, the Netherlands Registered in The Netherlands, Registered no. 39060160. Unless otherwise agreed by BMT ARGOSS in writing, all work, services, goods or products supplied by BMT ARGOSS shall be subject to and governed by BMT ARGOSS' own terms and conditions which are available for inspection from BMT ARGOSS on request. E-mail confidentiality notice and disclaimer: The contents of this e-mail and any attachments are intended for the use of the mail addressee(s) shown. If you are not that person, you are not allowed to read it, to take any action based upon it or to copy it, forward, distribute or disclose the contents of it and you should please delete it from your system. BMT ARGOSS does not accept liability for any errors or omissions in the context of this e-mail or its attachments which arise as a result of internet transmission, nor accept liability for statements which are those of the author and clearly not made on behalf of BMT ARGOSS.
thredds
archives: