NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Agreed. I said similar things at the meeting. But I was starting from the discussions at the meeting, which was the CDM as implemented in THREDDS. So I want to get clear in my mind what TDS will be able to do for observational data first before looking for alternatives. -Roy On Dec 2, 2010, at 2:42 PM, Upendra Dadi wrote: > Roy, > With regard WODB, the users want to be able to search not just using physical > coordinates like lat-lon bounding box, but also using other attributes like > institution, project, platform etc which are usually not represented using > coordinate variables. So grouping the data based on physical coordinates > might cause performance issues when searching using other criteria. Would it > be possible to have some additional structures, something like a database > index, but without a full-fledge DBMS? Does TDS allow such structures? > > Upendra > > > Roy Mendelssohn wrote: >> Hi John: >> >> Since the UAF meeting in Seattle I have been giving some thought about how >> to serve some large, important datasets, such as the raw ICOADS observations >> or the WODB observations. While reading over the PointObservation >> Conventions proposal on the CF site, while the proposal makes it clear how I >> might put data into a netcdf file, it doesn't make clear what the interplay >> might be with a service in TDS, and how a possible service might be affected >> by a very large dataset without further structure. >> >> So it seems pretty clear that the ICOADS would be points. From the example: >> >> dimensions: >> obs = 1234 ; >> >> variables: >> double time(obs) ; >> time:long_name = "time of measurement" ; >> time:units = "days since 1970-01-01 00:00:00" ; >> float lon(obs) ; >> lon:long_name = "longitude of the observation"; >> lon:units = "degrees_east"; >> float lat(obs) ; >> lat:long_name = "latitude of the observation" ; >> lat:units = "degrees_north" ; >> float alt(obs) ; >> alt:long_name = "vertical distance above the surface" ; >> alt:standard_name = "height" ; >> alt:units = "m"; >> alt:positive = "up"; >> alt:axis = "Z"; >> >> float humidity(obs) ; >> humidity:long_name = "specific humidity" ; >> humidity:coordinates = "time lat lon alt" ; >> float temp(obs) ; >> temp:long_name = "temperature" ; >> temp:units = "Celsius" ; >> temp:coordinates = "time lat lon alt" ; >> >> attributes: >> :CF\:featureType = "point"; >> Now I am assuming that in a TDS implementation of a service, I will be able >> to select on the coordinate variables, is that correct? Even so, for >> something like ICOADS, obs is quite large and that extract could be quite >> slow unless either there is additional structure or the TDS pre-fetches the >> coordinate variables much as the present Dapper server does. >> >> Other options would be to say have a file for each 10-degree block, and then >> have TDS aggregate over the files - would this be possible. Then the search >> would a lot faster when people want time series in a region as opposed to >> more synoptic extractions. Would the TDS service be supporting such an >> option? Or, as netcdf-4 supports groups, to have 10-degree groups with >> 2-degree subgroups, which would work as far as netcdf-4 is concerned, but >> that is not the same as TDS knowing what to do with the hierarchy or to take >> advantage of the structure. >> >> My questions for Profiles (that is for the WODB) are pretty much the same. >> I assume that the TDS service will be able to search on the coordinate >> variables, is that correct? And I have the issue with the fact that the >> profile dimension variable will get quite large and without further >> structure the search could be very slow. Adding the same types of >> structures mentioned above would provide possible solutions, but only if >> TDS, as opposed to netcdf4, supported them. >> >> As you may have guessed, these are not theoretical questions - I would >> really like to see ICOADS and WODB served as part of the year 2 UAF effort. >> So now is a good time to start thinking about how to do it correctly and >> what the service will be able to do. >> >> Thoughts? >> >> Thanks, >> >> -Roy >> >> >> >> >> >> >> >> ********************** >> "The contents of this message do not reflect any position of the U.S. >> Government or NOAA." >> ********************** >> Roy Mendelssohn >> Supervisory Operations Research Analyst >> NOAA/NMFS >> Environmental Research Division >> Southwest Fisheries Science Center >> 1352 Lighthouse Avenue >> Pacific Grove, CA 93950-2097 >> >> e-mail: Roy.Mendelssohn@xxxxxxxx <mailto:Roy.Mendelssohn@xxxxxxxx> (Note new >> e-mail address) >> voice: (831)-648-9029 >> fax: (831)-648-8440 >> www: http://www.pfeg.noaa.gov/ >> >> "Old age and treachery will overcome youth and skill." >> "From those who have been given much, much will be expected" >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> thredds mailing list >> thredds@xxxxxxxxxxxxxxxx >> For list information or to unsubscribe, visit: >> http://www.unidata.ucar.edu/mailing_lists/ > ********************** "The contents of this message do not reflect any position of the U.S. Government or NOAA." ********************** Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center 1352 Lighthouse Avenue Pacific Grove, CA 93950-2097 e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address) voice: (831)-648-9029 fax: (831)-648-8440 www: http://www.pfeg.noaa.gov/ "Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected"
thredds archives: