NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi Sean- On 7/2/13 7:07 PM, Sean Arms wrote:
Is there a reason to use the NAM CONUS 20km grids over the 12km grids? I do believe these files were being FTP'd to Unidata servers, and were not being transmitted via IDD, so perhaps something changed on the FTP server. At any rate, no one mentioned any issue with accessing these files (again, the other machines with much longer scour times were empty too, so it's been down for quite some time), so I made the call to remove them from the catalog. Perhaps it was the wrong call for me to make.
I don't use the 20km NAM, but to answer your more general question of why would I use a lower resolution grid, it would be for speed of download. For example, I use the 1.0 degree GFS for "quick, large scale looks", but if I want a little more detail over a particular area, I use the .5 degree grid. Same with the 40 vs. 13 km RAP - 40 km is good for CONUS, 13 is better for smaller area analysis.
I would prefer to see these changes announced before the fact rather than after or on a web page that one has to search around for.That would be the ideal situation! The idea of this page is that it will be "the place" to look. No need to search for it if you bookmark it (and if it's prominently linked off of our main page). What do you feel would be the appropriate venue other than a status page? I'm assuming some soft of real time notification? RSS, twitter (which can be used like an rss feed), mailing list (which already exist, like nws-changes), the blink html tag in the thredds catalogs? These have seriously crossed my mind. Ok, the blink tag might be a bit much (and it's not being support much anymore), but in all honestly, what would be most beneficial for you (or anyone else - feel free to chime in!).
You already have a thredds mailing list and a datastream mailing list. And you already have a community RSS feed. All I'm asking is that you use them. It's a matter of push vs. pull. For news, I like the push model where I find out ahead of time what's going on rather than having to check (pull) every day a web page that may or may not be updated ahead of a change.
For example, the recent change from storing 21 days to 15 days impacted my work and I only found out about it after when I tried to display the data.That change was a bit difficult - we could not give much of a lead time as our disk space was ran low quick. This is in part due to us including the new FNMOC NAVGEM 0.5 degree global model output, but level II radar data volumes are also getting tough to manage (and continue to grow). Also, as you mentioned in your support message, you understand that Unidata isn't a data archive center.
However, the UPC has maintained a valuable resource in the "motherlode" TDS and ADDE servers which many in the community have come to rely on.
> You also mention NOMADS, which is a
model data archive (oversimplified statement). I do believe they are testing out the use of TDS 4.3, which will ease the issues you have with the way they serve out the individual forecast grids. Have you tried to contact them to see what their plans are?
I have not, but in the past, the way they store their data is not conducive to the types of queries I can make to thredds.ucar.edu. Datasets are not aggregated very well for multi-day access.
...shameless plug... Not necessairly news for you, but for the benefit of others...This is also a good time to note that while the thredds servers at Unidata (particularly the one housed on the motherlode machine) has always been a "demonstration" installation (even though it has been "good enough" for people to heavily rely upon). We encourage people in the community to apply for Unidata equipment awards to setup their own "motherlode"-like servers: http://www.unidata.ucar.edu/community/equipaward/
I'm all for a community network of servers, but the reality is that most Unidata sites do not have the resources to maintain the type of datasets that motherlode provides. And community servers tend to come and go, whereas motherlode has been a pretty consistent source of near realtime data.
Also, if there are real-time datasets people would like to see being served by Unidata, the best way to make the request is through our Users Committee: http://www.unidata.ucar.edu/committees/usercom/ We've relied on them in the past (and will continue to do so) to get a sense of what model output are important to the community, what model output could be cut with minimal impact, and what would be beneficial to add, say, to the CONDUIT feed (it should be noted that NCEP has been awesome about listening to our community!).
Were they consulted on the recent changes?
...end shameless plug...What is the status of the longer term LEAD archive? Will it be upgraded to TDS 4.3 in the near future so I can switch to that for my longer "real-time" archive needs?We are discussing how to best proceed with our other servers. Do we simply make motherlode clones and distribute the load across them? How do we best allow clients, like the IDV, to have one access url that can point to other resources behind the scenes (as to prevent bundle breaking if a server goes down). It does not make sense to blast forward at this point, but. we are talking about these things internally
That's good to know. In terms of LEAD, it is not a clone, but a longer term storage of some of the datasets. The big problem for me is that I had to redo all my access to account for the new names of the GRIB variables when thredds.ucar.edu went to 4.3 and if I wanted to go to the lead machine which is still running 4.2, I'd have to redo them again to go back to the old names. And when the lead machine goes to 4.3, I'd have to do that again.
Unidata provides a valuable service with the thredds.ucar.edu dataset to the IDV and other communities, so even minor changes have impacts that would be nice to know of before they happen.Again, some changes we can announce with some lead-time, but these are mostly limited to NWS or CONDUIT feed changes (for which email lists already exist).
NWS and CONDUIT change messages do not easily translate changes in the user will experience for the data access. It would be nice if the UPC would provide the translation and disseminate that information. An example of where this worked well was the move to the RAP.
Don -- Don Murray NOAA/ESRL/PSD and CIRES 303-497-3596 http://www.esrl.noaa.gov/psd/people/don.murray/
thredds archives: