NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Dear Peter,I read all the e-mails from this thread and checked the statistics of your machines, including the latencies. ldmingest01.nwc.ou.edu [6.8.1] <http://www.unidata.ucar.edu/cgi-bin/rtstats/siteindex?ldmingest01.nwc.ou.edu> ldmingest02.nwc.ou.edu [6.8.1] <http://www.unidata.ucar.edu/cgi-bin/rtstats/siteindex?ldmingest02.nwc.ou.edu>
See that you are losing data, more on ingest02 than ingest01, and that the latencies on both machines reach a delay of 3600s many times, when your LDM "gives up", forget about the old products and request for the newer ones. Everytime this happen you lose data.
To me seems more a situation where the LDM data-requests are more than the current capability of your network to serve your LDMs.
I am used to similar situations. So, I suggest:1) check the network connection to idd.unidata.ucar.edu, your feeder. Traceroute is ok, but I prefer to use the "mtr":
mtr idd.unidata.ucar.edukeep watching for 5 minutes. If any "packet loss" appears on the path, you found where the problem is. These things will keep your network working, perhaps a bit slower, but only you (data hunger) will notice anything...
2) Your LDM machines are getting a lot of data. Someone in your path could have set a kind of "packet shaping", "QoS" or anything else to reduce the priority of who is getting too much data (you). For the others the network will be nice and faster, and only your LDMs will suffer. If this is the case, there are tricks to cheat these things, like to replace one request line for several ones. This worked fine to improve the massive LDM feeds to South America.
replace: REQUEST ANY ".*" idd.unidata.ucar.edu By: REQUEST NEXRAD2 ".*" idd.unidata.ucar.edu REQUEST CONDUIT ".*" idd.unidata.ucar.edu REQUEST FNMOC ".*" idd.unidata.ucar.edu REQUEST NGRID ".*" idd.unidata.ucar.edu REQUEST NEXRAD3 ".*" idd.unidata.ucar.edu REQUEST HDS ".*" idd.unidata.ucar.edu and so on...Check to see if this improves the output. At this moment your statistics are:
Data Volume Summary for ldmingest01.nwc.ou.edu Maximum hourly volume 12967.169 M bytes/hour Average hourly volume 3520.793 M bytes/hour Data Volume Summary for ldmingest02.nwc.ou.edu Maximum hourly volume 5016.503 M bytes/hour Average hourly volume 845.772 M bytes/hour Cheers, Waldenio. -- Waldenio Gambi de Almeida Instituto Nacional de Pesquisas Espaciais Centro de Previsão de Tempo e Estudos Climáticos Divisão de Operacoes Rodovia Pres. Dutra, km 40 Cachoeira Paulista/SP - Brasil 12630-000 Phone : (12)3186-8541 e-mail : waldenio.almeida@xxxxxxxxxxxxx
ldm-users archives: