NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Tim, > ... If I understand things correctly, > there is an RPC call for _every_ product sent down the line. While this > makes sense for large products, it seems like we are essentially doubling > the communication overhead for the smaller products. Is there a way that > the data can be held at the upstream host (maybe at the root host even), > disseminated in larger bundles to the downstream host, where the data > stream could be unbundled? The bundles could be defined either in terms of > number of products or duration of time... where the later would seem to > make more sense. Yes, there's even a name for this: batched RPC. A previous version of the LDM, version 4, used batched RPC to try to minimize communication overhead. Here's what the Site Manager's Guide said about this: Previous versions of the LDM protocol relied on batched RPC. This is simply a remote procedure call that is made over a reliable connection and expects no return value. The advantage to this system is that it allows asynchronous processing to occur and the system has more freedom to use buffering to increase performance. But using batched RPC over wide area networks caused some problems, so we no longer use batched RPC in LDM version 5; all the remote procedure calls have a return value. Setting the "way-back machine", I seem to remember the problems with this mechanism included: - It was not possible to get back a status for whether individual products were successfully delivered, so the granularity of failure handling increased and the reliability of the LDM suffered as a result. - Batching interfered with timely delivery of products. If there is a period in a product stream during which no data is injected, the data that is still batched is "stuck in the pipe" so may not be delivered as soon it is available, increasing product latencies. There are ways to solve this with timeouts, but they complicate the protocol and architecture of the LDM. - Some vendors implementations of batched RPC was buggy, and we had to find workarounds for the bugs. - Using the "rpcgen" tool to generate RPC stubs doesn't work for batched RPC, so it all has to be hand-coded, which is difficult and error-prone. --Russ _____________________________________________________________________ Russ Rew UCAR Unidata Program russ@xxxxxxxxxxxxxxxx http://www.unidata.ucar.edu
ldm-users archives: