NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi, re: > Note, Tom suggested I change: > request CONDUIT|NGRID ".*.gfs*" idd.unidata.ucar.edu > request CONDUIT|NGRID ".*.GFS*" idd.unidata.ucar.edu > request NGRID|CONDUIT "(gfs|GFS)" idd.unidata.ucar.edu > > Yes - the first two were commented out during the test. Very good. One of the primary reasons for the change was a ".*" at the beginning of a regular expression is not needed/redundant. This is what is referred to as a pathalogical regular expression. Here is Steve's explanation: A pathological regular expression is one that starts with ".*". Such a prefix changes nothing (the same product-identifiers are matched either way) but for some regular expression libraries, the matching can be orders of magnitude slower. re: > I'm not sure it is a coincidence, but ldm connections greatly increase > after that change I don't understand what you are saying here: "ldm connections greatly increase after the change" What greatly increased? - the number of connections? This should not be the case as the change combined two REQUESTs in one. - the latency of product receipt? The rtstats plot for latencies for NGRID on on-dsserve do not look like this was the case: https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NGRID+on-dcserve.ssec.wisc.edu - the volume of data being received in the NGRID feed? The volume rtstats volume plot for on-dsserve _does_ show this: https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?NGRID+on-dcserve.ssec.wisc.edu It should not be the case that combining two REQUESTs into one where the regular expression used in the combined REQUEST is, in essence, the same as the union of the regular expressions used in the two would result in more data being received! re: > and we dropped a lot of grib data that come in on NGRID. I don't know what this means either. Do you mean that it was not processed by XCD, or are you saying that a lot of grib data was not received? The rtstats volume plot for NGRID on on-dsserve indicates that you received _more_ data (and all of it is in GRIB2 format) in NGRID than before the change. re: > Our grib data is processed through XCD, so there maybe another > underlying problem. Yes, it is (near) impossible for us to make any comments about any data processing "downstream" of the LDM with the information at hand. re: > This may be of interest. > https://qcweb.ssec.wisc.edu/web/xcd/2021239/index.html It is interesting that data missing in the XCD processing pipeline is well documented. re: > As Tom mentioned, it's hard for you to comment if problems are XCD related. Correct. My comment should have been more generic - it is (near) impossible to make any specific comments about processing that is done downstream of the LDM receipt of data. All I can suggest is that someone there (you?) spend the time comparing what was actually received by the LDM (mine the log file for the 'notifyme' invocation that is looking at the local LDM) and want ended up being processed by your XCD processing pipeline. re: > I'm going back to the previous version of ldmd.conf to catch the > incoming 12Z run ... Cheers, Tom -- **************************************************************************** Unidata User Support UCAR Unidata Program (303) 497-8642 P.O. Box 3000 address@hidden Boulder, CO 80307 ---------------------------------------------------------------------------- Unidata HomePage http://www.unidata.ucar.edu **************************************************************************** Ticket Details =================== Ticket ID: CZE-486853 Department: Support LDM Priority: Normal Status: Closed =================== NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.