[ldm-users] 20080318: LDM and duplicate product rejection

• To: "Rout, Minaketan" <MRout@xxxxxxxxxxx>
• Subject: [ldm-users] 20080318: LDM and duplicate product rejection
• From: Tom Yoksas <yoksas@xxxxxxxxxxxxxxxx>
• Date: Tue, 18 Mar 2008 16:10:24 -0600

Hi Min,

re:
>I was hoping LDM should de-dup identical NWS text bulletins but does not
>look like. I am attaching 2 bulletins which are very identical and it
>came through LDM.

As a follow-on to the reference Patrick provided earlier (a response
by Steve Chiswell (Chiz) to a user's inquiry), I can add:

- the LDM will reject identical products _if_ the original is still
  in the LDM queue

- products are considered identical _if_ their MD5 signatures are the
  same

If you can catch an instance where you have received what you believe
to be an identical duplicate of a product, you should run the LDM
'pqcat' utility to list out the original product's and its identical
twin's MD5 signatures:

<as 'ldm'>
pqcat -vxl- -f <feedtype> -p 'product ID matching regular expression' > 
/dev/null

The listing of the product(s) whose IDs match the '-p' regular expression
will contain the MD5 signatures for the product(s).  Comparisons of the
MD5 signatures should show that the products are different even though
the text looks the same.

Cheers,

Tom
--
+----------------------------------------------------------------------------+
* Tom Yoksas                                            UCAR Unidata Program *
* (303) 497-8642 (last resort)                                 P.O. Box 3000 *
* yoksas@xxxxxxxxxxxxxxxx                                  Boulder, CO 80307 *
* Unidata WWW Service                            http://www.unidata.ucar.edu/*
+----------------------------------------------------------------------------+

• References:
  • [ldm-users] LDM and de-dup
    • From: Rout, Minaketan