Method, system, apparatus, program code and means for determining a redundancy of information

• US 8,442,953 B2
• Filed: 09/15/2004
• Issued: 05/14/2013
• Est. Priority Date: 07/02/2004
• Status: Expired due to Fees

• Current Assignee: Regulatory DataCorp, Inc. (Moody's Corporation)
• Original Assignee: Goldman Sachs & Company (Goldman Sachs Group Incorporated)
• Inventors: Lawrence, David, Nitze, Peter, MacDonald, Alasdair
• Primary Examiner(s): Ruiz, Angelica
• Application Number:
  US10/941,584
• Publication Number:
  US 20060004878A1
• Time in Patent Office: 3,163 Days
• Field of Search: 705/38, 707/203, 707/204, 707/4, 711/161, 711/162
• US Class Current: 707/690
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/24578 using ranking

Abstract

Some embodiments include a system, method, apparatus and means for determining that first information in an input data packet is not redundant with second information previously stored in a database system, includes receiving the input data packet, generating a database query based on one or more tagged portions in the input data packet, comparing second information retrieved by the database query with the first information to identify at least a first portion of the first information that is different than the second information, and causing storage of the at least a first portion of the first information in the database system.

Claims

1. A processor-implemented method for determining that first information in an input data packet is not redundant with second information previously stored in a database system, the method comprising:

receiving an input data packet comprising first information, wherein the first information comprises one or more portions;

processing the received input data packet in accordance with coarse and fine political risk data filters to identify one or more political risk relevant portions;

determining via a processor portion scores, each portion score associated with each of the political risk relevant portions;

applying tags to at least one of the political risk relevant portions;

determining a first score comprising a total of the portion scores associated with the political risk relevant tagged portions;

generating a database query based on the political risk relevant tagged portions in said input data packet and on the first score;

receiving second information based on said database query;

comparing said second information with said first information to identify at least a first portion of said first information that is different than said second information based on the first score and a redundancy limit; and

facilitating storage of said at least a first portion of said first information in said database system based on the comparison of the first score and the redundancy limit.

2. The method of claim 1, wherein said tagged portions include tags identifying one or more relevant terms.

3. The method of claim 1, wherein said comparing second information further comprises identifying a second portion of said first information that is similar to said second information.

4. The method of claim 3, further comprising:

removing said second portion from said first information prior to said causing storage of said at least first portion.

7. The method of claim 1, further comprising spell checking each portion of the one or more portions of the input data packet.

8. The method of claim 1, wherein the tags are applied to at least one of the political risk relevant portions based on rules.

9. The method of claim 1, further comprising identifying presence or absence of predetermined keywords and associated terms in the received input data packet.

10. The method of claim 1, wherein the tags identify and associate the political risk relevant portions with the portion scores.

11. The method of claim 3, further comprising associating said second portion of the said information with a redundancy metric.

5. A system, comprising:

a processor; and

a storage device in communication with said processor and storing instructions adapted to be executed by said processor to;

receive an input data packet comprising first information, wherein the first information comprises one or more portions;

process the received input data packet in accordance with coarse and fine political risk data filters to identify one or more political risk relevant portions;

determine portion scores, each portion score associated with each of the political risk relevant portions;

apply tags to at least one of the political risk relevant portions;

determine a first score comprising a total of the portion scores associated with the political risk relevant tagged portions;

generate a database query based on the political risk relevant tagged portions in said input data packet and on the first score;

receive second information based on said database query;

compare said second information with said first information to identify at least a first portion of said first information that is different than said second information based on the first score and a redundancy limit; and

facilitate storage of said at least a first portion of said first information in said database system based on the comparison of the first score and the redundancy limit.

12. The system of claim 5, further comprising instructions adapted to be executed by the processor to compare said second information with said first information to identify a second portion of said first information that is similar to said second information.

13. The system of claim 12, further comprising instructions adapted to be executed by the processor to associate said second portion of said first information with a redundancy metric.

14. The system of claim 5, further comprising instructions adapted to be executed by the processor to identify presence or absence of predetermined keywords and associated terms in the received input data packet.

6. A processor-readable medium comprising a storage element, the storage element storing processor-issuable instructions to:

receive an input data packet comprising first information, wherein the first information comprises one or more portions;

process the received input data packet in accordance with coarse and fine political risk data filters to identify one or more political risk relevant portions;

determine portion scores, each portion score associated with each of the political risk relevant portions;

apply a tag to at least one of the political risk relevant portions;

determine a first score comprising a total of the of the portion scores associated with the political risk relevant tagged portions;

generate a database query based on the political risk relevant tagged portions in said input data packet and on the first score;

receive second information based on said database query;

compare said second information with said first information to identify at least a first portion of said first information that is different than said second information based on the first score and a redundancy limit; and

store said at least a first portion of said first information in said database system based on the comparison of the first score and the redundancy limit.