Method, system, apparatus, program code and means for determining a redundancy of information

• US 9,063,985 B2
• Filed: 05/10/2013
• Issued: 06/23/2015
• Est. Priority Date: 07/02/2004
• Status: Active Grant

• Current Assignee: Regulatory DataCorp, Inc. (Moody's Corporation)
• Original Assignee: Goldman Sachs & Company (Goldman Sachs Group Incorporated)
• Inventors: Lawrence, David, Nitze, Peter, MacDonald, Alasdair
• Primary Examiner(s): Ruiz, Angelica
• Application Number:
  US13/892,059
• Publication Number:
  US 20130246401A1
• Time in Patent Office: 774 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/24578 using ranking

Abstract

Some embodiments include a system, method, apparatus and means for determining that first information in an input data packet is not redundant with second information previously stored in a database system, includes receiving the input data packet, generating a database query based on one or more tagged portions in the input data packet, comparing second information retrieved by the database query with the first information to identify at least a first portion of the first information that is different than the second information, and causing storage of the at least a first portion of the first information in the database system.

Claims

1. A system, comprising:

an information feed;

a first device to store information; and

a second device in communication with the information feed and the first device, the second device configured to;

receive first information comprising one or more portions from the information feed,

determine a redundancy portion score associated with each of the one or more portions, wherein the redundancy portion score associated with the each of the one or more portions indicates a degree of redundancy of the each of the one or more portions with second information,

determine a first score comprising a total of the redundancy portion scores associated with the each of the one or more portions,

compare the first score with a redundancy limit, and

cause non-redundant portions of the first information to be stored in the first device.

2. The system of claim 1, wherein the second device includes:

a component to extract the second information from the first device.

3. The system of claim 2, wherein the second device further includes:

a component to compare the first information to the second information to identify the non-redundant portions of the first information.

4. The system of claim 3, wherein the second device further includes:

a component to insert the identified portions of the first information into the first device.

5. The system of claim 1, wherein the first device includes a data structure arranged to facilitate identification of risk.

6. The system of claim 1, wherein the second device is further configured to spell check the first information.

7. The system of claim 1, wherein the second device is further configured to cause redundant portions of the first information to be stored in the first device store.

8. The system of claim 1, wherein the second device is further configured to tag the each of the one or more portions with the corresponding redundancy portion score.

9. A method, comprising:

receiving first information comprising one or more portions from an information feed;

determining, by a processor, a redundancy portion score associated with each of the one or more portions, wherein the redundancy portion score associated with the each of the one or more portions indicates a degree of redundancy of the each of the one or more portions with second information;

determining, by the processor, a first score comprising a total of the redundancy portion scores associated with the each of the one or more portions;

comparing, by the processor, the first score with a redundancy limit; and

storing non-redundant portions of the first information in a database.

10. The method of claim 9, further comprising extracting the second information from the database.

11. The method of claim 10, further comprising:

comparing the first information to the second information to identify the non-redundant portions of the first information that are not redundant.

12. The method of claim 11, further comprising inserting the identified portions of the first information into the database.

13. The method of claim 9, further comprising arranging a data structure to facilitate identification of risk.

14. The method of claim 9, further comprising spell checking the first information.

15. A non-transitory, computer-readable medium comprising computer-readable instructions which when executed by a processor program the processor determine redundant information, the instructions comprising instructions for:

receiving first information comprising one or more portions from an information feed;

determining a redundancy portion score associated with each of the one or more portions, wherein the redundancy portion score associated with the each of the one or more portions indicates a degree of redundancy of the each of the one or more portions with second information;

determining a first score comprising a total of the redundancy portion scores associated with the each of the one or more portions;

comparing the first score with a redundancy limit; and

storing non-redundant portions of the first information in a database.

16. The non-transitory, computer-readable medium of claim 15, wherein the instructions further comprise instructions for extracting the second information from the database.

17. The non-transitory, computer-readable medium of claim 16, wherein the instructions further comprise instructions for comparing the first information to the second information to identify the non-redundant portions of the first information.

18. The non-transitory, computer-readable medium of claim 17, wherein the instructions further comprise instructions for inserting the identified portions of the first information into the database.

19. The non-transitory, computer-readable medium of claim 15, wherein the instructions further comprise instructions for arranging a data structure to facilitate identification of risk.

20. The non-transitory, computer-readable medium of claim 15, wherein the instructions further comprise instructions for spell checking the first information.