STATISTICAL MEASURE AND CALIBRATION OF SEARCH CRITERIA WHERE ONE OR BOTH OF THE SEARCH CRITERIA AND DATABASE IS INCOMPLETE

• US 20100005090A1
• Filed: 07/02/2009
• Published: 01/07/2010
• Est. Priority Date: 07/02/2008
• Status: Active Grant

• Current Assignee: LexisNexis Risk Solutions, Inc. (RELX PLC)
• Original Assignee: LexisNexis Risk & Information Analytics Group, Inc. (RELX PLC)
• Inventors: BAYLISS, David Alan
• Granted Patent: US 8,090,733 B2
• Time in Patent Office: Days
• US Class Current: 707/5
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 16/24578 using ranking

Abstract

Disclosed is a system for, and method of, identifying an entity representation. In some embodiments, search criteria are used to identify an entity representation in a universal database, and this identification is then used to identify a corresponding entity representation in a foreign database. Certain embodiments provide assurance, with a know probability of error, that the entity representation identified in the universal database is correct.

Claims

1. A method of identifying an entity representation in an electronic universal database that corresponds to an entity representation in an electronic foreign database, each database comprising a plurality of entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight, the method comprising:

electronically storing a plurality of field tables, each field table corresponding to a particular field, each field table comprising field value weights for each unique pair consisting of an arbitrary entity representation from the universal database and a field value appearing in the particular field of a record in the arbitrary entity representation from the universal database;

receiving a plurality of search criteria field values identifying an entity representation in the foreign database;

for each search criteria field value, fetching a weight from an associated field table corresponding to the search criteria field value;

summing results of the step of fetching according to entity representations from the universal database, resulting in a plurality of summed weights, one summed weight for each of a plurality of entity representations from the universal database;

ranking entity representations according to the plurality of summed weights;

determining a highest ranked entity representation;

calculating a confidence level reflecting a likelihood that the highest ranked entity representation corresponds to the entity representation identified by the search criteria field values; and

outputting, if the confidence level exceeds a predetermined threshold, an identifier for the highest ranked entity representation.

2. The method of claim 1, wherein each field value weight for the universal database comprises a logarithm of a probability that an arbitrary entity representation in the universal database comprises a corresponding field value in a field of a record in the arbitrary entity representation.

3. The method of claim 1, wherein the calculating a confidence level comprises calculating a difference between a summed weight for the highest ranked entity reference and a summed weight for a next highest ranked entity reference, and wherein the threshold comprises a logarithm of a term comprising a confidence level.

4. The method of claim 1, wherein the calculating a confidence level comprises a sum of terms, each term comprising an exponent, each exponent comprising a difference between a summed weight for the highest ranked entity reference and a summed weight for another entity representation.

5. The method of claim 4, wherein the sum is over an initial portion of highest ranked entity representations.

6. The method of claim 1, wherein the calculating a confidence level comprises calculating a difference between a summed weight for the highest ranked entity reference and a term comprising a difference between a number of entities reflected in the universal database and a number of entities reflected in the foreign database, and wherein the threshold comprises a logarithm of a term comprising a confidence level.

7. The method of claim 1, wherein the outputting an identifier comprises outputting a social security number.

14. The method of claim 1, wherein the outputting an identifier comprises outputting a social security number.

8. A method of identifying an entity representation in an electronic universal database that corresponds to an entity representation in an electronic foreign database, each database comprising a plurality of entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight, the method comprising:

receiving a plurality of search criteria field values identifying an entity representation in the foreign database;

ranking entity representations in the universal database according to summed field value weights, wherein each summed field value weight comprises weights corresponding to field values, from records within a same entity representation from the universal database, that match a search criteria field value, and wherein, for each summed field value weight, each weight corresponding to field values is counted at most once for each search criteria field value;

determining a highest ranked entity representation;

calculating a confidence level reflecting a likelihood that the highest ranked entity representation corresponds to the entity representation identified by the search criteria field values; and

outputting, if the confidence level exceeds a predetermined threshold, an identifier for the highest ranked entity representation.

9. The method of claim 8, wherein each field value weight for the universal database comprises a logarithm of a probability that an arbitrary entity representation in the universal database comprises a corresponding field value in a field of a record in the arbitrary entity representation.

10. The method of claim 8, wherein the calculating a confidence level comprises calculating a difference between a summed weight for the highest ranked entity reference and a summed weight for a next highest ranked entity reference, and wherein the threshold comprises a logarithm of a term comprising a confidence level.

11. The method of claim 8, wherein the calculating a confidence level comprises a sum of terms, each term comprising an exponent, each exponent comprising a difference between a summed weight for the highest ranked entity reference and a summed weight for another entity representation.

12. The method of claim 11, wherein the sum is over an initial portion of highest ranked entity representations.

13. The method of claim 8, wherein the calculating a confidence level comprises calculating a difference between a summed weight for the highest ranked entity reference and a term comprising a difference between a number of entities reflected in the universal database and a number of entities reflected in the foreign database, and wherein the threshold comprises a logarithm of a term comprising a confidence level.

15. A system for identifying an entity representation in an electronic universal database that corresponds to an entity representation in an electronic foreign database, each database comprising a plurality of entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight, the system comprising:

an electronic universal database comprising a plurality of electronically stored entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight;

a processor programmed to form and store a plurality of field tables, each field table corresponding to a particular field, each field table comprising field value weights for each unique pair consisting of an arbitrary entity representation from the universal database and a field value appearing in the particular field of a record in the arbitrary entity representation from the universal database;

an electronic memory storing a plurality of search criteria field values identifying an entity representation in the foreign database;

a processor programmed to, for each search criteria field value, fetch a weight from an associated field table corresponding to the search criteria field value;

a processor programmed to sum the fetched weights according to entity representations from the universal database, resulting in a stored plurality of summed weights, one summed weight for each of a plurality of entity representations from the universal database;

a processor configured to rank entity representations according to the plurality of summed weights;

a processor programmed to determine a highest ranked entity representation;

a processor programmed to calculate a confidence level reflecting a likelihood that the highest ranked entity representation corresponds to the entity representation identified by the search criteria field values; and

a processor programmed to output, if the confidence level exceeds a predetermined threshold, an identifier for the highest ranked entity representation.

16. The system of claim 15, wherein each field value weight for the universal database comprises a logarithm of a probability that an arbitrary entity representation in the universal database comprises a corresponding field value in a field of a record in the arbitrary entity representation.

17. The system of claim 15, wherein the processor programmed to calculate a confidence level is further configured to calculate a difference between a summed weight for the highest ranked entity reference and a summed weight for a next highest ranked entity reference, and wherein the threshold comprises a logarithm of a term comprising a confidence level.

18. The system of claim 15, wherein the processor programmed to calculate a confidence level is further configured to calculate a sum of terms, each term comprising an exponent, each exponent comprising a difference between a summed weight for the highest ranked entity reference and a summed weight for another entity representation.

19. The system of claim 18, wherein the sum is over an initial portion of highest ranked entity representations.

20. The system of claim 15, wherein the processor programmed to calculate a confidence level is further configured to calculate a difference between a summed weight for the highest ranked entity reference and a term comprising a difference between a number of entities reflected in the universal database and a number of entities reflected in the foreign database, and wherein the threshold comprises a logarithm of a term comprising a confidence level.

21. The system of claim 15, wherein the identifier comprises a social security number.

22. A system for identifying an entity representation in an electronic universal database that corresponds to an entity representation in an electronic foreign database, each database comprising a plurality of entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight, the system comprising:

an electronic universal database comprising a plurality of electronically stored entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight;

an electronic memory storing a plurality of search criteria field values identifying an entity representation in the foreign database;

a processor programmed to rank entity representations in the universal database according to summed field value weights, wherein each summed field value weight comprises weights corresponding to field values, from records within a same universal entity representation, that match a search criteria field value, and wherein, for each summed field value weight, each weight corresponding to field values is counted at most once for each search criteria field value;

a processor programmed to determine a highest ranked universal entity representation;

a processor programmed to calculate a confidence level reflecting a likelihood that the highest ranked entity representation corresponds to the entity representation identified by the search criteria field values; and

a processor programmed to output, if the confidence level exceeds a predetermined threshold, an identifier for the highest ranked entity representation.

23. The system of claim 22, wherein each field value weight for the universal database comprises a logarithm of a probability that an arbitrary entity representation in the universal database comprises a corresponding field value in a field of a record in the arbitrary entity representation.

24. The system of claim 22, wherein the processor configured to calculate a confidence level is further configured to calculate a difference between a summed weight for the highest ranked entity reference and a summed weight for a next highest ranked entity reference, and wherein the threshold comprises a logarithm of a term comprising a confidence level.

25. The system of claim 22, wherein the processor configured to calculate a confidence level is further configured to calculate a sum of terms, each term comprising an exponent, each exponent comprising a difference between a summed weight for the highest ranked entity reference and a summed weight for another entity representation.

26. The system of claim 25, wherein the sum is over an initial portion of highest ranked entity representations.

27. The system of claim 22, wherein the processor configured to calculate a confidence level is further configured to calculate a difference between a summed weight for the highest ranked entity reference and a term comprising a difference between a number of entities reflected in the universal database and a number of entities reflected in the foreign database, and wherein the threshold comprises a logarithm of a term comprising a confidence level.

28. The system of claim 22, wherein the identifier comprises a social security number.