Statistical measure and calibration of search criteria where one or both of the search criteria and database is incomplete

• US 8,090,733 B2
• Filed: 07/02/2009
• Issued: 01/03/2012
• Est. Priority Date: 07/02/2008
• Status: Active Grant

• Current Assignee: LexisNexis Risk Solutions, Inc. (RELX PLC)
• Original Assignee: LexisNexis Risk & Information Analytics Group, Inc. (RELX PLC)
• Inventors: Bayliss, David Alan
• Primary Examiner(s): Betit, Jacob F
• Assistant Examiner(s): HUANG, MIRANDA M
• Application Number:
  US12/496,861
• Publication Number:
  US 20100005090A1
• Time in Patent Office: 915 Days
• Field of Search: 707/748, 707/758
• US Class Current: 707/758
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 16/24578 using ranking

Abstract

Disclosed is a system for, and method of, identifying an entity representation. In some embodiments, search criteria are used to identify an entity representation in a universal database, and this identification is then used to identify a corresponding entity representation in a foreign database. Certain embodiments provide assurance, with a know probability of error, that the entity representation identified in the universal database is correct.

Claims

1. A method of identifying an entity representation in an electronic universal database that corresponds to an entity representation in an electronic foreign database, each database comprising a plurality of entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight, the method comprising:

electronically storing a plurality of field tables, each field table corresponding to a particular field, each field table comprising field value weights for each unique pair consisting of an arbitrary entity representation from the universal database and a field value appearing in the particular field of a record in the arbitrary entity representation from the universal database, wherein each field value weight comprises a logarithm of a probability that an arbitrary entity representation in the universal database comprises a corresponding field value in a field of a record in the arbitrary entity representation, wherein each probability comprise a ratio of entity representation in the universal database that contain a corresponding field value to a total number of entity representations in the universal database;

receiving a plurality of search criteria field values identifying an entity representation in the foreign database;

performing a fetch operation from an associated field table for each search criterion, by for each search criteria field value, fetching a field value weight from an the associated field table corresponding to the search criteria field value;

summing results of the step of fetching field value weights for each field from the fetch operation according to entity representations from the universal database, resulting in a plurality of summed weights, one summed weight for each of a plurality of entity representations from the universal database;

ranking entity representations according to the plurality of summed weights;

determining a highest ranked entity representation;

calculating a confidence level reflecting a likelihood that the highest ranked entity representation corresponds to the entity representation identified by the search criteria field values, wherein the calculation is based on the summed field value weight; and

outputting, if the confidence level exceeds a predetermined threshold, wherein the threshold comprises a logarithm of a term comprising a confidence level, an identifier for the highest ranked entity representation.

2. The method of claim 1, wherein the calculating a confidence level comprises calculating a difference between a summed weight for the highest ranked entity reference and a summed weight for a next highest ranked entity reference, and wherein the threshold comprises a logarithm of a term comprising a confidence level.

3. The method of claim 1, wherein the calculating a confidence level comprises a sum of terms, each term comprising an exponent, each exponent comprising a difference between a summed weight for the highest ranked entity reference and a summed weight for another entity representation.

4. The method of claim 3, wherein the sum is over an initial portion of highest ranked entity representations.

5. The method of claim 1, wherein the calculating a confidence level comprises calculating a difference between a summed weight for the highest ranked entity reference and a term comprising a difference between a number of entities reflected in the universal database and a number of entities reflected in the foreign database, and wherein the threshold comprises a logarithm of a term comprising a confidence level.

6. The method of claim 1, wherein the outputting an identifier comprises outputting a social security number.

7. A system for identifying an entity representation in an electronic universal database that corresponds to an entity representation in an electronic foreign database, each database comprising a plurality of entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight, the system comprising:

an electronic universal database comprising a plurality of electronically stored entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight;

a processor programmed to form and store a plurality of field tables, each field table corresponding to a particular field, each field table comprising field value weights for each unique pair consisting of an arbitrary entity representation from the universal database and a field value appearing in the particular field of a record in the arbitrary entity representation from the universal database, wherein each field value weight comprises a logarithm of a probability that an arbitrary entity representation in the universal database comprises a corresponding field value in a field of a record in the arbitrary entity representation, wherein each probability comprises a ratio of entity representations in the universal database that contain a corresponding field value to a total number of entity representations in the universal database;

an electronic memory storing a plurality of search criteria field values identifying an entity representation in the foreign database;

a processor programmed to, for each search criteria field value, perform a fetch operation from an associated field table for each search criteria, fetch a field value weight from an the associated field table corresponding to the search criteria field value;

a processor programmed to sum the fetched weights field value weights for each field from the fetch operation according to entity representations from the universal database, resulting in a stored plurality of summed weights, one summed weight for each of a plurality of entity representations from the universal database;

a processor configured to rank entity representations according to the plurality of summed weights;

a processor programmed to determine a highest ranked entity representation;

a processor programmed to calculate a confidence level reflecting a likelihood that the highest ranked entity representation corresponds to the entity representation identified by the search criteria field values, wherein the calculation is based on the summed field value weight; and

a processor programmed to output, if the confidence level exceeds a predetermined threshold, wherein the threshold comprises a logarithm of a term comprising a confidence level, an identifier for the highest ranked entity representation.

8. The system of claim 7, wherein the processor programmed to calculate a confidence level is further configured to calculate a difference between a summed weight for the highest ranked entity reference and a summed weight for a next highest ranked entity reference, and wherein the threshold comprises a logarithm of a term comprising a confidence level.

9. The system of claim 7, wherein the processor programmed to calculate a confidence level is further configured to calculate a sum of terms, each term comprising an exponent, each exponent comprising a difference between a summed weight for the highest ranked entity reference and a summed weight for another entity representation.

10. The system of claim 9, wherein the sum is over an initial portion of highest ranked entity representations.

11. The system of claim 7, wherein the processor programmed to calculate a confidence level is further configured to calculate a difference between a summed weight for the highest ranked entity reference and a term comprising a difference between a number of entities reflected in the universal database and a number of entities reflected in the foreign database, and wherein the threshold comprises a logarithm of a term comprising a confidence level.

12. The system of claim 7, wherein the identifier comprises a social security number.