STATISTICAL MEASURE AND CALIBRATION OF INTERNALLY INCONSISTENT SEARCH CRITERIA WHERE ONE OR BOTH OF THE SEARCH CRITERIA AND DATABASE IS INCOMPLETE

• US 20100005057A1
• Filed: 07/02/2009
• Published: 01/07/2010
• Est. Priority Date: 07/02/2008
• Status: Active Grant

• Current Assignee: LexisNexis Risk Solutions, Inc. (RELX PLC)
• Original Assignee: LexisNexis Risk & Information Analytics Group, Inc. (RELX PLC)
• Inventors: BAYLISS, David Alan
• Granted Patent: US 8,572,070 B2
• Time in Patent Office: Days
• US Class Current: 707/2
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 16/24578 using ranking

Abstract

Disclosed is a system for, and method of, searching for and identifying an entity representation. Some embodiments permit search criteria that are internally inconsistent. Such internally inconsistent criteria may include, for example, a maiden last name and a married last name. Certain embodiments account for such criteria in an intelligent manner and identify matching entity representations with a known confidence level of accuracy.

Claims

1. A method of identifying, using an internally inconsistent search criteria, an entity representation in an electronic universal database that corresponds to an entity representation in an electronic foreign database, each database comprising a plurality of entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight, the method comprising:

receiving a plurality of search criteria field values, each search criteria field value associated with a field, wherein at least two search criteria field values are associated with a same field, wherein the at least two search criteria field values are not identical;

receiving at least one match template specifying an ordered plurality of fields;

forming and electronically storing, for each match template, a table comprising field value weights for a plurality of matches between a search criteria field value and a field value appearing in a record in the universal database, the field value weights in each table arranged according to entity representation, wherein the at least two search criteria field values match field values in two records in a same entity representation, and wherein at least one table comprises a sum of at least a portion of field value weights for the at least two search criteria field values that match field values in records in the same entity representation;

merging the tables according to entity representation, resulting in a merged table;

summing field value weights according to entity representation in the merged table, resulting in a plurality of summed weights, one summed weight for each entity representation;

ranking entity representations in the merged table according to the plurality of summed weights;

determining a highest ranked entity representation;

calculating a confidence level reflecting a likelihood that the highest ranked entity representation corresponds to the plurality of query field values; and

outputting, if the confidence level exceeds a predetermined threshold, an identifier for the highest ranked entity representation.

2. The method of claim 1, wherein each field value weight comprises a logarithm of a probability that an arbitrary entity representation in the universal database comprises a corresponding field value in a field of a record in the arbitrary entity representation.

3. The method of claim 1, wherein at least one table comprising field value weights for a plurality of matches between a search criteria field value and a field value appearing in a record in the universal database comprises a field value weight for a nonidentical match between a search criteria field value and a field value appearing in a record in the universal database, and wherein each summed weight comprises at most one field value weight for each of the at least two search criteria field values.

4. The method of claim 1, wherein at least one match template comprises a fixed field portion and an optional field portion.

5. The method of claim 4, wherein an arbitrary record matches an arbitrary search criteria if all fixed fields of the arbitrary record are populated with field values that match corresponding fixed field values of the arbitrary search criteria and if all optional fields of the arbitrary record are each populated with one of a null field value and a field value that matches a corresponding optional field value of the arbitrary search criteria.

6. The method of claim 1, wherein the identifier comprises a social security number.

7. A method of identifying, using an internally inconsistent search criteria, an entity representation in an electronic universal database that corresponds to an entity representation in an electronic foreign database, each database comprising a plurality of entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight, the method comprising:

receiving a plurality of search criteria field values, each search criteria field value associated with a field, wherein at least two search criteria field values are associated with a same field, wherein the at least two search criteria field values are not identical;

forming and electronically storing a table comprising field value weights for a plurality of matches between a search criteria field value and a field value appearing in a record in the universal database, the field value weights in each table arranged according to entity representation, wherein the at least two search criteria field values match field values in two records in a same entity representation, and wherein at least one table comprises a sum of at least a portion of field value weights for the at least two search criteria field values that match field values in records in the same entity representation;

merging the tables according to entity representation, resulting in a merged table;

summing field value weights according to entity representation in the merged table, resulting in a plurality of summed weights, one summed weight for each entity representation;

ranking entity representations in the merged table according to the plurality of summed weights;

determining a highest ranked entity representation;

calculating a confidence level reflecting a likelihood that the highest ranked entity representation corresponds to the plurality of query field values; and

outputting, if the confidence level exceeds a predetermined threshold, an identifier for the highest ranked entity representation.

8. The method of claim 7, wherein each field value weight comprises a logarithm of a probability that an arbitrary entity representation in the universal database comprises a corresponding field value in a field of a record in the arbitrary entity representation.

9. The method of claim 7, wherein at least one table comprising field value weights for a plurality of matches between a search criteria field value and a field value appearing in a record in the universal database comprises a field value weight for a non-identical match between a search criteria field value and a field value appearing in a record in the universal database, and wherein each summed weight comprises at most one field value weight for each of the at least two search criteria field values.

10. The method of claim 7, wherein the identifier comprises a social security number.

11. A system for identifying, using an internally inconsistent search criteria, an entity representation in an electronic universal database that corresponds to an entity representation in an electronic foreign database, each database comprising a plurality of entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight, the system comprising:

an electronic universal database comprising a plurality of electronically stored entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight;

an electronic memory storing a plurality of search criteria field values, each search criteria field value associated with a field, wherein at least two search criteria field values are associated with a same field, wherein the at least two search criteria field values are not identical;

an electronic memory storing at least one match template specifying an ordered plurality of fields;

a processor programmed to form and electronically store, for each match template, a table comprising field value weights for a plurality of matches between a search criteria field value and a field value appearing in a record in the universal database, the field value weights in each table arranged according to entity representation, wherein the at least two search criteria field values match field values in two records in a same entity representation, and wherein at least one table comprises a sum of at least a portion of field value weights for the at least two search criteria field values that match field values in records in the same entity representation;

a processor programmed to merge and electronically store the tables according to entity representation, resulting in a stored merged table;

a processor programmed to sum and electronically store field value weights according to entity representation in the merged table, resulting in a plurality of stored summed weights, one summed weight for each entity representation;

a processor programmed to rank entity representations in the merged table according to the plurality of summed weights;

a processor programmed to determine a highest ranked entity representation;

a processor programmed to calculate a confidence level reflecting a likelihood that the highest ranked entity representation corresponds to the plurality of query field values; and

an output configured to output, if the confidence level exceeds a predetermined threshold, an identifier for the highest ranked entity representation.

12. The system of claim 11, wherein each field value weight comprises a logarithm of a probability that an arbitrary entity representation in the universal database comprises a corresponding field value in a field of a record in the arbitrary entity representation.

13. The system of claim 11, wherein at least one table comprising field value weights for a plurality of matches between a search criteria field value and a field value appearing in a record in the universal database comprises a field value weight for a non-identical match between a search criteria field value and a field value appearing in a record in the universal database, and wherein each summed weight comprises at most one field value weight for each of the at least two search criteria field values.

14. The system of claim 11, wherein at least one match template comprises a fixed field portion and an optional field portion.

15. The system of claim 14, wherein an arbitrary record matches an arbitrary search criteria if all fixed fields of the arbitrary record are populated with field values that match corresponding fixed field values of the arbitrary search criteria and if all optional fields of the arbitrary record are each populated with one of a null field value and a field value that matches a corresponding optional field value of the arbitrary search criteria.

16. The system of claim 11, wherein the identifier comprises a social security number.

17. A system for identifying, using an internally inconsistent search criteria, an entity representation in an electronic universal database that corresponds to an entity representation in an electronic foreign database, each database comprising a plurality of entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight, the system comprising:

an electronic universal database comprising a plurality of electronically stored entity representations, each entity representation comprising a plurality of linked records, each record comprising a plurality of fields, each field capable of containing a field value, each field value associated with a field value weight;

an electronic memory storing a plurality of search criteria field values, each search criteria field value associated with a field, wherein at least two search criteria field values are associated with a same field, wherein the at least two search criteria field values are not identical;

a processor programmed to form and electronically store a table comprising field value weights for a plurality of matches between a search criteria field value and a field value appearing in a record in the universal database, the field value weights in each table arranged according to entity representation, wherein the at least two search criteria field values match field values in two records in a same entity representation, and wherein at least one table comprises a sum of at least a portion of field value weights for the at least two search criteria field values that match field values in records in the same entity representation;

a processor programmed to merge and electronically store the tables according to entity representation, resulting in a stored merged table;

a processor programmed to sum and electronically store field value weights according to entity representation in the merged table, resulting in a plurality of stored summed weights, one summed weight for each entity representation;

a processor programmed to rank entity representations in the merged table according to the plurality of summed weights;

a processor programmed to determine a highest ranked entity representation;

a processor programmed to calculate a confidence level reflecting a likelihood that the highest ranked entity representation corresponds to the plurality of query field values; and

an output configured to output, if the confidence level exceeds a predetermined threshold, an identifier for the highest ranked entity representation.

18. The system of claim 17, wherein each field value weight comprises a logarithm of a probability that an arbitrary entity representation in the universal database comprises a corresponding field value in a field of a record in the arbitrary entity representation.

19. The system of claim 17, wherein at least one table comprising field value weights for a plurality of matches between a search criteria field value and a field value appearing in a record in the universal database comprises a field value weight for a non-identical match between a search criteria field value and a field value appearing in a record in the universal database, and wherein each summed weight comprises at most one field value weight for each of the at least two search criteria field values.

20. The system of claim 17, wherein the identifier comprises a social security number.