SYSTEM AND METHOD FOR ASSOCIATING RELATED RECORDS TO COMMON ENTITIES ACROSS MULTIPLE LISTS

• US 20170046400A1
• Filed: 10/31/2016
• Published: 02/16/2017
• Est. Priority Date: 12/15/2014
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: ERENRICH, Daniel, TESSIER-LAVIGNE, Christian
• Granted Patent: US 10,242,072 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/24558 Binary matching operations
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/2462 Approximate or statistical queries
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases
• CPC Class Codes: G06F 16/285 Clustering or classification

Abstract

Computer implemented systems and methods are disclosed for associating records across lists, wherein the lists include a plurality of records and the plurality of records is associated with a respective entity. In accordance with some embodiments, the systems and methods further comprise grouping one or more records from a first list into a first group based on fields of the records in the first list, grouping one or more records from a second list into a second group based on fields of the records in the second list, pairing a record from the first group with a record from the second group, assessing each pair of records based on an evaluation of the respective pair according to fields of the pair, and associating records from the first group and records of the second group with an entity based on the assessment.

Claims

1. A system for associating records across a first list and a second list to a common entity, the system comprising:

one or more memory devices configured to store;

a set of instructions; and

a first list and a second list, wherein the first list and the second list both include a plurality of records, and wherein the plurality of records includes records that are associated with a respective entity and includes one or more fields;

one or more processors configured to execute the set of instructions that cause the one or more processors to;

obtain a sample of a plurality of pairs of records, wherein each pair of the plurality of pairs of records includes a record from the first list and a record from the second list;

label a subset of pairs of the sample of the plurality of pairs;

generate a statistical model based on the labeled pairs;

evaluate pairs of the plurality of pairs based on the statistical model;

output, based on the statistical model, scores representing the likelihood that the records in each pair are related to a similar entity;

based on the scores, resolve similar entities; and

associate one or more records of the first list and one or more of the records of the second list with the similar entities based on the resolution.

2. The system of claim 1, further comprising one or more processors configured to execute a set of instructions that cause the one or more processors to filter pairs having a score above a threshold score.

3. The system of claim 2, wherein the threshold score is determined based on the evaluation of the pairs of the plurality of pairs.

4. The system of claim 1, further comprising one or more processors configured to update the statistical model based on the output of the evaluated pairs.

5. The system of claim 1, wherein the sample is chosen algorithmically.

6. The system of claim 1, wherein the sample is chosen randomly.

7. The system of claim 1, wherein the sample is chosen by a user.

8. The system of claim 1, wherein the label for a record of the plurality of records is a matching label or a non-matching label.

9. The system of claim 8, wherein a determination of the label is based on highest combined evaluations or lowest combined evaluations of a specified field of the record.

10. A method for associating records across a first list and a second list to a common entity, the method comprising:

obtaining a first list and a second list, wherein the first list and the second list both include a plurality of records, and wherein the plurality of records includes records that are associated with a respective entity and includes one or more fields;

obtaining a sample of a plurality of pairs of records wherein each pair of the plurality of pairs of records includes a record from the first list and a record from the second list;

labeling a subset of pairs of the sample of the plurality of pairs;

generating a statistical model based on the labeled pairs;

evaluating pairs of the plurality of pairs based on the statistical model;

outputting, based on the statistical model, scores representing the likelihood that the records in each pair are related to a similar entity;

based on the scores, resolving similar entities; and

associating one or more records of the first list and one or more of the records of the second list with the similar entities based on the resolution.

11. The method of claim 10, further comprising filtering pairs having a score above a threshold score.

12. The method of claim 11, wherein the threshold score is determined based on the evaluation of the pairs of the plurality of pairs.

13. The method of claim 10, further comprising updating the statistical model based on the output of the evaluated pairs.

14. The method of claim 10, wherein the label for a record of the plurality of records is a matching label or a non-matching label.

15. The method of claim 14, wherein a determination of the label is based on highest combined evaluations or lowest combined evaluations of a specified field of the record.

16. A non-transitory computer-readable medium storing a set of instructions that are executable by one or more processors to cause the one or more processors to perform a method to associate related records across a first list and a second list to a common entity, the method comprising:

obtaining a first list and a second list, wherein the first list and the second list both include a plurality of records, and wherein the plurality of records includes records that are associated with a respective entity and includes one or more fields;

obtaining a sample of a plurality of pairs of records wherein each pair of the plurality of pairs of records includes a record from the first list and a record from the second list;

labeling a subset of pairs of the sample of the plurality of pairs;

generating a statistical model based on the labeled pairs;

evaluating pairs of the plurality of pairs based on the statistical model;

outputting, based on the statistical model, scores representing the likelihood that the records in each pair are related to a similar entity;

based on the scores, resolving similar entities; and

associating one or more records of the first list and one or more of the records of the second list with the similar entities based on the resolution.

17. The non-transitory computer readable medium of claim 16 further comprising a set of instructions that are executable by one or more processors to cause the one or more processors to perform filtering pairs having a score above a threshold score, wherein the threshold score is determined based on the evaluation of the pairs of the plurality of pairs.

19. The non-transitory computer readable medium of claim 16, wherein the label for a record of the plurality of records is a matching label or a non-matching label.

20. The non-transitory computer readable medium of claim 16, wherein a determination of the label is based on highest combined evaluations or lowest combined evaluations of a specified field of the record.

18. The non-transitory computer readable medium of claim 18, further comprising updating the statistical model based on the output of the evaluated pairs.