System and method for performing configurable matching of similar data in a data repository

• US 7,542,973 B2
• Filed: 05/01/2006
• Issued: 06/02/2009
• Est. Priority Date: 05/01/2006
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP AG (SAP SE)
• Inventors: Segal, Anat, Cohen, Ronen
• Primary Examiner(s): Wong; Don
• Assistant Examiner(s): DANG, THANH HA T
• Application Number:
  US11/416,466
• Publication Number:
  US 20070276844A1
• Time in Patent Office: 1,128 Days
• Field of Search: 707/2, 707/3, 707/5, 707/6, 707/7, 707/100, 707/102, 707/104.1, 707/4
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24558 Binary matching operations
• CPC Class Codes: G06F 16/2458 Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99937 Sorting
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

Adaptive matching of similar data in a data repository to determine if two or more data items are related in accordance with configurable criteria. Matches are adapted by learning and presenting appropriate match criteria based on previous user input. The system can merge the data items into one master data item, group similar items and perform further processing based on the result. The configurable match criteria presented to a user are adapted by the system based on previous interactions of the system with users. Matching is performed by selecting data items to match, removing frequently used strings, normalizing data, tokenizing multi-word data items, assigning weights to each token, calculating a score using the assigned weights, generating groups of similar records, assigning thresholds for match levels. Adapting choices of match criteria for a user based on past interaction allows for rapid match creation and match maintenance that optimizes data integrity across an enterprise.

Claims

1. A computer program product for adaptive matching of records in a data repository comprising:

a computer usable memory medium having computer readable program code embodied therein wherein said computer readable program code comprises a matching executable unit configured to;

present at least one field common to a first record and a second record wherein said at least one field is used to perform a match between said first record and said second record and wherein said at least one field is presented to a user;

obtain a first selected field from said first record and a second selected field from said second record wherein said first selected field and said second selected field is obtained from said user;

obtain a first data entry in said first selected field for said first record, said first data entry comprising a first string;

tokenize said first string to retrieve a first tokenized data entry string;

obtain a second data entry in said second selected field for said second record, said second data entry comprising a second string;

tokenize said second string to retrieve a second tokenized data entry string;

exclude at least one character from said first tokenized data entry string for utilization in said match that involves said first selected field and said second selected field;

exclude at least one different character with respect to said at least one character from said second tokenized data entry string for utilization in said match that involves said first selected field and said second selected field;

remove frequently used strings from said first tokenized data entry string and from said second tokenized data entry string;

normalize data from said first selected field and from said second selected field to cleanse strings;

accept a first list of tokens desired for said match to occur utilizing said first selected field;

accept a second list of tokens desired for said match to occur utilizing said second selected field;

assign weights to each token in said first list of tokens and each token in said second list of tokens;

calculate a score for said match through summation of said weights for each token occurring in said first tokenized data entry string and said first record and for each token that occurs in said second tokenized data entry string and said second record;

generate a group of similar records when said score is above a threshold;

display said group of similar records to said user;

learn at least one token that is relevant;

learn at least one weight that results in a match; and

learn at least one match criteria appropriate for said match for use in future matching based on user input regarding said score.

2. The computer program product of claim 1 wherein said computer readable program code is further configured to:

said exclude further configured to present a list of excluded characters most often excluded in matches involving said first field.

3. The computer program product of claim 1 wherein said computer readable program code is further configured to:

said remove said frequently used strings further configured to present a list of frequently used strings most often removed in matches involving said first field.

4. The computer program product of claim 1 wherein said computer readable program code is further configured to:

said normalize said data from said first field further configured to present a list of tokens most often normalized in matches involving said first field.

5. The computer program product of claim 1 wherein said computer readable program code is further configured to:

said assign said weights to each token in said first list of tokens further configured to present a list of tokens and corresponding weight values most often chosen in matches involving said first field.

6. The computer program product of claim 1 wherein said computer readable program code is further configured to:

accept input that signifies if said first list of tokens is required to match in sequential order.

7. The computer program product of claim 1 wherein said computer readable program code is further configured to:

accept input that signifies if said first list of tokens is required to match in non-sequential order.

8. The computer program product of claim 1 wherein said computer readable program code is further configured to:

accept a weight value associated with a field.

9. The computer program product of claim 1 wherein said computer readable program code is further configured to:

present a list of tokens previously used by a user to define a match involving said first selected field.

10. The computer program product of claim 1 wherein said computer readable program code is further configured to:

alter a list of tokens presented to a user to define a match involving said first selected field when said user selects said second selected field.

11. A computer program product for adaptive matching of records in a data repository comprising:

a computer usable memory medium having computer readable program code embodied therein wherein said computer readable program code comprises a matching executable unit configured to;

present at least one field common to a first record and a second record wherein said at least one field is used to perform a match between said first record and said second record and wherein said at least one field is presented to a user;

obtain a first selected field from said first record and a second selected field from said second record wherein said first selected field and said second selected field is obtained from said user;

obtain a first data entry in said first selected field for said first record, said first data entry comprising a first string;

tokenize said first string to retrieve a first tokenized data entry string;

obtain a second data entry in said second selected field for said second record, said second data entry comprising a second string;

tokenize said second string to retrieve a second tokenized data entry string;

exclude at least one character from said first tokenized data entry string for utilization in said match that involves said first selected field and said second selected field;

exclude at least one different character with respect to said at least one character from said second tokenized data entry string for utilization in said match that involves said first selected field and said second selected field;

remove frequently used strings from said first tokenized data entry string and from said second tokenized data entry string;

normalize data from said first selected field and from said second selected field to cleanse strings;

alter a list of tokens presented to a user to define said match involving said first selected field when said user selects said second selected field;

accept a first list of tokens desired for said match to occur utilizing said first selected field;

accept a second list of tokens desired for said match to occur utilizing said second selected field;

assign weights to each token in said first list of tokens and each token in said second list of tokens;

calculate a score for said match through summation of said weights for each token that occurs in said first tokenized data entry string and said first record and for each token that occurs in said second tokenized data entry string and said second record;

generate a group of similar records when said score is above a threshold;

display said group of similar records to said user;

learn at least one token that is relevant;

learn at least one weight that results in a match; and

learn at least one match criteria appropriate for said match for use in future matching based on user input regarding said score.

12. The computer program product of claim 11 wherein said computer readable program code is further configured to:

said exclude further configured to present a list of excluded characters most often excluded in matches involving said first field.

13. The computer program product of claim 11 wherein said computer readable program code is further configured to:

said remove said frequently used strings further configured to present a list of frequently used strings most often removed in matches involving said first field.

14. The computer program product of claim 11 wherein said computer readable program code is further configured to:

said normalize said data from said first field further configured to present a list of tokens most often normalized in matches involving said first field.

15. The computer program product of claim 11 wherein said computer readable program code is further configured to:

said assign said weights to each token in said first list of tokens further configured to present a list of tokens and corresponding weight values most often chosen in matches involving said first field.

16. The computer program product of claim 11 wherein said computer readable program code is further configured to:

accept input that signifies if said first list of tokens is required to match in sequential order.

17. The computer program product of claim 11 wherein said computer readable program code is further configured to:

accept input that signifies if said first list of tokens is required to match in non-sequential order.

18. The computer program product of claim 11 wherein said computer readable program code is further configured to:

accept a weight value associated with a field.

19. The computer program product of claim 11 further comprising:

present a list of tokens previously used by a user to define a match involving said first selected field.

20. A computer system for adaptive matching of records in a data repository comprising:

a computer system comprising a display device and a data repository, said computer system configured to;

present on said display device at least one field common to a first record and a second record wherein said at least one field is used to perform a match between said first record and said second record and wherein said at least one field is presented to a user on said display device;

obtain a first selected field from said first record and a second selected field from said second record wherein said first selected field and said second selected field is obtained from said user;

obtain a first data entry in said first selected field for said first record, said first data entry comprising a first string;

tokenize said first string to retrieve a first tokenized data entry string;

obtain a second data entry in said second selected field for said second record, said second data entry comprising a second string;

tokenize said second string to retrieve a second tokenized data entry string;

exclude at least one character from said first tokenized data entry string for utilization in said match that involves said first selected field and said second selected field;

exclude at least one different character with respect to said at least one character from said second tokenized data entry string for utilization in said match that involves said first selected field and said second selected field;

remove frequently used strings from said first tokenized data entry string and from said second tokenized data entry string;

normalize data from said first selected field and from said second selected field to cleanse strings;

accept a first list of tokens desired for said match to occur utilizing said first selected field;

accept a second list of tokens desired for said match to occur utilizing said second selected field;

assign weights to each token in said first list of tokens and each token in said second list of tokens;

calculate a score for said match through summation of said weights for each token that occurs in said first tokenized data entry string and said second record and for each token that occurs in said second tokenized data entry and said second record;

generate a group of similar records when said score is above a threshold;

display said group of similar records to said user;

learn at least one token that is relevant;

learn at least one weight that results in a match; and

learn at least one match criteria appropriate for said match for use in future matching based on user input regarding said score.