System and method for building and maintaining a database

• US 20020035555A1
• Filed: 08/03/2001
• Published: 03/21/2002
• Est. Priority Date: 08/04/2000
• Status: Active Grant

• Current Assignee: Fair Isaac Corporation
• Original Assignee: Infoglide Software Corp. (Fair Isaac Corporation)
• Inventors: Wheeler, David B., Wotring, Steven C.
• Granted Patent: US 6,795,819 B2
• Time in Patent Office: Days
• US Class Current: 707/1
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: Y10S 707/917 Text
• CPC Class Codes: Y10S 707/954 Relational
• CPC Class Codes: Y10S 707/956 Hierarchical
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

The invention relates generally to similarity scoring engines. More particularly, the invention is a computer-implemented similarity scoring system and method that allows for using similarity scoring to store and maintain a unique product and retailer inventory database in a database. It enables users of a database management system to be presented with data that maintains a high level of accuracy and reliability and minimizes the occurrences of redundant data. Users of the databases may then be presented with unique instances of data rather than product and inventory data that appears to be different but are actually duplicates of data already entered into the database.

Claims

1. A computer implemented method for combining data elements to build and maintain a unique database comprised of data entries, comprising:

a. using at least one candidate data element that is a candidate to be added to existing data elements in the unique database, performing a comparison between the candidate data element and the existing data elements in the unique database; and

b. computing a similarity score that represents a similarity between the at least one candidate data element and the existing data elements in the unique database.

2. The method of claim 1 further comprising determining if the candidate data element should be entered into the unique database based on the similarity score.

3. The method of claim 1 further comprising rejecting the candidate data element for entry into the unique database if the similarity score is greater than a similarity score threshold.

4. The method of claim 1 further comprising selecting the candidate data element as a candidate for entry into the unique database if the similarity score is equal to or less than a similarity score threshold.

5. The method of claim 4 wherein the candidate data element is entered into the unique database.

6.he. The method of claim 1 wherein the computing a score comprises:

a. using a schema that represents the organization or structure of the unique database;

b. developing a query containing at least one field, based on the schema;

c. examining candidate data elements, wherein the candidate data element comprises at least one data field, and inputting at least one data field of the candidate data element into the query;

d. searching the unique database for the contents of the query; and

e. computing the similarity score.

7. The method of claim 1 wherein the computing the similarity score comprises:

a. separating the unique database into at least one selected categories;

b. developing a schema for the at least one selected categories;

c. developing a query containing at least one field based on the schema for the at least one selected categories;

d. assigning the candidate data element to at least one of the selected categories;

e. inputting at least one data field of the candidate data element into the query;

f. searching the unique database for the contents of the query; and

g. computing the similarity score.

8. The method of claim 1 wherein the computing a similarity score comprises:

a. separating the unique database with data elements or objects into at least one selected categories;

b. developing a schema for the at least one selected categories;

c. identifying the candidate data element as at least one of the selected categories;

d. comparing the candidate data element to at least one of the selected categories of data elements of the unique database;

e. computing the similarity score; and

f. using the resulting similarity score to determine if the candidate data element should be added to the unique database.

9. The method of claim 1 wherein the computing a similarity score comprises:

a. separating the unique database into at least one selected category;

b. developing a schema for the selected categories;

c. assigning the candidate data element to at least one of the selected categories;

d. formulating a similarity score command for each candidate data element based on the selected categories to which the candidate data element belongs;

e. sending the similarity score command to a similarity score function; and

f. performing a search using the score command and the unique database whereby a similarity score result is returned from the search function that represents the similarity score between the at least one candidate data element and the existing data elements in the unique database.

10. The method of claim 9 wherein if the similarity score result indicates a similarity score exceeding the similarity score threshold level, the candidate data element is not added to the unique database.

11. The method of claim 9 wherein if the similarity score result indicates a similarity score less than or equal to a similarity score threshold level, the candidate data element is added to the unique database.

12. The method of claim 9 wherein if the search result indicates a match less than a threshold level, the candidate data entry is annotated as a unique element to be added to the unique database.

13. The method of claim 12 wherein the candidate data entry is added to the unique database.

14. The method of claim 12 further comprising selecting the similarity score result from the group consisting of similar and dissimilar.

15. The method of claim 14 further comprising if the similarity score result is similar, the candidate data element is not added to the database.

16. The method of claim 14 further comprising if the similarity score result is dissimilar, processing further comprises performing a validation to verify the candidate data element is unique and should be added to the unique database.

17. The method of claim 16 wherein the validation is selected form the group consisting of a manual validation and a computer automated validation.

18. The method of claim 16 further comprising if results of the validation indicate the candidate data element is unique, the candidate data element is added to the unique database according to its selected categories.

19. The method of claim 16 further comprising if results of the validation indicate the candidate data element is not unique, the candidate data element is not added to the unique database.

20. The method of claim 9 wherein the similarity score results, the candidate data element and the unique database are available for display to a user through a graphical user interface.

21. The method of claim 9 wherein the formulating, sending and performing occur in near real-time.

22. The method of claim 9 wherein the formulating, sending and performing occur in a background mode.

23. The method of claim 1 wherein the candidate data elements are in a format selected from the group consisting of text format, relational database format, hierarchical database format, extensible markup language (XML) format and hypertext markup language (HTML) format.

24. The method of claim 9 wherein the data elements are formulated in XML.

25. The method of claim 9 further comprising creating at least one XML similarity score command to compare the candidate data elements to the unique database and executing the similarity score command.

26. The method of claim 25 wherein execution of the similarity score command comprises:

a. interpreting the score command by a similarity score function; and

b. executing the similarity score command by the similarity score function and returning the similarity score results in a markup language.

27. The method of claim 25 further comprising processing a similarity score result set by interpreting the similarity score results and applying a set of programmed logic functions, including filtering mechanisms to filter the result set.

28. The method of claim 27 wherein the filtering mechanisms comprise a set of threshold tolerances selected from the group consisting of a threshold match, a proximity match and a non-threshold match.

29. The method of claim 28 wherein the threshold match occurs when the similarity score results returned is greater than the sum of a predefined percentage and a proximity match score.

30. The method of claim 28 wherein the proximity mach score is equal to the union of the set in which the similarity score result is less than a similarity score tolerance percentage (plus an additional percentage) and the set in which the similarity score results is greater than the similarity score tolerance percentage minus the additional percentage.

31. The method of claim 28 wherein the non-threshold match occurs when the similarity score result is less than the difference between the similarity score tolerance and the proximity search tolerance.

32. The method of claim 28 further comprising performing a threshold similarity score comparison match action that is selected from the group consisting of discarding the candidate data element, indicating the candidate data element is not to be added to the database and performing a secondary validation of the similarity score search result.

33. The method of claim 28 wherein performing a secondary validation comprises repeating the search with tighter comparison match parameters and returning a second similarity score result.

34. A computer-readable media containing instructions for controlling a computer system to implement the method of claim 1.

35. A computer-readable media containing instructions for controlling a computer system to implement the method of claim 9.

37. The system of claim 36, wherein the candidate data elements are entered into the database if the similar scoring engine determines a similarity score result set that is less than or equal to a predetermined threshold value.

38. The system of claim 37, further comprising means for performing a secondary similar score validation check on the candidate data elements prior to entering the candidate data elements into the database.

39. The system of claim 36, wherein the candidate data elements are not entered into the database if the similar scoring engine determines a similarity score result set that is greater than a predetermined threshold value.

40. The system of claim 36, wherein the candidate data elements comprise manufacturers product listings and retailer product inventory.

36. A system for combining data elements to build and maintain a unique database, comprising:

a. candidate data elements;

b. unique data elements contained in a database;

c. a similar scoring engine for comparing the candidate data elements with the unique data elements; and

d. means for entering the candidate data elements into the database based on the comparison of the similar scoring engine.