Software tool for training and testing a knowledge base

• US 8,495,002 B2
• Filed: 04/29/2004
• Issued: 07/23/2013
• Est. Priority Date: 05/06/2003
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Nelken, Yoram, Hod, Ronen, Navot, Amir, Michelson, Sam, Jessee, Randy, Shacham, Tsachy, Lanin, Beth, Hajaj, Nissan, Margalit, Avi, Magdalen, Josemina, Cohen, Dani
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): COUGHLAN, PETER D
• Application Number:
  US10/835,694
• Publication Number:
  US 20040225653A1
• Time in Patent Office: 3,372 Days
• Field of Search: 706/1, 706/15, 706/45, 706/18, 706/17, 706/16, 706/12, 706/20, 706/13, 706/21, 706/25, 706/55, 706/2, 706/3, 706/47, 706/6, 706/10, 706/48, 700/1, 700/90, 395/22, 707/100, 707/5, 273/54, 435/40.5, 382/195, 342/450, 702/19, 702/20, 702/185, 702/130
• US Class Current: 706/62
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

A software tool for creating, training and testing a knowledge base of a computerized customer relationship management system is disclosed. The software tool includes corpus editing processes for displaying and editing text-based corpus items, and assigning selected categories to individual corpus items. Knowledge base construction processes construct a knowledge base by analyzing a first subset of the corpus items, and testing processes test the knowledge base on a second subset of the corpus items. Reporting processes generate reports containing indicia representative of the testing results, which may be utilized to edit the corpus items and retrain the knowledge base so as to improve performance.

Claims

1. A computer-implemented software tool for training and testing a knowledge base of a computerized customer relationship management system, comprising:

corpus editing processes, performed on one or more computers, for displaying and editing corpus items belonging to a corpus, and for assigning a suitable category from a set of predefined categories to individual corpus items;

knowledge base building processes, performed on the one or more computers, for building a knowledge base of a computerized customer relationship management system by performing natural language and semantic analysis of a first subset of the corpus items and thereby deriving semantic and statistical information from the corpus items that are associated with nodes in the knowledge base;

knowledge base testing processes, performed on the one or more computers, for testing the knowledge base of the computerized customer relationship management system on a second subset of the corpus items by extracting concepts from the corpus items of the second subset, performing statistical pattern matching to generate a set of match scores for each corpus item of the second subset, wherein each match score in the match score set represents a confidence level for classifying each corpus item of the second subset into at least one of the predefined categories using the semantic and statistical information associated with the nodes in the knowledge base of the computerized customer relationship management system; and

reporting processes, performed on the one or more computers, for generating reports based on results produced by the knowledge base testing processes and causing the reports to be displayed to a user of the computerized customer relationship management system to gauge performance of the knowledge base, so that appropriate adjustments are made to improve the performance of the knowledge base.

2. The software tool of claim 1, wherein the knowledge base testing processes calculate a set of match scores for each corpus item in the second subset, each match score from the calculated set of match scores being associated with a corresponding category and being representative of a confidence that the corpus item belongs to the corresponding category.

3. The software tool of claim 1, wherein the reporting processes generate a report relating to a single selected category.

4. The software tool of claim 1, wherein the reporting processes generate a cumulative report relating to a plurality of categories.

5. The software tool of claim 1, wherein the reporting processes calculate and display, for a selected category, a match score based on user input consisting of one of a precision value, a recall value, false positive rate, false negative rate, automation ratio or a cost ratio.

6. The software tool of claim 1, wherein the reporting processes calculate and display, for a selected category, a precision value and a recall value based on a match score input by the user.

7. The software tool of claim 1, wherein the reporting processes calculate precision as a function of recall and cause a graph to be displayed depicting the relationship between precision and recall.

8. The software tool of claim 1, wherein the reporting processes generate and display a graph depicting cumulative success over time, the graph showing, for a plurality of groups of corpus items each having a common time parameter, the fraction of corpus items in the group that were appropriately classified.

9. The software tool of claim 1, wherein the reporting processes generate and display a report showing, for each of a plurality of pairs of categories, a percentage of corpus items initially assigned to a first category of the pair of categories that were erroneously classified into a second category of the pair of categories.

10. The software tool of claim 1, wherein the reporting processes generate and display a scoring report showing, for a selected category, the match scores for each corpus item in the second subset, the match scores being representative of the relevance of the selected category to the corpus item.

11. The software tool of claim 1, wherein the first and second subsets of corpus items are selected in accordance with user input.

12. The software tool of claim 1, wherein the knowledge base building processes and the knowledge base testing processes use a modeling engine to analyze and classify corpus items.

13. The software tool of claim 12, wherein the modeling engine includes a natural language processing engine and a semantic modeling engine.

14. The software tool of claim 1, wherein the reporting processes are configured to allow a user to select a report to be generated from a plurality of available reports.

15. The software tool of claim 1, wherein the corpus items comprise customer communications.

31. The software tool of claim 1, wherein the corpus items include structured and unstructured information.

16. A computer-implemented method for training and testing a knowledge base of a computerized customer relationship management system, comprising:

collecting, on one or more computers, corpus items into a corpus;

assigning, on the one or more computers, a category from a set of predefined categories to individual corpus items;

building, on the one or more computers, a knowledge base of a computerized customer relationship management system by performing natural language and semantic analysis of a first subset of corpus items and thereby deriving semantic and statistical information from the corpus items that are associated with nodes in the knowledge base;

testing, on the one or more computers, the knowledge base of the computerized customer relationship management system on a second subset of corpus items by extracting concepts from the corpus items of the second subset, performing statistical pattern matching to generate a set of match scores for each corpus item of the second subset, wherein each match score in the match score set represents a confidence level for classifying each corpus item of the second subset into at least one of the predefined categories using the semantic and statistical information associated with the nodes in the knowledge base of the computerized customer relationship management system; and

generating and displaying, on the one or more computers, a report based on results produced by the testing step to a user of the computerized customer relationship management system to gauge performance of the knowledge base, so that appropriate adjustments are made to improve the performance of the knowledge base.

17. The method of claim 16, wherein the step of testing the knowledge base includes calculating a set of match scores for each corpus item in the second subset, each match score from the calculated set of match scores being associated with a corresponding category and being representative of a confidence that the corpus item belongs to the corresponding category.

18. The method of claim 16, wherein the step of generating and displaying a report includes generating a report relating to a single selected category.

19. The method of claim 16, wherein the step of generating and displaying a report includes generating a cumulative report relating to a plurality of categories.

20. The method of claim 16, wherein the step of generating and displaying a report includes:

receiving user input specifying one of a precision value, a recall value, false positive rate, false negative rate, automation ratio or a cost ratio; and

calculating and displaying, for a selected category, a match score based on the user input.

21. The method of claim 16, wherein the step of generating and displaying a report includes:

receiving user input specifying a match score; and

calculating and displaying, for a selected category, a precision value and a recall value based on the user input.

22. The method of claim 16, wherein the step of generating and displaying a report includes calculating precision as a function of recall and causing a graph to be displayed depicting the relationship between precision and recall.

23. The method of claim 16, wherein the step of generating and displaying a report includes generating and displaying a graph depicting cumulative success over time, the graph showing, for a plurality of groups of corpus items each having a common time parameter, the fraction of corpus items in the group that were appropriately classified.

24. The method of claim 16, wherein the step of generating and displaying a report includes generating and displaying a report showing, for each of a plurality of pairs of categories, a percentage of corpus items initially assigned to a first category of the pair of categories that were erroneously classified into a second category of the pair of categories.

25. The method of claim 16, wherein the step of generating and displaying a report includes generating and displaying a scoring report showing, for a selected category, the match scores for each corpus item in the second subset, the match scores being representative of the relevance of the selected category to the corpus item.

26. The method of claim 16, wherein the first and second subsets of corpus items are selected in accordance with user input.

27. The method of claim 16, wherein the steps of use building and testing the knowledge base include using a modeling engine to analyze and classify corpus items.

28. The method of claim 16, wherein the step of generating and displaying a report includes selecting a report from a plurality of available reports in response to user input.

29. The method of claim 16, wherein the corpus items comprise customer communications.

30. The method of claim 16, wherein the corpus items include structured and unstructured information.

32. A computer-readable non-transitory medium embodying instructions executable by a computer for performing the steps of:

collecting corpus items into a corpus;

assigning a category from a set of predefined categories to individual corpus items;

building a knowledge base of a computerized customer relationship management system by performing natural language and semantic analysis of a first subset of corpus items and thereby deriving semantic and statistical information from the corpus items that are associated with nodes in the knowledge base;

testing the knowledge base of a computerized customer relationship management system on a second subset of corpus items by extracting concepts from the corpus items of the second subset, performing statistical pattern matching to generate a set of match scores for each corpus item of the second subset, wherein each match score in the match score set represents a confidence level for classifying each corpus item of the second subset into at least one of the predefined categories using the semantic and statistical information associated with the nodes in the knowledge base of a computerized customer relationship management system; and

generating and displaying, on a computer, a report based on results produced by the testing step to a user of the computerized customer relationship management system to gauge performance of the knowledge base, so that appropriate adjustments are made to improve the performance of the knowledge base.