Confidence indicators for automated suggestions

• US 7,512,580 B2
• Filed: 08/04/2005
• Issued: 03/31/2009
• Est. Priority Date: 08/04/2005
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP AG (SAP SE)
• Inventors: Ronnewinkel, Christopher
• Primary Examiner(s): Vincent; David R
• Assistant Examiner(s): Tran; Mai T
• Application Number:
  US11/196,968
• Publication Number:
  US 20070094217A1
• Time in Patent Office: 1,335 Days
• Field of Search: 706/52, 706/20, 707/6
• US Class Current: 706/52
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

Systems, computer program products, and associated methods provide for selecting between manual and automatic operating modes based upon previous experience as quantified by confidence indicators. Confidence indicators are values that may each represent a likelihood that an associated candidate class or candidate object matches an incoming message, for example. Confidence indicator values may be updated according to previous manual selections or rejections of the associated class or object, for example. In an exemplary response management system, the selections may be for modes of operation that relate to classification of an incoming message, and/or suggestion of objects, for example.

Claims

1. A method, comprising:

determining a confidence indicator (CI) for at least one candidate class for an incoming message received in a response management system, each CI being a function of a comparison between previous selections of the candidate class and previous suggestions of the candidate class for previous messages having one or more similar features, wherein each determined CI represents a likelihood that the corresponding candidate class matches the incoming message, wherein the response management system comprises at least one computer and at least one knowledge base, wherein the at least one knowledge base comprises information related to previous selections of the candidate class and previous suggestions of the candidate class for previous messages having one or more similar features, and wherein determining the CI is performed at the at least one computer using the at least one knowledge base;

identifying at the at least one computer one or more candidate classes that match the incoming message based on class weights that are functions of each of the determined CIs;

calculating at the at least one computer a classification accuracy measure that is a function of the class weights;

selecting at the at least one computer between a manual and an automatic classification mode of operation by comparing the calculated classification accuracy measure to a predetermined threshold; and

generating at the at least one computer an electronic message for transmission by the response management system in response to the received incoming message, wherein the generated electronic message includes information related to at least one of the identified candidate classes.

2. The method of claim 1, further comprising automatically sending at least one of the identified candidate classes if the automatic classification mode of operation is selected.

3. The method of claim 1, further comprising providing information about candidate classes to a user if the manual classification mode of operation is selected.

4. The method of claim 3, further comprising:

in response to the user selecting a candidate class, updating the CI for the selected candidate class to represent an increased likelihood that the selected candidate class will match a future incoming message having one or more features that are similar to features of the present incoming message.

5. The method of claim 4, further comprising:

for each candidate class provided to the user and not selected by the user, updating the CI for that candidate class to represent a decreased likelihood that the candidate class will match a future incoming message having one or more features that are similar to features of the present incoming message.

6. The method of claim 1, wherein identifying one or more matching candidate classes further comprises identifying candidate classes associated with the highest CI from among the candidate classes.

7. The method of claim 1, wherein the class weights are further functions of class scores associated with text-mining.

8. The method of claim 1, wherein the candidate classes are arranged in a hierarchy.

9. The method of claim 8, wherein identifying at least one sufficiently accurate candidate class further comprises identifying a candidate class that is at the lowest level relative to the hierarchy levels of the other candidate classes.

10. A method, comprising:

determining a confidence indicator (CI) for at least one candidate object associated with a candidate class for an incoming message received in a response management system, each CI being a function of a comparison between previous selections of the object and previous suggestions of the object for previous messages having the same class as the corresponding candidate class, wherein each determined CI represents a likelihood that a corresponding object matches the incoming message, wherein the response management system comprises at least one computer and at least one knowledge base, wherein the at least one knowledge base comprises information related to previous selections of the object and previous suggestions of the object for previous messages having the same class as the corresponding candidate class, and wherein determining the CI is performed at the at least one computer using the at least one knowledge base;

identifying at the at least one computer one or more candidate objects that match the incoming message based on object weights that are functions of each of the determined CIs;

calculating at the at least one computer a suggestion accuracy measure that is a function of the object weights;

selecting at the at least one computer between a manual and an automatic suggestion mode of operation by comparing the calculated suggestion accuracy measure to a predetermined threshold; and

generating at the at least one computer an electronic message for transmission by the response management system in response to the received incoming message, wherein the generated electronic message includes information related to at least one of the identified candidate objects.

11. The method of claim 10, further comprising automatically sending a response based on one or more of the identified objects if the automatic suggestion mode of operation is selected.

12. The method of claim 10, further comprising providing information about candidate objects to a user if the manual suggestion mode of operation is selected.

13. The method of claim 12, further comprising:

in response to the user selecting one of the candidate objects, updating the CI for the selected object to represent an increased likelihood that the selected object will match a future incoming message having the same class as the present incoming message.

14. The method of claim 13, further comprising:

for each of the candidate objects provided to the user and not selected by the user, updating the CI to represent a decreased likelihood that the object will match a future incoming message having the same class as the present incoming message.

15. The method of claim 10, wherein identifying one or more candidate objects further comprises identifying a candidate object associated with the highest CI from among the candidate objects.

16. The method of claim 10, wherein the object weights are further functions of class confidence indicators (CCIs) and class scores associated with text-mining,

wherein each CCI is a function of a comparison between previous selections of the candidate class and previous suggestions of the candidate class for previous messages having one or more similar features, and each CCI represents a likelihood that the corresponding candidate class matches the incoming message.

17. The method of claim 10, wherein the candidate objects comprise information that may be contained in a response message.

18. The method of claim 10, wherein the candidate objects comprise an instruction to contact to an expert.

19. A computer program product tangibly embodied in a machine-readable storage medium and containing instructions that, when executed on a processor, cause the processor to perform operations, the operations comprising:

determining a class confidence indicator (CCI) for at least one candidate class for an incoming message received in a response management system, each CCI being a function of a comparison between previous selections of the candidate class and previous suggestions of the candidate class for previous messages having one or more similar features, wherein each determined CCI represents a likelihood that a corresponding candidate class matches the incoming message, wherein the response management system comprises at least one computer and at least one knowledge base, wherein the at least one knowledge base comprises information related to previous selections of the candidate class and previous suggestions of the candidate class for previous messages having one or more similar features, and wherein determining the CCI is performed at the at least one computer using the at least one knowledge base;

identifying at the at least one computer one or more candidate classes that match the incoming message based on class weights that are functions of each of the determined CCIs;

calculating at the at least one computer a classification accuracy measure that is a function of the class weights;

selecting at the at least one computer between a manual and an automatic classification mode of operation by comparing the calculated classification accuracy measure to a predetermined threshold; and

generating at the at least one computer an electronic message for transmission by the response management system in response to the received incoming message, wherein the generated electronic message includes information related to at least one of the identified candidate classes.

20. The computer program product of claim 19, the operations further comprising:

determining an object confidence indicator (OCI) for at least one candidate object associated with one of the candidate classes, each OCI being a function of a comparison between previous selections of the object and previous suggestions of the object for previous messages having the same class as the corresponding candidate class, wherein each determined OCI represents a likelihood that a corresponding object matches the incoming message;

identifying one or more candidate objects that match the incoming message based on object weights that are functions of each of the determined OCIs;

calculating a suggestion accuracy measure that is a function of the object weights; and

selecting between a manual and an automatic suggestion mode of operation by comparing the calculated suggestion accuracy measure to a predetermined threshold.