Natural language processing review and override based on confidence analysis

• US 10,592,502 B2
• Filed: 10/25/2016
• Issued: 03/17/2020
• Est. Priority Date: 10/25/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Erpenbach, Eric L., Lavery, Andrew J., Stevens, Richard J., Suarez Saiz, Fernando J.
• Primary Examiner(s): Raab, Christopher J
• Application Number:
  US15/333,793
• Publication Number:
  US 20180113867A1
• Time in Patent Office: 1,239 Days
• Field of Search: None
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 40/10 Text processing natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40

Abstract

Natural language processing includes processing unstructured data in response to a request for a conclusion to a line of inquiry to produce structured information that conforms to a computer readable information schema. Values obtained from the unstructured data are assigned to attributes of the structured information and determining a candidate conclusion to the line of inquiry based on the value-assigned attributes. A confidence score is computed that is indicative of a sufficiency of the candidate conclusion as the conclusion to the line of inquiry. One or more of the attributes are displayed to a user for modification responsive to the confidence score of the corresponding conclusion failing to meet a confidence criterion. The displayed attributes are ranked by relevance of the attribute to the corresponding conclusion. The conclusion to the line of inquiry is determined based on the modified values of the user-selected attributes.

Claims

1. A method of natural language processing comprising:

processing unstructured data in response to a request for a conclusion to a line of inquiry to produce structured information that conforms to a computer readable information schema;

assigning values obtained from the unstructured data to attributes of the structured information and determining a candidate conclusion to the line of inquiry based on the value-assigned attributes;

computing a confidence score indicative of a sufficiency of the candidate conclusion as the conclusion to the line of inquiry;

displaying one or more of the attributes to a user for modification responsive to the confidence score of the corresponding conclusion failing to meet a confidence criterion, the displayed attributes being ranked by relevance of the attribute to the corresponding conclusion, wherein displaying one or more of the attributes comprises;

collecting the attributes for presentation to the user based on the confidence of the conclusion on which the conclusion is based;

excluding the attributes that are not relevant to the conclusion from the attributes collected for presentation;

generating a user interface component to accept modifications to values assigned to a user-selected one or more of the collected attributes;

populating the user interface component with representations of the attributes collected for presentation in accordance with the respective rankings thereof;

generating an override interface component responsive to user selection of one of the representations of the attributes; and

accepting the modifications to the values assigned to the collected attributes through the override interface component; and

determining the conclusion to the line of inquiry based on the modified values of the attributes.

2. The method of claim 1, wherein generating the override interface component comprises:

presenting alternative values for the selected attribute in the override interface component; and

replacing the value of the attribute with one of the alternative values selected by the user through an override control.

3. The method of claim 2, further comprising:

determining the alternative values from the confidence scores of conclusions based on different interpretations of ambiguous attributes.

4. The method of claim 2, further comprising:

determining the alternative values from the confidence scores of conclusions based on recall or precision scores of suspect attributes.

5. The method of claim 2, further comprising:

determining the alternative values from the confidence scores of source documents containing the unstructured data.

6. A system for natural language processing comprising:

a processor configured to;

process unstructured data in response to a request for a conclusion to a line of inquiry to produce structured information that conforms to a computer readable information schema;

assign values obtained from the unstructured data to attributes of the structured information and determining a candidate conclusion to the line of inquiry based on the value-assigned attributes;

compute a confidence score indicative of a sufficiency of the candidate conclusion as the conclusion to the line of inquiry;

display one or more of the attributes to a user for modification responsive to the confidence score of the corresponding conclusion failing to meet a confidence criterion, the displayed attributes being ranked by relevance of the attribute to the corresponding conclusion wherein displaying one or more of the attributes comprises;

collecting the attributes for presentation to the user based on the confidence of the conclusion on which the conclusion is based;

excluding the attributes that are not relevant to the conclusion from the attributes collected for presentation;

generating a user interface component to accept modifications to values assigned to a user-selected one or more of the collected attributes;

populating the user interface component with representations of the attributes collected for presentation in accordance with the respective rankings thereof;

generating an override interface component responsive to user selection of one of the representations of the attributes; and

accepting the modifications to the values assigned to the collected attributes through the override interface component; and

determine the conclusion to the line of inquiry based on the modified values of the attributes.

7. The system of claim 6, wherein the processor is further configured to:

present alternative values for the selected attribute in the override interface component; and

replace the value of the attribute with one of the alternative values selected by the user through an override control.

8. The system of claim 7, wherein the processor is further configured to:

determine the alternative values from the confidence scores of conclusions based on different interpretations of ambiguous attributes.

9. The system of claim 7, wherein the processor is further configured to:

determine the alternative values from the confidence scores of conclusions based on recall or precision scores of suspect attributes.

10. The system of claim 7, wherein the processor is further configured to:

determine the alternative values from the confidence scores of source documents containing the unstructured data.

11. A computer program product for natural language processing, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:

process unstructured data in response to a request for a conclusion to a line of inquiry to produce structured information that conforms to a computer readable information schema;

assign values obtained from the unstructured data to attributes of the structured information and determining a candidate conclusion to the line of inquiry based on the value-assigned attributes;

compute a confidence score indicative of a sufficiency of the candidate conclusion as the conclusion to the line of inquiry;

display one or more of the attributes to a user for modification responsive to the confidence score of the corresponding conclusion failing to meet a confidence criterion, the displayed attributes being ranked by relevance of the attribute to the corresponding conclusion wherein displaying one or more of the attributes comprises;

collecting the attributes for presentation to the user based on the confidence of the conclusion on which the conclusion is based;

excluding the attributes that are not relevant to the conclusion from the attributes collected for presentation;

generating a user interface component to accept modifications to values assigned to a user-selected one or more of the collected attributes;

populating the user interface component with representations of the attributes collected for presentation in accordance with the respective rankings thereof;

generating an override interface component responsive to user selection of one of the representations of the attributes; and

accepting the modifications to the values assigned to the collected attributes through the override interface component; and

determine the conclusion to the line of inquiry based on the modified values of the attributes.

12. The computer program product of claim 11, wherein the program instructions include further program instructions executable by the processor to cause the processor to:

present alternative values for the selected attribute in the override interface component; and

replace the value of the attribute with one of the alternative values selected by the user through an override control.

13. The computer program product of claim 12, wherein the program instructions include further program instructions executable by the processor to cause the processor to:

determine the alternative values from the confidence scores of conclusions based on different interpretations of ambiguous attributes.

14. The computer program product of claim 12, wherein the program instructions include further program instructions executable by the processor to cause the processor to:

determine the alternative values from the confidence scores of conclusions based on recall or precision scores of suspect attributes.

15. The computer program product of claim 12, wherein the program instructions include further program instructions executable by the processor to cause the processor to:

determine the alternative values from the confidence scores of source documents containing the unstructured data.