Measuring mutual understanding in human-computer conversation

• US 10,303,772 B2
• Filed: 07/29/2016
• Issued: 05/28/2019
• Est. Priority Date: 07/29/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hosn, Rafah A., Moore, Robert J.
• Primary Examiner(s): Singh, Satwant K
• Application Number:
  US15/223,508
• Publication Number:
  US 20180032504A1
• Time in Patent Office: 1,033 Days
• Field of Search: None
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation

Abstract

Technical solutions are described method for analyzing a natural language conversation-generating machine. An example computer implemented method includes determining, from a plurality of adjacency pairs in a conversation, a number of base pairs. The computer implemented method also includes determining, from the plurality of adjacency pairs, a number of expansion pairs. The computer implemented method also includes computing a efficiency score for the conversation by computing a ratio of the number of base pairs and the number of expansion pairs.

Claims

1. A computer implemented method for analyzing a natural language conversation-generating machine, the method comprising:

determining, from a plurality of adjacency pairs in a conversation, a number of base pairs;

determining, from the plurality of adjacency pairs, a number of expansion pairs;

computing a efficiency score for the conversation by computing a ratio of the number of base pairs and the number of expansion pairs;

identifying, from the conversation, a specific utterance of an adjacency pair wherein the adjacency pair is associated with an expansion pair, the utterance being a message that is played back by the machine in response to a statement by a user;

incrementing a counter corresponding to the utterance; and

in response to the counter crossing a predetermined threshold, sending a message to notify a developer to redesign the specific utterance.

2. The computer implemented method of claim 1, further comprising:

determining a type of outcome of the conversation; and

determining an outcome-efficiency type of the conversation based on the efficiency score and the type of the outcome.

3. The computer implemented method of claim 2, wherein the outcome-efficiency type is determined from a predetermined group of outcome-efficiency types consisting of compact-failure, compact-success, expanded-failure, and expanded-success.

4. The computer implemented method of claim 2, wherein the type of the outcome is determined based on an outcome indicator of a base pair of the conversation, wherein the outcome indicator is identified based on a second pair-part of the base pair being one of a predetermined set of response types.

5. The computer implemented method of claim 1, wherein the message is initiated further in response to the adjacency pair being expanded using a specific response type.

6. The computer implemented method of claim 5, wherein the specific response type is one from a group of response types consisting of a definition request, a repeat request, a partial repeat request, and an elaboration request.

7. A system for analyzing a natural language conversation-generating machine, the system comprising:

a memory; and

a processor coupled with the memory, the processor being configured to;

determine, from a plurality of adjacency pairs in a conversation, a number of base pairs;

determine, from the plurality of adjacency pairs, a number of expansion pairs;

compute a efficiency score for the conversation by computing a ratio of the number of base pairs and the number of expansion pairs;

identify, from the conversation, a specific utterance of an adjacency pair wherein the adjacency pair is associated with an expansion pair, the utterance being a message that is played back by the machine in response to a statement by a user;

increment a counter corresponding to the utterance; and

in response to the counter crossing a predetermined threshold, send a message to notify a developer to redesign the specific utterance.

8. The system of claim 7, wherein the processor is further configured to:

determine a type of outcome of the conversation; and

determine an outcome-efficiency type of the conversation based on the efficiency score and the type of the outcome.

9. The system of claim 8, wherein the outcome-efficiency type is determined from a predetermined group of outcome-efficiency types consisting of compact-failure, compact-success, expanded-failure, and expanded-success.

10. The system of claim 8, wherein the type of the outcome is determined based on an outcome indicator of a base pair of the conversation, wherein the outcome indicator is identified based on a second pair-part of the base pair being one of a predetermined set of response types.

11. The system of claim 7, wherein the message is initiated further in response to the adjacency pair being expanded using a specific response type.

12. The system of claim 11, wherein the specific response type is one from a group of response types consisting of a definition request, a repeat request, a partial repeat request, and an elaboration request.

13. A computer program product for analyzing a natural language conversation-generating machine, the computer program product comprising a computer readable storage medium, the computer readable storage medium comprising computer executable instructions, wherein the computer readable storage medium comprises instructions to:

determine, from a plurality of adjacency pairs in a conversation, a number of base pairs;

determine, from the plurality of adjacency pairs, a number of expansion pairs;

compute a efficiency score for the conversation by computing a ratio of the number of base pairs and the number of expansion pairs;

identify, from the conversation, a specific utterance of an adjacency pair wherein the adjacency pair is associated with an expansion pair, the utterance being a message that is played back by the machine in response to a statement by a user;

increment a counter corresponding to the utterance; and

in response to the counter crossing a predetermined threshold, send a message to notify a developer to redesign the specific utterance.

14. The computer program product of claim 13, wherein the computer readable storage medium further comprises instructions to:

determine a type of outcome of the conversation; and

determine an outcome-efficiency type of the conversation based on the efficiency score and the type of the outcome.

15. The computer program product of claim 14, wherein the outcome-efficiency type is determined from a predetermined group of outcome-efficiency types consisting of compact-failure, compact-success, expanded-failure, and expanded-success.

16. The computer program product of claim 14, wherein the type of the outcome is determined based on an outcome indicator of a base pair of the conversation, wherein the outcome indicator is identified based on a second pair-part of the base pair being one of a predetermined set of response types.

17. The computer program product of claim 13, wherein the message is initiated further in response to the adjacency pair being expanded using a specific response type, and wherein the specific response type is one from a group of response types consisting of a definition request, a repeat request, a partial repeat request, and an elaboration request.