Clarifying natural language input using targeted questions

• US 9,805,718 B2
• Filed: 04/19/2013
• Issued: 10/31/2017
• Est. Priority Date: 04/19/2013
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Ayan, Necip Fazil, Mandal, Arindam, Zheng, Jing
• Primary Examiner(s): Yen, Eric
• Application Number:
  US13/866,509
• Publication Number:
  US 20140316764A1
• Time in Patent Office: 1,656 Days
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: H04M 3/4936 Speech interaction details speech recognition per se G10L15/00

Abstract

A dialog assistant embodied in a computing system can present a clarification question based on a machine-readable version of human-generated conversational natural language input. Some versions of the dialog assistant identify a clarification target in the machine-readable version, determine a clarification type relating to the clarification target, present the clarification question in a conversational natural language manner, and process a human-generated conversational natural language response to the clarification question.

Claims

1. A computer-implemented method, comprising:

receiving, by a computing device, input, wherein the input is associated with an interaction with the computing device, and wherein the input includes dialog input, application input, or sensor input;

determining an interaction history, wherein determining includes adding the input to the interaction history, wherein adding includes using an artificial intelligence-based model to combine the input with prior interaction history;

determining a current context, wherein the current context is associated with an interaction with the computing device, and wherein the interaction includes dialog input, application input, or sensor input;

receiving verbal input, wherein the verbal input includes natural language, and wherein the verbal input includes a plurality of words;

interpreting the verbal input, wherein interpreting the verbal input includes assigning one or more attributes to the plurality of words;

identifying an attribute from the one or more attributes, wherein the identified attribute is associated with one or more words from the plurality of words, wherein the identified attribute indicates that the one or more words are unclear, and wherein a word is unclear when the computing system cannot recognize the word;

determining that the current context and the interaction history do not clarify the one or more words;

determining a type of information needed to recognize the one or more words, wherein determining includes using the one or more attributes assigned to the one or more words to identify missing information;

determining a clarification question for the one or more words, wherein determining includes using the type of information, the current context, and the interaction history, wherein the clarification question is formatted to request the missing information, and wherein the format is customized using the current context and the interaction history; and

outputting the clarification question.

2. The computer-implemented method of claim 1, wherein application input includes data or instructions output by an application executing on the computing device.

3. The computer-implemented method of claim 1, wherein the sensor input is generated automatically.

4. The computer-implemented method of claim 1, further comprising:

receiving non-verbal input, wherein the non-verbal input includes a physical motion interpreted from a sensor associated with the computing device, and wherein the current context includes the non-verbal input.

5. The computer-implemented method of claim 1, further comprising:

receiving additional verbal input;

determining that the additional verbal input includes the missing information.

6. The computer-implemented method of claim 1, further comprising:

receiving additional verbal input;

filtering one or more non-responsive portions of the additional verbal input; and

extracting the missing information from the filtered additional verbal input.

7. The computer-implemented method of claim 1, further comprising

receiving additional verbal input;

filtering one or more non-responsive portions of the additional verbal input; and

determining that no part of the filtered additional verbal input includes the missing information.

8. The computer-implemented method of claim 7, further comprising:

determining a second clarification question, wherein the second clarification question is different from the clarification question for the one or more words; and

outputting the second clarification question.

9. The computer-implemented method of claim 1, further comprising:

receiving additional verbal input;

using the additional verbal input, the current context, and the interaction history to determine an action; and

executing the action, wherein the action modifies an operation of the computing device.

10. The computer-implemented method of claim 9, wherein using the additional verbal input includes modifying the verbal input to include the additional verbal input.

11. The computer-implemented method of claim 9, wherein using the additional verbal input includes generating new verbal input, wherein the new verbal input combines the verbal input and the additional verbal input.

12. The computer-implemented method of claim 1, wherein the interaction history includes a clarification history, wherein the clarification history includes one or more previous clarification questions, and wherein determining the clarification question further includes using the clarification history.

13. The computer-implemented method of claim 1, further comprising:

determining one or more additional attributes from the one or more attributes, wherein the one or more attributes indicate that one or more sets of words from the plurality of words are unclear; and

determining an order for the identified attribute and the one or more additional attributes, wherein the order determines an order in which to clarify the identified attribute and the one or more attributes.

14. A computing device, comprising:

one or more processors; and

a non-transitory computer-readable medium including instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including;

receiving input, wherein the input is associated with an interaction with the computing device, and wherein the input includes dialog input, application input, or sensor input;

determining an interaction history, wherein determining includes adding the input to the interaction history, wherein adding includes using an artificial intelligence-based model to combine the input with prior interaction history;

determining a current context, wherein the current context is associated with an interaction with the computing device, and wherein the interaction includes dialog input, application input, or sensor input;

receiving verbal input, wherein the verbal input includes natural language, and wherein the verbal input includes a plurality of words;

interpreting the verbal input, wherein interpreting the verbal input includes assigning one or more attributes to the plurality of words;

identifying an attribute from the one or more attributes, wherein the identified attribute is associated with one or more words from the plurality of words, wherein the identified attribute indicates that the one or more words are unclear, and wherein a word is unclear when the computing system cannot recognize the word;

determining that the current context and the interaction history do not clarify the one or more words;

determining a type of information needed to recognize the one or more words, wherein determining includes using the one or more attributes assigned to the one or more words to identify missing information;

determining a clarification question for the one or more words, wherein determining includes using the type of information, the current context, and the interaction history, wherein the clarification question is formatted to request the missing information, and wherein the format is customized using the current context and the interaction history; and

outputting the clarification question.

15. The computing device of claim 14, wherein application input includes data or instructions output by an application executing on the computing device.

16. The computing device of claim 14, wherein the sensor input is generated automatically.

17. The computing device of claim 14, wherein the non-transitory computer-readable medium further includes instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

receiving non-verbal input, wherein the non-verbal input includes a physical motion interpreted from a sensor associated with the computing device, and wherein the current context includes the non-verbal input.

18. The computing device of claim 14, wherein the non-transitory computer-readable medium further includes instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

receiving additional verbal input;

determining that the additional verbal input includes the missing information.

19. The computing device of claim 14, wherein the non-transitory computer-readable medium further includes instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

receiving additional verbal input;

filtering one or more non-responsive portions of the additional verbal input; and

extracting the missing information from the filtered additional verbal input.

20. The computing device of claim 14, wherein the non-transitory computer-readable medium further includes instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

receiving additional verbal input;

filtering one or more non-responsive portions of the additional verbal input; and

determining that no part of the filtered additional verbal input includes the missing information.

21. The computing device of claim 20, wherein the non-transitory computer-readable medium further includes instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

determining a second clarification question, wherein the second clarification question is different from the clarification question for the one or more words; and

outputting the second clarification question.

22. The computing device of claim 14, wherein the non-transitory computer-readable medium further includes instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

receiving additional verbal input;

using the additional verbal input, the current context, and the interaction history to determine an action; and

executing the action, wherein the action modifies an operation of the computing device.

23. The computing device of claim 22, wherein using the additional verbal input includes modifying the verbal input to include the additional verbal input.

24. The computing device of claim 22, wherein using the additional verbal input includes generating new verbal input, wherein the new verbal input combines the verbal input and the additional verbal input.

25. The computing device of claim 14, wherein the interaction history includes a clarification history, wherein the clarification history includes one or more previous clarification questions, and wherein determining the clarification question further includes using the clarification history.

26. The computing device of claim 14, wherein the non-transitory computer-readable medium further includes instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

determining one or more additional attributes from the one or more attributes, wherein the one or more attributes indicate that one or more sets of words from the plurality of words are unclear; and

determining an order for the identified attribute and the one or more additional attributes, wherein the order determines an order in which to clarify the identified attribute and the one or more attributes.

27. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions that, when executed by one or more processors, cause the one or more processors to:

receive input, wherein the input is associated with an interaction with a computing device, and wherein the input includes dialog input, application input, or sensor input;

determine an interaction history, wherein determining includes adding the input to the interaction history, wherein adding includes using an artificial intelligence-based model to combine the input with prior interaction history;

determine a current context, wherein the current context is associated with an interaction with the computing device, and wherein the interaction includes dialog input, application input, or sensor input;

receive verbal input, wherein the verbal input includes natural language, and wherein the verbal input includes a plurality of words;

interpret the verbal input, wherein interpreting the verbal input includes assigning one or more attributes to the plurality of words;

identify an attribute from the one or more attributes, wherein the identified attribute is associated with one or more words from the plurality of words, wherein the identified attribute indicates that the one or more words are unclear, and wherein a word is unclear when the computing system cannot recognize the word;

determine that the current context and the interaction history do not clarify the one or more words;

determine a type of information needed to recognize the one or more words, wherein determining includes using the one or more attributes assigned to the one or more words to identify missing information;

determine a clarification question for the one or more words, wherein determining includes using the type of information, the current context, and the interaction history, wherein the clarification question is formatted to request the missing information, and wherein the format is customized using the current context and the interaction history; and

output the clarification question.

28. The computer-program product of claim 27, wherein application input includes data or instructions output by an application executing on the computing device.

29. The computer-program product of claim 27, wherein the sensor input is generated automatically.

30. The computer-program product of claim 27, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to:

receive non-verbal input, wherein the non-verbal input includes a physical motion interpreted from a sensor associated with the computing device, and wherein the current context includes the non-verbal input.

31. The computer-program product of claim 27, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to:

receive additional verbal input;

determine that the additional verbal input includes the missing information.

32. The computer-program product of claim 27, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to:

receive additional verbal input;

filter one or more non-responsive portions of the additional verbal input; and

extract the missing information from the filtered additional verbal input.

33. The computer-program product of claim 27, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to:

receive additional verbal input;

filter one or more non-responsive portions of the additional verbal input; and

determine that no part of the filtered additional verbal input includes the missing information.

34. The computer-program product of claim 33, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to:

determine a second clarification question, wherein the second clarification question is different from the clarification question for the one or more words; and

output the second clarification question.

35. The computer-program product of claim 27, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to:

receive additional verbal input;

use the additional verbal input, the current context, and the interaction history to determine an action; and

execute the action, wherein the action modifies an operation of the computing device.

36. The computer-program product of claim 35, wherein using the additional verbal input includes modifying the verbal input to include the additional verbal input.

37. The computer-program product of claim 35, wherein using the additional verbal input includes generating new verbal input, wherein the new verbal input combines the verbal input and the additional verbal input.

38. The computer-program product of claim 27, wherein the interaction history includes a clarification history, wherein the clarification history includes one or more previous clarification questions, and wherein determining the clarification question further includes using the clarification history.

39. The computer-program product of claim 27, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to:

determine one or more additional attributes from the one or more attributes, wherein the one or more attributes indicate that one or more sets of words from the plurality of words are unclear; and

determine an order for the identified attribute and the one or more additional attributes, wherein the order determines an order in which to clarify the identified attribute and the one or more attributes.