Generic virtual personal assistant platform

• US 10,163,440 B2
• Filed: 01/05/2017
• Issued: 12/25/2018
• Est. Priority Date: 12/08/2011
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Yadgar, Osher, Yorke-Smith, Neil, Peintner, Bart, Tur, Gokhan, Ayan, Necip Fazil, Wolverton, Michael J., Acharya, Girish, Parimi, Venkatarama Satyanarayana, Mark, William S., Wang, Wen, Kathol, Andreas, Vincent, Regis, Franco, Horacio E.
• Primary Examiner(s): Azad, Abul
• Application Number:
  US15/399,070
• Publication Number:
  US 20170116989A1
• Time in Patent Office: 719 Days
• Field of Search: 704270, 704275
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04M 3/4936 Speech interaction details speech recognition per se G10L15/00

Abstract

A method for assisting a user with one or more desired tasks is disclosed. For example, an executable, generic language understanding module and an executable, generic task reasoning module are provided for execution in the computer processing system. A set of run-time specifications is provided to the generic language understanding module and the generic task reasoning module, comprising one or more models specific to a domain. A language input is then received from a user, an intention of the user is determined with respect to one or more desired tasks, and the user is assisted with the one or more desired tasks, in accordance with the intention of the user.

Claims

1. A method for assisting a user with one or more desired tasks within a domain, the method comprising:

receiving, by a computing system comprising one or more computing devices, a verbal language input and at least one of a plurality of different kinds of non-verbal input from the user;

determining, by the computing system, from the verbal language input and the at least one of a plurality of different kinds of non-verbal input, an intention of the user with respect to the one or more desired tasks, by an executable generic language understanding module and a run-time specification comprising a model configured to a specific field of use; and

performing, by the computing system, a domain-specific task in accordance with the intention of the user, by an executable generic task reasoning module and a run-time specification comprising a task flow configured to the specific field of use.

2. The method of claim 1, wherein the verbal language input comprises a portion of a dialog with the computer processing system having multiple rounds, and wherein the determining the intention of the user includes considering, during a given round of the dialog, a previous intention of the user determined during one or more previous rounds of the dialog.

3. The method of claim 1, wherein the plurality of different kinds of nonverbal inputs comprises a gesture.

4. The method of claim 1 wherein the plurality of different kinds of nonverbal inputs comprises video input.

5. The method of claim 1 wherein the plurality of different kinds of nonverbal inputs comprises a click.

6. The method of claim 1 wherein the plurality of different kinds of nonverbal inputs comprises text.

7. The method of claim 1, wherein the plurality of different kinds of nonverbal inputs comprises a multimedia input having a video portion and an audio portion.

8. The method of claim 7, further comprising parsing the multimedia input to extract the audio portion and recognizing language in the audio portion by automatic speech recognition.

9. The method of claim 7, further comprising parsing the multimedia input to extract the video portion and recognizing gestures in the video portion.

10. The method of claim 1, wherein the model configured to a specific field of use utilized by the generic language understanding module includes a language model.

11. The method of claim 10, wherein the language model comprises a grammar and an ontology.

12. The method of claim 1, wherein the model configured to a specific field of use that are utilized by the generic task reasoning module includes a task flow model.

13. The method of claim 1, wherein determining the intention of the user includes considering feedback from the generic task reasoning module.

14. The method of claim 1, wherein the generic language understanding module comprises an utterance-level understanding component.

15. The method of claim 1, wherein the generic language understanding module comprises a context-level understanding component.

16. The method of claim 1, further comprising providing an executable speech recognition engine for execution on the computer processing system;

and wherein receiving the verbal language input from the user includes receiving voice input from the user, and converting the voice input to text using the speech recognition engine.

17. The method of claim 1, wherein the verbal language input comprises a natural language input.

18. The method of claim 1, further comprising receiving an additional input including at least one of:

a current geographical location of the user, an interaction context of the user within a web page, an image input or a non-natural language command; and

wherein determining the intention of the user includes considering the additional input.

19. The method of claim 1, wherein the desired tasks comprise at least one of:

content search;

content retrieval; and

transaction completion.

20. The method of claim 1, further comprising:

executing the one or more desired tasks with the computer processing system.

21. The method of claim 1, further comprising:

generating the model configured to a specific field of use via a computer-assisted development tool to convert a graphical representation of at least one task flow into the set of run-time specifications.

22. A virtual personal assistant platform embodied in one or more non-transitory computer-readable media, the virtual personal assistant platform configured to cause a computing system comprising one or more computing devices to:

receive a verbal language input from the user and the at least one of a plurality of different kinds of non-verbal input;

determine, from the verbal language input and the at least one of a plurality of different kinds of non-verbal input, an intention of the user with respect to the one or more desired tasks, by an executable generic language understanding module and a run-time specification comprising a model configured to a specific field of use; and

perform, by the computing system, a domain-specific task in accordance with the intention of the user, by an executable generic task reasoning module and a run-time specification comprising a task flow configured to the specific field of use.

23. The platform of claim 22, wherein the verbal language input of the user comprises a portion of a dialog with the computer processing system having multiple rounds, and wherein the determining the intention of the user includes considering, during a given round of the dialog, a previous intention of the user determined during one or more previous rounds of the dialog.

24. The platform of claim 22, comprising an input module configured to input a verbal language input and at least one of a plurality of different kinds of non-verbal input.

25. The platform of claim 24 wherein the input module is further configured to convert the different kinds of non-verbal input to text.

26. The platform of claim 24, wherein the plurality of different kinds of nonverbal input includes audio input and the input module comprises a speech recognition tool configured to recognize speech in an audio input.

27. The platform of claim 22, comprising a sentence level understanding module configured to derive an understanding of a fragment of the verbal language input.

28. The platform of claim 27, wherein the sentence level understanding module comprises a rule-based grammar module configured to parse the verbal language input, and a statistical parser module configured to parse the verbal language input if the rule-based grammar module fails to confidently parse the verbal language input.

29. The platform of claim 28, comprising an interpreter module configured to analyze a user intent argument output by the sentence level understanding module and output an interpreter intent argument.

30. The platform of claim 29, comprising a reasoner module configured to use the task flow configured to the specific field of use and the interpreter intent argument to determine a task to be performed by the computing system.

31. The platform of claim 30, wherein (i) the reasoner module is configured to output, to the interpreter module, a reasoner intent argument indicative of the determined task to be performed, and (ii) the interpreter module is configured to use the reasoner intent argument to analyze the user intent argument output by the sentence level understanding module.

32. The platform of claim 30, wherein the interpreter module is configured to compute a confidence level indicative of a level of certainty associated with the interpreter intent argument and output the confidence level to the reasoner module.

33. The platform of claim 30, wherein the reasoner module is configured to learn a preference of the user over time and use the learned preference to determine the task to be performed by the computing system.

34. The platform of claim 30, wherein the reasoner module is configured to output a reasoner intent argument indicative of the determined task to be performed, and the platform comprises a natural language generation module configured to generate a natural language response to the verbal language input of the user and the plurality of difference kinds of non-verbal input based on the reasoner intent argument.

35. The platform of claim 30, wherein the reasoner module is configured to output a reasoner intent argument indicative of the determined task to be performed to a service module, and wherein the service module is configured to interface with a server to retrieve information from a database.

36. The platform of claim 30, wherein the reasoner module is configured to (i) determine that a parameter needed to perform the task is missing from the verbal language input or the plurality of difference kinds of non-verbal input and (ii) cause the user to be prompted for the missing parameter.

37. The platform of claim 30, wherein the reasoner module is configured to utilize the plurality of difference kinds of non-verbal input to determine the task to be performed by the computing system.

38. A computing system for assisting a user with one or more desired tasks within a domain, the computing system comprising one or more computing devices configured to:

receive, by a computing system comprising one or more computing devices, a verbal language input from the user and a plurality of difference kinds of non-verbal input from the user;

determine, by the computing system, from the verbal language input and the plurality of difference kinds of non-verbal input, an intention of the user with respect to the one or more desired tasks, by an executable generic language understanding module and a run-time specification comprising a domain-specific plug-in model configured to a specific field of use; and

perform a domain-specific task in accordance with the intention of the user, by an executable generic task reasoning module and a run-time specification comprising a domain-specific plug-in task flow configured to the specific field of use.