VPA WITH INTEGRATED OBJECT RECOGNITION AND FACIAL EXPRESSION RECOGNITION

• US 20170160813A1
• Filed: 10/24/2016
• Published: 06/08/2017
• Est. Priority Date: 12/07/2015
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Divakaran, Ajay, Tamrakar, Amir, Kalns, Edgar, Mark, William, Acharya, Girish, Ho, Greg, Huang, Jihua, Salter, David, Wessel, Michael, Carpenter, James, Yin, Min, Mombourquette, Brent, Law, Eric, Frandsen, Michael, Kim, Hyong-Gyun, Albright, Cory, Tsiartas, Andreas, Shriberg, Elizabeth, Nitz, Kenneth
• Granted Patent: US 10,884,503 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

Methods, computing devices, and computer-program products are provided for implementing a virtual personal assistant. In various implementations, a virtual personal assistant can be configured to receive sensory input, including at least two different types of information. The virtual personal assistant can further be configured to determine semantic information from the sensory input, and to identify a context-specific framework. The virtual personal assistant can further be configured to determine a current intent. Determining the current intent can include using the semantic information and the context-specific framework. The virtual personal assistant can further be configured to determine a current input state. Determining the current input state can include using the semantic information and one or more behavioral models. The behavioral models can include one or more interpretations of previously-provided semantic information. The virtual personal assistant can further be configured to determine an action using the current intent and the current input state.

Claims

1. A method, comprising:

receiving, by an integrated circuit in a computing device, sensory input, wherein the sensory input includes at least two different types of information;

determining semantic information from the sensory input, wherein the semantic information provides an interpretation of the sensory input;

identifying a context-specific framework, wherein the context-specific framework includes a cumulative sequence of one or more previous intents;

determining a current intent, wherein determining the current intent includes using the semantic information and the context-specific framework;

determining a current input state, wherein determining the current input state includes using the semantic information and one or more behavioral models, and wherein the behavioral models include one or more interpretations of previously-provided semantic information; and

determining an action, wherein determining the action includes using the current intent and the current input state.

2. The method of claim 1, wherein types of information include speech information, graphical information, audio input, image input, or tactile input.

3. The method of claim 1, wherein an emotional state is capable of being derived from audio input.

4. The method of claim 1, wherein physical gestures are capable of being derived from image input.

5. The method of claim 1, wherein an emotional state is capable of being derived from image input.

6. The method of claim 1, wherein behavioral models include a configurable preference model, and wherein the configurable preference model includes one or more previous input states and semantic information associated with the one or more previous input states.

7. The method of claim 1, wherein the context-specific framework includes a dynamic ontology, wherein the dynamic ontology includes a cumulative sequence of one or more previous intents derived from a context-specific ontology.

8. A virtual personal assistant device, comprising:

one or more processors; and

a non-transitory computer-readable medium including instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including;

receiving sensory input, wherein the sensory input includes at least two different types of information;

determining semantic information from the sensory input, wherein the semantic information provides an interpretation of the sensory input;

identifying a context-specific framework, wherein the context-specific framework includes a cumulative sequence of one or more previous intents;

determining a current intent, wherein determining the current intent includes using the semantic information and the context-specific framework;

determining a current input state, wherein determining the current input state includes using the semantic information and one or more behavioral models, and wherein the behavioral models include one or more interpretations of previously-provided semantic information; and

determining an action, wherein determining the action includes using the current intent and the current input state.

9. The virtual personal assistant device of claim 8, wherein types of information include speech information, graphical information, audio input, image input, or tactile input.

10. The virtual personal assistant device of claim 8, wherein an emotional state is capable of being derived from audio input.

11. The virtual personal assistant device of claim 8, wherein physical gestures are capable of being derived from image input.

12. The virtual personal assistant device of claim 8, wherein an emotional state is capable of being derived from image input.

13. The virtual personal assistant device of claim 8, wherein behavioral models include a configurable preference model, and wherein the configurable preference model includes one or more previous input states and semantic information associated with the one or more previous input states.

14. The virtual personal assistant device of claim 8, wherein the context-specific framework includes a dynamic ontology, wherein the dynamic ontology includes a cumulative sequence of one or more previous intents derived from a context-specific ontology.

15. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions that, when executed by one or more processors, cause the one or more processors to:

receive, by an integrated circuit in a computing device, sensory input, wherein the sensory input includes at least two different types of information;

determine semantic information from the sensory input, wherein the semantic information provides an interpretation of the sensory input;

identify a context-specific framework, wherein the context-specific framework includes a cumulative sequence of one or more previous intents;

determine a current intent, wherein determining the current intent includes using the semantic information and the context-specific framework;

determine a current input state, wherein determining the current input state includes using the semantic information and one or more behavioral models, and wherein the behavioral models include one or more interpretations of previously-provided semantic information; and

determine an action, wherein determining the action includes using the current intent and the current input state.

16. The computer-program product of claim 15, wherein types of information include speech information, graphical information, audio input, image input, or tactile input.

17. The computer-program product of claim 15, wherein an emotional state is capable of being derived from audio input.

18. The computer-program product of claim 15, wherein physical gestures are capable of being derived from image input.

19. The computer-program product of claim 15, wherein an emotional state is capable of being derived from image input.

20. The computer-program product of claim 15, wherein behavioral models include a configurable preference model, and wherein the configurable preference model includes one or more previous input states and semantic information associated with the one or more previous input states.