USING NONVERBAL COMMUNICATION IN DETERMINING ACTIONS

• US 20140191939A1
• Filed: 01/09/2013
• Published: 07/10/2014
• Est. Priority Date: 01/09/2013
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chambers, Robert, Shriberg, Elizabeth, Penn, Daniel J., Hanson, Mark
• Application Number:
  US13/737,542
• Publication Number:
  US 20140191939A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection

Abstract

Nonverbal communication is used when determining an action to perform in response to received user input. The received input includes direct input (e.g. speech, text, gestures) and indirect input (e.g. nonverbal communication). The nonverbal communication includes cues such as body language, facial expressions, breathing rate, heart rate, well as vocal cues (e.g. prosodic and acoustic cues) and the like. Different nonverbal communication cues are monitored such that performed actions are personalized. A direct input specifying an action to perform (e.g. “perform action 1”) may be adjusted based on one or more indirect inputs (e.g. nonverbal cues) received. Another action may also be performed in response to the indirect inputs. A profile may be associated with the user such that the responses provided by the system are determined using nonverbal cues that are associated with the user.

Claims

1. A method for using nonverbal communication to determine an intended action, comprising:

receiving user interaction comprising direct input specifying an intended action and indirect input comprising nonverbal communication;

determining the direct input using at least one of;

speech input, gesture input, and textual input;

determining the indirect input comprising the nonverbal communication;

using the indirect communication in addition to the intended action determined from the direct input to determine an action to perform; and

performing the action.

2. The method of claim 1, further comprising determining a user satisfaction using received nonverbal communication after performing the action.

3. The method of claim 1, further comprising performing an additional action in response to determining a user satisfaction using received nonverbal communication after performing the action.

4. The method of claim 3, wherein performing the additional action in response to determining the user satisfaction comprises requesting a clarification to the intended action.

5. The method of claim 1, wherein performing the additional action in response to determining the user satisfaction comprises changing content displayed on a user interface.

6. The method of claim 1, wherein determining the user satisfaction using received nonverbal communication after performing the action comprises determining a facial expression.

7. The method of claim 1, wherein the nonverbal communication comprises a voice cue comprising one or more of:

a voice level, a spacing of speech, and a rate of speech.

8. The method of claim 1, further comprising accessing a profile of a user associated with the input that includes nonverbal communication information that is associated with the user.

9. The method of claim 1, wherein the nonverbal communication comprises one or more of:

a vocal cue, a heart rate, a breathing rate, a facial expression, a body movement, and a posture.

10. A computer-readable medium storing computer-executable instructions for using nonverbal communication, comprising:

receiving user interaction comprising direct input specifying an intended action and indirect input comprising nonverbal communication;

determining the direct input using at least one of;

speech input, gesture input, and textual input;

determining the indirect input comprising the nonverbal communication that comprises one or more of;

a vocal cue, a heart rate, a breathing rate, a facial expression, a body movement, and a posture;

accessing a profile that includes information relating to a baseline of nonverbal communication cues associated with a user;

determining changes from the baseline using the determined indirect communication;

using the indirect communication and determined changes in addition to the intended action determined from the direct input to determine an action to perform; and

performing the action.

11. The computer-readable medium of claim 10, further comprising determining a user satisfaction using received nonverbal communication after performing the action.

12. The computer-readable medium of claim 10, further comprising performing an additional action in response to determining a user satisfaction using received nonverbal communication after performing the action.

13. The computer-readable medium of claim 12, wherein performing the additional action in response to determining the user satisfaction comprises requesting a clarification to the intended action.

14. The computer-readable medium of claim 10, wherein performing the additional action in response to determining the user satisfaction comprises changing content displayed on a user interface.

15. The computer-readable medium of claim 10, wherein determining the user satisfaction using received nonverbal communication after performing the action comprises determining a facial expression.

16. The computer-readable medium of claim 10, wherein the nonverbal communication comprises a voice cue comprising one or more of:

a voice level, a spacing of speech, and a rate of speech.

17. A system for using nonverbal communication, comprising:

a camera that is configured to detect movements;

a microphone that is configured to receive speech input;

a processor and memory;

an operating environment executing using the processor;

a display; and

an understanding manager that is configured to perform actions comprising;

receiving user interaction comprising direct input specifying an intended action and indirect input comprising nonverbal communication;

determining the direct input using at least one of;

speech input, gesture input, and textual input;

determining the indirect input comprising the nonverbal communication that comprises one or more of;

a vocal cue, a heart rate, a breathing rate, a facial expression, a body movement, and a posture;

accessing a profile that includes information relating to a baseline of nonverbal communication cues associated with a user;

determining changes from the baseline using the determined indirect communication;

using the indirect communication and determined changes in addition to the intended action determined from the direct input to determine an action to perform; and

performing the action.

18. The system of claim 17, further comprising determining a user satisfaction using received nonverbal communication after performing the action and performing an additional action in response to determining a user satisfaction using received nonverbal communication after performing the action.

19. The system of claim 17, wherein determining the user satisfaction using received nonverbal communication after performing the action comprises determining a facial expression.

20. The system of claim 17, wherein the nonverbal communication comprises a voice cue comprising one or more of:

a voice level, a spacing of speech, and a rate of speech.