Wearable emotion detection and feedback system

• US 9,019,174 B2
• Filed: 10/31/2012
• Issued: 04/28/2015
• Est. Priority Date: 10/31/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Jerauld, Robert
• Primary Examiner(s): Pervan, Michael
• Application Number:
  US13/665,477
• Publication Number:
  US 20140118225A1
• Time in Patent Office: 909 Days
• Field of Search: 345 7- 9
• US Class Current: 345/8
• CPC Class Codes: A61B 5/11 Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb for measuring pulse A61B5/02 ;A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/486 Bio-feedback A61B5/375 takes precedence
• CPC Class Codes: A61B 5/7278 Artificial waveform generation or derivation, e.g. synthesising signals from measured signals A61B5/327 takes precedence
• CPC Class Codes: A61B 5/742 using visual displays A61B5/339 takes precedence
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

A see-through, head mounted display and sensing devices cooperating with the display detect audible and visual behaviors of a subject in a field of view of the device. A processing device communicating with display and the sensors monitors audible and visual behaviors of the subject by receiving data from the sensors. Emotional states are computed based on the behaviors and feedback provided to the wearer indicating computed emotional states of the subject. During interactions, the device, recognizes emotional states in subjects by comparing detected sensor input against a database of human/primate gestures/expressions, posture, and speech. Feedback is provided to the wearer after interpretation of the sensor input.

Claims

1. A see through head mounted display apparatus, comprising:

a see through, head mounted display;

a plurality of sensors cooperating with the display to detect audible and visual behaviors of a subject in a field of view of the apparatus;

one or more processing devices in communication with display and the sensors, the one or more processing devices automatically

monitor the audible and visual behaviors of the subject by receiving data from the sensors;

determine a scenario of use for the see-through, head mounted display;

compute an emotional state of the subjects based on the audible and visual behaviors weighted relative to the scenario;

provide feedback to the display indicating computed emotional states of the subject.

2. The apparatus of claim 1 wherein the one or more processing devices compute the emotional state by comparing recognized audible and visual sensor data to a set of associated emotional states represented by the behaviors exhibited in the behaviors.

3. The apparatus of claim 1 wherein the one or more processing devices store computed emotional states relative to recognized audible and visual sensor data for each subject.

4. The apparatus of claim 3 wherein the one or more processing devices compute the emotional state of a subject by comparing recognized audible and visual sensor data to a set of associated emotional states represented by the behaviors exhibited in the behaviors and historical data on computed emotional states stored relative to the subject.

5. The apparatus of claim 1 wherein the one or more processors identify one or more subjects compute emotional states for each subject and associate the computed emotional states with each subject.

6. The apparatus of claim 1 wherein the display includes an audible output and the feedback comprises audible feedback through the audible output.

7. The apparatus of claim 1 wherein the feedback comprises a visual indication of a computed emotional state in the display.

8. The apparatus of claim 1 wherein feedback to the display is a visual indication comprising is one of a specific indication of an emotional state or a positive or negative general indication of a change in emotional state.

9. The apparatus of claim 1 further including a communications interface connecting the apparatus to an analysis and feedback service, the interface outputting sensor data to the service, the interface receiving computed emotional states based on the data from the service.

10. A computer storage device including instructions operable on a processing device to perform a method comprising the steps of:

receiving input from a plurality of sensors mounted on a see through head mounted display device, the sensors mounted to detect audible and visual actions of a subject in a field of view of the see through head mounted display device;

computing an emotional state of the subject based on the audible and visual actions, the computing including determining a scenario of use, recognizing a behavior comprising any of one of a posture, expression, gesture, audible expression or word in the input from the sensors and comparing posture, expression, gesture, audible expression or word to a set of associated emotional states represented by the behavior relative to the scenario to determine a likely emotional state; and

providing feedback to a wearer indicating computed emotional states of the subject.

11. The computer storage device of claim 10 wherein the steps of receiving, continually computing and providing feedback are repeated in sequence in real time.

12. The computer storage device of claim 10 wherein the instructions further include providing the wearer with a selection of a social scenario, the selection of the social scenario biasing the comparing of the posture, expression, gesture, audible expression or word to associated emotional states.

13. The computer storage device of claim 10 wherein the instructions further include storing computed emotional states indexed to recognized posture, expression, gesture, audible expression or word data for each subject during an interaction.

14. The computer storage device of claim 10 wherein providing feedback includes rendering in the display one of:

a specific indication of an emotional state, or a positive or negative general indication of a change in emotional state.

15. A method for providing emotional state feedback of subjects in a field of view of a see through head mounted display system, comprising:

determining an environment and orientation of the system, the system includes one or more sensors and a see-through display;

determining a social scenario for a wearer of the see through head mounted display;

identifying one or more subjects in the field of view, the one or more subjects participating in the social scenario with the wearer;

receiving input from a plurality of sensors mounted on the see through head mounted display device;

recognizing an action of the subject comprising any of one of a posture, expression, gesture, audible expression or word in the input from the sensors and comparing posture, expression, gesture, audible expression or word to a set of associated emotional states likely in the social scenario;

computing an emotional state based on the action recognized relative to the social scenario; and

providing feedback to the wearer indicating emotional state of the one or more subjects.

16. The method of claim 15 wherein identifying comprises receiving a manual selection of subjects for whom emotional state feedback is desired.

17. The method of claim 16 wherein identifying comprises selecting subjects for whom emotional state feedback is desired based on user gaze.

18. The method of claim 17 wherein determining a social scenario comprises determining the scenario based on the environment, location and orientation of the system.

19. The method of claim 18 wherein providing feedback includes rendering in the display one of:

a specific indication of an emotional state including a visual representation specifying the state, or a positive or negative general indication of a change in emotional state based on tinting the display.

20. The method of claim 19 further including outputting sensor data to an analysis and feedback service and receiving computed emotional states based on the data from the service.