Wearable emotion detection and feedback system

• US 9,824,698 B2
• Filed: 11/29/2016
• Issued: 11/21/2017
• Est. Priority Date: 10/31/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Jerauld, Robert
• Primary Examiner(s): Pervan, Michael
• Application Number:
  US15/364,037
• Publication Number:
  US 20170117005A1
• Time in Patent Office: 357 Days
• Field of Search: 345 7- 9
• CPC Class Codes: A61B 5/11 Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb for measuring pulse A61B5/02 ;A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/486 Bio-feedback A61B5/375 takes precedence
• CPC Class Codes: A61B 5/7278 Artificial waveform generation or derivation, e.g. synthesising signals from measured signals A61B5/327 takes precedence
• CPC Class Codes: A61B 5/742 using visual displays A61B5/339 takes precedence
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

A see-through, head mounted display and sensing devices cooperating with the display detect audible and visual behaviors of a subject in a field of view of the device. A processing device communicating with display and the sensors monitors audible and visual behaviors of the subject by receiving data from the sensors. Emotional states are computed based on the behaviors and feedback provided to the wearer indicating computed emotional states of the subject. During interactions, the device, recognizes emotional states in subjects by comparing detected sensor input against a database of human/primate gestures/expressions, posture, and speech. Feedback is provided to the wearer after interpretation of the sensor input.

Claims

1. A see through head mounted display apparatus, comprising:

a see-through, head mounted display;

a plurality of sensors cooperating with the see-through, head mounted display to detect at least one of audible and visual behaviors of a subject in a field of view of the apparatus; and

one or more processing devices in communication with the see-through, head mounted display and the sensors, the one or more processing devices;

monitor the at least one of audible and visual behaviors of the subject by receiving data from the sensors;

receive supplemental data relating to an emotional state of the subject from a second see-through, head mounted display also monitoring at least one of audible and visual behaviors of the subject;

compute an emotional state of the subject based on the at least one of audible and visual behaviors monitored by see-through, head mounted display device and the supplemental data received from the second see-through, head mounted display; and

provide at least one of audible and visible feedback to a computing device indicating computed emotional states of the subject.

2. The apparatus of claim 1 wherein to compute an emotional state, the one or more processing devices:

recognize emitted characteristics of the subject comprising each one of an emitted posture, expression, gesture, audible expression and word from the subject in the input from the sensors and compare the posture, expression, gesture, audible expression and word to a set of emotional states associated with any one or more similar posture, expression, gesture, audible expression and word.

3. The apparatus of claim 2 wherein the one or more processing devices store computed emotional states based on the characteristics relative to recognized audible and visual sensor data for each subject identified.

4. The apparatus of claim 3 wherein the one or more processing devices compute the emotional state of a subject by identifying the subject, retrieving emotional states stored relative to the subject, and comparing recognized audible and visual sensor data to a set of associated emotional states represented by the behaviors exhibited in the behaviors and the computed emotional states stored relative to the subject.

5. The apparatus of claim 1 wherein the one or more processors identify one or more subjects as an unknown subject, determine that no stored emotional state data exists for the unknown subject, and compute emotional states for each unknown subject, and associate the computed emotional states with each unknown subject thereby creating a known subject.

6. The apparatus of claim 1 wherein the one or more processors prompt a wearer of the display for identification of the subjects and are operable to receiving identification of the subjects.

7. The apparatus of claim 1 wherein the one or more processors are operable to determine the emotional state based on scenario information.

8. The apparatus of claim 7 wherein the scenario information is determined by providing the user with a selection of scenarios and receiving a selection of the scenarios.

9. The apparatus of claim 1 further including a communications interface connecting the apparatus to an analysis and feedback service, the interface operable to output sensor data to the service, the interface operable to receive computed emotional states for identified individuals from the service.

10. The apparatus of claim 1, wherein the computing device receiving the feedback is the head mounted display device.

11. The apparatus of claim 1, wherein the computing device receiving the feedback is a computing device remote from the head mounted display device.

12. A computer storage device including instructions operable on a processing device to perform a method comprising the steps of:

receiving input from a plurality of sensors mounted on a see through head mounted display device, the sensors operable to detect at least one of audible and visual actions of one or more subjects in a field of view of the see through head mounted display device;

computing an emotional state of the subject based on the at least one of audible and visual actions, the computing including determining for each of the one or more subjects whether the one or more subjects is a known subject having historical emotional state data and if so, computing the emotional state based at least in part on the emotional historical state data and if not, computing the emotional state data based on unknown subject data, wherein the historical emotional state data comprises emotional state data captured at an earlier time and stored for later use; and

providing at least one of audible and visible feedback to a computing device indicating computed emotional states of the subject.

13. The computer storage device of claim 12 wherein recognizing a behavior comprising any of one of a posture, expression, gesture, audible expression or word in the input from the sensors and comparing posture, expression, gesture, audible expression or word to a set of associated emotional states represented by the behavior to determine a likely emotional state.

14. The computer storage device of claim 12 wherein the instructions further include providing the wearer with a selection of a social scenario, the selection of the social scenario biasing the comparing of the posture, expression, gesture, audible expression or word to associated emotional states.

15. The computer storage device of claim 12 wherein the instructions further include storing computed emotional states indexed to recognized posture, expression, gesture, audible expression or word data for each subject during an interaction.

16. The computer storage device of claim 12 wherein the steps of receiving, continually computing and providing feedback are repeated in sequence in real time.

17. The computer storage device of claim 12, wherein the computing device to which the feedback is provided is the head mounted display device.

18. The computer storage device of claim 12, wherein the computing device to which the feedback is provided is a computing device remote from the head mounted display device.

19. A method for providing emotional state feedback of subjects in a field of view of a see through head mounted display system, comprising:

determining an environment and orientation of the system, the system includes a plurality of sensors and a see-through display device;

identifying a crowd of subjects in the field of view;

receiving input from the plurality of sensors mounted on the see through head mounted display device;

recognizing actions of the crowd of subjects comprising any of one or more of a posture, expression, gesture, audible expression and words in the input for the subjects in the crowd of subject from the sensors and comparing recognized actions to a set of emotional states determined based on recognized actions performed in combination with each other;

computing an emotional state for the crowd of subjects as a whole based on the recognized actions; and

providing feedback to a computing device indicating emotional state of the crowd of subjects.

20. The method of claim 19 further including determining a social scenario based on the environment, location and orientation of the system and wherein the set of emotional states is determined relative to the social scenario.

21. The method of claim 20 wherein providing feedback includes rendering in the display one of:

a specific indication of an emotional state including a visual representation specifying the state, or a positive or negative general indication of a change in emotional state based on tinting the display.

22. The method of claim 21 further including outputting sensor data to an analysis and feedback service and receiving computed emotional states based on the data from the service.

23. The method of claim 19, wherein the step of providing feedback to a computing device comprises the step of providing feedback to the head mounted display device.

24. The method of claim 19, wherein the step of providing feedback to a computing device comprises the step of providing feedback to a computing device remote from the head mounted display device.