Technique for controlling virtual image generation system using emotional states of user

• US 10,540,004 B2
• Filed: 07/20/2017
• Issued: 01/21/2020
• Est. Priority Date: 07/21/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Sanger, George Alistair, Miller, Samuel A., Devine, Graeme John
• Primary Examiner(s): Cuff, Michael A
• Application Number:
  US15/655,563
• Publication Number:
  US 20180024626A1
• Time in Patent Office: 915 Days
• Field of Search: 463 13
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/822 Strategy games; Role-playing games A63F13/825, A63F13/828 take precedence
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 16/56 having vectorial format
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 40/175 : Static expression

Abstract

A method of operating a virtual image generation system comprises allowing an end user to interact with a three-dimensional environment comprising at least one virtual object, presenting a stimulus to the end user in the context of the three-dimensional environment, sensing at least one biometric parameter of the end user in response to the presentation of the stimulus to the end user, generating biometric data for each of the sensed biometric parameter(s), determining if the end user is in at least one specific emotional state based on the biometric data for the each of the sensed biometric parameter(s), and performing an action discernible to the end user to facilitate a current objective at least partially based on if it is determined that the end user is in the specific emotional state(s).

Claims

1. A method of operating an augmented reality (AR) display system that includes a wearable display device, the method comprising:

presenting an interactive three-dimensional environment to a user through the wearable display device, the interactive three-dimensional environment comprising a real-world scene and at least one virtual object that is generated by the AR display system and presented within the real-world scene;

responsive to presenting a stimulus in the three-dimensional environment, sensing at least one biometric parameter of the user, the at least one biometric parameter sensed using at least one sensor that is attached to the wearable display device;

generating biometric data for each of the at least one sensed biometric parameter;

determining at least one emotional state of the user based on the biometric data for the each of the at least one sensed biometric parameter; and

performing an action through the AR display system based at least partly on determining that the user is in the at least one emotional state, wherein the action includes a modification of the at least one virtual object while it is being presented to the user through the wearable display device, the modification of the at least one virtual object being selected to alter the at least one emotional state of the user.

2. The method of claim 1, further comprising determining the at least one emotional state of the user for a duration, wherein the action is performed through the AR display system based at least partially on determining that the user is in the at least one emotional state for the duration.

3. The method of claim 2, wherein the at least one emotional state comprises a plurality of different emotional states.

4. The method of claim 1, wherein the stimulus comprises a video game, program, musical background piece, reward, virtual object, or alert, and performing the action comprises making the video game, program, musical background piece, reward, virtual object, or alert more available or less available to the user.

5. The method of claim 1, wherein the stimulus comprises a feature of a video game, program, musical background piece, reward, virtual object, or alert, and performing the action comprises making the video game, program, musical background piece, reward, or alert more available or less available to the user.

6. The method of claim 1, wherein the stimulus comprises a strategy video game, and performing the action comprises making a game move using the at least one emotional state of the user.

7. The method of claim 6, wherein the strategy video game is a playing card video game having one or more simulated players that play against the user, the at least one emotional state is one that indicates whether or not the user is bluffing, and performing the action comprises the one or more simulated players playing cards against the user using knowledge of whether or not the user is bluffing.

8. The method of claim 7, wherein the playing card video game is a poker video game.

9. The method of claim 1, wherein the stimulus comprises a question presented to the user, and performing the action comprises informing the user whether or not the user is lying using the at least one emotional state.

10. The method of claim 1, wherein the stimulus comprises a plurality of options presented to the end user, and performing the action comprises informing the user which option the user prefers using the at least one emotional state.

11. The method of claim 1, wherein the stimulus comprises a simulated face, and performing the action comprises determining if the end user is about to laugh using the specific emotional state determination, and modifying the simulated face to make the user laugh.

12. The method of claim 1, wherein the at least one biometric parameter of the user is sensed at a plurality of different times in response to the presenting the stimulus, and the biometric data is generated at these different times.

13. The method of claim 1, wherein the stimulus is visually presented to the user through the wearable display device.

14. The method of claim 1, wherein the at least one sensed biometric parameter comprises a plurality of different sensed biometric parameters.

15. The method of claim 14, wherein determining the at least one specific emotional state of the user comprises performing a pattern recognition analysis on the generated biometric data.

16. The method of claim 1, wherein the at least one sensed biometric parameter comprises at least one facial expression.

17. The method of claim 16, wherein the least one facial expression is one or both of an attitude of the mouth and crow'"'"'s feet around the eyes of the end user.

18. The method of claim 17, wherein the at least one emotional state comprises happiness.

19. The method of claim 1, wherein the at least one sensed biometric parameter comprises at least one of a facial expression, hunching of shoulders, respiration rate, heart rate, body temperature, blood pressure, frequency and/or location of hand movements, frequency and/or location of body twitches, and elapsed time between eye movements.

20. The method of claim 1, wherein the at least one sensed biometric parameter comprises at least one micro-expression.

21. The method of claim 1, wherein the at least one emotional state comprises at least one of anger, contempt, disgust, fear, happiness, sadness, surprise, confusion, shame, attentiveness, exhaustion, relaxation, frustration, boredom, embarrassment.

22. The method of claim 1, wherein determining at least one emotional state of the user comprises comparing the generated biometric data to reference biometric data correlated to the at least one emotional state.

23. The method of claim 22, wherein the generated biometric data for one of the at least one sensed biometric parameter is a biometric scalar data value, the reference biometric data comprises a reference biometric value range, and comparing the generated biometric data to the reference biometric data comprises determining whether the biometric scalar data value falls within the reference biometric value range.

24. The method of claim 22, wherein the generated biometric data for one of the at least one sensed biometric parameter is a biometric multi-dimensional data vector, the reference biometric data comprises a reference biometric multi-dimensional data vector, and comparing the generated biometric data to the reference biometric data comprises performing a correlation function between the generated biometric multi-dimensional data vector and the reference biometric multi-dimensional data vector.

25. The method of claim 22, wherein determining the at least one emotional state of the user further comprises retrieving the reference biometric data from a custom emotional state profile of the end user.

26. The method of claim 1, wherein presenting the three-dimensional environment to the user through the wearable display device comprises rendering a plurality of synthetic image frames of a three-dimensional environment, and sequentially displaying the synthetic image frames to the user through wearable display device.

27. The method of claim 26, wherein the synthetic image frames are projected from a transparent display surface in the field of view of the user via a frame structure mounted to the head of the user.

28. The method of claim 26, wherein the synthetic image frames are superimposed over a real scene visualized by the user.

29. The method of claim 1, wherein the AR display system also includes a speaker coupled to the wearable display device, and

wherein the stimulus is aurally presented through the speaker.

30. The method of claim 1, wherein the AR display system also includes a mechanical actuator, and

wherein the stimulus is tactilely presented through the mechanical actuator.

31. The method of claim 1, wherein the at least one sensor comprises a rearward facing camera coupled to the wearable display device, and

wherein the at least one biometric parameter comprises a facial movement of the user.

32. The method of claim 1, wherein the at least one sensor comprises a microphone coupled to the wearable display device.

33. The method of claim 1, wherein the at least one sensor comprises an accelerometer coupled to a body part of the user, and

wherein the at least one biometric parameter comprises a movement of the body part of the user.

34. The method of claim 1, wherein when the determined at least one emotional state of the user is happiness or excitement, the modification of the at least one virtual object increases the presented stimulus.

35. The method of claim 1, wherein when the determined at least one emotional state of the user is frustration or boredom, the modification of the at least one virtual object decreases the presented stimulus.