TECHNIQUE FOR CONTROLLING VIRTUAL IMAGE GENERATION SYSTEM USING EMOTIONAL STATES OF USER

• US 20180024626A1
• Filed: 07/20/2017
• Published: 01/25/2018
• Est. Priority Date: 07/21/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Sanger, George Alistair, Miller, Samuel A., Devine, Graeme John
• Granted Patent: US 10,540,004 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/822 Strategy games; Role-playing games A63F13/825, A63F13/828 take precedence
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 16/56 having vectorial format
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 40/175 : Static expression

Abstract

A method of operating a virtual image generation system comprises allowing an end user to interact with a three-dimensional environment comprising at least one virtual object, presenting a stimulus to the end user in the context of the three-dimensional environment, sensing at least one biometric parameter of the end user in response to the presentation of the stimulus to the end user, generating biometric data for each of the sensed biometric parameter(s), determining if the end user is in at least one specific emotional state based on the biometric data for the each of the sensed biometric parameter(s), and performing an action discernible to the end user to facilitate a current objective at least partially based on if it is determined that the end user is in the specific emotional state(s).

Claims

1. A method of operating a virtual image generation system, comprising:

allowing an end user to interact with a three-dimensional environment comprising at least one virtual object;

presenting a stimulus to the end user in the context of the three-dimensional environment;

sensing at least one biometric parameter of the end user in response to the presentation of the stimulus to the end user;

generating biometric data for each of the at least one sensed biometric parameter;

determining if the end user is in at least one specific emotional state based on the biometric data for the each of the at least one sensed biometric parameter; and

performing an action discernible to the end user to facilitate a current objective at least partially based on if it is determined that the end user is in the at least one specific emotional state.

2. The method of claim 1, wherein the current objective is to evoke a desired emotional state in the end user for a desired duration, or to not evoke an undesired emotional state in the end user for a desired duration, or to evoke a plurality of different desired emotional states in the end user for a desired duration.

3. The method of claim 2, wherein the desired duration is for one time in a relevant period of time, a plurality of times in a relevant period of time, or continuously over a relevant period of time.

4. The method of claim 2, wherein the current objective is to evoke a desired emotional state in the end user for a desired duration, the at least one specific emotional state is consistent with the desired emotional state, the end user is determined to be in the at least one specific emotional state for the desired duration, and performing the action comprises making the stimulus more available to the end user.

5. The method of claim 2, wherein the current objective is to evoke a desired emotional state in the end user for a desired duration, the at least one specific emotional state is consistent with the desired emotional state, the end user is determined to not be in the at least one specific emotional state for the desired duration, and performing the action comprises making the stimulus less available to the end user.

6. The method of claim 2, wherein the current objective is to not evoke an undesired emotional state in the end user for a desired duration, the at least one specific emotional state is consistent with the undesired emotional state, the end user is determined to not be in the at least one specific emotional state for the desired duration, and performing the action comprises making the stimulus more available to the end user.

7. The method of claim 2, wherein the current objective is to not evoke an undesired emotional state in the end user for a desired duration, the at least one specific emotional state is consistent with the undesired emotional state, the end user is determined to be in the at least one specific emotional state for the desired duration, and performing the action comprises making the stimulus less available to the end user.

8. The method of claim 2, wherein the at least one specific emotional state comprises a plurality of different emotional states.

9. The method of claim 8, wherein the current objective is to evoke a plurality of different desired emotional states in the end user for a desired duration, the plurality of different emotional states is consistent with the plurality of desired emotional states, the end user is determined to be in the plurality of different emotional states for the desired duration, and performing the action comprises making the stimulus more available to the end user.

10. The method of claim 8, wherein the current objective is to evoke a plurality of different desired emotional states in the end user for a desired duration, the plurality of different emotional states is consistent with the plurality of desired emotional states, the end user is determined to not be in the plurality of different emotional states for the desired duration, and performing the action comprises making the stimulus less available to the end user.

11. The method of claim 2, wherein the stimulus takes the form of a video game, program, musical background piece, reward, virtual object, or alert, and performing the action comprises making the video game, program, musical background piece, reward, virtual object, or alert more available or less available to the end user.

12. The method of claim 2, wherein the stimulus takes the form of a feature of a video game, program, musical background piece, reward, virtual object, or alert, and performing the action comprises making the video game, program, musical background piece, reward, or alert more available or less available to the end user.

13. The method of claim 1, wherein the stimulus takes the form of a strategy video game, the current objective is to beat the end user at the strategy video game, and performing the action comprises making a game move using the specific emotional state determination against the end user.

14. The method of claim 13, wherein the strategy video game is a playing card video game having one or more simulated players that play against the end user, the at least one specific emotional state is one that indicates whether or not the end user is bluffing, and performing the action comprises the one or more simulated players playing cards against the end user using knowledge of whether or not the end user is bluffing.

15. The method of claim 14, wherein the playing card video game is a poker video game.

16. The method of claim 1, wherein the stimulus takes the form of a question presented to the end user, the current objective is to determine whether or not the end user is lying in response to the question, and performing the action comprises informing the end user whether or not the end user is lying using the specific emotional state determination.

17. The method of claim 1, wherein the stimulus takes the form of a plurality of options presented to the end user, the current objective is to determine which one of the plurality of objects the end user is thinking of, and performing the action comprises informing the end user which object the end user is thinking of using the specific emotional state determination.

18. The method of claim 1, wherein the stimulus takes the form of a simulated face, the current objective is to make the end user laugh, and performing the action comprises determining if the end user is about to laugh using the specific emotional state determination, and modifying the simulated face to make the end user laugh.

19. The method of claim 1, wherein the at least one biometric parameter of the end user is sensed at plurality of different times in response to the presentation of the stimulus, and the biometric data is generated at these different times.

20. The method of claim 1, wherein the stimulus is visually presented or aurally presented to the end user.

21. The method of claim 1, wherein the at least one sensed biometric parameter comprises a plurality of different sensed biometric parameters.

22. The method of claim 21, wherein determining if the end user is in the at least one specific emotional state comprises performing a pattern recognition analysis on the generated biometric data.

23. The method of claim 1, wherein the at least one sensed biometric parameter comprises at least one facial expression.

24. The method of claim 23, wherein the least one facial expression is one or both of an attitude of the mouth and crow'"'"'s feet around the eyes of the end user.

25. The method of claim 24, wherein the at least one specific emotional state comprises happiness.

26. The method of claim 1, wherein the at least one sensed biometric parameter comprises at least one of a facial expression, hunching of shoulders, respiration rate, heart rate, body temperature, blood pressure, frequency and/or location of hand movements, frequency and/or location of body twitches, and elapsed time between eye movements.

27. The method of claim 1, wherein the at least one sensed biometric parameter comprises at least one micro-expression.

28. The method of claim 1, wherein the at least one specific emotional state comprises at least one of anger, contempt, disgust, fear, happiness, sadness, surprise, confusion, shame, attentiveness, exhaustion, relaxation, frustration, boredom, embarrassment.

29. The method of claim 1, wherein determining if the end user is in the at least one specific emotional state comprises comparing the generated biometric data to reference biometric data correlated to the at least one specific emotional state.

30. The method of claim 29, wherein the generated biometric data for one of the at least one sensed biometric parameter is a biometric scalar data value, the reference biometric data comprises a reference biometric value range, and comparing the generated biometric data to the reference biometric data comprises determining whether the biometric scalar data value falls within the reference biometric value range.

31. The method of claim 29, wherein the generated biometric data for one of the at least one sensed biometric parameter is a biometric multi-dimensional data vector, the reference biometric data comprises a reference biometric multi-dimensional data vector, and comparing the generated biometric data to the reference biometric data comprises performing a correlation function between the generated biometric multi-dimensional data vector and the reference biometric multi-dimensional data vector.

32. The method of claim 29, wherein determining if the end user is in the at least one specific emotional state further comprises retrieving the reference biometric data from a custom emotional state profile of the end user.

33. The method of claim 1, wherein allowing the end user to interact with the three-dimensional environment comprises rendering a plurality of synthetic image frames of a three-dimensional environment, and sequentially displaying the synthetic image frames to the end user.

34. The method of claim 33, wherein the synthetic image frames are projected from a transparent display surface in the field of view of the end user via a frame structure mounted to the head of the end user.

35. The method of claim 33, wherein the synthetic image frames are superimposed over a real scene visualized by the end user.

36. A method of operating a virtual image generation system, comprising:

allowing an end user to interact with a three-dimensional environment comprising at least one virtual object;

presenting a stimulus to the end user in the context of the three-dimensional environment;

sensing at least one sensed biometric parameter of the end user in response to the presented stimulus;

generating biometric data for each of the at least one sensed biometric parameter;

correlating the generated biometric data to a specific emotional state of the end user;

deriving reference biometric data from the correlated biometric data; and

storing the reference biometric data in correlation with the specific emotional state in a custom profile of the end user.

37-58. -58. (canceled)