Method and system for physiologically modulating action role-playing open world video games and simulations which use gesture and body image sensing control input devices

• US 9,084,933 B1
• Filed: 03/15/2013
• Issued: 07/21/2015
• Est. Priority Date: 06/22/2012
• Status: Active Grant

• Current Assignee: United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration
• Original Assignee: The United States of America As Represented By The Secretary of Agriculture
• Inventors: Pope, Alan T., Stephens, Chad L., Jones, Christopher A.
• Primary Examiner(s): Elisca, Pierre E
• Application Number:
  US13/838,260
• Time in Patent Office: 858 Days
• Field of Search: 463/31, 463/32
• US Class Current: 1/1
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/219 for aiming at specific areas on the display, e.g. light-guns
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle

Abstract

Method for physiologically modulating videogames and simulations includes utilizing input from a motion-sensing video game system and input from a physiological signal acquisition device. The inputs from the physiological signal sensors are utilized to change the response of a user'"'"'s avatar to inputs from the motion-sensing sensors. The motion-sensing system comprises a 3D sensor system having full-body 3D motion capture of a user'"'"'s body. This arrangement encourages health-enhancing physiological self-regulation skills or therapeutic amplification of healthful physiological characteristics. The system provides increased motivation for users to utilize biofeedback as may be desired for treatment of various conditions.

Claims

1. A method of modifying the effects of sensor input from a 3D sensor system having full-body 3D motion capture of a user'"'"'s body on a interactive device to encourage the self-regulation of at least one physiological activity by the user, the interactive device comprising a display area which depicts game or simulation images, an apparatus for receiving at least one input from the sensor system to thus permit the user to control and interact with at least some of the depicted images, the method for modifying the effects comprising the steps of:

measuring at least one physiological activity of the user to obtain at least one physiological signal having a value indicative of the level of the at least one physiological activity;

modifying the effects of the user'"'"'s input on at least some of depicted game or simulation images by modifying the effects of the input from the sensor system in response to changes in the at least one physiological signal, wherein the input is based on a full-body 3D motion capture of a user'"'"'s body.

2. The method of claim 1, wherein

at least one of the images comprises a user'"'"'s avatar that is controlled by the user based on inputs from the 3D sensor system;

at least one of the avatar'"'"'s abilities is modified to thereby modify the effects of the user'"'"'s input on the user'"'"'s avatar.

3. The method of claim 2, wherein the at least one of the avatar'"'"'s abilities comprises a one-handed sword skill and a corresponding amount of damage that one swing of the avatar'"'"'s sword will cause to an at least one enemy or at least one object in the display area of the interactive device, constituting a game or simulation environment.

4. The method of claim 3, wherein the avatar'"'"'s one-handed sword skill and the corresponding amount of damage that one swing of the avatar'"'"'s sword can cause to the enemies or the objects in the display area of the interactive device is inversely proportional to a difference between a player'"'"'s current momentary physiological signal value and a pre-selected target value for the physiological signal.

5. The method of claim 2, wherein the at least one of the avatar'"'"'s abilities comprises an archery skill and a corresponding amount of damage that one arrow fired from the avatar'"'"'s bow will cause to the at least one enemy or the at least one object in the display area of the interactive device, constituting a game or a simulation environment.

6. The method of claim 5, wherein the avatar'"'"'s archery skill and the corresponding amount of damage that one arrow fired from the avatar'"'"'s bow will cause to the enemies or the objects in the display area of the interactive device is inversely proportional to a difference between a player'"'"'s current momentary physiological signal value and a pre-selected target value for the physiological signal.

7. The method of claim 2, wherein the at least one of the avatar'"'"'s abilities comprises an ability to zoom-in on a target when drawing an arrow and a corresponding accuracy of hitting a target with an arrow fired from the avatar'"'"'s bow in the display area of the interactive device, constituting a game or simulation environment.

8. The method of claim 7, wherein the avatar'"'"'s ability to zoom-in on a target when drawing an arrow and the corresponding accuracy of hitting a target in the display area of the interactive device with an arrow fired from the avatar'"'"'s bow is inversely proportional to the difference between the player'"'"'s current momentary physiological signal value and a pre-selected target value for the physiological signal.

9. The method of claim 2, wherein the at least one of the avatar'"'"'s abilities comprises a sneak skill and a corresponding ability to remain out of sight to an at least one enemy when in close proximity to the enemy in the display area of the interactive device, constituting a game or simulation environment.

10. The method of claim 9, wherein the avatar'"'"'s sneak skill and the corresponding ability to remain out of sight to the at least one enemy when in close proximity to the enemy in the display area of the interactive device is inversely proportional to the difference between the player'"'"'s current momentary physiological signal value and a pre-selected target value for the physiological signal.

11. The method of claim 2, wherein the at least one of the avatar'"'"'s abilities comprises a magic destruction skill and a corresponding amount of damage that a stream of damaging fire that the avatar can shoot from their hands can cause to at least one enemy or at least one object in the display area of the interactive device, constituting a game or simulation environment.

12. The method of claim 1, wherein the avatar'"'"'s magic destruction skill and the corresponding amount of damage that the stream of damaging tire that the avatar can shoot from their hands can cause to the enemy or the object in the display area of the interactive device is inversely proportional to a difference between a player'"'"'s current momentary physiological signal value and a pre-selected target value for the physiological signal.

13. The method of claim 1, wherein the step of measuring at least one physiological activity comprises measuring at least one of

an autonomically-mediated physiological activity;

a functional near infrared spectroscopic signals from a brain;

a transcranial Doppler bloodflow signals from a brain;

a brainwave electrical activity, and

at least one facial expression pattern for inferring operator state.

14. The method of claim 13 wherein the autonomically-mediated physiological activity comprises at least one of

a) a skin temperature;

b) a skin conductance;

c) a galvanic skin response;

d) a electrical activity of muscle;

e) a blood flow through the skin;

f) a heart electrical activity;

g) a heart rate;

h) a heart rate variability;

i) a blood pressure;

j) a respiratory rate; and

;

k) any combinations of a)-j).

15. The method of claim 13 wherein the brainwave electrical activity comprises at least one of

at least one event-related electrical brain potential;

and

at least one brainwave frequency band.

16. The method of claim 15, wherein the at least one brainwave frequency band comprising at least one of

a) delta;

b) theta;

c) alpha;

d) sensory-motor rhythm (SMR);

e) beta;

f) gamma, and;

g) any combination of a)-f).

17. The method of claim 1, including utilizing the measured physiological activity of the user to generate an index relating to a user'"'"'s affective state or cognitive state.

18. The method of claim 17, wherein the user'"'"'s affective state or cognitive state comprises at least one of a user'"'"'s engagement/boredom, frustration, meditation, and excitement and wherein the index comprises at least one of an index of at least one of a user'"'"'s state.

19. An apparatus for modifying the effect of a 3D sensor system having full body 3D motion capture of a user on an interactive device, the interactive device comprising a display area, or image-generating feature for depicting images upon the display area, constituting a game or simulation environment, and a feature that permits receiving control input from the 3D sensor system, the control input permitting the user to control and interact with at least some of the depicted images by movement of the user'"'"'s body to provide control input, the modifying apparatus comprising:

a sensor that measures at least one physiological activity of the user;

a converter that converts the at least one measured physiological activity into at least one physiological signal having a value indicative of the level of the at least one physiological activity; and

a program that modifies the user'"'"'s control input to the 3D sensor system in response to changes in the at least one physiological signal, to thus modify the effects of the user'"'"'s input on at least one of the depicted images in response to changes in the at least one physiological signal.

20. The modifying apparatus according to claim 19, wherein the interactive device comprises software for controlling the displayed images, the control input comprises control signals, and wherein the modifying apparatus comprises an arrangement to transform the control signals in response to changes in the at least one physiological signal prior to the control signals being used by the interactive device software.

21. The modifying apparatus according to claim 19, wherein

the interactive device comprises software for controlling the displayed images, the control input comprises control signals, and wherein:

the modifying apparatus comprises an arrangement to change the response of a user'"'"'s avatar within the game to inputs from the motion-sensing sensors in response to changes in the at least one physiological signal.

22. The modifying apparatus of claim 19, wherein

the at least one physiological signal comprises at least one of:

an autonomically-mediated physiological activity, a functional near infrared spectroscopic signals from brain, a transcranial Doppler bloodflow signals from brain, a brainwave electrical activity, and at least one facial expression pattern for inferring operator state.

23. The modifying apparatus of claim 22, wherein the autonomically-mediated physiological activity comprises at least one of:

a) a skin temperature;

b) a skin conductance;

c) a galvanic skin response;

d) an electrical activity of muscle;

e) a blood flow through the skin;

f) a heart electrical activity;

g) a heart rate;

h) a heart rate variability;

i) a blood pressure;

j) a respiratory rate; and

;

k) any combination of a)-j).

24. The modifying apparatus of claim 22, wherein the brainwave electrical activity comprises at least one of

event-related electrical brain potentials;

and

at least one brainwave frequency band.

25. The modifying apparatus of claim 24, wherein:

the at least one brainwave frequency band comprises at least one of;

a) delta;

b) theta;

c) alpha;

d) sensory-motor rhythm (SMR);

e) beta;

f) gamma; and

;

g) any combination of a)-f).