Method and system for physiologically modulating videogames and simulations which use gesture and body image sensing control input devices

• US 9,630,093 B2
• Filed: 03/15/2013
• Issued: 04/25/2017
• Est. Priority Date: 06/22/2012
• Status: Active Grant

• Current Assignee: United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration
• Original Assignee: United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration
• Inventors: Pope, Alan T., Stephens, Chad L., Habowski, Tyler
• Primary Examiner(s): D'Agostino, Paul A
• Assistant Examiner(s): GRAY, BRANDON RAMON
• Application Number:
  US13/836,609
• Publication Number:
  US 20150258429A1
• Time in Patent Office: 1,502 Days
• Field of Search: 463 29, 463 30, 463 31, 463 32
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/219 for aiming at specific areas on the display, e.g. light-guns
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle

Abstract

Method for physiologically modulating videogames and simulations includes utilizing input from a motion-sensing video game system and input from a physiological signal acquisition device. The inputs from the physiological signal sensors are utilized to change the response of a user'"'"'s avatar to inputs from the motion-sensing sensors. The motion-sensing system comprises a 3D sensor system having full-body 3D motion capture of a user'"'"'s body. This arrangement encourages health-enhancing physiological self-regulation skills or therapeutic amplification of healthful physiological characteristics. The system provides increased motivation for users to utilize biofeedback as may be desired for treatment of various conditions.

Claims

1. A method of modifying the effects of sensor input from a first sensor system having full-body 3D motion capture of a user'"'"'s body on an interactive device to encourage self-regulation of at least one physiological activity by the user in a biofeedback process, the interactive device comprising a display area which depicts game or simulation images, an apparatus for receiving at least one input from the first sensor system to thus permit the user to control and interact with at least some of the depicted images, the method for modifying the effects comprising the steps of:

utilizing the full-body 3D motion capture of the first sensor system to provide user input corresponding to at least one of a user'"'"'s position and movement;

measuring at least one physiological activity of the user utilizing a second sensor comprising a physiological signal sensor to obtain at least one physiological signal indicative of the level of the at least one physiological activity, wherein the at least one physiological activity comprises a physiological process about which a user would not normally be aware;

modifying the effects of the user'"'"'s input on at least some of depicted game or simulation images by modifying the effects of the input from the first sensor system in response to changes in the at least one physiological signal from the second sensor to provide biofeedback enabling a user to self-regulate physiological activity.

2. The method of claim 1, wherein

at least one of the images comprises a user'"'"'s avatar that is controlled by the user based on inputs from the 3D sensor system;

at least one of the avatar'"'"'s abilities is modified to thereby modify the effects of the user'"'"'s input on the user'"'"'s avatar.

3. The method of claim 2, wherein the at least one of the avatar'"'"'s abilities comprises a speed of the avatar'"'"'s movement.

4. The method of claim 3, wherein the speed of the avatar'"'"'s movements is inversely proportional or directly proportional to a difference between a user'"'"'s current momentary physiological signal from the second sensor and a preselected target value for the physiological signal.

5. The method of claim 2, wherein the at least ne of the avatar'"'"'s abilities comprises a size of the avatar.

6. The method of claim 5, wherein the size of the avatar is inversely proportional or directly proportional to a difference between a user'"'"'s current momentary physiological signal from the second sensor and a preselected target value for the physiological signal.

7. The method of claim 2, wherein the at least one of the avatar'"'"'s abilities comprises the movement of the avatar through the display area of the interactive device in both time and space relative to that of at least one enemy.

8. The method of claim 7, wherein the movement of the avatar through the display area of the interactive device in both time and space relative to that of the at least one enemy is made inversely proportional to the difference between the player'"'"'s current momentary physiological signal from the second sensor and a pre-selected target value for the physiological signal.

9. The method of claim 2, wherein the at least one of the avatar'"'"'s abilities comprises a steadiness of the avatar'"'"'s hand or arm.

10. The method of claim 9, wherein the steadiness of the avatar'"'"'s hand or arm is inversely proportional to the difference between the player'"'"'s current momentary physiological signal from the second sensor and a pre-selected target value for the physiological signal.

11. The method of claim 2, wherein the at least one of the avatar'"'"'s abilities comprises an ability of the avatar to grasp an object in the display area of the interactive device.

12. The method of claim 11, wherein the ability of an avatar to grasp an object in the display area of the interactive device is inversely proportional to the difference between the player'"'"'s current momentary physiological signal from die second sensor and a pre-selected target value for the physiological signal.

13. The method of claim 2, wherein the at least one of the avatar'"'"'s abilities comprises a degree of the effect of avatar movements upon objects in the display area of the interactive device.

14. The method of claim 13, wherein the degree of the effect of avatar movements upon objects in the display area of the interactive device is inversely proportional to the difference between the player'"'"'s current momentary physiological signal from the second sensor and a pre-selected target value for the physiological signal.

15. The method of claim 2, wherein the at least one of the avatar'"'"'s abilities comprises an avatar'"'"'s opacity and the probability of the avatar being visible to other characters in the display area of the interactive device.

16. The method of claim 15, wherein the avatar'"'"'s opacity and the probability of the avatar being visible to other characters in the display area of the interactive device is proportional to the difference between the player'"'"'s current momentary physiological signal from the second sensor and a pre-selected target value for the physiological signal.

17. The method of claim 2, wherein the at least one of the avatar'"'"'s abilities comprises an intensity of a light source emanating from a player'"'"'s avatar or an object the avatar is wielding.

18. The method of claim 17, wherein the intensity of a light source emanating from a player'"'"'s avatar or an object the avatar is wielding is inversely proportional to the difference between the player'"'"'s current momentary physiological signal from the second sensor and a pre-selected target value for the physiological signal.

19. The method of claim 2, wherein the at least one of the avatar'"'"'s abilities comprises a speed with which an avatar can reload weapons.

20. The method of claim 19, wherein the speed with which an avatar can reload weapons is inversely proportional to the difference between the player'"'"'s current momentary physiological signal from the second sensor and a pre-selected target value for the physiological signal.

21. The method of claim 2, wherein the at least one of the avatar'"'"'s abilities comprises an amplitude of the avatar'"'"'s voice-based weapon and a corresponding amount of damage the voice-based weapon can cause to enemies and objects in the display area of the interactive device, constituting a game or simulation environment.

22. The method of claim 21, wherein the amplitude of the avatar'"'"'s voice-based weapon and the corresponding amount of damage it can cause to enemies and objects in the display area of the interactive device are inversely proportional to the difference between the player'"'"'s current momentary physiological signal from the second sensor and a pre-selected target value for the physiological signal.

23. The method of claim 1, wherein the step of measuring at least one physiological activity comprises measuring at least one of:

an autonomically-mediated physiological activity;

a functional near infrared spectroscopic signal from a brain;

a transcranial Doppler bloodflow signal from a brain; and

a brainwave electrical activity.

24. The method of claim 23, wherein the autonomically-mediated physiological activity comprises at least one of:

a) a skin temperature;

b) a skin conductance;

c) a galvanic skin response;

d) a electrical activity of muscle;

e) a blood flow through the skin;

f) a heart electrical activity;

g) a heart rate;

h) a heart rate variability;

i) a blood pressure;

j) a respiratory rate; and

k) any combination of a)-j).

25. The method of claim 23, wherein the brainwave electrical activity comprises at least one of:

at least one event-related electrical brain potential; and

at least one brainwave frequency band.

26. The method of claim 25, wherein at least one brainwave frequency band comprising at least one of:

a) delta;

b) theta;

c) alpha;

d) sensory-motor rhythm (SMR);

e) beta;

f) gamma, and;

g) any combination of a)-f).

27. The method of claim 1, further comprising utilizing the measured physiological activity of the user from the second sensor to generate an index relating to a user'"'"'s affective state or cognitive state.

28. The method of claim 27, wherein the user'"'"'s affective state or cognitive state comprises at least one of a user'"'"'s engagement/boredom, frustration, meditation, and excitement and wherein the index comprises at least one of an index of at least one of a user'"'"'s state.

29. An apparatus for modifying the effect of a first sensor system having full body 3D motion capture of a user on an interactive device, the interactive device comprising a display area, or image-generating feature for depicting images upon the display area, constituting a game or simulation environment, and a feature that permits receiving control input from the first sensor system, the control input permitting the user to control and interact with at least some of the depicted images by movement of the user'"'"'s body to provide control input, the modifying apparatus comprising:

a second sensor that measures at least one physiological activity of the user, wherein the at least one physiological activity comprises a physiological process about which a user would not normally be aware;

a converter that converts the at least one measured physiological activity into at least one physiological signal indicative of the level of the at least one physiological activity; and

a program that modifies the user'"'"'s control input to the first sensor system in response to changes in the at least one physiological signal from the second sensor, to thus modify the effects of the user'"'"'s input on at least one of the depicted images in response to changes in the at least one physiological signal to provide biofeedback enabling a user to self-regulate the physiological activity.

30. The modifying apparatus according to claim 29, wherein the interactive device comprises software for controlling the displayed images, the control input comprises control signals, and wherein the modifying apparatus comprises an arrangement to transform the control signals in response to changes in the at least one physiological signal measured by the second sensor prior to the control signals being used by the interactive device software.

31. The modifying apparatus of claim 29 wherein the at least one physiological signal comprises at least one of:

an autonomically-mediated physiological activity, a functional near infrared spectroscopic signal from brain, a transcranial Doppler bloodflow signal from brain, and a brainwave electrical activity.

32. The modifying apparatus of claim 31 wherein the autonomically-mediated physiological activity comprises at least one of

a) a skin temperature;

b) a skin conductance;

c) a galvanic skin response;

d) an electrical activity of muscle;

e) a blood flow through the skin;

f) a heart electrical activity;

g) a heart rate;

h) a heart rate variability;

i) a blood pressure;

j) a respiratory rate; and

k) any combination of a)-j).

33. The modifying apparatus of claim 31 wherein the brainwave electrical activity comprises at least one of:

event-related electrical brain potentials; and

at least one brainwave frequency band.

34. The modifying apparatus of cairn 33, wherein the at least one brainwave frequency band comprises at least one of:

a) delta;

b) theta;

c) alpha;

d) sensory-motor rhythm (SMR);

e) beta;

f) gamma; and

;

g) any combination of a)-f).