SYSTEM AND METHOD FOR ENHANCED TRAINING USING A VIRTUAL REALITY ENVIRONMENT AND BIO-SIGNAL DATA

• US 20160077547A1
• Filed: 09/11/2015
• Published: 03/17/2016
• Est. Priority Date: 09/11/2014
• Status: Active Grant

• Current Assignee: InteraXon Inc.
• Original Assignee: InteraXon Inc.
• Inventors: TADICH, PAUL V., Aimone, Christopher Allen, Coleman, Trevor, Garten, Ariel Stephanie, Vidyarthi, Kapil Jay Mishra, Pino, Locillo (Lou) Giuseppe, CHABIOR, MICHAEL APOLLO, Baranowski, Paul Harrison, Rupsingh, Raul Rajiv, ASHBY, MADELINE
• Granted Patent: US 10,120,413 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A61B 2503/12 Healthy persons not otherwise provided for, e.g. subjects of a marketing survey
• CPC Class Codes: A61B 2560/0475 Special features of memory means, e.g. removable memory cards
• CPC Class Codes: A61B 2560/0493 controlled by voice
• CPC Class Codes: A61B 2562/0219 Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
• CPC Class Codes: A61B 2562/125 characterised by the manufacture of electrodes
• CPC Class Codes: A61B 2562/164 the sensor is mounted in or on a conformable substrate or carrier
• CPC Class Codes: A61B 2562/166 the sensor is mounted on a specially adapted printed circuit board
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 5/0006 ECG or EEG signals
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/0024 for multiple sensor units attached to the patient, e.g. using a body or personal area network
• CPC Class Codes: A61B 5/1114 Tracking parts of the body
• CPC Class Codes: A61B 5/16 Devices for psychotechnics using teaching or educational appliances G09B1/00 - G09B7/00; Testing reaction times ; Devices for evaluating the psychological state
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/318 Heart-related electrical modalities, e.g. electrocardiography [ECG]
• CPC Class Codes: A61B 5/369 Electroencephalography [EEG] devices for psychotechnics A61B5/16
• CPC Class Codes: A61B 5/378 Visual stimuli
• CPC Class Codes: A61B 5/38 Acoustic or auditory stimuli
• CPC Class Codes: A61B 5/486 Bio-feedback A61B5/375 takes precedence
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/6814 : Head
• CPC Class Codes: A61B 5/6821 : Eye
• CPC Class Codes: A61B 5/6831 : Straps, bands or harnesses
• CPC Class Codes: A61B 5/7203 : for noise prevention, reduction or removal
• CPC Class Codes: A61B 5/7264 : Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/744 : Displaying an avatar, e.g. an animated cartoon character
• CPC Class Codes: A61B 5/7445 : Display arrangements, e.g. multiple display units
• CPC Class Codes: G06F 1/163 : Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 2203/011 : Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 : Head tracking input arrangements
• CPC Class Codes: G06F 3/015 : Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/016 : Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G16H 40/67 : for remote operation

Abstract

A training apparatus has an input device and a wearable computing device with a bio-signal sensor and a display to provide an interactive virtual reality (“VR”) environment for a user. The bio-signal sensor receives bio-signal data from the user. The user interacts with content that is presented in the VR environment. The user interactions and bio-signal data are scored with a user state score and a performance scored. Feedback is given to the user based on the scores in furtherance of training. The feedback may update the VR environment and may trigger additional VR events to continue training.

Claims

1. A training apparatus comprising:

an input device and a wearable computing device with a bio-signal sensor and a display to provide an interactive virtual reality (“

VR”

) environment for a user, the bio-signal sensor receives bio-signal data from the user, the bio-signal sensor comprising a brainwave sensor;

the computing device having or in communication with a processor configured to;

as part of the interactive VR environment, present content on the display where the content has a VR event, desired user states, and desired effects;

receive user manual inputs from the input device which have effects in the interactive VR environment including during the VR event;

receive the bio-signal data of the user from the bio-signal sensor during the VR event;

process the bio-signal data to determine user states of the user, including brain states, during the VR event, the user states are processed using a user profile stored in a data storage device accessible by the processor and the user states include brain states;

determine a user state score by comparing the user states of the user to the desired user states during the course of the VR event;

determine a performance score by comparing the effects to the desired effects during the course of the VR event; and

provide feedback to the user wherein the feedback is based on a combination of the user state score and the performance score.

2. The apparatus of claim 1 where the wearable computing device further comprises

an inertial sensor and the bio-signal sensor further comprises a facial bio-signal sensor, and where the bio-signal data further comprises data from the facial bio-signal sensor and the inertial sensor;

the computing device is further configured to;

receive the bio-signal data from the facial bio-signal sensor and the inertial sensor for a user head and eye gaze direction

wherein the user states further comprises the user head and eye gaze direction, and the desired user states further comprises a desired user head and eye gaze direction.

3. The apparatus of claim 1 wherein the brain states comprises one or more of ability of operator to learn;

prediction error; and

emotional state leading to impaired thinking.

4. The apparatus of claim 1 further comprising presenting the user state score and the performance score synchronized with the content and the VR event to assist the user to better attain the desired user states and desired manual inputs on the input device.

5. The apparatus of claim 4 where

the computing device is further configured to:

revise the content in response to the feedback provided to the user where the user is further trained on the revised content.

6. The apparatus of claim 4 wherein the user state score further comprises failure brain states.

7. The apparatus of claim 1 where the display is a stereoscopic display.

8. The apparatus of claim 1 further comprising

a second display for presenting the content and a visual representation of the user states and manual inputs of the user in real time.

9. The apparatus of claim 1 wherein the computing device having or in communication with the processor is further configured to:

provide real time or near real time feedback to the user during the presentation of the content.

10. The apparatus of claim 1, wherein the VR event is associated with event time data and a portion of bio-signal data is associated with bio-signal time data corresponding to the event time data, wherein the processor is configured to identify a portion of the bio-signal data based on the event time data and process the portion of the bio-signal data to determine the user states during the VR event, the bio-signal time data synchronized to the event time data.

11. The apparatus of claim, wherein the VR event is associated with event time data and the bio-signal data is associated with bio-signal time data, and wherein the processor is configured to identify a time interval based on an expected response time for the VR event and the event time data, identify a portion of the bio-signal data based on the time interval and the bio-signal time data, and process the portion of the bio-signal data to determine the user states during the VR event, the bio-signal time data synchronized to the event time data.

12. A training method implemented using an input device and a wearable computing device having or in communication with a processor, a bio-signal sensor and a display to provide an interactive virtual reality (“

VR”

) environment for a user, the bio-signal sensor receives bio-signal data from the user, the bio-signal sensor comprising a brainwave sensor;

the training method comprising;

as part of the interactive VR environment, presenting content on the display where the content has a VR event, desired user states, and desired effects;

receiving user manual inputs from the input device which have effects in the interactive VR environment including during the VR event;

receiving the bio-signal data of the user from the bio-signal sensor during the VR event;

processing the bio-signal data to determine user states of the user, including brain states, during the VR event, the user states are processed using a user profile stored in a data storage device accessible by the processor and the user states include brain states;

determining a user state score by comparing the user states of the user to the desired user states during the course of the VR event;

determining a performance score by comparing the effects to the desired effects during the course of the VR event; and

providing feedback to the user wherein the feedback is based on a combination of the user state score and the performance score.

13. The training method of claim 12, where the wearable computing device further comprises an inertial sensor and the bio-signal sensor further comprises a facial bio-signal sensor, and where the bio-signal data further comprises data from the facial bio-signal sensor and the inertial sensor;

the method further comprising

receiving the bio-signal data from the facial bio-signal sensor and the inertial sensor for a user head and eye gaze direction;

wherein the user states further comprises the user head and eye gaze direction, and the desired user states further comprises a desired user head and eye gaze direction.

14. The training method of claim 12, wherein the brain states comprises one or more of ability of operator to learn;

prediction error; and

emotional state leading to impaired thinking.

15. The training method of claim 12, further comprising post presenting the user state score and the performance score synchronized with the content and the VR event to assist the user to better attain the desired user states and desired manual inputs on the input device.

16. The training method of claim 15, further comprising

revising the content in response to the feedback provided to the user where the user is further trained on the revised content.

17. The training method of claim 15, wherein the user state score further comprises failure brain states.

18. The training method of claim 12, where the display is a stereoscopic display.

19. The training method of claim 15, further comprising

presenting the content and the user states and manual inputs of the user in real time on a second display.

20. The training method of claim 15, further comprising

providing real time feedback to the user during the presentation of the content.