System and method for training of state-classifiers

• US 10,192,173 B2
• Filed: 04/18/2017
• Issued: 01/29/2019
• Est. Priority Date: 07/02/2010
• Status: Active Grant

• Current Assignee: United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration
• Original Assignee: United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration
• Inventors: Stephens, Chad L., Harrivel, Angela R., Pope, Alan T., Prinzel, III, Lawrence J
• Primary Examiner(s): Fletcher, Jerry-Daryl
• Application Number:
  US15/490,130
• Publication Number:
  US 20170220956A1
• Time in Patent Office: 651 Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/67 adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

Method and systems are disclosed for training state-classifiers for classification of cognitive state. A set of multimodal signals indicating physiological responses of an operator are sampled over a time period. A depiction of operation by the operator during the time period is displayed. In response to user input selecting a cognitive state for a portion of the time period, the one or more state-classifiers are trained. In training the state-classifiers, the set of multimodal signals sampled in the portion of the time period are used as input to the one or more state-classifiers and the selected one of the set of cognitive states is used as a target result to be indicated by the one or more state-classifiers.

Claims

1. A system, comprising:

a set of sensors configured to provide a set of multimodal signals indicating physiological responses of an operator, of a vehicle or device, to stimuli in a first time period;

a processing circuit configured to train one or more state-classifiers to map the set of multimodal signals to a set of cognitive states; and

a display configured to depict operation of the vehicle or device by the operator in the first time period;

a data storage circuit coupled to the processing circuit and configured to store the one or more state-classifiers;

a second processing circuit coupled to the data storage and configured to determine a cognitive state of an operator using the one or more state-classifiers and the set of multimodal signals as input to the one or more state-classifiers;

a third processing circuit configured to perform an action specified in a settings file, stored in the data storage, in response to the determined cognitive state of the operator satisfying a set of criteria specified in the settings file;

wherein the action specified in the settings file includes at least one of a set of actions including adjusting operation of the vehicle or device, providing an alert to the operator, and sending an alert message; and

wherein the processing circuit is further configured to, in response to user input selecting one of the set of cognitive states and a portion of the first time period, perform the training of the one or more state-classifiers using the set of multimodal signals sampled in the portion of the time period as input to the one or more state-classifiers and the selected one of the set of cognitive states as a target result to be mapped to by the one or more state-classifiers.

2. The system of claim 1, further comprising a second display configured to provide the stimuli configured to induce a set of cognitive states in the operator.

3. The system of claim 1, further comprising a circuit coupled to the set of sensors and configured to sample outputs of the set of sensors to produce the set of multimodal signals.

4. The system of claim 1, wherein the display is configured to provide a graphical user interface for selection of the portion of the time period and selection of the one of the set of cognitive states.

5. The system of claim 1, wherein the third processing circuit is configured to:

determine a level of confidence in the cognitive state determined by second processing circuit; and

in response to the level of confidence being less than a threshold level specified in the set of criteria, cause the first-mentioned processing circuit to retrain the one or more state-classifiers.

6. The system of claim 1, wherein

in response to a first user input, the third processing circuit is configured to provide automated feedback to the operator in further response to the determined cognitive state of the operator being a performance limiting cognitive state;

and

in response to a second user input, the third processing circuit is configured to disable the automated feedback to the operator.

7. The system of claim 1, wherein the third processing circuit is configured to trigger an audible alert or engage an autonomous driving system of the vehicle in response to the determined cognitive state of the operator indicating the operator is in an inattentive or unresponsive cognitive state.

8. A method, comprising:

sampling a set of multimodal signals indicating physiological responses of an operator of a vehicle or device over a time period;

retrieving one or more state-classifiers from a data storage, wherein the one or more state-classifiers are configured to map the set of multimodal signals to a set of cognitive states;

determining respective probabilities of the operator being in the set of cognitive states using the sampled set of multimodal signals as input to the one or more state-classifiers; and

in response to the determined respective probabilities satisfying a set of criteria stored in a settings file performing one or more of a set of actions including adjusting operation of the vehicle or device, providing an alert to the operator, or sending an alert message.

9. The method of claim 8, wherein the performing of one or more of the set of actions provides an audible alert to the operator or engages an autonomous driving system of the vehicle and is performed in response to the determined respective probabilities indicating the operator is in an inattentive or unresponsive cognitive state.

10. The method of claim 8, wherein the performing of one or more of the set of actions prevents operation of the vehicle and is performed in response to the determined respective probabilities indicating the operator is in a cognitive state indicative of the operator being under an influence of alcohol.

11. The method of claim 8, wherein the performing of one or more of the set of actions disables a radio or stereo system and is performed in response to the determined respective probabilities indicating the operator is in an inattentive cognitive state.

12. The method of claim 8, wherein the performing of one or more of the set of actions sends an alert message in response to the determined respective probabilities indicating the operator is in a particular cognitive state specified in the settings file.

13. A method, comprising:

providing a simulation environment including visual, audible, or tactile stimuli, or a combination thereof, configured to induce a target cognitive state in an operator;

sampling a set of multimodal signals indicating physiological responses of the operator of a vehicle or device over a lime period;

retrieving one or more state-classifiers from a data storage, wherein the one or more state-classifiers are configured to map the set of multimodal signals to a set of cognitive states, the set of cognitive states including the target state;

determining respective probabilities of the operator being in the set of cognitive states using the sampled set of multimodal signals as input to the one or more state-classifiers; and

in response to the determined probabilities satisfying a set of criteria specified in a settings file, performing one or more of a set of actions, the set of actions including adjusting the visual, audible, or tactile stimuli provided to the operator by the simulation environment and providing an alert to a trainer or supervisor.

14. The method of claim 13, further comprising,

displaying the depiction of operation in a time period;

providing a graphical user interface for selecting portions of the time period and selection of respective ones of cognitive states for the selected portions; and

in response to selection of a portion of the time period and one of the cognitive states, training the one or more state-classifiers to map the set of multimodal signals for the selected portion to the selected one of the cognitive states.

15. The method of claim 14, wherein the displaying the depiction of operation in the time period includes displaying a video recording of the operator in the time period.

16. The method of claim 14, wherein the displaying the depiction of operation in the time period further includes displaying respective probabilities that the operator is in each one of the set of cognitive states during the time period.

17. The method of claim 14, further comprising, prior to the displaying the depiction of operation in the time period, performing initial training of the one or more state-classifiers to map the set of multimodal signals to the set of cognitive states.

18. The method of claim 17, wherein for each of one or more of the set of cognitive states, the initial training includes:

presenting stimuli configured to induce the cognitive state in the operator for a respective portion of the time period;

sampling the set of multimodal signals in the portion; and

mapping the one or more of the physiological responses indicated by the multimodal signals in the portion to the cognitive state.

19. The method of claim 18, wherein capturing of the set of multimodal signals includes:

converting one or more analog signals to digital signals;

aligning the digital signals in time; and

removing noise artifacts from one or more of the digital signals.

20. The method of claim 18, wherein, presenting stimuli includes simulating an operation scenario in a simulator, presenting a video depicting operation by another operator in the operation scenario, or a combination thereof.