MEASURING COGNITIVE LOAD

• US 20100217097A1
• Filed: 06/28/2006
• Published: 08/26/2010
• Est. Priority Date: 06/29/2005
• Status: Active Grant

• Current Assignee: National ICT Australia Limited
• Original Assignee: National ICT Australia Limited
• Inventors: Ruiz, Natalie, Chen, Fang, Choi, Eric
• Granted Patent: US 9,189,596 B2
• Time in Patent Office: Days
• US Class Current: 600/301
• CPC Class Codes: A61B 5/16 Devices for psychotechnics using teaching or educational appliances G09B1/00 - G09B7/00; Testing reaction times ; Devices for evaluating the psychological state
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/4088 Diagnosing of monitoring cognitive diseases, e.g. Alzheimer, prion diseases or dementia
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

This invention concerns a method for measuring cognitive load of a person in performing a task. In other aspects the invention can be expressed as a computer and as software that are used to perform the method. The computer (50) has an interface having a plural number of unimodal input (20, 30 to 40) and output devices, and a cognitive load analyzer (50). The analyzer comprises a receiver to receive input data signal streams (25, 35 and 45) from respective devices (20, 30 and 40). A classifier (56 to 59), (66 to 69) and (76 to 79) is also provided to identify predetermined “meta-interaction patterns” from the streams (25, 35 and 45), and to weight the identified predetermined “meta-interaction patterns” to produce respective weighted outputs. A combiner (80) to fuse the outputs to produce a measure indicating the person'"'"'s cognitive load.

Claims

1. A method for measuring a person'"'"'s cognitive load, comprising the steps of:

receiving multiple input data streams from a multimodal interface being used by a person while performing a task;

identifying predetermined “

meta-interaction patterns”

from the streams;

weighting the identified predetermined “

meta-interaction patterns”

to produce respective outputs; and

fusing the outputs to produce a measure indicating the person'"'"'s cognitive load.

2. A method for measuring a person'"'"'s cognitive load according to claim 1, wherein the received input data streams involve one or more of the following types of information:

visual data, including data that captures eye movements, facial expressions, head, body and hand gestures;

speech data, including pitch and fundamental frequency and its associated higher order statistics, sound emissions, speech rate; and

information throughput;

interaction events such as mouse hovering, mouse trajectory, menu and button selections and typing and input method speed;

task characteristics such as task complexity or difficulty, current stage or state of the task, error rate, completion rate, and latency to respond;

person characteristics such as gender, age and education; and

physiological data such as heart rate and perspiration.

3. A method for measuring a person'"'"'s cognitive load according to claim 1, wherein the step of identifying predetermined “

meta-interaction patterns”

from the streams comprises the steps of identifying predetermined “

meta-interaction features”

in each input data stream and then identifying patterns among the features.

4. A method for measuring a person'"'"'s cognitive load according to claim 3, wherein the patterns are identified using one of more of the following models:

gaussian mixture models;

hidden Markov models;

linear discriminant analysis;

decision trees; and

artificial neural networks.

5. A method for measuring a person'"'"'s cognitive load according to claim 1, wherein the method further comprises the steps of any one or more of selecting input data streams, “

meta-interaction patterns,”

method of performing the step of weighting and method of performing the step of fusing that are specific to the person and/or specific to the task.

10. A method for measuring a person'"'"'s cognitive load according to claim 1, wherein the method further comprises a calibration phase where the person is given a set of standard tasks to perform that vary in complexity, starting with simple tasks and gradually becoming more difficult.

11. A method for measuring a person'"'"'s cognitive load according to claim 10, wherein the output measure from each task is considered and, depending on the proximity of this measure to an optimized pre-set cognitive load target level for that task and person context, the next task is verified for appropriateness or changed.

12. A method for measuring a person'"'"'s cognitive load according to claim 11, wherein the method further comprises a measurement phase where the person is asked to perform a task and the resulting output produced by the system is compared to the calibrated results to produce a measure of cognitive load.

13. A method for measuring a person'"'"'s cognitive load according to claim 1, wherein the weightings used in the measurement phase is person and/or task specific.

14. A method for measuring a person'"'"'s cognitive load according to claim 1, wherein patterns that are more pronounced in reflecting actual cognitive loads are given a heavier weighting.

15. A method for measuring a person'"'"'s cognitive load according to claim 1, wherein machine learning is applied to weight features more heavily for particular persons if they give a better indication of the person'"'"'s cognitive load fluctuations.

16. A method for measuring a person'"'"'s cognitive load according to claim 1, wherein dynamic learning is applied to improve the individual patterns for new persons.

17. A method for measuring a person'"'"'s cognitive load according to claim 1, wherein parameters of the individual classifiers and the fusion can be adjusted, such as through Bayesian Predictive Learning, to better reflect the characteristics of the new modality data acquired from the user.

18. A method for measuring a person'"'"'s cognitive load according to claim 1, wherein the input streams are collected from the user in an unobtrusive manner.

19. A method for measuring a person'"'"'s cognitive load according to claim 5, wherein each selected input data stream specific to a particular task corresponds to a deliberate input by the person to the multimodal interface in the course of performing the task.

30. Software operable when installed on a computer that can interface with a person to perform the method of measuring the person'"'"'s cognitive load according to claim 1.

6-9. -9. (canceled)

20. A multimodal computer to measure a person'"'"'s cognitive load while performing a task comprising an interface having a plural number of unimodal input and output devices, and a cognitive load analyzer comprising:

a receiver to receive input data signal streams from respective devices; and

a classifier to identify predetermined “

meta-interaction patterns”

from the streams, and to weight the identified predetermined “

meta-interaction patterns”

to produce respective weighted outputs; and

,

a combiner to fuse the outputs to produce a measure indicating the person'"'"'s cognitive load.

21. A multimodal computer to measure a person'"'"'s cognitive load according to claim 20, wherein one or more of the following input devices is used by the interface:

video cameras;

infrared cameras;

eye gaze trackers;

accelerometers;

input device trackers and piezoelectric transducers;

sound capture transducers; and

ultrasound capture devices.

22. A multimodal computer to measure a person'"'"'s cognitive load according to claim 20, wherein the interface also having a set of feature extractors to extract predetermined “

meta-interaction features”

from the incoming data streams such that the classifiers identify predetermined “

meta-interaction patterns”

from the features.

23. A multimodal computer to measure a person'"'"'s cognitive load according to claim 20, wherein the classifiers identify the patterns using one of more of the following models:

Gaussian mixture models;

hidden Markov models;

linear discriminant analysis;

decision trees; and

artificial neural networks.

24. A multimodal computer to measure a person'"'"'s cognitive load according to claim 23, wherein the classifiers are not of the same model type such that a combination of model types within each feature stream and among different classifiers is used.

25. A multimodal computer to measure a person'"'"'s cognitive load according to claim 20, wherein the interface is able to adapt its behavior to implement strategies that reduce cognitive load from use of the interface.

26. A multimodal computer to measure a person'"'"'s cognitive load according to claim 25, wherein the behavioral adaptation includes any one or more changes to:

the materials used in the interaction;

the volume of information flow;

the format of communication; and

the range of tasks presented to the person during periods of both high and low cognitive load.

27. A multimodal computer to measure a person'"'"'s cognitive load according to claim 20, wherein input data signals streams received by the receiver, the “

meta-interaction patterns” and

weights of the classifier and fusion of the combiner are specific to the person and/or task.

29. A multimodal computer to measure a person'"'"'s cognitive load according to claim 20, wherein the input devices used by the interface are those input devices deliberately used by the person in performing the task.

28. (canceled)