Measuring cognitive load

• US 9,189,596 B2
• Filed: 06/28/2006
• Issued: 11/17/2015
• Est. Priority Date: 06/29/2005
• Status: Active Grant

• Current Assignee: National ICT Australia Limited
• Original Assignee: National ICT Australia Limited
• Inventors: Chen, Fang, Ruiz, Natalie, Choi, Eric
• Primary Examiner(s): Thomson, William
• Assistant Examiner(s): JIAN, SHIRLEY XUEYING
• Application Number:
  US11/993,008
• Publication Number:
  US 20100217097A1
• Time in Patent Office: 3,429 Days
• US Class Current: 1/1
• CPC Class Codes: A61B 5/16 Devices for psychotechnics using teaching or educational appliances G09B1/00 - G09B7/00; Testing reaction times ; Devices for evaluating the psychological state
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/4088 Diagnosing of monitoring cognitive diseases, e.g. Alzheimer, prion diseases or dementia
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

This invention concerns a method for measuring cognitive load of a person in performing a task. In other aspects the invention can be expressed as a computer and as software that are used to perform the method. The computer (50) has an interface having a plural number of unimodal input (20, 30 to 40) and output devices, and a cognitive load analyzer (50). The analyzer comprises a receiver to receive input data signal streams (25, 35 and 45) from respective devices (20, 30 and 40). A classifier (56 to 59), (66 to 69) and (76 to 79) is also provided to identify predetermined “meta-interaction patterns” from the streams (25, 35 and 45), and to weight the identified predetermined “meta-interaction patterns” to produce respective weighted outputs. A combiner (80) to fuse the outputs to produce a measure indicating the person'"'"'s cognitive load.

Claims

1. A computer implemented method for measuring a person'"'"'s cognitive load, comprising the steps of:

receiving two or more input data streams from an interface being used by a person while performing a task;

identifying predetermined “

meta-interaction features”

based on the task from the two or more input data streams;

identifying predetermined “

meta-interaction patterns”

from the “

meta-interaction features”

from two or more selected input data streams, including at least a first “

meta-interaction pattern”

identified from a predetermined combination of co-occurring “

meta-interaction features”

from two or more selected input data streams to form the predetermined combination;

weighting each of the identified predetermined “

meta-interaction patterns”

based on the task to produce respective outputs; and

fusing the outputs to produce a measure indicating the person'"'"'s cognitive load.

2. A method for measuring a person'"'"'s cognitive load according to claim 1, wherein the received two or more input data streams involve one or more of the following types of information:

visual data, including data that captures one or more of eye movements, facial expressions, head, body and hand gestures;

speech data, including one or more of pitch and fundamental frequency and its associated higher order statistics, sound emissions, speech rate, and information throughput;

interaction events including one or more of mouse hovering, mouse trajectory, menu and button selections and typing and input method speed;

task characteristics including one or more of task complexity or difficulty, current stage or state of the task, error rate, completion rate, and latency to respond;

person characteristics including one or more of gender, age and education; and

physiological data including one or more of heart rate and perspiration.

3. The method for measuring a person'"'"'s cognitive load according to claim 1, wherein the patterns are identified using one of more of the following models:

gaussian mixture models;

hidden Markov models;

linear discriminant analysis;

decision trees; and

artificial neural networks.

4. The method for measuring a person'"'"'s cognitive load according to claim 1, wherein any one or more of the following steps are specific to one or more of the person and task:

a further step of selecting two or more input data streams,

identifying predetermined “

meta-interaction patterns,”

weighting the identified predetermined “

meta-interaction patterns,” and

fusing the outputs.

5. The method for measuring a person'"'"'s cognitive load according to claim 4, wherein each selected input data stream specific to the task corresponds to a deliberate input by the person to the interface in performing the task.

6. The method for measuring a person'"'"'s cognitive load according to claim 1, wherein the method further comprises a calibration phase where the person is given a set of standard tasks to perform that vary in complexity.

7. The method for measuring a person'"'"'s cognitive load according to claim 6, wherein an output measure from each task of the calibration phase is considered and, depending on the proximity of this measure to an optimized pre-set cognitive load target level for that task and person context, the next task is verified for appropriateness or changed.

8. The method for measuring a person'"'"'s cognitive load according to claim 7, wherein the method further comprises a measurement phase where the person is asked to perform the task and the outputs produced by the method are compared to output measures of the calibration phase to produce a measure of cognitive load.

9. The method for measuring a person'"'"'s cognitive load according to claim 1, wherein the weightings used in a measurement phase is person and/or task specific.

10. The method for measuring a person'"'"'s cognitive load according to claim 1, wherein patterns that are more pronounced in reflecting actual cognitive loads are given a heavier weighting.

11. The method for measuring a person'"'"'s cognitive load according to claim 1, wherein machine learning is applied to weight features more heavily for particular persons if they give a better indication of the person'"'"'s cognitive load fluctuations.

12. The method for measuring a person'"'"'s cognitive load according to claim 1, wherein dynamic learning is applied to improve the individual patterns for new persons.

13. The method for measuring a person'"'"'s cognitive load according to claim 1, wherein parameters of individual classifiers and the fusion can be adjusted, to better reflect the characteristics of the new modality data acquired from the user.

14. The method for measuring a person'"'"'s cognitive load according to claim 1, wherein one or more input streams are collected from the user in an unobtrusive manner.

15. Software operable when installed on a computer that can interface with a person to perform the method of measuring the person'"'"'s cognitive load according to claim 1.

16. The method for measuring a person'"'"'s cognitive load according to claims 1, wherein the interface is a multimodal interface.

17. A computer to measure a person'"'"'s cognitive load while performing a task comprising an interface having unimodal input and output devices, the computer configured to execute computer readable medium, wherein the computer readable medium stores software instructions for a cognitive load analyzer, the cognitive load analyzer comprising:

a receiver to receive two or more input data streams from the unimodal input devices;

a set of feature extractors to extract predetermined “

meta-interaction features”

from the two or more input data streams;

a classifier to identify predetermined “

meta-interaction patterns”

based on the task from “

meta-interaction features,”

wherein “

meta-interaction patterns”

have “

meta-interaction features”

from two or more data input streams, including at least a first “

meta-interaction pattern”

identified from a predetermined combination of co-occurring “

meta-interaction features”

selected from two or more input data steams to form the predetermined combination and to weight the identified predetermined “

meta-interaction patterns”

based on the task to produce respective weighted outputs; and

a combiner to fuse the outputs to produce a measure indicating the person'"'"'s cognitive load.

18. A computer to measure a person'"'"'s cognitive load according to claim 17, wherein one or more of the following input devices is used by the interface:

video cameras;

infrared cameras;

eye gaze trackers;

accelerometers;

input device trackers and piezoelectric transducers;

sound capture transducers; and

ultrasound capture devices.

19. The computer to measure a person'"'"'s cognitive load according to claim 17, wherein the classifier identifies the patterns using one of more of the following models:

Gaussian mixture models;

hidden Markov models;

linear discriminant analysis;

decision trees; and

artificial neural networks.

20. The computer to measure a person'"'"'s cognitive load according to claim 19, wherein multiple classifiers are used to identify the patterns and the classifiers are not of the same model type such that a combination of model types for each data stream and among different classifiers is used.

21. The computer to measure a person'"'"'s cognitive load according to claim 17, wherein the interface is able to adapt to implement strategies that reduce cognitive load from use of the interface.

22. The computer to measure a person'"'"'s cognitive load according to claim 21, wherein the behavioral adaptation includes any one or more changes to:

the materials used in the interaction;

the volume of information flow;

the format of communication; and

the range of tasks presented to the person during periods of both high and low cognitive load.

23. The computer to measure a person'"'"'s cognitive load according to claim 17, wherein any one or more of the:

input data streams, “

meta-interaction patterns,”

weight of the classifier, and fusion of the combiner, are specific to one or more of the person and task.

24. The computer to measure a person'"'"'s cognitive load according to claim 17, wherein the one or more input devices used by the interface are those deliberately used by the person in performing the task.