METHOD AND SYSTEM FOR REAL-TIME AND OFFLINE ANALYSIS, INFERENCE, TAGGING OF AND RESPONDING TO PERSON(S) EXPERIENCES

• US 20110263946A1
• Filed: 04/22/2010
• Published: 10/27/2011
• Est. Priority Date: 04/22/2010
• Status: Abandoned Application

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: MIT Media Lab
• Inventors: Picard, Rosalind W., Kashef, Youssef, Mahmoud, Abdelrahman N., Madsen, Miriam Anna Rimm, el Kaliouby, Rana, Mikhail, Mina
• Application Number:
  US12/765,555
• Publication Number:
  US 20110263946A1
• Time in Patent Office: Days
• US Class Current: 600/300
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/16 Devices for psychotechnics using teaching or educational appliances G09B1/00 - G09B7/00; Testing reaction times ; Devices for evaluating the psychological state
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/168 Evaluating attention deficit, hyperactivity
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G16H 50/70 for mining of medical data, e.g. analysing previous cases of other patients

Abstract

A digital computer and method for processing data indicative of images of facial and head movements of a subject to recognize at least one of said movements and to determine at least one mental state of said subject is provided. The outputting instructions for providing to a user information relating to at least one said mental state. A further processing data reflective of input from a user, and based at least in part on said input, confirming or modifying said determination and generating with a transducer an output of humanly perceptible stimuli indicative of said at least one mental state.

Claims

1. A method comprising:

a digital computer;

processing data indicative of images of facial and head movements of a subject to recognize at least one of said movements and to determine at least one mental state of said subject,

outputting instructions for providing to a user information relating to at least one said mental state, and

further processing data reflective of input from a user, and based at least in part on said input, confirming or modifying said determination, and

generating with a transducer an output of humanly perceptible stimuli indicative of said at least one mental state.

2. The method of claim 1, wherein processing with said computer comprises calculating a value indicative of certainty, or of a range of certainties, or of probability, or of a range of probabilities, in case regarding said at least one mental state.

3. The method of claim 1, wherein outputting instructions comprises providing to a user substantially real time information regarding said at least one mental state.

4. The method of claim 3, wherein said computer is adapted to recognize a set of the mental states that comprises at least seven elements, at least one of which is a mental state other than a “

basic emotions”

of happiness, sadness, anger, fear, surprise and disgust.

5. The method of claim 3, wherein said computer is adapted to recognize at least two types of events, and wherein at least one said type of event has a shorter time duration than at least one other said type of event.

6. The method of claim 3, wherein said computer is adapted to recognize facial or head movements that are asynchronous or overlapping.

7. The method of claim 3, wherein a plurality of recognized facial or head movements is mapped to a single mental state.

8. The method of claim 1, wherein at least one said transducer comprises a graphical user interface.

9. The method of claim 1, wherein outputting instructions with said computer comprises providing to the user one or more images of said facial or head movements and substantially concurrently providing to said user information regarding said at least one mental state associated with said movements.

10. The method of claim 1, wherein processing includes using data consciously inputted or provided by said subject.

11. The method of claim 1, wherein processing includes using physiological data regarding said subject.

12. The method of claim 1, wherein at least part of said computer is remote from said user.

13. The method of claim 1, wherein outputting instructions comprises providing to a user a summary of mental states inferred from facial and head movements over a period of time.

14. The method of claim 1, further comprising associating, with said computer, said at least one mental state with at least two events, wherein at least one of said events is indicated by said data indicative of images of facial and head movements and wherein at least one other of said events is indicated by another data set, which other data set comprises content provided to said subject or data recorded about said subject.

15. The method of claim 1, further comprising processing data indicative of images of facial and head movements of a plurality of subjects to determine mental states of the plurality of subjects.

16. The method of claim 3, wherein said real time information is provided to a plurality of, users and input from a plurality of users is processed.

17. A method comprising:

with a digital computer;

processing data indicative of images of facial and head movements of a subject to determine at least one mental state of said subject, and

associating said at least one mental state with at least two events, wherein at least one of said events is indicated by said data indicate of images of facial and head movements and wherein at least one other of said events is indicated by another data set, which other data set comprises content provided to said subject or data recorded about said subject.

18. The method of claim 17, wherein said association employs at least one time stamp, frame number or other value indicative of temporal order.

19. The method of claim 17, wherein said content provided to said subject comprises the display of an audio or visual content.

20. The method of claim 17, wherein processing comprise processing physiologic data recorded about said subject.

21. The method of claim 17, wherein processing comprises processing data recorded relating to said subject'"'"'s interaction with a graphical user interface.

22. The method of claim 17, further comprising, with said computer outputting instructions providing to a user substantially real time information relating to said at least one mental state.

23. The method of claim 17, further comprising, with said computer analyzing data reflective of input from a user, and based at least in part on said analysis of said input, to change or confirm at least one said determination.

24. An apparatus comprising:

at least one camera for capturing images of facial and head movements of a subject; and

at least one computer adapted for;

analyzing data indicative of said images and determining one or more mental states of said subject,

outputting digital instructions for providing a user substantially real time information relating to said at least one mental state,

analyzing data reflective of input from a user, and based at least in part on said user input data analysis, changing or confirming said determination.

25. An article of manufacture, comprising a machine-accessible medium having instructions encoded thereon for enabling a computer to perform the operations of:

processing data indicative of images of facial and head movements of a subject to recognize at least one said movement and to determine at least one mental state of said subject,

outputting instructions for providing to a user information relating to said at least one mental state, and

processing data reflective of input from a user, and based in least in part on said input, confirm or modify said determination.