Learning emotional states using personalized calibration tasks

• US 9,767,349 B1
• Filed: 05/09/2016
• Issued: 09/19/2017
• Est. Priority Date: 05/09/2016
• Status: Expired due to Fees

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Shreve, Matthew Adam, Kumar, Jayant, Bala, Raja, Emmett, Phillip J., Clar, Megan, Subramanian, Jeyasri, Harte, Eric
• Primary Examiner(s): Seth, Manav
• Application Number:
  US15/149,284
• Time in Patent Office: 498 Days
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A method for determining an emotional state of a subject taking an assessment. The method includes eliciting predicted facial expressions from a subject administered questions each intended to elicit a certain facial expression that conveys a baseline characteristic of the subject; receiving a video sequence capturing the subject answering the questions; determining an observable physical behavior experienced by the subject across a series of frames corresponding to the sample question; associating the observed behavior with the emotional state that corresponds with the facial expression; and training a classifier using the associations. The method includes receiving a second video sequence capturing the subject during an assessment and applying features extracted from the second image data to the classifier for determining the emotional state of the subject in response to an assessment item administered during the assessment.

Claims

1. A method for determining an emotional state of a subject taking an assessment, the method comprising:

generating a calibration task to elicit predicted responses from an associated subject administered the task, wherein each portion of the task is intended to elicit a certain emotional response that conveys a baseline characteristic of the associated subject;

receiving video data capturing the associated subject performing the calibration task, wherein each frame of the video data is synchronized within the task to correspond to a portion of the task;

processing the video data for determining an observable physical behavior experienced by the associated subject across a series of frames during the each portion of the task;

detecting an emotional response experienced by the associated subject across the series of frames corresponding to the each portion of the task;

associating (or tagging) the observed behavior with one of multiple emotional categories (or labels), wherein each category (or label) corresponds with one of the emotional responses; and

,

training a classifier using the features extracted from the image data, where each class is one of the categories associated with the observed behavior.

2. The method of claim 1, wherein the calibration task that is used for training includes:

a first class including a set of elicited behaviors associate associated with a neutral and relaxed emotional state; and

a second class that includes all other elicited behaviors,

wherein a baseline is established for the associated subject by training the classifier on the features extracted from the image data from both the first and second classes to learn the neutral and relaxed emotional state of the associated subject.

3. The method of claim 1, wherein the calibration task includes multiple questions across a range of difficulty levels, wherein at least two questions are different levels of difficulty, wherein each level of difficulty corresponds with a different one of the predicted responses and the each portion of the task is intended to elicit the certain emotional response.

4. The method of claim 1 further comprising:

using a mobile computing device including the image capture device, monitoring the associated subject during an administration of the calibration task to the associated subject.

5. The method of claim 1, wherein each certain emotional response is a predicted facial expression that conveys the emotional state of the associated subject.

6. The method of claim 1 further comprising:

receiving second image data capturing the associated subject during an assessment administered after the calibration task;

applying features extracted from the second image data to the classifier for determining the emotional state of the associated subject in response to an assessment item administered during the assessment.

7. The method of claim 1 further comprising:

using a sensor, monitoring the associated subject during the calibration task, wherein the sensor measures a biometric indicative of the emotional state of the associated subject.

8. The method of claim 1 further comprising:

receiving the image data capturing at least two associated subjects performing the calibration task, wherein the calibration task includes questions covering at least two topics;

detecting observable physical behaviors experienced by the at least two associated subjects during the each portion of the task;

in response to determining a pattern of similarity between the observable physical behaviors, associating (or tagging) the observed behavior with one of multiple emotional categories (or labels), wherein each category (or label) corresponds with one of the intended emotional responses;

training the classifier using the associations.

9. The method of claim 1, wherein the processing the image data includes:

detecting the associated subject in an image frame of the image data;

applying a landmark detection (or Viola-Jones facial detector) algorithm to the image frame to determine a facial region;

applying a deformable part-based model (DPM) to determine a region of interest in the facial region corresponding to a facial landmark of the associated subject;

extracting features in the region of interest;

associating the features with an emotional category; and

training the classifier using the association.

10. The method of claim 9, further comprising:

generating feature histograms from one or more of the extracted features;

concatenating the extracted features to generate a final feature descriptor; and

,

training the classifier using the final feature descriptor.

11. The method of claim 9, wherein the processing the image data further includes:

performing a coordinate transform on the region of interest across multiple image frames.

12. A method for determining an emotional state of a subject taking an assessment, the method comprising:

generating sample questions to elicit predicted facial expressions from an associated subject administered the questions, wherein each question is intended to elicit a certain facial expression that conveys a baseline characteristic of the associated subject;

receiving a video sequence capturing the associated subject answering the questions, wherein each frame of the video sequence is synchronized within a sample question;

determining an observable physical behavior experienced by the associated subject across a series of frames corresponding to the sample question;

detecting a facial expression conveyed by the associated subject across the series of frames corresponding to the question;

associating (or tagging) the observed behavior with the emotional state that corresponds with the facial expression;

training a classifier using the associations;

receiving a second video sequence capturing the associated subject during an assessment administered after the sample questions;

applying features extracted from the second image data to the classifier for determining the emotional state of the associated subject in response to an assessment item administered during the assessment.

13. A calibration system for determining an emotional state of a subject taking an assessment, the system comprising:

a processor, and

a non-transitory computer readable memory storing instructions that are executable by the processor to perform the operations of;

generating a calibration task to elicit predicted responses from an associated subject administered the task, wherein each portion of the task is intended to elicit a certain emotional response that conveys a baseline characteristic of the associated subject;

receiving image data from an image capture device capturing the associated subject performing the calibration task, wherein each frame of the image data is synchronized to correspond to a portion of the task;

determining an observable physical behavior experienced by the associated subject across a series of frames during the portion of the task;

detecting an emotional response experienced by the associated subject across the series of frames corresponding to the portion of the task;

associating (or tag) the observed behavior with one of multiple emotional categories (or labels), wherein each category (or label) corresponds with one of the emotional responses;

training a classifier using the associations;

receiving second image data capturing the associated subject during an assessment administered after the calibration task; and

,

applying features extracted from the second image data to the classifier for determining the emotional state of the associated subject in response to an assessment item administered during the assessment.

14. The system of claim 13, wherein the calibration task includes multiple questions across a range of difficulty levels, wherein at least two questions are different levels of difficulty, wherein each level of difficulty corresponds with a different one of the predicted responses and the each portion of the task is intended to elicit the certain emotional response.

15. The system of claim 13 further comprising:

a mobile computing device including the image capture device for monitoring the associated subject during an administration of the calibration task to the associated subject.

16. The system of claim 13, wherein each certain emotional response is a predicted facial expression that conveys the emotional state of the associated subject.

17. The system of claim 13 further comprising:

a sensor for monitoring the associated subject during the calibration task, wherein the sensor measures a biometric indicative of the emotional response is a measurement of stress.

18. The system of claim 13, wherein the processor is further programmed to perform the operations of:

receiving image data capturing at least two associated subjects performing the calibration task, wherein the calibration task includes questions covering at least two topics;

determining observable physical behaviors experienced by the at least two associated subjects during the each portion of the task;

in response to determining a pattern of similarity between the observable physical behaviors, associating (or tagging) the observed behavior with one of multiple emotional categories (or labels), wherein each category (or label) corresponds with one of the intended emotional responses; and

,

training a classifier using the associations.

19. The system of claim 13, wherein the processor is further programmed to perform the operations of:

detecting the associated subject in an image frame of the image data;

applying a landmark detection (or Viola-Jones facial detector) algorithm to the image frame to determine a facial region;

applying a deformable part-based model (DPM) to determine a region of interest in the facial region corresponding to a facial landmark of the associated subject;

extracting features in the region of interest;

associating the features with an emotional category; and

training the classifier using the association.

20. The system of claim 18, wherein the processor is further programmed to perform the operations of:

generating a Histogram of Features (HoG) using the extracted features;

concatenating the gradients to generate a final feature vector; and

,

training the classifier using the final feature vector.

21. The system of claim 18, wherein the processor is further programmed to perform the operation of:

performing a coordinate transform on the region of interest across multiple image frames (or align the facial landmark across the multiple frames using a geometric transform).