Annotating stimulus based on determined emotional response

• US 8,126,220 B2
• Filed: 05/03/2007
• Issued: 02/28/2012
• Est. Priority Date: 05/03/2007
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Greig, Darryl
• Primary Examiner(s): ZARKA, DAVID PETER
• Application Number:
  US11/799,780
• Publication Number:
  US 20080275830A1
• Time in Patent Office: 1,762 Days
• Field of Search: 382/118
• US Class Current: 382/118
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A method of annotating audio-visual data is disclosed. The method includes detecting a plurality of facial expressions in an audience based on a stimulus, determining an emotional response to the stimulus based on the facial expressions and generating at least one annotation of the stimulus based on the determined emotional response.

Claims

1. A method of indexing a stimulus comprising:

measuring a plurality of facial expressions in an audience of the stimulus;

interpreting measurements of the facial expressions using a processor, wherein the processor interpreting the measurements produces a result estimating a mood for a time period of the stimulus, and wherein interpreting measurements of the facial expressions comprises;

classifying each of the facial expressions as corresponding to an emotion from a set of emotions; and

statistically analyzing the respective emotions corresponding to the plurality of facial expressions to estimate the mood;

generating an annotation of the stimulus, wherein the annotation is generated based on the mood estimated;

repeating the measuring, interpreting, and generating steps to produce a plurality of the annotations that respectively correspond to different time periods of the stimulus; and

indexing the annotations according to the respective moods, wherein the indexing provides a two-way linkage between the annotations and the time periods of the stimulus.

2. The method of claim 1, wherein the stimulus comprises an audio-video stimulus.

3. The method of claim 1, wherein measuring the plurality of facial expressions comprises:

scanning the audience in real time; and

detecting face locations in the audience.

4. The method of claim 1, wherein interpreting the measurements further comprises:

classifying the moods in real time from the plurality of facial expressions.

5. The method of claim 1, wherein each of the annotations indicates how many people are in the audience.

6. The method of claim 1, wherein the set of emotions includes joy, sadness, surprise, anger, fear, disgust, and neutral.

7. A computer program product for annotating a stimulus, the computer program product comprising a non-transitory computer usable medium having computer readable program means for causing a computer to perform the steps of:

detecting a plurality of facial expressions in an audience of the stimulus;

interpreting the facial expressions to estimate a mood for a time period of the stimulus, wherein interpreting the facial expressions comprises;

classifying each of the facial expressions as corresponding to an emotion from a set of emotions; and

statistically analyzing the respective emotions corresponding to the plurality of facial expressions to estimate the mood;

generating an annotation of the stimulus based on the mood estimated;

repeating the detecting, interpreting, and generating steps to produce a plurality of the annotations that respectively correspond to different time periods of the stimulus; and

indexing the annotations according to the respective moods, wherein the indexing provides a two-way linkage between the annotations and the time periods of the stimulus.

8. The computer program product of claim 7, wherein the stimulus comprises an audio-video stimulus.

9. The computer program product of claim 7, wherein detecting the plurality of facial expressions comprises:

scanning the audience in real time; and

detecting face locations in the audience.

10. The computer program product of claim 7, wherein determining a mood comprises:

analyzing the plurality of facial expressions; and

classifying real time emotional responses from the plurality of facial expressions.

11. A system for annotating a stimulus comprising:

an image capture device; and

a computer coupled to the image capture device, wherein the computer executes an annotation generation module that interacts with the image capture device, and wherein the annotation generation module comprises logic for performing the steps of;

detecting a plurality of facial expressions in an audience of the stimulus;

interpreting the facial expressions to estimate a mood for a time period of the stimulus, wherein interpreting the facial expressions comprises;

classifying each of the facial expressions as corresponding to an emotion from a set of emotions; and

statistically analyzing the respective emotions corresponding to the plurality of facial expressions to estimate the mood;

generating an annotation of the stimulus based on the mood estimated;

repeating the measuring, interpreting, and generating steps to produce a plurality of the annotations that respectively correspond to different time periods of the stimulus; and

indexing the annotations according to the respective moods, wherein the indexing provides a two-way linkage between the annotations and the time periods of the stimulus.

12. The system of claim 11, further comprising a display that presents at least a portion of the stimulus to the audience.

13. The system of claim 11, wherein the stimulus comprises an audio-video stimulus.

14. The system of claim 11, wherein detecting a plurality of facial expressions comprises:

scanning the audience in real time; and

detecting face locations in the audience.

15. The system of claim 11, wherein interpreting the facial expressions comprises:

analyzing the plurality of facial expressions; and

classifying a mood in real time from the plurality of facial expressions.

16. A method of indexing a stimulus, comprising:

measuring a plurality of facial expressions in an audience of the stimulus;

interpreting measurements of the facial expressions using a processor, wherein the processor interpreting the measurements produces a result estimating a mood for a time period of the stimulus;

generating an annotation of the stimulus, wherein the annotation is generated based on the mood estimated;

repeating the measuring, interpreting, and generating steps to produce a plurality of the annotations that respectively correspond to different time periods of the stimulus;

indexing the annotations according to the respective moods, wherein the indexing provides a two-way linkage between the annotations and the time periods of the stimulus employing a query that indicates a mood; and

using the indexing of the annotations to find in the stimulus a time period that is linked to the mood indicated in the query.