Method and system for measuring human response to visual stimulus based on changes in facial expression

• US 20090285456A1
• Filed: 05/19/2008
• Published: 11/19/2009
• Est. Priority Date: 05/19/2008
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Videomining Corporation
• Inventors: Jung, Namsoon, Sharma, Rajeev, Moon, Hankyu
• Granted Patent: US 8,462,996 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The present invention is a method and system for measuring human emotional response to visual stimulus, based on the person'"'"'s facial expressions. Given a detected and tracked human face, it is accurately localized so that the facial features are correctly identified and localized. Face and facial features are localized using the geometrically specialized learning machines. Then the emotion-sensitive features, such as the shapes of the facial features or facial wrinkles, are extracted. The facial muscle actions are estimated using a learning machine trained on the emotion-sensitive features. The instantaneous facial muscle actions are projected to a point in affect space, using the relation between the facial muscle actions and the affective state (arousal, valence, and stance). The series of estimated emotional changes renders a trajectory in affect space, which is further analyzed in relation to the temporal changes in visual stimulus, to determine the response.

Claims

1. A method for determining a person'"'"'s emotional response to a visual stimulus, based on the person'"'"'s facial expressions, comprising the following steps of:

a) detecting and tracking a face from input images of the person captured by at least a means for capturing images, and localizing the face and facial features,

b) extracting emotion-sensitive features of the face from the face by applying emotion-sensitive feature filters to localized facial features and transient features,

c) determining facial muscle actions of the face based on the emotion-sensitive features,

d) finding the changes in affective state, called an emotion trajectory, of the person based on the facial muscle actions, and

e) determining the response of the person to the visual stimulus, by analyzing the emotion trajectory.

2. The method according to claim 1, wherein the method further comprises a step of training a plurality of first machines for localizing the face.

3. The method according to claim 1, wherein the method further comprises a step of training a plurality of second machines for localizing facial features in the face.

4. The method according to claim 1, wherein the method further comprises a step of deriving emotion-sensitive feature filters, by first generating emotion-sensitive candidate filters and determining the emotion-sensitive feature filters by choosing the filters from the emotion-sensitive candidate filters that yield high responses to a sufficient number of facial images.

5. The method according to claim 4, wherein the method further comprises a step of generating emotion-sensitive candidate filters by constructing a set of filters where each filter is designed to collect gradient response of a specific facial feature or a transient feature.

6. The method according to claim 1, wherein the method further comprises a step of training a third machine so that the machine accepts the emotion-sensitive features as an input and computes a vector output where each component of the vector is the likelihood of the emotion-sensitive features arising from a specific facial muscle action.

7. The method according to claim 6, wherein the method further comprises a step of estimating the Action Units in the Facial Action Coding System as facial muscle actions.

8. The method according to claim 1, wherein the method further comprises a step of determining an affective state of the face by using the mapping from the facial muscle actions to the six emotional categories.

9. The method according to claim 1, wherein the method further comprises a step of determining an affective state of the face by using the coordinates of six emotional categories in the affect space.

10. The method according to claim 1, wherein the method further comprises a step of determining the response of the person to the visual stimulus, by comparing an initial state and a terminal state.

11. The method according to claim 1, wherein the method further comprises a step of determining the response of the person to the visual stimulus, by analyzing the emotion trajectory in affect space.

12. The method according to claim 1, wherein the method further comprises a step of segmenting the visual stimulus in terms of the visual content.

13. The method according to claim 12, wherein the method further comprises a step of determining the response of the person to the visual stimulus, by analyzing the emotion trajectory in relation to the segmented contents in visual stimulus.

14. The method according to claim 12, wherein the method further comprises a step of segmenting the emotion trajectory so that each segment in the emotion trajectory is an emotional response to a segment in the visual stimulus.

15. The method according to claim 14, wherein the method further comprises a step of normalizing the emotion trajectory so that the temporal length of each visual stimulus segments has a fixed standard size.

16. The method according to claim 1, wherein the method further comprises a step of determining the response of the person to the visual stimulus, by training a fourth machine using a plurality of emotion trajectories and ratings data, so that the machine can predict the rating based on the emotion trajectory.

17. An apparatus for determining a person'"'"'s emotional response to a visual stimulus, based on the person'"'"'s facial expressions, comprising:

a) means for detecting and tracking a face from input images of the person captured by at least a means for capturing images, and localizing the face and facial features,

b) means for extracting emotion-sensitive features of the face from the face by applying emotion-sensitive feature filters to localized facial features and transient features,

c) means for determining facial muscle actions of the face based on the emotion-sensitive features,

d) means for finding the changes in affective state, called an emotion trajectory, of the person based on the facial muscle actions, and

e) means for determining the response of the person to the visual stimulus, by analyzing the emotion trajectory.

18. The apparatus according to claim 17, wherein the apparatus further comprises means for training a plurality of first machines for localizing the face.

19. The apparatus according to claim 17, wherein the apparatus further comprises means for training a plurality of second machines for localizing facial features in the face.

20. The apparatus according to claim 17, wherein the apparatus further comprises means for deriving emotion-sensitive feature filters, by first generating emotion-sensitive candidate filters and determining the emotion-sensitive feature filters by choosing the filters from the emotion-sensitive candidate filters that yield high responses to a sufficient number of facial images.

21. The apparatus according to claim 20, wherein the apparatus further comprises means for generating emotion-sensitive candidate filters by constructing a set of filters where each filter is designed to collect gradient response of a specific facial feature or a transient feature.

22. The apparatus according to claim 17, wherein the apparatus further comprises means for training a third machine so that the machine accepts the emotion-sensitive features as an input and computes a vector output where each component of the vector is the likelihood of the emotion-sensitive features arising from a specific facial muscle action.

23. The apparatus according to claim 22, wherein the apparatus further comprises means for estimating the Action Units in the Facial Action Coding System as facial muscle actions.

24. The apparatus according to claim 17, wherein the apparatus further comprises means for determining an affective state of the face by using the mapping from the facial muscle actions to the six emotional categories.

25. The apparatus according to claim 17, wherein the apparatus further comprises means for determining an affective state of the face by using the coordinates of six emotional categories in the affect space.

26. The apparatus according to claim 17, wherein the apparatus further comprises means for determining the response of the person to the visual stimulus, by comparing an initial state and a terminal state.

27. The apparatus according to claim 17, wherein the apparatus further comprises means for determining the response of the person to the visual stimulus, by analyzing the emotion trajectory in affect space.

28. The apparatus according to claim 17, wherein the apparatus further comprises means for segmenting the visual stimulus in terms of the visual content.

29. The apparatus according to claim 28, wherein the apparatus further comprises means for determining the response of the person to the visual stimulus, by analyzing the emotion trajectory in relation to the segmented contents in visual stimulus.

30. The apparatus according to claim 28, wherein the apparatus further comprises means for segmenting the emotion trajectory so that each segment in the emotion trajectory is an emotional response to a segment in the visual stimulus.

31. The apparatus according to claim 30, wherein the apparatus further comprises means for normalizing the emotion trajectory so that the temporal length of each visual stimulus segment has a fixed standard size.

32. The apparatus according to claim 17, wherein the apparatus further comprises means for determining the response of the person to the visual stimulus, by training a fourth machine using a plurality of emotion trajectories and ratings data, so that the machine can predict the rating based on the emotion trajectory.