Image analysis using a semiconductor processor for facial evaluation

• US 10,474,875 B2
• Filed: 11/20/2015
• Issued: 11/12/2019
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: Pitre, Boisy G, el Kaliouby, Rana, Turcot, Panu James
• Primary Examiner(s): Tucker, Wesley J
• Application Number:
  US14/947,789
• Publication Number:
  US 20160078279A1
• Time in Patent Office: 1,453 Days
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/6898 Portable consumer electronic devices, e.g. music players, telephones, tablet computers
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 50/30 for calculating health indices; for individual health risk assessment
• CPC Class Codes: G16H 50/70 for mining of medical data, e.g. analysing previous cases of other patients

Abstract

Image analysis for facial evaluation is performed using logic encoded in a semiconductor processor. The semiconductor chip analyzes video images that are captured using one or more cameras and evaluates the videos to identify one or more persons in the videos. When a person is identified, the semiconductor chip locates the face of the evaluated person in the video. Facial regions of interest are extracted and differences in the regions of interest in the face are identified. The semiconductor chip uses classifiers to map facial regions for emotional response content and evaluate the emotional response content to produce an emotion score. The classifiers provide gender, age, or ethnicity with an associated probability. Localization logic within the chip is used to localize a second face when one is evaluated in the video. The one or more faces are tracked, and identifiers for the faces are provided.

Claims

1. An apparatus for analysis comprising:

a device containing image analysis logic encoded in a semiconductor chip comprising;

evaluation logic that evaluates bodies of one or more persons in videos;

localization logic that performs localization of a face within the videos where the face is from one of the bodies of the one or more persons;

feature extraction logic that performs extraction of regions of interest on the face;

identification logic that provides identification of differences in the regions of interest within the face;

classifier logic that employs image classifiers to map the regions within the face for emotional response content;

scoring logic that evaluates the emotional response content to produce an emotion score based on the face, wherein the emotion score provides information on concentration based on the regions of the face including eyebrows where the eyebrows are furrowed; and

ranking logic that ranks an intensity of a plurality of facial expressions as a function of the emotion score.

2. The apparatus of claim 1 wherein the localization logic further performs localization of a second face within the videos.

3. The apparatus of claim 2 further comprising tracking logic for tracking the face and the second face.

4. The apparatus of claim 3 wherein the tracking logic provides an identifier for the face and the second face.

5. The apparatus of claim 4 wherein the tracking logic identifies that a face has left a video frame.

6. The apparatus of claim 5 wherein the tracking logic identifies that the face has returned to the video frame and associates information previously collected about the face from before the face left the video frame.

7. The apparatus of claim 1 wherein the identification logic further extracts one or more histogram-of-gradient (HoG) features from the regions of interest (RoI).

8. The apparatus of claim 7 wherein the scoring logic produces the emotion score based on the histogram-of-gradient features.

9. The apparatus in claim 1 wherein the extraction of regions is based on one or more of texture or shape.

10. The apparatus in claim 1 wherein the extraction of regions is based on motion within the face.

11. The apparatus in claim 1 wherein the identification of differences includes evaluation of eyebrow locations, eye locations, or mouth locations.

12. The apparatus in claim 1 wherein the classifiers are utilized by a support vector machine analysis to identify the emotional response content.

13. The apparatus of claim 1 wherein the classifier logic further identifies a gender, age, or ethnicity for the face.

14. The apparatus of claim 13 wherein the gender, age, or ethnicity is provided with an associated probability.

15. The apparatus in claim 1 wherein the identification of differences includes landmark detection within the face.

16. The apparatus in claim 1 further comprising storage memory coupled to the device.

17. The apparatus in claim 16 wherein the storage memory stores videos for analysis by the device to evaluate moods for people in the videos.

18. The apparatus in claim 16 wherein the storage memory stores classifier information used by the classifier logic.

19. The apparatus in claim 16 wherein the videos are retrieved from the storage memory for evaluation by the evaluation logic.

20. The apparatus in claim 1 wherein the emotion score includes a facial expression and a probability of occurrence of the facial expression.

21. The apparatus in claim 1 wherein the device further performs image correction for the videos including one or more of lighting correction, contrast correction, or noise filtering smoothing of the emotion score.

22. The apparatus in claim 1 wherein the emotion score is augmented by physiological information.

23. The apparatus in claim 1 wherein the emotion score is used to track a mood of a person with the face that is localized.

26. The apparatus of claim 1 wherein the scoring logic further determines an affect valence for a presentation based on the emotion score.

27. The apparatus of claim 26 wherein the affect valence describes an emotional reaction of an individual to the presentation.

28. The apparatus of claim 27 wherein the affect valence is positive or negative.

24. A computer program product embodied in a non-transitory computer readable medium for image analysis, the computer program product comprising:

code for executing on a device containing image analysis logic encoded in a semiconductor chip where the code causes;

evaluation logic to evaluate bodies of one or more persons in videos;

localization logic to perform localization of a face within the videos where the face is from one of the bodies of the one or more persons;

feature extraction logic to perform extraction of regions of interest on the face;

identification logic to provide identification of differences in the regions of interest within the face;

classifier logic to employ image classifiers to map the regions within the face for emotional response content;

scoring logic to evaluate the emotional response content to produce an emotion score based on the face, wherein the emotion score provides information on concentration based on the regions of the face including eyebrows where the eyebrows are furrowed; and

ranking logic to rank an intensity of a plurality of facial expressions as a function of the emotion score.

25. A processor-implemented method for analysis comprising:

using a device containing image analysis logic encoded in a semiconductor chip to perform;

evaluating of bodies of one or more persons in videos;

localizing of a face within the videos where the face is from one of the bodies of the one or more persons;

feature extraction of regions of interest on the face;

identifying of differences in the regions of interest within the face;

mapping, using image classifiers, the regions within the face for emotional response content;

scoring the emotional response content to produce an emotion score based on the face, wherein the emotion score provides information on concentration based on the regions of the face including eyebrows where the eyebrows are furrowed; and

ranking an intensity of a plurality of facial expressions as a function of the emotion score.