HEAD-POSE INVARIANT RECOGNITION OF FACIAL ATTRIBUTES

• US 20150324632A1
• Filed: 07/17/2015
• Published: 11/12/2015
• Est. Priority Date: 07/17/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Emotient, Inc. (Apple Inc.)
• Inventors: Whitehill, Jacob, Movellan, Javier R., Fasel, Ian
• Granted Patent: US 9,547,808 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/175 Static expression

Abstract

A system facilitates automatic recognition of facial expressions or other facial attributes. The system includes a data access module and an expression engine. The expression engine further includes a set of specialized expression engines, a pose detection module, and a combiner module. The data access module accesses a facial image of a head. The set of specialized expression engines generates a set of specialized expression metrics, where each specialized expression metric is an indication of a facial expression of the facial image assuming a specific orientation of the head. The pose detection module determines the orientation of the head from the facial image. Based on the determined orientation of the head and the assumed orientations of each of the specialized expression metrics, the combiner module combines the set of specialized expression metrics to determine a facial expression metric for the facial image that is substantially invariant to the head orientation.

Claims

1. A computer-implemented method for training a specialized recognition engine to recognize a preselected attribute of a facial image, the specialized recognition engine specialized for a range of head poses, the method comprising:

for each of a plurality of facial images of heads that are within the range of head poses for the specialized recognition engine;

identifying a pair comprising a known good facial image and a known good specialized recognition metric, wherein the known good facial image is for a same head as said facial image and has the same preselected attribute as said facial image but for a head pose within a range of head poses that is different than the range of head poses for the specialized recognition engine, and the known good specialized recognition metric is indicative of the preselected attribute of the known good facial image; and

associating a specialized recognition metric with said facial image, wherein the specialized recognition metric is derived from the known good specialized recognition metric; and

training the specialized recognition engine by using pairs comprising the plurality of facial images and the associated specialized recognition metrics as a training set for supervised learning.

2. The computer-implemented method of claim 1 wherein the known good facial images are frontal views of the head and the plurality of facial images are non-frontal views of the head.

3. The computer-implemented method of claim 1 wherein the range of head poses for the specialized recognition engine spans at least +/−

ten degrees with respect to at least one of Euler angles (yaw, pitch).

4. The computer-implemented method of claim 1 wherein, for at least some of the plurality of facial images, the head pose for said facial image differs by at least ten degrees from the head pose for the known good facial image.

5. The computer-implemented method of claim 1 wherein, for at least some of the plurality of facial images, the specialized recognition metric associated said facial image is equal to the known good specialized recognition metric.

6. The computer-implemented method of claim 1 wherein the pairs of known good facial images and known good specialized recognition metrics are outputs of a different, previously trained specialized recognition engine.

7. A computer-implemented method for training a plurality of specialized recognition engines to recognize a preselected attribute of a facial image, wherein each specialized recognition engine is specialized for a different range of head poses, the method comprising repeating the computer-implemented method of claim 1 for each of the plurality of specialized recognition engines.

8. The computer-implemented method of claim 7 wherein the known good facial images are frontal views of the head and all of the specialized recognition engines are specialized for non-frontal head poses.

9. The computer-implemented method of claim 7 wherein the plurality of specialized recognition engines are specialized for ranges of head poses that differ by at least ten degrees with respect to at least one of Euler angles (yaw, pitch).

10. The computer-implemented method of claim 7 wherein each specialized recognition engine is specialized over a range of at least +/−

ten degrees with respect to at least one of Euler angles (yaw, pitch).

11. The computer-implemented method of claim 7 wherein the pairs of known good facial images and known good specialized recognition metrics are outputs of a previously trained specialized recognition engine.

12. The computer-implemented method of claim 7 wherein the specialized recognition engines are machine learning engines.

13. The computer-implemented method of claim 7 wherein the plurality of specialized recognition engines includes at least fifteen specialized recognition engines.

14. The computer-implemented method of claim 7 wherein an aggregate of the ranges for the plurality of specialized recognition engines spans at least +/−

forty-five degrees with respect to Euler angle (yaw).

15. The computer-implemented method of claim 7 wherein an aggregate of the ranges for the plurality of specialized recognition engines spans at least forty-five degrees with respect to Euler angle (pitch).

16. The computer-implemented method of claim 1 wherein the specialized recognition metric comprises a confidence level of occurrence of the preselected attribute in said facial image.

17. The computer-implemented method of claim 1 wherein the preselected attribute is facial expression.

18. The computer-implemented method of claim 1 wherein the preselected attribute is gender, age, ethnicity or race.

19. The computer-implemented method of claim 1 wherein the preselected attribute is a lifestyle attribute, a wealth attribute, a health attribute, or a personality attribute.

20. A non-transitory computer readable medium containing instructions that, when executed by a processor, execute a method for training a specialized recognition engine to recognize a preselected attribute of a facial image, the specialized recognition engine specialized for a range of head poses, the method comprising:

for each of a plurality of facial images of heads that are within the range of head poses for the specialized recognition engine;

identifying a pair comprising a known good facial image and a known good specialized recognition metric, wherein the known good facial image is for a same head as said facial image and has the same preselected attribute as said facial image but for a head pose within a range of head poses that is different than the range of head poses for the specialized recognition engine, and the known good specialized recognition metric is indicative of the preselected attribute of the known good facial image; and

associating a specialized recognition metric with said facial image, wherein the specialized recognition metric is derived from the known good specialized recognition metric; and

training the specialized recognition engine by using pairs comprising the plurality of facial images and the associated specialized recognition metrics as a training set for supervised learning.