Identifying facial expressions in acquired digital images

• US 8,488,023 B2
• Filed: 05/20/2010
• Issued: 07/16/2013
• Est. Priority Date: 05/20/2009
• Status: Expired due to Fees

• Current Assignee: FotoNation Limited (Adeia Inc.), National University of Ireland Galway
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.), National University of Ireland Galway
• Inventors: Bacivarov, Ioana, Corcoran, Peter
• Primary Examiner(s): Pasiewicz, Daniel M
• Application Number:
  US12/784,418
• Publication Number:
  US 20110007174A1
• Time in Patent Office: 1,153 Days
• Field of Search: 348 77- 78, 348/239, 348/222.1, 382/118
• US Class Current: 348/239
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/175 Static expression
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body

Abstract

A face is detected and identified within an acquired digital image. One or more features of the face is/are extracted from the digital image, including two independent eyes or subsets of features of each of the two eyes, or lips or partial lips or one or more other mouth features and one or both eyes, or both. A model including multiple shape parameters is applied to the two independent eyes or subsets of features of each of the two eyes, and/or to the lips or partial lips or one or more other mouth features and one or both eyes. One or more similarities between the one or more features of the face and a library of reference feature sets is/are determined. A probable facial expression is identified based on the determining of the one or more similarities.

Claims

1. A method of recognizing a facial expression, comprising:

using a processor;

acquiring a sequence of digital images;

detecting a facial region within a first digital image;

applying a facial model, comprising both shape and texture components, to said facial region to obtain a first detailed match to the facial region;

applying a feature model, comprising both shape and texture components, to a feature within the facial region to obtain a first detailed match to said feature;

adjusting the facial model, responsive to the first detailed match of the second model to said feature, to obtain an improved match of the facial model to the facial region;

adjusting the feature model, responsive to the improved match of the facial model to said facial region, to obtain an improved match to the feature;

extracting principal shape components from both the facial model and the feature model; and

characterizing said principal shape components to determine a best matching facial expression of a discrete set of facial expressions.

2. The method of claim 1, further comprising recognizing another facial expression within one or more additional digital images from said sequence.

3. The method of claim 1, wherein the feature comprises one or two eyes, an eye feature, one or two lips or partial lips, or one or more other mouth features, or combinations thereof.

4. The method of claim 1, wherein the feature comprises a geometric feature.

5. The method of claim 4, wherein the geometric feature comprises a shape, deformation or location of one or more facial components, or a pose variation, or combinations thereof.

6. The method of claim 1, wherein the feature comprises an appearance feature.

7. The method of claim 6, wherein the appearance feature comprises a skin texture change.

8. The method of claim 7, wherein the skin texture change comprises a furrow, a bulge, an expression wrinkle, or an illumination variation, or blushing, or combinations thereof.

9. The method of claim 1, further comprising categorizing the best matching facial expression as indicating surprise, fear, happiness, anger, neutral, sadness, or disgust.

10. The method of claim 1, further comprising adapting a game difficulty based on the best matching facial expression.

11. The method of claim 1, further comprising adapting a game workflow based on the best matching facial expression.

12. The method of claim 1, further comprising mirroring the best matching facial expression in an avatar within a game display.

13. The method of claim 1, wherein the facial model comprises an Active Appearance Model (AAM).

14. The method of claim 1, wherein the principal shape components comprise seven or less principal shape components.

15. The method of claim 1, wherein the principal shape components comprise six or seven principal shape components.

16. One or more non-transitory computer readable media having code embedded therein for programming a processor to perform a method of recognizing a facial expression within a facial region detected within a digital image acquired among a sequence of digital images, wherein the method comprises:

applying a facial model, comprising both shape and texture components, to said facial region to obtain a first detailed match to the facial region;

applying a feature model, comprising both shape and texture components, to a feature within the facial region to obtain a first detailed match to said feature;

adjusting the facial model, responsive to the first detailed match of the second model to said feature, to obtain an improved match of the facial model to the facial region;

adjusting the feature model, responsive to the improved match of the facial model to said facial region, to obtain an improved match to the feature;

extracting principal shape components from both the facial model and the feature model; and

characterizing said principal shape components to determine a best matching facial expression of a discrete set of facial expressions.

17. The one or more non-transitory computer readable media of claim 16, wherein the method further comprises recognizing another facial expression within one or more additional digital images from said sequence.

18. The one or more non-transitory computer readable media of claim 16, wherein the feature comprises one or two eyes, an eye feature, one or two lips or partial lips, or one or more other mouth features, or combinations thereof.

19. The one or more non-transitory computer readable media of claim 16, wherein the feature comprises a geometric feature.

20. The one or more non-transitory computer readable media of claim 19, wherein the geometric feature comprises a shape, deformation or location of one or more facial components, or a pose variation, or combinations thereof.

21. The one or more non-transitory computer readable media of claim 16, wherein the feature comprises an appearance feature.

22. The one or more non-transitory computer readable media of claim 21, wherein the appearance feature comprises a skin texture change.

23. The one or more non-transitory computer readable media of claim 22, wherein the skin texture change comprises a furrow, a bulge, an expression wrinkle, or an illumination variation, or blushing, or combinations thereof.

24. The one or more non-transitory computer readable media of claim 16, wherein the method further comprises categorizing the best matching facial expression as indicating surprise, fear, happiness, anger, neutral, sadness, or disgust.

25. The one or more non-transitory computer readable media of claim 16, wherein the method further comprises adapting a game difficulty based on the best matching facial expression.

26. The one or more non-transitory computer readable media of claim 16, wherein the method further comprises adapting a game workflow based on the best matching facial expression.

27. The one or more non-transitory computer readable media of claim 16, wherein the method further comprises mirroring the best matching facial expression in an avatar within a game display.

28. The one or more non-transitory computer readable media of claim 16, wherein the facial model comprises an Active Appearance Model (AAM).

29. The one or more non-transitory computer readable media of claim 16, wherein the principal shape components comprise seven or less principal shape components.

30. The one or more non-transitory computer readable media of claim 16, wherein the principal shape components comprise six or seven principal shape components.

31. A digital image acquisition device configured to recognize a facial expression, comprising:

a lens and image sensor for capturing a digital image;

a processor;

a face detection module configured to detect a facial region within the digital image;

a feature extraction module configured to program the processor to extract a feature within the facial region

a facial model application module configured to program the processor to apply a facial model, comprising both shape and texture components, to said facial region to obtain a first detailed match to the facial region;

a feature model application module configured to program the processor to apply a feature model, comprising both shape and texture components, to the extracted feature to obtain a first detailed match to said feature;

a facial model adjustment module configured to program the processor to adjust the facial model, responsive to the first detailed match of the second model to said feature, to obtain an improved match of the facial model to the facial region;

a feature model adjustment module configured to program the processor to adjust the feature model, responsive to the improved match of the facial model to said facial region, to obtain an improved match to the feature;

a classification module configured to program the processor to extract principal shape components from both the facial model and the feature model; and

a facial expression module configured to program the processor to characterize said principal shape components to determine a best matching facial expression of a discrete set of facial expressions.

32. The device of claim 31, wherein the feature comprises one or two eyes, an eye feature, one or two lips or partial lips, or one or more other mouth features, or combinations thereof.

33. The device of claim 31, wherein the feature comprises a geometric feature.

34. The device of claim 33, wherein the geometric feature comprises a shape, deformation or location of one or more facial components, or a pose variation, or combinations thereof.

35. The device of claim 31, wherein the feature comprises an appearance feature.

36. The device of claim 35, wherein the appearance feature comprises a skin texture change.

37. The device of claim 36, wherein the skin texture change comprises a furrow, a bulge, an expression wrinkle, or an illumination variation, or blushing, or combinations thereof.

38. The device of claim 31, wherein the best matching facial expression comprises an indication of surprise, fear, happiness, anger, neutral, sadness, or disgust.

39. The device of claim 31, further comprising a game adaption module configured to program the processor to adapt a game difficulty based on the best matching facial expression.

40. The device of claim 39, wherein the game adaptation module is further programmed to adapt a game workflow based on the best matching facial expression.

41. The device of claim 31, further comprising a game adaption module configured to program the processor to adapt a game workflow based on the best matching facial expression.

42. The device of claim 31, further comprising a game adaption module configured to program the processor to mirror the best matching facial expression in an avatar within a game display.

43. The device of claim 31, wherein the facial model comprises an Active Appearance Model (AAM).

44. The device of claim 31, wherein the principal shape components comprise seven or less principal shape components.

45. The device of claim 31, wherein the principal shape components comprise six or seven principal shape components.