DETECTING FACIAL EXPRESSIONS IN DIGITAL IMAGES

• US 20090190803A1
• Filed: 01/15/2009
• Published: 07/30/2009
• Est. Priority Date: 01/29/2008
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Ireland Limited (Adeia Inc.)
• Inventors: Zarakov, Eric, David, Emilian, Neghina, Catalina, Steinberg, Eran, Petrescu, Stefan, Gangea, Mihnea, Bigioi, Petronel
• Granted Patent: US 8,750,578 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/175 Static expression
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body

Abstract

A method and system for detecting facial expressions in digital images and applications therefore are disclosed. Analysis of a digital image determines whether or not a smile and/or blink is present on a person'"'"'s face. Face recognition, and/or a pose or illumination condition determination, permits application of a specific, relatively small classifier cascade.

Claims

1. A method of in-camera processing of a still image including one or more faces as part of an acquisition process, comprising:

identifying a group of pixels that correspond to a face within at least one digitally-acquired image on a portable camera;

generating in-camera, capturing or otherwise obtaining in-camera a collection of relatively lower resolution images including a face;

tracking said face within said collection of relatively lower resolution images;

acquiring cropped versions of multiple images of the collection of relatively lower resolution images including the face;

accumulating smile state information of the face over the multiple images;

classifying a statistical smile state of the face based on the accumulated smile state information; and

initiating one or more smile state-dependent operations selected based upon results of the analyzing.

2. The method of claim 1, further comprising applying face recognition to one or more of the multiple images.

3. The method of claim 1, further comprising determining a pose or illumination condition, or both, for the face, and training a specific set of face classifiers adjusted based on the determined pose or illumination or both.

4. The method of claim 1, wherein the classifying of the statistical smile state comprises assigning a chain of Haar and/or census features.

5. The method of claim 1, wherein the cropped versions each comprise substantially only a region of the image that includes the face.

6. The method of claim 1, wherein the cropped versions each comprise substantially only a region of the image that includes a mouth region of the face.

7. The method of claim 1, wherein the classifying comprises thresholding, such that a classifying result comprises smile, no smile or inconclusive.

8. The method of claim 1, wherein the classifying further comprises calculating a statistical smile difference vector between frames of said collection of relatively lower resolution images, and determining that a certain threshold or more of difference corresponds to a sudden change in pose, illumination, or other image parameter, or to a changing smile state, and wherein the classifying further comprises confirming a particular cause of the certain threshold or more of difference.

9. The method of claim 1, wherein multiple faces are identified and tracked, and smile state information for each of the multiple faces is classified, and the method further comprises initiating a smile-dependent group shot operation if more than a first threshold number of faces is classified as no smile or if less than a second threshold number of faces is classified as smile, or both.

10. The method of claim 1, further comprising compositing a best smile image including combining one or more face regions of the at least one digitally-acquired images with a best smile region of one or more of the images of the collection of relatively lower resolution images.

12. The device of claim 1 further comprising applying face recognition to one or more of the multiple images.

13. The device of claim 12, further comprising training a relatively short classifier cascade of images that each include a specifically-recognized person'"'"'s face.

14. The device of claim 13, wherein the relatively short classifier cascade comprises different poses or illuminations, or both, of the specifically-recognized person'"'"'s face.

15. The device of claim 12, further comprising determining a pose or illumination condition, or both, and adjusting the relatively short classifier cascade base on the determined pose or illumination or both.

16. The device of claim 12, further comprising initiating or delaying an image acquisition when the face is classified as smile or no-smile, or combinations thereof.

17. The device of claim 12, further comprising delaying an image acquisition when the face is not recognized as a specifically-recognized person or is not classified as a smile.

18. The device of claim 12, further comprising initiating an image acquisition when the face is recognized as a specifically-recognized person and is classified as a smile.

20. The device of claim 1 wherein the classifying of the statistical smile state comprises assigning a chain of Haar and/or census features.

25. The device of claim 1 wherein the classifying further comprises calculating a statistical smile difference vector between frames of said collection of relatively lower resolution images, and determining that a certain threshold or more of difference corresponds to a sudden change in pose, illumination, or other image parameter, or to a changing smile state, and wherein the classifying further comprises confirming a particular cause of the certain threshold or more of difference.

11. A portable digital image acquisition device, including a lens and image sensor for acquiring digital images, a processor, and one or more processor-readable media have code embedded therein for programming the processor to perform a method of in-camera processing of a still image including one or more faces as part of an acquisition process, wherein the method comprises:

identifying a group of pixels that correspond to a face within at least one digitally-acquired image on a portable camera;

generating in-camera, capturing or otherwise obtaining in-camera a collection of relatively lower resolution images including a face;

tracking said face within said collection of relatively lower resolution images;

acquiring cropped versions of multiple images of the collection of relatively lower resolution images including the face;

accumulating smile state information of the face over the multiple images;

classifying a statistical smile state of the face based on the accumulated smile state information; and

initiating one or more smile state-dependent operations selected based upon results of the analyzing.

19. The device of claim 11, further comprising determining a pose or illumination condition, or both, for the face, and training a specific set of face classifiers adjusted based on the determined pose or illumination condition, or both.

22. The device of claim 11, wherein the cropped versions each comprise substantially only a region of the image that includes the face.

23. The device of claim 11, wherein the cropped versions each comprise substantially only a region of the image that includes a mouth region of the face.

24. The device of claim 11, wherein the classifying comprises thresholding, such that a classifying result comprises smile, no smile or inconclusive.

27. The device of claim 11, wherein multiple faces are identified and tracked, and smile state information for each of the multiple faces is classified, and the method further comprises initiating a smile-dependent group shot operation if more than a first threshold number of faces is classified as no smile or if less than a second threshold number of faces is classified as smile, or both.

28. The device of claim 27, wherein the smile-dependent group shot operation comprises triggering a warning signal to a user or delaying acquisition of a group shot until determining that less than the first threshold number of faces is classified as no smile or that more than the second threshold number of faces is classified as smile, or both.

29. The device of claim 11, further comprising compositing a best smile image including combining one or more face regions of the at least one digitally-acquired images with a best smile region of one or more of the images of the collection of relatively lower resolution images.

30. The device of claim 29, wherein the best smile region comprises a mouth region with a highest probability of being classified as a smile.

31. One or more processor-readable media having code embedded therein for programming a processor to perform a method of in-camera processing of a still image including one or more faces as part of an acquisition process, wherein the method comprises:

identifying a group of pixels that correspond to a face within at least one digitally-acquired image on a portable camera;

generating in-camera, capturing or otherwise obtaining in-camera a collection of relatively lower resolution images including a face;

tracking said face within said collection of relatively lower resolution images;

acquiring cropped versions of multiple images of the collection of relatively lower resolution images including the face;

accumulating smile state information of the face over the multiple images;

classifying a statistical smile state of the face based on the accumulated smile state information; and

initiating one or more smile state-dependent operations selected based upon results of the analyzing.

32. The one or more processor-readable media of claim 31, wherein the method further comprises applying face recognition to one or more of the multiple images.

33. The one or more processor-readable media of claim 32, wherein the method further comprises training a relatively short classifier cascade of images that each include a specifically-recognized person'"'"'s face.

34. The one or more processor-readable media of claim 33, wherein the relatively short classifier cascade comprises different poses and illuminations of the specifically-recognized person'"'"'s face.

35. The one or more processor-readable media of claim 32, wherein the method further comprises determining a pose or illumination condition, or both, and adjusting the relatively short classifier cascade base on the determined pose or illumination or both.

36. The one or more processor-readable media of claim 32, wherein the method further comprises initiating or delaying an image acquisition when the face is or is not recognized as one of one or more specific persons, or combinations thereof.

37. The one or more processor-readable media of claim 32, wherein the method further comprises initiating or delaying an image acquisition when the face is classified as smile or no-smile, or combinations thereof.

38. The one or more processor-readable media of claim 32, wherein the method further comprises delaying an image acquisition when the face is not recognized as a specifically-recognized person or is not classified as a smile.

39. The one or more processor-readable media of claim 32, wherein the method further comprises initiating an image acquisition when the face is recognized as a specifically-recognized person and is classified as a smile.

40. The one or more processor-readable media of claim 31, wherein the method further comprises determining a pose or illumination condition, or both, for the face, and training a specific set of face classifiers adjusted based on the determined pose or illumination or both.

41. The one or more processor-readable media of claim 31, wherein the classifying of the statistical smile state comprises assigning a chain of Haar and/or census features.

42. The one or more processor-readable media of claim 31, wherein the identifying of the group of pixels that correspond to a face comprises applying approximately the same Haar and/or census features as the classifying.

43. The one or more processor-readable media of claim 31, wherein the cropped versions each comprise substantially only a region of the image that includes the face.

44. The one or more processor-readable media of claim 31, wherein the cropped versions each comprise substantially only a region of the image that includes a mouth region of the face.

45. The one or more processor-readable media of claim 31, wherein the classifying comprises thresholding, such that a classifying result comprises smile, no smile or inconclusive.

46. The one or more processor-readable media of claim 45, wherein the thresholding comprises comparing the statistical smile state to a first threshold between 60%-90% likely to be a smile, or to a second threshold of 10%-40% likely to be a smile, or both, with said 60%-90% or more corresponding to a smile result, and with said 10%-40% or less corresponding to a no smile result, and with between said 10%-40% and said 60%-90% corresponding to an inconclusive result.

47. The one or more processor-readable media of claim 46, wherein said first threshold comprises approximately 80% and said second threshold comprises approximately 20%.

48. The one or more processor-readable media of claim 31, wherein the classifying further comprises calculating a statistical smile difference vector between frames of said collection of relatively lower resolution images, and determining that a certain threshold or more of difference corresponds to a sudden change in pose, illumination, or other image parameter, or to a changing smile state, and wherein the classifying further comprises confirming a particular cause of the certain threshold or more of difference.

49. The one or more processor-readable media of claim 31, wherein multiple faces are identified and tracked, and smile state information for each of the multiple faces is classified, and the method further comprises initiating a smile-dependent group shot operation if more than a first threshold number of faces is classified as no smile or if less than a second threshold number of faces is classified as smile, or both.

50. The one or more processor-readable media of claim 49, wherein the smile-dependent group shot operation comprises triggering a warning signal to a user or delaying acquisition of a group shot until determining that less than the first threshold number of faces is classified as no smile or that more than the second threshold number of faces is classified as smile, or both.

51. The one or more processor-readable media of claim 31, wherein the method further comprises compositing a best smile image including combining one or more face regions of the at least one digitally-acquired images with a best smile region of one or more of the images of the collection of relatively lower resolution images.

52. The one or more processor-readable media of claim 51, wherein the best smile region comprises a mouth region with a highest probability of being classified as a smile.