Detecting facial expressions in digital images

• US 8,750,578 B2
• Filed: 01/15/2009
• Issued: 06/10/2014
• Est. Priority Date: 01/29/2008
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.)
• Inventors: Gangea, Mihnea, Petrescu, Stefan, Bigioi, Petronel, David, Emilian, Zarakov, Eric, Steinberg, Eran, Neghina, Catalina
• Primary Examiner(s): Li, Ruiping
• Application Number:
  US12/354,707
• Publication Number:
  US 20090190803A1
• Time in Patent Office: 1,972 Days
• Field of Search: 382/118, 382/203, 348/207.99, 348/E05.024
• US Class Current: 382/118
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/175 Static expression
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body

Abstract

A method and system for detecting facial expressions in digital images and applications therefore are disclosed. Analysis of a digital image determines whether or not a smile and/or blink is present on a person'"'"'s face. Face recognition, and/or a pose or illumination condition determination, permits application of a specific, relatively small classifier cascade.

Claims

1. A method of in-camera processing of a still image including one or more faces as part of an acquisition process, comprising:

identifying a group of pixels that correspond to a face within at least one digitally-acquired image on a portable camera;

acquiring a stream of images including the face;

performing the following steps in real time as the stream of images is acquired by the portable camera;

tracking said face within said stream of images;

for each interval of a plurality of intervals within the stream of images, performing;

determining a smile classification for the interval; and

based on the smile classification, updating a confidence parameter;

after updating the confidence parameter for each of the plurality of intervals, making a smiling decision based, at least in part, on the sign and absolute value of the confidence parameter;

wherein each interval of the plurality of intervals includes one or more imam from the stream of images;

initiating one or more smile state-dependent operations selected based upon results of the smiling decision.

2. The method of claim 1, further comprising applying face recognition to the face within one or more frames of the stream of images.

3. The method of claim 1, further comprising determining a pose or illumination condition, or both, for the face, and training a specific set of face classifiers adjusted based on the determined pose or illumination or both.

4. The method of claim 1, wherein the step of making a smiling decision comprises assigning a chain of Haar and/or census features.

5. The method of claim 1, further comprising acquiring cropped versions of the face within each of multiple frames of the stream of images including substantially only a region of the image that includes the face.

6. The method of claim 5, wherein the cropped versions each comprise substantially only a region of the image that includes a mouth region of the face.

7. The method of claim 1, wherein the step of making a smiling decision comprises thresholding, such that a smiling decision result comprises smile, no smile or inconclusive.

8. The method of claim 1. wherein the step of making a smiling decision further comprises calculating a statistical smile difference vector between multiple frames of said stream of images, and determining that a certain threshold or more of difference corresponds to a sudden change in pose, illumination, or other image parameter, or to a changing smile state, and wherein the step of making a smiling decision further comprises confirming a particular cause of the certain threshold or more of difference.

9. The method of claim 1, wherein multiple faces are identified and tracked, and a smiling decision for each of the multiple faces is made, and the method further comprises initiating a smile-dependent group shot operation if the smiling decision for more than a first threshold number of faces is no smile or if the smiling decision for less than a second threshold number of faces is smile, or both.

10. The method of claim 1, further comprising compositing a best smile image including combining one or more face regions of the stream of digitally acquired images with a best smile region of one or more of the stream of images.

51. The method of claim 1, wherein each interval of the plurality of intervals includes a single frame.

52. The method of claim 1, wherein the step of making a smiling decision is performed after each interval of the plurality of intervals.

53. The method of claim 1, wherein updating the confidence parameter based on the smile classification made for an interval includes:

incrementing the confidence parameter if the smile classification made for the interval indicates a positive smile response; and

decrementing the confidence parameter if the smile classification made for the interval indicates a negative smile response.

11. A portable digital image acquisition device, including a lens and image sensor for acquiring digital images, a processor, and one or more processor-readable media have code embedded therein for programming the processor to perform a method of in-camera processing of a still image including one or more faces as part of an acquisition process, wherein the method comprises:

identifying a group of pixels that correspond to a face within at least one digitally-acquired image on a portable camera;

acquiring a stream of images including the face;

performing the following steps in real time as the stream of images is acquired by the portable camera;

tracking said face within said stream of images;

for each interval of a plurality of intervals within the stream of images, performing;

determining a smile classification for the interval; and

based on the smile classification, updating a confidence parameter;

after updating the confidence parameter for each of the plurality of intervals, making a smiling decision based, at least in part, on the sign and absolute value of the confidence parameter;

wherein each interval of the plurality of intervals includes one or more imam from the stream of images;

initiating one or more smile state-dependent operations selected based upon results of the smiling decision.

12. The device of claim 11, wherein the method further comprises applying face recognition to the face within one or more frames of the stream of images.

13. The device of claim 12, wherein the method further comprises training a relatively short classifier cascade of images that each include a specifically-recognized person'"'"'s face.

14. The device of claim 13, wherein the relatively short classifier cascade comprises different poses or illuminations, or both, of the specifically-recognized person'"'"'s face.

15. The device of claim 14, wherein the method further comprises determining a pose or illumination condition, or both, and adjusting the relatively short classifier cascade based on the determined pose or illumination condition or both.

16. The device of claim 12, wherein the method further comprises initiating or delaying an image acquisition when the smiling decision for the face is smile or no-smile, or combinations thereof.

17. The device of claim 12, wherein the method further comprises delaying an image acquisition when the face is not recognized as a specifically-recognized person or the smiling decision for the face is not smile.

18. The device of claim 12, wherein the method further comprises initiating an image acquisition when the face is recognized as a specifically-recognized person and the smiling decision for the face is smile.

19. The device of claim 11, wherein the method further comprises determining a pose or illumination condition, or both, for the face, and training a specific set of face classifiers adjusted based on the determined pose or illumination condition, or both.

20. The device of claim 11, wherein the step of making a smiling decision comprises assigning a chain of Haar and/or census features.

21. The device of claim 11, further comprising acquiring cropped versions of the face within each of multiple frames of the stream of images including substantially only a region of the image that includes the face.

22. The device of claim 21, wherein each of the cropped versions comprise substantially only a region of the image that includes a mouth region of the face.

23. The device of claim 11, wherein the step of making a smiling decision comprises thresholding, such that a smiling decision result comprises smile, no smile or inconclusive.

24. The device of claim 11, wherein the step of making a smiling decision further comprises calculating a statistical smile difference vector between multiple frames of said stream of images, and determining that a certain threshold or more of difference corresponds to a sudden change in pose, illumination, or other image parameter, or to a changing smile state, and wherein the step of making decision further comprises confirming a particular cause of the certain threshold or more of difference.

25. The device of claim 11, wherein multiple faces are identified and tracked, and a smiling decision for each of the multiple faces is made, and the method further comprises initiating a smile-dependent group shot operation if the smiling decision for more than a first threshold number of faces is no smile or if the smiling decision for less than a second threshold number of faces is smile, or both.

26. The device of claim 25, wherein the smile-dependent group shot operation comprises triggering a warning signal to a user or delaying acquisition of a group shot until determining that the smiling decision for less than the first threshold number of faces is no smile or that the smiling decision for more than the second threshold number of faces is smile, or both.

27. The device of claim 11, wherein the method further comprises compositing a best smile image including combining one or more face regions of the stream of digitally-acquired images with a best smile region of one or more of the stream of images.

28. The device of claim 27, wherein the best smile region comprises a mouth region with a highest probability of being classified as a smile.

54. The device of claim 11, wherein each interval of the plurality of intervals includes a single frame.

55. The device of claim 11, wherein the step of making a smiling decision is performed after each interval of the plurality of intervals.

56. The device of claim 11, wherein updating the confidence parameter based on the smile classification made for an interval includes:

incrementing the confidence parameter if the smile classification made for the interval indicates a positive smile response; and

decrementing the confidence parameter if the smile classification made for the interval indicates a negative smile response.

29. One or more non-transitory processor-readable media having code embedded therein for programming a processor to perform a method of in-camera processing of a still image including one or more faces as part of an acquisition process, wherein the method comprises:

identifying a group of pixels that correspond to a face within at least one digitally-acquired image on a portable camera;

acquiring a stream of images including the face;

performing the following steps in real time as the stream of images is acquired by the portable camera;

tracking said face within said stream of images;

for each interval of a plurality of intervals within the stream of images, performing;

determining a smile classification for the interval; and

based on the smile classification, updating a confidence parameter;

after updating the confidence parameter for each of the plurality of intervals, making a smiling decision based, at least in part, on the sign and absolute value of the confidence parameter;

wherein each interval of the plurality of intervals includes one or more imam from the stream of images;

initiating one or more smile state-dependent operations selected based upon results of the smiling decision.

30. The one or more processor-readable media of claim 29, wherein the method further comprises applying face recognition to the face within one or more frames of the stream of images.

31. The one or more processor-readable media of claim 30, wherein the method further comprises training a relatively short classifier cascade of images that each include a specifically-recognized person'"'"'s face.

32. The one or more processor-readable media of claim 31, wherein the relatively short classifier cascade comprises different poses and illuminations of the specifically-recognized person'"'"'s face.

33. The one or more processor-readable media of claim 32, wherein the method further comprises determining a pose or illumination condition, or both, and adjusting the relatively short classifier cascade based on the determined pose or illumination or both.

34. The one or more processor-readable media of claim 30, wherein the method further comprises initiating or delaying an image acquisition when the face is or is not recognized as one of one or more specific persons, or combinations thereof.

35. The one or more processor-readable media of claim 30, wherein the method further comprises initiating or delaying an image acquisition when the smiling decision for the face is a smile or no-smile, or combinations thereof.

36. The one or more processor-readable media of claim 30, wherein the method further comprises delaying an image acquisition when the face is not recognized as a specifically-recognized person or the smiling decision for the face is not a smile.

37. The one or more processor-readable media of claim 30, wherein the method further comprises initiating an image acquisition when the face is recognized as a specifically-recognized person and the smiling decision for the face is a smile.

38. The one or more processor-readable media of claim 29, wherein the method further comprises determining a pose or illumination condition, or both, for the face, and training a specific set of face classifiers adjusted based on the determined pose or illumination or both.

39. The one or more processor-readable media of claim 29, wherein the making of a smiling decision comprises assigning a chain of Haar and/or census features.

40. The one or more processor-readable media of claim 39, wherein the identifying of the group of pixels that correspond to the face comprises applying approximately same Haar and/or census features as the making of a smiling decision.

41. The one or more processor-readable media of claim 29, further comprising acquiring cropped versions of the face within each of multiple frames of the stream of images including substantially only a region of the image that includes the face.

42. The one or more processor-readable media of claim 41, wherein the cropped versions each comprise substantially only a region of the image that includes a mouth region of the face.

43. The one or more processor-readable media of claim 41, wherein the step of making a smiling decision comprises thresholding, such that a smiling decision result comprises smile, no smile or inconclusive.

44. The one or more processor-readable media of claim 43, wherein the thresholding comprises comparing the confidence parameter to a first threshold between 60%-90% likely to be a smile, or to a second threshold of 10%-40% likely to be a smile or both, with said 60%-90% or more corresponding to a smile result, and with said 10%-40% or less corresponding to a no smile result, and with between said 10%-40% and said 60%-90% corresponding to an inconclusive result.

45. The one or more processor-readable media of claim 44, wherein said first threshold comprises approximately 80% and said second threshold comprises approximately 20%.

46. The one or more processor-readable media of claim 29, wherein the step of making a smiling decision further comprises calculating a statistical smile difference vector between multiple frames of said stream of images and determining that a certain threshold or more of difference corresponds to a sudden change in pose, illumination, or other image parameter, or to a changing smile state, and wherein the step of making a smiling decision further comprises confirming a particular cause of the certain threshold or more of difference.

47. The one or more processor-readable media of claim 29, wherein multiple faces are identified and tracked, and a smiling decision for each of the multiple faces is made, and the method further comprises initiating a smile-dependent group shot operation if the smiling decision for more than a first threshold number of faces is no smile or if the smiling decision for less than a second threshold number of faces is smile, or both.

48. The one or more processor-readable media of claim 47, wherein the smile-dependent group shot operation comprises triggering a warning signal to a user or delaying acquisition of a group shot until determining that the smiling decision for less than the first threshold number of faces is no smile or that the smiling decision for more than the second threshold number of faces is smile, or both.

49. The one or more processor-readable media of claim 29, wherein the method further comprises compositing a best smile image including combining one or more face regions of the stream of digitally-acquired images with a best smile region of one or more of the stream of images.

50. The one or more processor-readable media of claim 49, wherein the best smile region comprises a mouth region with a highest probability of being classified as a smile.

57. The one or more processor-readable media of claim 29, wherein each interval of the plurality of intervals includes a single frame.

58. The one or more processor-readable media of claim 29, wherein the step of making a smiling decision is performed after each interval of the plurality of intervals.

59. The one or more processor-readable media of claim 29, wherein updating the confidence parameter based on the smile classification made for an interval includes:

incrementing the confidence parameter if the smile classification made for the interval indicates a positive smile response; and

decrementing the confidence parameter if the smile classification made for the interval indicates a negative smile response.