Detecting facial expressions in digital images

• US 9,462,180 B2
• Filed: 06/09/2014
• Issued: 10/04/2016
• Est. Priority Date: 01/27/2008
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Neghina, Catalina, Gangea, Mihnea, Petrescu, Stefan, David, Emilian, Bigioi, Petronel, Zarakov, Eric, Steinberg, Eran
• Primary Examiner(s): Li, Ruiping
• Application Number:
  US14/300,150
• Publication Number:
  US 20140347514A1
• Time in Patent Office: 848 Days
• Field of Search: 348/222.1
• US Class Current: 1/1
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/175 Static expression
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body

Abstract

A method and system for detecting facial expressions in digital images and applications therefore are disclosed. Analysis of a digital image determines whether or not a smile and/or blink is present on a person'"'"'s face. Face recognition, and/or a pose or illumination condition determination, permits application of a specific, relatively small classifier cascade.

Claims

1. A method of in-camera processing to create a photographic record of a captured image of a group of faces by increasing smiling faces in the image, the method comprising:

acquiring the captured image with a camera having an image sensor for acquiring digital images;

providing one or more digitally-acquired first images of the group of faces in addition to the captured image;

from among the group of faces, identifying a number of non-smiling faces in one of the images;

based, at least in part, on the number of non-smiling faces, initiating a plurality of operations including, when a predetermined number of non-smiling faces are present in said one of the images;

(i) delaying acquisition and recording of the captured image to allow time for reducing non-smiling faces in the captured image relative to said number of non-smiling faces in said one of the images; and

(ii) replacing a face region in the captured image by combining a face region in at least one of the one or more digitally-acquired images into the captured image; and

storing the captured image with the replaced face region as digital content in a mass storage device.

2. The method of claim 1 wherein the operations are initiated in response to the number of non-smiling faces being above a threshold number.

3. The method of claim 1 wherein:

the method further comprises determining that a particular percentage of the plurality of faces are smiling; and

the operations are initiated in response to determining that the particular percentage is less than a threshold percentage.

4. The method of claim 1 wherein the operations include issuing a warning beep.

5. The method of claim 2 wherein delaying acquisition and recording of the captured image is until the particular number is no longer above the threshold number.

6. The method of claim 1 wherein the one or more digitally-acquired first images are a constant stream of preview images.

7. The method of claim 6, wherein the captured image is of higher resolution than the constant stream of preview images.

22. The method of claim 6 wherein the face region of said at least one digitally-acquired image includes a mouth region with a higher calculated probability of being classified as a smile than the replaced face region.

8. A portable digital image acquisition device for creating a photographic record of a captured image of a group of faces by increasing smiling faces in an image of a group of faces, the image acquisition device including a lens and an image sensor for acquiring digital images, a processor, and one or more processor-readable media having code embedded therein for programming the processor to perform a method of in-camera processing, wherein the method comprises:

acquiring the captured image;

providing one or more digitally-acquired first images of the group of faces in addition to the captured image;

from among the group of faces, identifying a number of non-smiling faces in one of the images;

based, at least in part, on the number of non-smiling faces in said one of the images, initiating a plurality of operations including, when at least a predetermined number of non-smiling faces are present in said one of the images;

(i) delaying acquisition and recording of the captured image to allow time for reducing non-smiling faces in the captured image relative to the number of non-smiling faces in said one of the images; and

(ii) replacing a face region in the captured image by combining a face region in at least one of the one or more digitally-acquired images into the captured image.

9. The device of claim 8 wherein the one or more operations are initiated in response to the number being above a threshold number.

10. The device of claim 8 wherein:

the method further comprises determining that a particular percentage of the plurality of faces are smiling; and

the operations are initiated in response to determining that the particular percentage is less than a threshold percentage.

11. The device of claim 8 wherein the operations include issuing a warning beep.

12. The device of claim 9 wherein delaying acquisition and recording of the captured image is until the particular number is no longer above the threshold number.

13. The device of claim 8 wherein the one or more digitally-acquired first images are a constant stream of preview images.

14. The device of claim 13, wherein the captured image is of higher resolution than the constant stream of preview images.

21. The one or more processor-readable media of claim 13, wherein the captured image is of higher resolution than the constant stream of preview images.

23. The portable digital image acquisition device of claim 8 wherein the face region of said at least one digitally-acquired image includes a mouth region with a higher calculated probability of being classified as a smile than the replaced face region.

15. One or more non-transitory processor-readable media having code embedded therein for programming a processor to create a captured image of a group of faces with an image acquisition device according to a method of in-camera processing, the method comprising:

acquiring the captured image;

providing one or more digitally-acquired first images of the group of faces in addition to the captured image;

from among the group of faces, identifying a number of non-smiling faces in one of the images;

based, at least in part, on the number of non-smiling faces in said one of the images, initiating a plurality of operations including, when at least a predetermined number of non-smiling faces are present in said one of the first images;

(i) delaying acquisition and recording of the captured image to allow time for reducing non-smiling faces in the captured image relative to the number of non-smiling faces in said one of the images; and

(ii) replacing a face region in the captured image by combining a face region in at least one of the one or more digitally-acquired images into the captured image.

16. The one or more processor-readable media of claim 15 wherein the operations are initiated in response to said number being above a threshold number.

17. The one or more processor-readable media of claim 15 wherein:

the method further comprises determining that a particular percentage of the plurality of faces are smiling; and

the operations are initiated in response to determining that the particular percentage is less than a threshold percentage.

18. The one or more processor-readable media of claim 15 wherein the operations include issuing a warning beep.

19. The one or more processor-readable media of claim 16 wherein delaying acquisition of an image and recording of the captured image is until the particular number is no longer above the threshold number.

20. The one or more processor-readable media of claim 15 wherein the one or more digitally-acquired first images are a constant stream of preview images.

24. The one or more non-transitory processor-readable media of claim 15 wherein the face region of said at least one digitally-acquired image includes a mouth region with a higher calculated probability of being classified as a smile than the replaced face region.