Image processing method and apparatus

• US 8,494,232 B2
• Filed: 02/25/2011
• Issued: 07/23/2013
• Est. Priority Date: 05/24/2007
• Status: Expired due to Fees

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.)
• Inventors: Bigioi, Petronel, Corcoran, Peter, Steinberg, Eran
• Primary Examiner(s): TUCKER, WESLEY J
• Application Number:
  US13/034,707
• Publication Number:
  US 20110234847A1
• Time in Patent Office: 879 Days
• Field of Search: 382115-118, 382/173, 382/181, 382/282
• US Class Current: 382/118
• CPC Class Codes: G03B 19/02 Still-picture cameras
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes

Abstract

An image processing technique includes acquiring a main image of a scene and determining one or more facial regions in the main image. The facial regions are analyzed to determine if any of the facial regions includes a defect. A sequence of relatively low resolution images nominally of the same scene is also acquired. One or more sets of low resolution facial regions in the sequence of low resolution images are determined and analyzed for defects. Defect free facial regions of a set are combined to provide a high quality defect free facial region. At least a portion of any defective facial regions of the main image are corrected with image information from a corresponding high quality defect free facial region.

Claims

1. An image processing method comprising:

using a processor-based image acquisition device;

acquiring a main image of a scene and two or more further images approximately of a same scene within a time period including said acquiring of said main image;

determining one or more facial regions in said main image and said two or more further images, including temporarly storing two or more subsampled resolution images in a temporary storage directly from an image sensor, locating said one or more facial regions within each of said two or more subsampled resolution images and predicting said one or more facial regions within said main image based on the locating within the two or more temporarily stored subsampled resolution images;

analyzing said one or more facial regions to determine if any one of said facial regions includes a defect, including

(i) applying a model to each facial region,

(ii) analyzing one or more parameters of said model for each facial region to provide an indication of facial expression; and

(iii) segmenting from said set one or more facial regions indicative of a defective facial expression;

combining defect free facial regions form at least two of said two or more subsampled resolution images to provide a high quality defect free facial region; and

replacing at least a portion of a defective facial region of said main image with image information from a corresponding region of said high quality defect free facial region.

2. A method according to claim 1 wherein said time period includes one or both of a time period just preceding or a time period just following said time for acquiring said main image.

3. A method according to claim 1 wherein said applying a model comprises applying a model that comprises a plurality of vertices defining a periphery of a facial region to each high quality defect free facial region and a corresponding defective facial region;

and

mapping pixels of said high quality defect free facial region to said defective facial region according to the correspondence of vertices for said respective regions.

4. A method according to claim 3 wherein said model comprises an Active Appearance Model (AAM).

5. A method according to claim 1 wherein said main image is acquired at an exposure level different to the exposure level of said one or more further images and wherein said replacing comprises mapping luminance levels of said high quality defect free facial region to luminance levels of said defective facial region.

6. A method according to claim 1 further comprising storing one or more of said further images in an image header file of said main image.

7. A method according to claim 1 wherein said analysing comprises analyzing each facial region to determine if any eyes of said facial region are closed.

8. A method according to claim 1 wherein said analysing comprises analyzing each facial region to determine a configuration of a mouth of said facial region.

9. A method according to claim 1 further comprising:

prior to said combining, resizing and aligning faces containing said defect free facial regions.

10. An image acquisition and processing apparatus for providing images with enhanced facial regions in digital images, including a lens and image sensor, a processor and a memory having code embedded therein for programming the processor to control the apparatus to perform an image processing method that comprises:

a) acquiring a main image of a scene and two or more further images approximately of a same scene within a time period including said acquiring of said main image;

b) determining one or more facial regions in said main image and said two or more further images, including temporarily storing two or more subsampled resolution images in a temporary storage directly from an image sensor, locating said one or more facial regions within each of said two or more subsampled resolution images and predicting said one or more facial regions within said main image based on the locating within the two or more temporarily stored subsampled resolution images;

c) analyzing said one or more facial regions to determine if any one of said facial regions includes a defect, including

(i) applying a model to each facial region,

(ii) analyzing one or more parameters of said model for each facial region to provide an indication of facial expression; and

(iii) segmenting from said set one or more facial regions indicative of a defective facial expression;

d) combining defect free facial regions form at least two of said two or more subsampled resolution images to provide a high quality defect free facial region; and

e) replacing at least a portion of a defective facial region of said main image with image information from a corresponding region of said high quality defect free facial region.

11. An apparatus according to claim 10 wherein said time period includes one or both of a time period just preceding or a time period just following said time for acquiring said main image.

12. An apparatus according to claim 10 wherein said applying a model comprises applying a model that comprises a plurality of vertices defining a periphery of a facial region to each high quality defect free facial region and a corresponding defective facial region;

and

mapping pixels of said high quality defect free facial region to said defective facial region according to the correspondence of vertices for said respective regions.

13. An apparatus according to claim 12 wherein said model comprises an Active Appearance Model (AAM).

14. An apparatus according to claim 10 wherein said main image is acquired at an exposure level different to the exposure level of said one or more further images and wherein said replacing comprises mapping luminance levels of said high quality defect free facial region to luminance levels of said defective facial region.

15. An apparatus according to claim 10, the method further comprising storing one or more of said further images in an image header file of said main image.

16. An apparatus according to claim 10 wherein said analysing comprises analysing each facial region to determine if any eyes of said facial region are closed.

17. An apparatus according to claim 10 wherein said analysing comprises analysing each facial region to determine a configuration of a mouth of said facial region.

18. An apparatus according to claim 10, the method further comprising:

prior to said combining, resizing and aligning faces containing said defect free facial regions.

19. One or more non-transitory, processor-readable media having code embedded therein for programming a processor to control an image acquisition and processing apparatus to perform an image acquisition and processing method that comprises:

a) determining one or more facial regions within a main image of a scene and within two or more further images approximately of said same scene acquired within a time period including said acquiring of said main image, including temporarily storing two or more subsampled resolution images in a temporary storage directly from an image sensor, locating said one or more facial regions within said two or more subsampled resolution images and predicting said one or more facial regions within said main image based on the locating within the two or more temporarily stored subsampled resolution images;

b) analyzing said one or more facial regions to determine if any one of said facial regions includes a defect, including

(i) applying a model to each facial region,

(ii) analyzing one or more parameters of said model for each facial region to provide an indication of facial expression; and

(iii) segmenting from said set one or more facial regions indicative of a defective facial expression;

c) combining defect free facial regions from at least two of said two or more subsampled resolution images to provide a high quality defect free facial region; and

d) replacing at least a portion of a defective facial region of said main image with image information from a corresponding region of said high quality defect free facial region.

20. One or more non-transitory, processor-readable media according to claim 19, wherein said time period includes one or both of a time period just preceding or a time period just following said time for acquiring said main image.

21. One or more non-transitory, processor-readable media according to claim 19, wherein said applying a model comprises applying a model that comprises a plurality of vertices defining a periphery of a facial region to each high quality defect free facial region and a corresponding defective facial region;

and

mapping pixels of said high quality defect free facial region to said defective facial region according to the correspondence of vertices for said respective regions.

22. One or more non-transitory, processor-readable media according to claim 21, wherein said model comprises an Active Appearance Model (AAM).

23. One or more non-transitory, processor-readable media according to claim 19, wherein said main image is acquired at an exposure level different to the exposure level of said one or more further images and wherein said replacing comprises mapping luminance levels of said high quality defect free facial region to luminance levels of said defective facial region.

24. One or more non-transitory, processor-readable media according to claim 19, the method further comprising storing one or more of said further images in an image header file of said main image.

25. One or more non-transitory, processor-readable media according to claim 19, wherein said analysing comprises analysing each facial region to determine if any eyes of said facial region are closed.

26. One or more non-transitory, processor-readable media according to claim 19, wherein said analysing comprises analysing each facial region to determine a configuration of a mouth of said facial region.

27. One or more non-transitory, processor-readable media according to claim 19, the method further comprising:

prior to said combining, resizing and aligning faces containing said defect free facial regions.