Image processing method and apparatus

• US 8,515,138 B2
• Filed: 05/08/2011
• Issued: 08/20/2013
• Est. Priority Date: 05/24/2007
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia, Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia, Inc.)
• Inventors: Bigioi, Petronel, Corcoran, Peter, Steinberg, Eran
• Primary Examiner(s): TUCKER, WESLEY J
• Application Number:
  US13/103,077
• Publication Number:
  US 20110235912A1
• Time in Patent Office: 835 Days
• Field of Search: 382/100, 382115-118, 382/173, 382/181, 382/282
• US Class Current: 382/118
• CPC Class Codes: G03B 19/02 Still-picture cameras
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes

Abstract

An image processing technique includes acquiring a main image of a scene and determining one or more facial regions in the main image. The facial regions are analysed to determine if any of the facial regions includes a defect. A sequence of relatively low resolution images nominally of the same scene is also acquired. One or more sets of low resolution facial regions in the sequence of low resolution images are determined and analysed for defects. Defect free facial regions of a set are combined to provide a high quality defect free facial region. At least a portion of any defective facial regions of the main image are corrected with image information from a corresponding high quality defect free facial region.

Claims

1. An image processing method comprising:

using a processor-based image acquisition device;

acquiring a main image of a scene and two or more further images approximately of a same scene within a time period including said acquiring of said main image;

determining one or more regions of interest in said main image and said two or more further images, including temporarily storing two or more subsampled resolution images in a temporary storage directly from an image sensor, locating said one or more regions of interest within each of said two or more subsampled resolution images and predicting said one or more regions of interest within said main image based on the locating within the two or more temporarily stored subsampled resolution images;

analyzing said one or more regions of interest; and

determining to initiate a further action based on said analyzing of at least one region of interest, including

applying a model to said at least one region of interest;

analyzing one or more parameters of said model for said at least one region of interest to provide an indication that a characteristic of said at least one region of interest has at least a threshold value;

segmenting from said main image or said two or more further images, or both, said at least one region of interest;

comparing said at least one region of interest with at least one stored profile region also having at least said threshold value of said characteristic; and

initiating said further action based on said analyzing of said at least one region of interest and on said comparing.

2. A method according to claim 1 wherein said time period includes one or both of a time period just preceding or a time period just following said time for acquiring said main image.

3. A method according to claim 1 wherein said model comprises a plurality of vertices defining a periphery of the at least one region of interest, and said method comprises mapping pixels of said at least one stored profile region to said at least one region of interest according to a correspondence of vertices for said respective regions.

4. A method according to claim 3 wherein said model comprises an Active Appearance Model (AAM).

5. A method according to claim 1 wherein said main image is acquired at an exposure level different to the exposure level of said one or more further images and wherein said comparing comprises mapping luminance levels of said at least one stored profile region to luminance levels of said at least one region of interest.

6. A method according to claim 1 further comprising storing one or more of said further images in an image header file of said main image.

7. A method according to claim 1 further comprising:

prior to said comparing, resizing and aligning one or both of said at least one region of interest and said at least one stored profile region.

8. An image acquisition and processing apparatus, comprising:

a lens and image sensor for acquiring a main image of a scene and two or more further images approximately of a same scene within a time period including said acquiring of said main image;

a processor; and

a memory having code embedded therein for programming the processor to control the apparatus to perform an image processing method that comprises;

determining one or more regions of interest in said main image and said two or more further images, including temporarily storing two or more subsampled resolution images in a temporary storage directly from an image sensor, locating said one or more regions of interest within each of said two or more subsampled resolution images and predicting said one or more regions of interest within said main image based on the locating within the two or more temporarily stored subsampled resolution images;

analyzing said one or more regions of interest; and

determining to initiate a further action based on said analyzing of at least one region of interest, including

applying a model to said at least one region of interest;

analyzing one or more parameters of said model for said at least one region of interest to provide an indication that a characteristic of said at least one region of interest has at least a threshold value;

segmenting from said main image or said two or more further images, or both, said at least one region of interest;

comparing said at least one region of interest with at least one stored profile region also having at least said threshold value of said characteristic; and

initiating said further action based on said analyzing of said at least one region of interest and on said comparing.

9. An apparatus according to claim 8, wherein said time period includes one or both of a time period just preceding or a time period just following said time for acquiring said main image.

10. An apparatus according to claim 8, wherein said model comprises a plurality of vertices defining a periphery of the at least one region of interest, and said method comprises mapping pixels of said at least one stored profile region to said at least one region of interest according to a correspondence of vertices for said respective regions.

11. An apparatus according to claim 10, wherein said model comprises an Active Appearance Model (AAM).

12. An apparatus according to claim 8, wherein said main image is acquired at an exposure level different to the exposure level of said one or more further images and wherein said comparing comprises mapping luminance levels of said at least one stored profile region to luminance levels of said at least one region of interest.

13. An apparatus according to claim 8, the method further comprising storing one or more of said further images in an image header file of said main image.

14. An apparatus according to claim 8, the method further comprising:

prior to said comparing, resizing and aligning one or both of said at least one region of interest and said at least one stored profile region.

15. One or more non-transitory, processor-readable media having code embedded therein for programming a processor to control an image acquisition and processing apparatus to perform an image acquisition and processing method that comprises:

determining one or more regions of interest in said main image and said two or more further images, including temporarily storing two or more subsampled resolution images in a temporary storage directly from an image sensor, locating said one or more regions of interest within each of said two or more subsampled resolution images and predicting said one or more regions of interest within said main image based on the locating within the two or more temporarily stored subsampled resolution images;

analyzing said one or more regions of interest; and

determining to initiate a further action based on said analyzing of at least one region of interest, including

applying a model to said at least one region of interest;

analyzing one or more parameters of said model for said at least one region of interest to provide an indication that a characteristic of said at least one region of interest has at least a threshold value;

segmenting from said main image or said one two or more further images, or both, said at least one region of interest;

comparing said at least one region of interest with at least one stored profile region also having at least said threshold value of said characteristic; and

initiating said further action based on said analyzing of said at least one region of interest and on said comparing.

16. One or more non-transitory, processor-readable media according to claim 15, wherein said time period includes one or both of a time period just preceding or a time period just following said time for acquiring said main image.

17. One or more non-transitory, processor-readable media according to claim 15, wherein said model comprises a plurality of vertices defining a periphery of the at least one region of interest, and said method comprises mapping pixels of said at least one stored profile region to said at least one region of interest according to a correspondence of vertices for said respective regions.

18. One or more non-transitory, processor-readable media according to claim 17, wherein said model comprises an Active Appearance Model (AAM).

19. One or more non-transitory, processor-readable media according to claim 15, wherein said main image is acquired at an exposure level different to the exposure level of said one or more further images and wherein said comparing comprises mapping luminance levels of said at least one stored profile region to luminance levels of said at least one region of interest.

20. One or more non-transitory, processor-readable media according to claim 19, the method further comprising storing one or more of said further images in an image header file of said main image.

21. One or more non-transitory, processor-readable media according to claim 15, the method further comprising:

prior to said comparing, resizing and aligning one or both of said at least one region of interest and said at least one stored profile region.