Digital image enhancement with reference images

• US 8,213,737 B2
• Filed: 06/20/2008
• Issued: 07/03/2012
• Est. Priority Date: 06/21/2007
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Bigioi, Petronel, Zamfir, Adrian, Drimbarean, Alexandru, Corcoran, Peter
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): Patel, Nirav G
• Application Number:
  US12/142,773
• Publication Number:
  US 20080317379A1
• Time in Patent Office: 1,474 Days
• Field of Search: 382/275
• US Class Current: 382/275
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 2207/30216 Redeye defect
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 25/44 by partially reading an SSIS array
• CPC Class Codes: H04N 25/46 by combining or binning pixels
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/907 using static stores, e.g. storage tubes or semiconductor memories H04N5/91 takes precedence
• CPC Class Codes: H04N 9/8042 : involving data reduction

Abstract

A digital image processing technique detects and corrects visual imperfections using a reference image. A main image and one or more reference images having a temporal and/or spatial overlap and/or proximity with the original image are captured. Device information, image data and/or meta data are analyzed of the one or more reference images relating to a defect in the main image. The device corrects the defect based on the information, image data and/or meta-data to create an enhanced version of the main image.

Claims

1. A digital image processing method for detecting and correcting visual imperfections using a reference image, comprising:

capturing, on a hand-held or otherwise portable or spatial or temporal performance-based image capture device, an main image and one or more reference images having a temporal or spatial overlap or proximity with the main image, or combinations thereof;

analyzing on the device the one or more reference images for enhancing the main image;

correcting on the device one or more defects or other sub-optimal characteristics in the main image based on the analyzing of the one or more reference images to create a modified image comprising an enhanced version of the main image; and

displaying, printing, transmitting, or storing the modified image or a further processed version thereof;

whereby the correcting on the device of the one or more defects of other sub-optimal characteristics of the main image based on the analyzing of the one or more reference images produces at the device an enhanced main image from an original image in real-time with spatial economy and performance efficiency, and

wherein the method further comprises detecting a region containing a face within the original main image, and modifying the face region based on the one or more reference images,

wherein the one or more reference images comprise at least one region containing the face, and the method further comprises comparing said face region with the face region detected within the original main image, and replacing the face region in the main image with the face region of the one or more reference images,

wherein said face region of said one or more reference images is captured less than one second before or after said main image.

2. The method of claim 1, wherein the main image and at least one reference image comprise flash and non-flash versions of approximately a same scene.

3. The method of claim 1, wherein at least one reference image comprises a hi-speed capture to freeze motion of approximately a same scene as captured in the main image, and said one or more defects comprise a motion defect.

4. The method of claim 1, wherein at least one reference image comprises an infra-red version of approximately the same scene as captured in the main image to enhance heat emitting objects in the frame.

5. The method of claim 1, wherein the main image and at least one reference image comprise different focal distances.

6. The method of claim 1, wherein the one or more reference images comprise a plurality of images organized as a video clip.

7. The method of claim 1, further comprising segmenting the main image into foreground and background regions, and modifying one of the foreground and background regions based on the one or more reference images while not similarly modifying the other of the foreground and background regions.

8. The method of claim 1, further comprising correcting a red-eye defect within the face region of the main image based on the one or more reference images.

9. The method of claim 1, further comprising correcting a blink within the face region of the main image.

10. The method of claim 1, further comprising changing a facial expression within the face region of the main image.

11. The method of claim 1, further comprising correcting a blur defect in the main image based on the one or more reference images.

12. The method of claim 11, wherein said blur defect comprises a blur defect caused by an incorrect depth of field.

13. The method of claim 11, wherein said blur defect comprises a blur defect caused by motion during image capture.

14. The method of claim 1, wherein the one or more reference images are captured using one or more different capture parameters than the main image.

15. The method of claim 14, wherein the one or more different capture parameters include exposure time, dynamic range, contrast, sharpness, color balance, or white balance, or combinations thereof.

16. The method of claim 1, wherein the main image is captured using a primary optical system while the one or more reference images are captured using a secondary optical system differing from the primary optical system in focal length, depth of field, depth of focus, exit pupil, entry pupil, aperture, or lens coating or other optical parameter of a designed lens, or combinations thereof.

17. The method of claim 1, wherein the one or more reference images include at least one reference image that includes only one or more sub-component features of an overall scene captured in the main image.

18. The method of claim 17, wherein the one or more sub-component features comprise a center of the overall scene captured in the main image.

19. The method of claim 17, wherein the one or more sub-component features comprise a face, faces, or region or regions of a face or faces captured in the main image.

20. The method of claim 17, wherein the one or more sub-component features comprise a foreground region of the overall scene captured in the main image.

21. The method of claim 1, wherein said one or more reference images comprise a reference image including an eye region, and said correcting comprises replacing a corresponding eye region of the main image with said eye region of said reference image.

22. A hand-held or otherwise portable or spatial or temporal performance-based image capture device, comprising:

(a) one or more lenses, one or more apertures and one or more sensors for capturing a main image and one or more reference images, wherein the capturing of the one or more reference images has a temporal or spatial overlap or proximity or combinations thereof, with the capturing of the main image;

(b) a processor;

(c) one or more processor-readable media having embedded code therein for programming the processor to perform a digital image processing method that comprises;

analyzing on the device the one or more reference images for enhancing the main image;

correcting on the device one or more defects or other sub-optimal characteristics in the main image based on the analyzing of the one or more reference images to create a modified image comprising an enhanced version of the original main image; and

displaying, printing, transmitting, or storing the modified image or a further processed version thereof;

whereby the correcting on the device of the one or more defects of other sub-optimal characteristics of the main image based on the analyzing of the one or more reference images produces at the device an enhanced main image from an original image in real-time with spatial economy and performance efficiency, and

wherein the method further comprises detecting a region containing a face within the main image, and modifying the face region based on the analyzing of the one or more reference images,

wherein the one or more reference images comprise at least one region containing the face, and the method further comprises comparing said face region with the face region detected within the original main image, and replacing the face region in the main image with the face region of the one or more reference images,

wherein said face region of said one or more reference images is captured less than one second before or after said main image.

23. The image capture device of claim 22, wherein the main image and at least one reference image comprise flash and non-flash versions of approximately a same scene.

24. The image capture device of claim 23, wherein the method further comprises segmenting the main image into foreground and background regions, and modifying one of the foreground and background regions based on the one or more reference images while not similarly modifying the other of the foreground and background regions.

25. The image capture device of claim 22, wherein the modifying comprises correcting a red-eye defect within the face region of the main image.

26. The image capture device of claim 22, wherein the method further comprises correcting a blink within the face region of the main image.

27. The image capture device of claim 22, wherein the method further comprises changing a facial expression within the face region of the main image.

28. The image capture device of claim 22, wherein the method further comprises correcting a blur defect in the main image based on the analyzing of the one or more reference images.

29. The image capture device of claim 28, wherein said blur defect comprises a blur defect caused by incorrect depth of field.

30. The image capture device of claim 28, wherein said blur defect comprises a blur defect caused by motion during image capture.

31. The image capture device of claim 22, wherein at least one reference image comprises a hi-speed version to freeze motion of approximately a same scene a captured in the main image.

32. The image capture device of claim 22, wherein at least one reference image comprises an infra-red version of approximately the same scene to enhance heat emitting objects in the frame.

33. The image capture device of claim 22, wherein the main image and at least one reference image comprise different focal distances.

34. The image capture device of claim 22, wherein the one or more reference images comprises a plurality of images organized as a video clip.

35. The image capture device of claim 22, wherein said one or more reference images comprise a reference image including an eye region, and said correcting comprises replacing a corresponding eye region of the main image with said eye region of said reference image.

36. A hand-held or otherwise portable or spatial or temporal performance-based image capture device, comprising:

(a) one or more lenses, one or more apertures and one or more sensors for capturing a main image and one or more reference images, wherein the capturing of the one or more reference images has a temporal or spatial overlap or proximity or combinations thereof, with the capturing of the main image, or combinations thereof, and the one or more reference images are captured using one or more different capture parameters than the main image;

(b) a processor;

(c) one or more processor-readable media having embedded code therein for programming the processor to perform a digital image processing method that comprises;

analyzing on the device the one or more reference images for enhancing the main image;

correcting on the device one or more defects or other sub-optimal characteristics in the main image based on the analyzing of the one or more reference images to create a modified image comprising an enhanced version of the main image; and

displaying, printing, transmitting, or storing the modified image or a further processed version thereof;

whereby the correcting on the device of the one or more defects of other sub-optimal characteristics of the main image based on the analyzing of the one or more reference images produces at the device an enhanced main image from an original image in real-time with spatial economy and performance efficiency, and

wherein the method further comprises detecting a region containing a face within the original main image, and modifying the face region based on the one or more reference images,

wherein the one or more reference images comprise at least one region containing the face, and the method further comprises comparing said face region with the face region detected within the original main image, and replacing the face region in the main image with the face region of the one or more reference images,

wherein said face region of said one or more reference images is captured less than one second before or after said main image.

37. The device of claim 36, wherein the one or more different capture parameters include exposure time, dynamic range, contrast, sharpness, color balance, or white balance, or combinations thereof.

38. The image capture device of claim 36, wherein said one or more reference images comprise a reference image including an eye region, and said correcting comprises replacing a corresponding eye region of the main image with said eye region of said reference image.

39. A hand-held or otherwise portable or spatial or temporal performance-based image capture device, comprising:

(a) one or more lenses, one or more apertures and one or more sensors for capturing a main image and one or more reference images, wherein the capturing of the one or more reference images has a temporal or spatial overlap or proximity or combinations thereof, with the capturing of the main image, or combinations thereof, and the main image is captured using a primary optical system while the one or more reference images are captured using a secondary optical system differing from the primary optical system in focal length, depth of field, depth of focus, exit pupil, entry pupil, aperture, or lens coating or other optical parameter of a designed lens, or combinations thereof,

(b) a processor;

(c) one or more processor-readable media having embedded code therein for programming the processor to perform a digital image processing method that comprises;

analyzing on the device the one or more reference images for enhancing the main image;

correcting on the device one or more defects or other sub-optimal characteristics in the main image based on the analyzing of the one or more reference images to create a modified image comprising an enhanced version of the main image; and

displaying, printing, transmitting, or storing the modified image or a further processed version thereof;

whereby the correcting on the device of the one or more defects of other sub-optimal characteristics of the main image based on the analyzing of the one or more reference images produces at the device an enhanced main image from an original image in real-time with spatial economy and performance efficiency, and

wherein the method further comprises detecting a region containing a face within the original main image, and modifying the face region based on the one or more reference images,

wherein the one or more reference images comprise at least one region containing the face, and the method further comprises comparing said face region with the face region detected within the original main image, and replacing the face region in the main image with the face region of the one or more reference images,

wherein said face region of said one or more reference images is captured less than one second before or after said main image.

40. The image capture device of claim 39, wherein said one or more reference images comprise a reference image including an eye region, and said correcting comprises replacing a corresponding eye region of the main image with said eye region of said reference image.

41. A hand-held or otherwise portable or spatial or temporal performance-based image capture device, comprising:

(a) one or more lenses, one or more apertures and one or more sensors for capturing a main image and one or more reference images, wherein the capturing of the one or more reference images has a temporal or spatial overlap or proximity or combinations thereof with the capturing of the main image, wherein the one or more reference images include at least one reference image that includes only one or more sub-component features of an overall scene captured in the main image;

(b) a processor;

(c) one or more processor-readable media having embedded code therein for programming the processor to perform a digital image processing method that comprises;

analyzing on the device the one or more reference images for enhancing the main image;

correcting on the device one or more defects or other sub-optimal characteristics in the main image based on the analyzing of the one or more reference images to create a modified image comprising an enhanced version of the main image; and

displaying, printing, transmitting, or storing the modified image or a further processed version thereof;

whereby the correcting on the device of the one or more defects or other sub-optimal characteristics of the main image based on the analyzing of the one or more reference images produces at the device an enhanced main image from an original image in real-time with spatial economy and performance efficiency, and

wherein the method further comprises detecting a region containing a face within the original main image, and modifying the face region based on the one or more reference images,

wherein the one or more reference images comprise at least one region containing the face, and the method further comprises comparing said face region with the face region detected within the original main image, and replacing the face region in the main image with the face region of the one or more reference images,

wherein said face region of said one or more reference images is captured less than one second before or after said main image.

42. The device of claim 41, wherein the one or more sub-component features comprise a center of the overall scene captured in the main image.

43. The device of claim 41, wherein the one or more sub-component features comprise a face, faces, or region or regions of a face or faces captured in the main image.

44. The device of claim 41, wherein the one or more sub-component features comprise a foreground region of the overall scene captured in the main image.

45. The image capture device of claim 41, wherein said one or more reference images comprise a reference image including an eye region, and said correcting comprises replacing a corresponding eye region of the main image with said eye region of said reference image.

46. A hand-held or otherwise portable or spatial or temporal performance-based image capture device, comprising:

(a) one or more lenses, one or more apertures and one or more sensors for capturing a main image and one or more reference images, wherein the capturing of the one or more reference images has a temporal or spatial overlap or proximity or combinations thereof with the capturing of the main image;

(b) a processor;

(c) one or more processor-readable media having embedded code therein for programming the processor to perform a digital image processing method that comprises;

storing the one or more reference images with main image data for post-processing enhancement of the main image;

correcting one or more defects or other sub-optimal characteristics in the main image based on analyzing the one or more reference images to create a modified image comprising an enhanced version of the main image; and

displaying, printing, transmitting, or storing the modified image or a further processed version thereof;

whereby the correcting of the one or more defects or other sub-optimal characteristics of the main image based on the analyzing of the one or more reference images in post-processing produces an enhanced main image from an original image,

wherein said one or more reference images comprise a reference image including an eye region, and said correcting comprises replacing a corresponding eye region of the main image with said eye region of said reference image, wherein said eye region in the main image that is replaced is determined to be closed or semi-closed,

wherein said reference image that comprises said eye region is captured less than one second before or after a blinking action.

47. A hand-held or otherwise portable or spatial or temporal performance-based image capture device, comprising:

(a) one or more lenses, one or more apertures and one or more sensors for capturing a main image and one or more reference images, wherein the capturing of the one or more reference images has a temporal or spatial overlap or proximity or combinations thereof with the capturing of the main image;

(b) a processor;

(c) one or more processor-readable media having embedded code therein for programming the processor to perform a digital image processing method that comprises;

storing the one or more reference images with main image data for post-processing enhancement of the main image;

correcting one or more defects or other sub-optimal characteristics in the main image based on analyzing the one or more reference images to create a modified image comprising an enhanced version of the main image; and

displaying, printing, transmitting, or storing the modified image or a further processed version thereof;

whereby the correcting of the one or more defects or other sub-optimal characteristics of the main image based on the analyzing of the one or more reference images in post-processing produces an enhanced main image from an original image,

wherein the one or more reference images comprise data of a region of interest (ROI) at a same or higher resolution as said main image,

wherein the ROI includes an eye region,

wherein said eye region is captured less than one second before or after a blink appearing in the main image.

48. The device of claim 47, wherein the ROI further includes a mouth region.

49. The device of claim 48, further comprising comparing said mouth region with a corresponding mouth region in the main image, and replacing the mouth region in the main image with the mouth region of the one or more reference images.

50. A hand-held or otherwise portable or spatial or temporal performance-based image capture device, comprising:

(a) one or more lenses, one or more apertures and one or more sensors for capturing a main image and one or more reference images, wherein the capturing of the one or more reference images has a temporal or spatial overlap or proximity or combinations thereof with the capturing of the main image;

(b) a processor;

(c) one or more processor-readable media having embedded code therein for programming the processor to perform a digital image processing method that comprises;

storing the one or more reference images with main image data for post-processing enhancement of the main image;

correcting one or more defects or other sub-optimal characteristics in the main image based on analyzing the one or more reference images to create a modified image comprising an enhanced version of the main image; and

displaying, printing, transmitting, or storing the modified image or a further processed version thereof;

whereby the correcting of the one or more defects or other sub-optimal characteristics of the main image based on the analyzing of the one or more reference images in post-processing produces an enhanced main image from an original image,

wherein the one or more reference images comprise data of a region of interest (ROI) at a same or higher resolution as said main image,

wherein the ROI includes a mouth region,

wherein the method further comprises comparing said mouth region with a corresponding mouth region in the main image, and replacing the mouth region in the main image with the mouth region of the one or more reference images,

wherein said mouth region of said one or more reference images is captured less than one second before or after said main image.