Digital image enhancement with reference images

• US 8,682,097 B2
• Filed: 06/16/2008
• Issued: 03/25/2014
• Est. Priority Date: 02/14/2006
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Bigioi, Petronel, Zamfir, Adrian, Drimbarean, Alexandru, Corcoran, Peter, Petrescu, Stefan, David, Emilian, Vertan, Constantin
• Primary Examiner(s): Patel, Nirav G
• Application Number:
  US12/140,048
• Publication Number:
  US 20080317378A1
• Time in Patent Office: 2,108 Days
• Field of Search: None
• US Class Current: 382/275
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/30216 Redeye defect
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/811 by dust removal, e.g. from surfaces of the image sensor or processing of the image signal output by the electronic image sensor
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/68 applied to defects
• CPC Class Codes: H04N 25/69 SSIS comprising testing or correcting structures for circuits other than pixel cells

Abstract

A digital image processing technique is for detecting and correcting visual imperfections using a reference image. A main image and one or more reference images having a temporal and/or spatial overlap and/or proximity with the original image are captured. Device information, image data and/or meta data are analyzed of the one or more reference images relating to a defect in the main image. The device corrects the defect based on the information, image data and/or meta-data to create an enhanced version of the main image.

Claims

1. A digital image processing method for detecting and correcting visual imperfections using a reference image, comprising:

capturing, on a hand-held or otherwise portable or spatial or temporal performance-based image capture device, an original main image and one or more reference images having a temporal or spatial overlap or proximity with the original image, or combinations thereof;

assessing on the device that the original main image includes one or more defects or otherwise sub-optimal characteristics;

analyzing on the device information, image data or meta data, or combinations thereof, of the one or more reference images relating to the one or more defects or otherwise sub-optimal characteristics of the original main image;

correcting on the device the one or more defects or other sub-optimal characteristics in the original main image based on the information, image data or meta data, or combinations thereof, of the one or more reference images to create a modified image comprising an enhanced versions of the original main image; and

rendering the modified image at a digital rendering device, display or printer, or combinations thereof, as output from the image capture device;

whereby the correcting of the one or more defects of other sub-optimal characteristics of the original main image based on the one or more reference images produces at the device the modified image from the original main image in real-time with spatial economy and performance efficiency;

aligning the reference and original main images globally to account for movement of the camera in the interval between the captures of the reference and original main images;

matching the foreground regions of the reference and original main images to account for localized movement of a subject or other region that defines at least a portion of the foreground region; and

calculating luminance histograms of reference and flash images and determining at least a portion of the foreground region based on differences between the reference and flash images;

wherein the method further comprising detecting a region containing a face within the original main image, and modifying the face region based on the one or more reference images.

2. The method of claim 1, wherein the original main image and at least one reference image comprise one or more hi-speed versions to freeze motion of a substantially same scene.

3. The method of claim 1, wherein at least one reference image comprises an infra-red version of substantially the same scene to enhance heat emitting objects in the frame.

4. The method of claim 1, wherein the original main image and at least one reference image comprise different focal distances.

5. The method of claim 1, wherein the reference image comprises a plurality of images organized as a video clip.

6. The method of claim 1, further comprising segmenting the original main image into foreground and background regions, and modifying one of the foreground and background regions based on the one or more reference images.

7. The method of claim 1, further comprising correcting a blur defect in the original main image based on the one or more reference images.

8. The method of claim 7, wherein said blur defect comprises a blur defect caused by an incorrect depth of field.

9. The method of claim 7, wherein said blur defect comprises a blur defect caused by motion during image capture.

10. The method of claim 1, further comprising correcting a dust artifact defect in the original main image based on the one or more reference images.

11. The method of claim 1, further comprising detecting a region containing a face within the original main image, and modifying the face region based on the one or more reference images.

12. The method of claim 11, further comprising correcting a redeye defect of with the face region of the original main image.

13. The method of claim 11, further comprising correcting a blink within the face region of the original main image.

14. The method of claim 11, further comprising changing a facial expression within the face region of the original main image.

15. A digital image processing method for detecting and correcting visual imperfections using a reference image, comprising:

capturing, on a hand-held or otherwise portable or spatial or temporal performance-based image capture device, an original main image and one or more reference images having a temporal or spatial overlap or proximity with the original image, or combinations thereof;

assessing on the device that the original main image includes one or more defects or otherwise sub-optimal characteristics;

analyzing on the device information, image data or meta data, or combinations thereof, of the one or more reference images relating to the one or more defects or otherwise sub-optimal characteristics of the original main image;

correcting on the device the one or more defects or other sub-optimal characteristics in the original main image based on the information, image data or meta data, or combinations thereof, of the one or more reference images to create a modified image comprising an enhanced versions of the original main image; and

rendering the modified image at a digital rendering device, display or printer, or combinations thereof, as output from the image capture device;

whereby the correcting of the one or more defects of other sub-optimal characteristics of the original main image based on the one or more reference images produces at the device the modified image from the original main image in real-time with spatial economy and performance efficiency, and

aligning the reference and original main images globally to account for movement of the camera in the interval between the captures of the reference and original main images;

matching the foreground regions of the reference and original main images to account for localized movement of a subject or other region that defines at least a portion of the foreground region; and

calculating luminance histograms of reference and flash images and determining at least a portion of the foreground region based on differences between the reference and flash images;

and the method further comprising detecting a region containing a face within the original main image, and modifying the face region based on the one or more reference images.

16. The method of claim 15, further comprising correcting a red-eye defect of with the face region of the original main image.

17. The method of claim 15, further comprising correcting a blink within the face region of the original main image.

18. The method of claim 15, further comprising changing a facial expression within the face region of the original main image.

19. A hand-held or otherwise portable or spatial or temporal performance-based image capture device, comprising:

one or more lenses, an aperture and a photodetector for capturing an original main image and one or more reference images having a temporal or spatial overlap or proximity with the original image, or combinations thereof;

a processor;

one or more non-transitory processor-readable media having embedded therein code for programming the processor to perform a digital image processing method that comprises;

assessing on the device that the original main image includes one or more defects or otherwise sub-optimal characteristics;

analyzing on the device information, image data or meta data, or combinations thereof, of the one or more reference images relating to the one or more defects or otherwise sub-optimal characteristics of the original main image;

correcting on the device the one or more defects or other sub-optimal characteristics in the original main image based on the information, image data or meta data, or combinations thereof, of the one or more reference images to create a modified image comprising an enhanced versions of the original main image; and

rendering the modified image at a digital rendering device, display or printer, or combinations thereof, as output from the image capture device;

whereby the correcting of the one or more defects of other sub-optimal characteristics of the original main image based on the one or more reference images produces at the device the modified image from the original main image in real-time with spatial economy and performance efficiency, and

aligning the reference and original main images globally to account for movement of the camera in the interval between the captures of the reference and original main images;

matching the foreground regions of the reference and original main images to account for localized movement of a subject or other region that defines at least a portion of the foreground region; and

calculating luminance histograms of reference and flash images and determining at least a portion of the foreground region based on differences between the reference and flash images;

wherein the method further comprising detecting a region containing a face within the original main image, and modifying the face region based on the one or more reference images.

20. The image capture device of claim 19, wherein the method further comprises segmenting the original main image into foreground and background regions, and modifying one of the foreground and background regions based on the one or more reference images.

21. The image capture device of claim 19, wherein the method further comprises correcting a red-eye defect of with the face region of the original main image.

22. The image capture device of claim 19, wherein the method further comprises correcting a blink within the face region of the original main image.

23. The image capture device of claim 19, wherein the method further comprises changing a facial expression within the face region of the original main image.

24. The image capture device of claim 19, wherein the method further comprises correcting a blur defect in the original main image based on the one or more reference images.

25. The image capture device of claim 24, wherein said blur defect comprises a blur defect caused by incorrect depth of field.

26. The image capture device of claim 24, wherein said blur defect comprises a blur defect caused by motion during image capture.

27. The image capture device of claim 19, wherein the method further comprises correcting a dust artifact defect in the original main image based on the one or more reference images.

28. The image capture device of claim 19, wherein the original main image and at least one reference image comprise one or more hi-speed versions to freeze motion of a substantially same scene.

29. The image capture device of claim 19, wherein at least one reference image comprises an infra-red version of substantially the same scene to enhance heat emitting objects in the frame.

30. The image capture device of claim 19, wherein the original main image and at least one reference image comprise different focal distances.

31. The image capture device of claim 19, wherein the reference image comprises a plurality of images organized as a video clip.

32. The image capture device of claim 19, wherein the method further comprises detecting a region containing a face within the original main image, and modifying the face region based on the one or more reference images.

33. The image capture device of claim 32, wherein the method further comprises correcting a red-eye defect of with the face region of the original main image.

34. The image capture device of claim 32, wherein the method further comprises correcting a blink within the face region of the original main image.

35. The image capture device of claim 32, wherein the method further comprises changing a facial expression within the face region of the original main image.

36. The image capture device of claim 19, wherein the method further comprises segmenting the original main image into foreground and background regions, and modifying one of the foreground and background regions based on the one or more reference Images.

37. The image capture device of claim 19, wherein the method further comprises correcting a blur defect in the original main image based on the one or more reference images.

38. The image capture device of claim 37, wherein said blur defect comprises a blur defect caused by incorrect depth of field.

39. The image capture device of claim 37, wherein said blur defect comprises a blur defect caused by motion during image capture.

40. The image capture device of claim 19, wherein the original main image and at least one reference image comprise one or more hi-speed versions to freeze motion of approximately a same scene.

41. The image capture device of claim 19, wherein at least one reference image comprises an infra-red version of approximately the same scene to enhance heat emitting objects in the frame.

42. The image capture device of claim 19, wherein the original main image and at least one reference image comprise different focal distances.

43. The image capture device of claim 19, wherein the reference image comprises a plurality of images organized as a video clip.

44. One or more non-transitory processor-readable media having embedded therein code for programming the processor to perform a digital image processing method involving a captured original main image and one or more reference images having a temporal or spatial overlap or proximity with the original image, or combinations thereof, wherein the method comprises:

assessing on the device that the original main image includes one or more defects or otherwise sub-optimal characteristics;

analyzing on the device information, image data or meta data, or combinations thereof, of the one or more reference images relating to the one or more defects or otherwise sub-optimal characteristics of the original main image;

correcting on the device the one or more defects or other sub-optimal characteristics in the original main image based on the information, image data or meta data, or combinations thereof, of the one or more reference images to create a modified image comprising an enhanced versions of the original main image; and

rendering the modified image at a digital rendering device, display or printer, or combinations thereof, as output from the image capture device;

wherein the correcting of the one or more defects of other sub-optimal characteristics of the original main image based on the one or more reference images produces at the device the modified image from the original main image in real-time with spatial economy and performance efficiency,

aligning the reference and original main images globally to account for movement of the camera in the interval between the captures of the reference and original main images;

matching the foreground regions of the reference and original main images to account for localized movement of a subject or other region that defines at least a portion of the foreground region; and

calculating luminance histograms of reference and flash images and determining at least a portion of the foreground region based on differences between the reference and flash images;

detecting a region containing a face within the original main image, and

modifying the face region based on the one or more reference images.

45. The one or more non-transitory storage devices of claim 44, wherein the original main image and at least one reference image comprise one or more hi-speed versions to freeze motion of a substantially same scene.

46. The one or more non-transitory storage devices of claim 44, wherein at least one reference image comprises an infra-red version of substantially the same scene to enhance heat emitting objects in the frame.

47. The one or more non-transitory storage devices of claim 44, wherein the original main image and at least one reference image comprise different focal distances.

48. The one or more non-transitory storage devices of claim 44, wherein the reference image comprises a plurality of images organized as a video clip.

49. The one or more non-transitory storage devices of claim 44, further comprising segmenting the original main image into foreground and background regions, and modifying one of the foreground and background regions based on the one or more reference images.

50. The one or more non-transitory storage devices of claim 44, further comprising correcting a blur defect in the original main image based on the one or more reference images.

51. The one or more non-transitory storage devices of claim 50, wherein said blur defect comprises a blur defect caused by an incorrect depth of field.

52. The one or more non-transitory storage devices of claim 50, wherein said blur defect comprises a blur defect caused by motion during image capture.

53. The one or more non-transitory storage devices of claim 44, further comprising correcting a dust artifact defect in the original main image based on the one or more reference images.

54. One or more non-transitory processor-readable media having embedded therein code for programming the processor to perform a digital image processing method involving a captured original main image and one or more reference images having a temporal or spatial overlap or proximity with the original image, or combinations thereof, wherein the method comprises:

assessing that the original main image includes one or more defects or otherwise sub-optimal characteristics;

analyzing information, image data or meta data, or combinations thereof, of the one or more reference images relating to the one or more defects or otherwise sub-optimal characteristics of the original main image;

correcting the one or more defects or other sub-optimal characteristics in the original main image based on the information, image data or meta data, or combinations thereof, of the one or more reference images to create a modified image comprising an enhanced versions of the original main image; and

wherein the correcting of the one or more defects of other sub-optimal characteristics of the original main image based on the one or more reference images produces at the device the modified image from the original main image in real-time with spatial economy and performance efficiency, and

aligning the reference and original main images globally to account for movement of the camera in the interval between the captures of the reference and original main images;

matching the foreground regions of the reference and original main images to account for localized movement of a subject or other region that defines at least a portion of the foreground region; and

calculating luminance histograms of reference and flash images and determining at least a portion of the foreground region based on differences between the reference and flash images;

detecting a region containing a face within the original main image, and

modifying the face region based on the one or more reference images.

55. The one or more non-transitory storage devices of claim 54, further comprising correcting a red-eye defect of with the face region of the original main image.

56. The one or more non-transitory storage devices of claim 54, further comprising correcting a blink within the face region of the original main image.

57. The one or more non-transitory storage devices of claim 54, further comprising changing a facial expression within the face region of the original main image.