Method of determining PSF using multiple instances of nominally scene

• US 7,660,478 B2
• Filed: 12/01/2006
• Issued: 02/09/2010
• Est. Priority Date: 11/10/2004
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Vision Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Prilutsky, Yury, Corcoran, Peter
• Primary Examiner(s): Mehta; Bhavesh M
• Assistant Examiner(s): Bloom; Nathan
• Application Number:
  US11/566,180
• Publication Number:
  US 20070058073A1
• Time in Patent Office: 1,166 Days
• Field of Search: 382/107, 348/280.99, 348/208.1, 348/208.4, 348/208.14, 348/208.16
• US Class Current: 382/255
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction

Abstract

A digital image acquisition system includes a portable apparatus for capturing digital images and a digital processing component for detecting, analyzing and informing the photographer regarding motion blur, and for reducing camera motion blur in an image captured by the apparatus. The digital processing component operates by comparing the image with at least one other image, for example a preview image, of nominally the same scene taken outside the exposure period of the main image. In one embodiment the digital processing component identifies at least one feature in a single preview image which is relatively less blurred than the corresponding feature in the main image, calculates a point spread function (PSF) in respect of such feature, and de-convolves the main image using the PSF. In another embodiment, the digital processing component calculates a trajectory of at least one feature in a plurality of preview images, extrapolates such feature on to the main image, calculates a PSF in respect of the feature, and de-convolves the main image using the PSF. In another embodiment the digital processing unit after determining the degree of blur notifies the photographer of the existing blur or automatically invokes consecutive captures.

Claims

1. A portable digital camera, comprising

(a) an apparatus for capturing digital images;

and

(b) a digital processing component for determining a camera motion blur function in a captured digital image based on a comparison of at least two sets of image data acquired at different times;

(c) wherein each of said sets of image data is acquired with a same sensor, such that at least a same portion of said same sensor captures at least a portion of each of said at least two sets of image data, within a temporal range that includes an exposure period of the captured digital image, and times proximately before and after said exposure period, and of nominally the same scene as that of the captured digital image, and

(d) wherein a first set of said sets of image data is compared to a second set of said sets of image data, the second set comprising a lower pixel resolution than the first set and the second set being taken outside the exposure period of the first set, and

(e) wherein said digital processing component is arranged to match the pixel resolutions of the first and second image data sets by at least one of up-sampling the first image data set and sub-sampling the second image data set; and

(f) wherein said digital processing component is arranged to align at least portions of the at least two sets of image data before they are compared, and

(g) wherein the digital processing component is further arranged to determine whether to de-blur said captured digital image depending on whether the camera motion blur function indicates that sufficient or insufficient blur exists.

2. A system according to claim 1 wherein the camera motion blur function is mathematically defined as a point spread function (PSF).

3. A system according to claim 1, wherein said digital processing component further de-blurs the captured image using said determined camera motion blur function.

4. A system according to claim 1, wherein at least one of the sets of image data that are compared comprises a preview image.

5. A system according to claim 1, wherein said digital processing component is arranged to align at least portions of the at least two image data sets that are compared.

6. A system according to claim 5, wherein the alignment is a global alignment on entire or substantially entire images.

7. A system according to claim 1, wherein said digital processing component is configured to inform the photographer if it determines that the captured image is blurred.

8. A system according to claim 7, wherein the digital image capturing apparatus is signaled to capture another image if the digital processing component determines that the captured image is blurred.

9. A system according to claim 1, wherein the digital image capturing apparatus is signaled to capture another image if the digital processing component determines that the captured image is blurred.

10. A system according to claim 1, wherein said at least two image data sets comprise said captured image and a reference image taken outside the exposure period of said captured image.

11. A system according to claim 10, wherein said reference image is taken before said exposure period of said captured image.

12. A system according to claim 10, wherein the camera motion blur function is mathematically defined as a point spread function (PSF).

13. A system according to claim 10, wherein said digital processing component further de-blurs the captured image using said determined camera motion blur function.

14. A system according to claim 10, wherein the reference image comprises a preview image.

15. A system according to claim 10, wherein said digital processing component is arranged to align at least portions of the captured image and the reference image.

16. A system according to claim 1, wherein said motion blur function determining occurs in real-time relative to the image capture.

17. A portable digital camera, comprising

(a) an apparatus for capturing digital images and

(b) a digital processing component for determining a camera motion blur function in a captured digital image based on a comparison of at least two sets of image data acquired at different times,

(c) wherein each of said sets of image data is acquired with a same sensor, such that at least a same portion of said same sensor captures at least a portion of each of said at least two sets of image data, within a temporal range that includes an exposure period of the captured digital image, and times proximately before and after said exposure period, and of nominally the same scene as that of the captured digital image, and

(d) wherein said digital processing component uses the captured digital image and a low resolution reference image among said sets of image data as the image data sets that are compared, the low resolution reference image being taken outside the exposure period of the captured digital image, and

(e) wherein said digital processing component identifies at least one characteristic of the reference image which is relatively less blurred than the corresponding characteristic of the captured image, and calculates said blur function in respect of said characteristic;

and

(f) wherein said digital processing component is arranged to align at least portions of the at least two sets of image data before they are compared, and

(g) wherein the digital processing component is further arranged to determine whether to de-blur said captured digital image depending on whether the camera motion blur function indicates that sufficient or insufficient blur exists.

18. A system according to claim 17, wherein said digital processing component de-blurs the captured image by de-convolving the captured image using the blur function.

19. A portable digital camera, comprising

(a) an apparatus for capturing digital images and

(b) a digital processing component for determining a camera motion blur function in a captured digital image based on a comparison of multiple sets of image data acquired at different times,

(c) wherein each of said sets of image data is acquired with a same sensor, such that at least a same portion of said same sensor captures at least a portion of each of said at least two sets of image data, within a temporal range that includes an exposure period of the captured digital image, and times proximately before and after said exposure period, and of nominally the same scene as that of the captured digital image, and

(d) wherein said digital processing component uses the captured image and multiple low resolution reference images among said sets of data taken at successive points in time, such that the multiple low resolution data sets are taken outside the exposure period of the captured digital image, and

(e) wherein said digital processing component calculates a trajectory of at least one characteristic in the reference images, extrapolates such characteristic on to the captured image, and calculates said blur function in respect of said characteristic;

and

(f) wherein said digital processing component is arranged to align at least portions of the multiple low resolution data sets and the captured digital image, and

(g) wherein the digital processing component is further arranged to determine whether to de-blur said captured digital image depending on whether the camera motion blur function indicates that sufficient or insufficient blur exists.

20. A system according to claim 19, wherein said digital processing component de-blurs the captured image by de-convolving the captured image using the blur function.