Method of Notifying Users Regarding Motion Artifacts Based on Image Analysis

• US 20110193989A1
• Filed: 04/17/2011
• Published: 08/11/2011
• Est. Priority Date: 11/10/2004
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Tessera Technologies Ireland Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Prilutsky, Yury, Corcoran, Peter
• Granted Patent: US 8,270,751 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/003 Deblurring; Sharpening vibration or motion blur correction for cameras comprising an electronic image sensor H04N23/682
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction

Abstract

A digital image acquisition system includes a portable apparatus for capturing digital images and a digital processing component for detecting, analyzing and informing the photographer regarding motion blur, and for reducing camera motion blur in an image captured by the apparatus. The digital processing component operates by comparing the image with at least one other image, for example a preview image, of nominally the same scene taken outside the exposure period of the main image. In one embodiment the digital processing component determines the degree of artefacts and whether to inform the user that the image is blurred by identifying at least one feature in a single preview image which is relatively less blurred than the corresponding feature in the main image. In another embodiment, the digital processing component calculates a trajectory of at least one feature in a plurality of preview images, extrapolates such feature on to the main image, calculates a PSF in respect of the feature, and informs the user based on the calculated PSF. In another embodiment the digital processing unit after determining the degree of blur notifies the photographer of the existing blur or automatically invokes consecutive captures. In another embodiment, the digital processing unit determines whether the image quality is acceptable from real time analysis of the captured image and provides this information to the user. Such real time analysis may use the auto focusing mechanism to qualitatively determine the PSF.

Claims

1. A digital image acquisition device, comprising:

an image capture component for capturing digital images;

an auto-focus mechanism configured to automatically focus a scene and to be used in determining a camera motion blur; and

a digital processing component, including;

a processor; and

embedded code for programming the processor to initiate notifying a photographer, de-blurring a digital image, or delaying or initiating a subsequent image capture, or combinations thereof, upon determining that a threshold camera motion blur has occurred in a captured digital image based on a real time analysis of the captured digital image and at least one further image each acquired with said digital image acquisition device, such that said same digital image acquisition device captures each of said captured digital image and said at least one further image, within a temporal range that includes an exposure period of the captured digital image, and times proximately before and after said exposure period, and of nominally the same scene as that of the captured digital image, and

wherein the camera motion blur is determined qualitatively using the auto-focus mechanism;

wherein said captured digital image and said at least one further image are compared, the at least one further image comprising a lower pixel resolution than the captured digital image; and

wherein the digital processing component is further arranged to determine whether to notify the photographer, de-blur said captured digital image or delay or initiate a subsequent image capture depending on whether the threshold camera motion blur has occurred in the captured digital image.

2. A device according to claim 1, wherein the determining that a threshold camera motion blur has occurred is based on qualitatively determining a motion blur.

3. A device according to claim 2, wherein the camera motion blur is mathematically defined as a point spread function (PSF).

4. A device according to claim 2, wherein said digital processing component further de-blurs the captured image using said determined camera motion blur function.

5. A device according to claim 2, wherein said qualitatively determining a motion blur is performed using an auto-focus mechanism of the camera as the digital processing component.

6. A device according to claim 1, wherein the digital image capturing apparatus is configured to capture another image if the digital processing component determines that the captured image is blurred.

7. A device according to claim 1, wherein said image capture component comprises a single sensor for capturing said captured digital image and said at least one further image.

8. A device according to claim 1, wherein said determining that a threshold camera motion blur has occurred in the captured digital image is further based on a comparison of said captured digital image with at least two further images each acquired with said same sensor, such that at least a same portion of said same sensor captures at least a portion of each of said captured digital image and said at least two further images, within a temporal range that includes an exposure period of the captured digital image, and times proximately before and after said exposure period, and of nominally the same scene as that of the captured digital image.

21. The device of claim 1, wherein said digital processing component is arranged to match the pixel resolutions of the captured digital image and the at least one further image by at least one of up-sampling the at least one further image and sub-sampling the captured digital image.

24. The device of claim 1, wherein said real time analysis occurs within the device and within a temporal duration commensurate either with an average delay between capture of consecutive images or with timing of an auto-focus mechanism of the device, or both.

27. The device of claim 1, wherein a local contrast gradient or edge detection auto-focus technique, or both, is used to qualitatively determine the camera motion blur.

30. The device of claim 1, wherein the threshold camera motion blur comprises a qualitative non-zero distortion amount around one or more points or lines or both in the image compared to one or more relatively sharp matched points or lines, or both, in the at least one further image.

9. One or more non-transitory computer readable media having digital code embedded therein for programming a processor to perform a method on a digital image acquisition device that comprises an image capture component for capturing digital images, a digital processing component and an auto-focus mechanism, wherein the method comprises:

qualitatively determining a camera motion blur at least in part using an auto focus mechanism of the device;

determining that a threshold camera motion blur has occurred in a captured digital image based on a real time analysis of the captured digital image and at least one further image each acquired with a same digital image acquisition device, such that said same digital image acquisition device captures each of said captured digital image and said at least one further image, within a temporal range that includes an exposure period of the captured digital image, and times proximately before and after said exposure period, and of nominally the same scene as that of the captured digital image;

comparing said captured digital image and said at least one further image, the at least one further image comprising a lower pixel resolution than the captured digital image;

determining whether to notify the photographer, de-blur said captured digital image, or delay or initiate a subsequent image capture depending on whether the threshold camera motion blur has occurred in the captured digital image;

notifying the photographer, de-blurring the captured digital image, or delaying or initiating said subsequent image capture, or determining to performing none of these operations, based on the determining depending on whether the threshold camera motion blur has occurred in the captured digital image; and

de-blurred or not, storing, transmitting or displaying the captured digital image or a further processed version, or combinations thereof.

10. One or more computer readable media according to claim 9, wherein the determining that a threshold camera motion blur has occurred is based on qualitatively determining a motion blur.

11. One or more computer readable media according to claim 10, wherein the camera motion blur is mathematically defined as a point spread function (PSF).

12. One or more computer readable media according to claim 10, wherein said qualitatively determining a motion blur is performed using an auto-focus mechanism of the camera as the digital processing component.

13. One or more computer readable media according to claim 9, wherein said digital image acquisition system comprises a portable digital camera comprising a single sensor for capturing said captured digital image and said at least one further image.

14. One or more computer readable media according to claim 9, wherein said determining that a threshold camera motion blur has occurred in the captured digital image is further based on a comparison of said captured digital image with at least two further images each acquired with said same sensor, such that at least a same portion of said same sensor captures at least a portion of each of said captured digital image and said at least two further images, within a temporal range that includes an exposure period of the captured digital image, and times proximately before and after said exposure period, and of nominally the same scene as that of the captured digital image.

22. One or more computer readable media according to claim 9, wherein the method further comprises matching pixel resolutions of the captured digital image and the at least one further image including at least one of up-sampling the at least one further image and sub-sampling the captured digital image.

25. One or more computer readable media according to claim 9, wherein said real time analysis occurs within the device and within a temporal duration commensurate either with an average delay between capture of consecutive images or with timing of an auto-focus mechanism of the device, or both.

28. One or more computer readable media according to claim 9, wherein a local contrast gradient or edge detection auto-focus technique, or both, is used to qualitatively determine the camera motion blur.

31. One or more computer readable media according to claim 9, wherein the threshold camera motion blur comprises a qualitative non-zero distortion amount around one or more points or lines or both in the image compared to one or more relatively sharp matched points or lines, or both, in the at least one further image.

15. On a digital image acquisition device that comprises an image capture component for capturing digital images and a digital processing component, a method of notifying a photographer, de-blurring a digital image, or delaying or initiating a subsequent image capture, or combinations thereof, depending on determining whether a threshold camera motion blur has occurred in the digital image, wherein the method comprises:

qualitatively determining a camera motion blur at least in part using an auto focus mechanism of the device;

determining that a threshold camera motion blur has occurred in a captured digital image based on a real time analysis of the captured digital image and at least one further image each acquired with a same digital image acquisition device, such that said same digital image acquisition device captures each of said captured digital image and said at least one further image, within a temporal range that includes an exposure period of the captured digital image, and times proximately before and after said exposure period, and of nominally the same scene as that of the captured digital image;

comparing said captured digital image and said at least one further image, the at least one further image comprising a lower pixel resolution than the captured digital image;

determining whether to notify the photographer, de-blur said captured digital image, or delay or initiate a subsequent image capture depending on whether the threshold camera motion blur has occurred in the captured digital image;

notifying the photographer, de-blurring the captured digital image, or delaying or initiating said subsequent image capture, or determining to performing none of these operations, based on the determining depending on whether the threshold camera motion blur has occurred in the captured digital image; and

de-blurred or not, storing, transmitting or displaying the captured digital image or a further processed version, or combinations thereof.

16. A method according to claim 15, wherein the determining that a threshold camera motion blur has occurred is based on qualitatively determining a motion blur.

17. A method according to claim 16, wherein the camera motion blur is mathematically defined as a point spread function (PSF).

18. A method according to claim 16, wherein said qualitatively determining a motion blur is performed using an auto-focus mechanism of the camera as the digital processing component.

19. A method according to claim 15, wherein said digital image acquisition system comprises a portable digital camera comprising a single sensor for capturing said captured digital image and said at least one further image.

20. A method according to claim 15, wherein said determining that a threshold camera motion blur has occurred in the captured digital image is further based on a comparison of said captured digital image with at least two further images each acquired with said same sensor, such that at least a same portion of said same sensor captures at least a portion of each of said captured digital image and said at least two further images, within a temporal range that includes an exposure period of the captured digital image, and times proximately before and after said exposure period, and of nominally the same scene as that of the captured digital image.

23. A method according to claim 15, further comprising matching pixel resolutions of the captured digital image and the at least one further image including at least one of up-sampling the at least one further image and sub-sampling the captured digital image.

26. A method according to claim 15, wherein said real time analysis occurs within the device and within a temporal duration commensurate either with an average delay between capture of consecutive images or with timing of an auto-focus mechanism of the device, or both.

29. A method according to claim 15, wherein a local contrast gradient or edge detection auto-focus technique, or both, is used to qualitatively determine the camera motion blur.

32. A method according to claim 15, wherein the threshold camera motion blur comprises a qualitative non-zero distortion amount around one or more points or lines or both in the image compared to one or more relatively sharp matched points or lines, or both, in the at least one further image.