Method of notifying users regarding motion artifacts based on image analysis

• US 20060098891A1
• Filed: 11/10/2004
• Published: 05/11/2006
• Est. Priority Date: 11/10/2004
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Vision Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Prilutsky, Yury, Corcoran, Peter
• Granted Patent: US 7,639,889 B2
• Time in Patent Office: Days
• US Class Current: 382/255
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction

Abstract

A digital image acquisition system includes a portable apparatus for capturing digital images and a digital processing component for detecting, analyzing and informing the photographer regarding motion blur, and for reducing camera motion blur in an image captured by the apparatus. The digital processing component operates by comparing the image with at least one other image, for example a preview image, of nominally the same scene taken outside the exposure period of the main image. In one embodiment the digital processing component determines the degree of artefacts and whether to inform the user that the image is blurred by identifying at least one feature in a single preview image which is relatively less blurred than the corresponding feature in the main image. In another embodiment, the digital processing component calculates a trajectory of at least one feature in a plurality of preview images, extrapolates such feature on to the main image, calculates a PSF in respect of the feature, and informs the user based on the calculated PSF. In another embodiment the digital processing unit after determining the degree of blur notifies the photographer of the existing blur or automatically invokes consecutive captures. In another embodiment, the digital processing unit determines whether the image quality is acceptable from real time analysis of the captured image and provides this information to the user. Such real time analysis may use the auto focusing mechanism to qualitatively determine the PSF.

Claims

1. A digital image acquisition system, comprising an apparatus for capturing digital images and a digital processing component for notifying the photographer upon determining that a threshold camera motion blur has occurred in a captured digital image based on a comparison of at least two sets of image data each acquired within a temporal range that includes an exposure period of the captured digital image, and times proximately before and after said exposure period, and of nominally the same scene as that of the captured digital image.

2. A system according to claim 1, wherein the determining that a threshold camera motion blur has occurred is based on determining a motion blur function.

3. A system according to claim 2 wherein the camera motion blur function is mathematically defined as a point spread function (PSF).

4. A system according to claim 2, wherein said digital processing component further de-blurs the captured image using said determined camera motion blur function.

5. A system according to claim 1, wherein at least one of the sets of image data that are compared comprises a preview image.

6. A system according to claim 1, wherein a first set of image data is compared to a second set of image data comprising a lower pixel resolution than the first set and wherein said digital processing component is arranged to match the pixel resolutions of the first and second image data sets.

7. A system according to claim 6, wherein said digital processing component is arranged to match the pixel resolutions of the first and second image data sets by at least one of up-sampling the first image data set and sub-sampling the second image data set.

8. A system according to claim 1, wherein said digital processing component is arranged to align at least portions of the at least two image data sets that are compared.

9. A system according to claim 8, wherein the alignment is a global alignment on entire or substantially entire images.

10. A system according to claim 1, wherein said digital processing component uses the captured image and a single reference image as the image data sets that are compared, and determines a motion blur function based on the comparison.

11. A system according to claim 10, wherein said digital processing component identifies at least one characteristic of the reference image which is relatively less blurred than the corresponding characteristic of the captured image, and calculates said blur function in respect of said characteristic.

12. A system according to claim 11, wherein said digital processing component de-blurs the captured image by de-convolving the captured image using the blur function.

13. A system according to claim 1, wherein said digital processing component uses the captured image and multiple reference images taken at successive points in time.

14. A system according to claim 13, wherein said digital processing component calculates a trajectory of at least one characteristic in the reference images, extrapolates such characteristic on to the captured image, and calculates a motion blur function in respect of said characteristic.

15. A system according to claim 14, wherein said digital processing component de-blurs the captured image by de-convolving the captured image using the blur function.

16. A system according to claim 1, wherein said digital processing component is configured to visually inform the photographer if it determines that the captured image is blurred.

17. A system according to claim 1, wherein said digital processing component is configured to inform the photographer with an audio signal if it determines that the captured image is blurred.

18. A system of according to claim 1, wherein the digital image capturing apparatus is signalled to capture another image if the digital processing component determines that the captured image is blurred.

19. A system according to claim 1, wherein said digital image acquisition system comprises a portable digital camera.

20. A system as claimed in claim 19, wherein said digital processing component is located in said portable digital camera.

21. A system according to claim 1, wherein said digital image acquisition system comprises a combination of a portable digital camera and an external processing device.

22. A system as claimed in claim 20, wherein said digital processing component is located in said external processing device.

23. A system as in claim 1, wherein said at least two image data sets comprise said captured image and a reference image taken outside the exposure period of said captured image.

24. A system as in claim 23, wherein said reference image is taken before said exposure period of said captured image.

25. A system according to claim 23 wherein the digital processing component determines a camera motion blur function based on the comparison, wherein the camera motion blur function is mathematically defined as a point spread function (PSF).

26. A system according to claim 25, wherein said digital processing component further de-blurs the captured image using said determined camera motion blur function.

27. A system according to claim 25, wherein the reference image comprises a preview image.

28. A system according to claim 23, wherein the reference image comprises a lower pixel resolution than the captured image, and wherein said digital processing component is arranged to match the pixel resolutions of the captured image and the reference image.

29. A system according to claim 28, wherein said digital processing component is arranged to match the pixel resolutions of the captured image and the reference mage by at least one of up-sampling the reference image and sub-sampling the captured image.

30. A system according to claim 23, wherein said digital processing component is arranged to align at least portions of the captured image and the reference image.

31. A system according to claim 23, wherein the motion blur determining is performed in real time relative to the capturing of the digital image.

32. A system according to claim 1, wherein said apparatus comprises an image sensor component and said at least two sets of image data are acquired by said same image sensor component.

33. A system according to claim 32, wherein said apparatus further comprises an optical system and said at least two sets of image data are acquired along said same optical path.

34. In a digital image acquisition system including an apparatus for capturing digital images, a method comprising notifying the photographer upon determining that a threshold camera motion blur has occurred in a captured digital image based on a comparison of at least two sets of image data each acquired within a temporal range that includes an exposure period of the captured digital image, and times proximately before and after said exposure period, and of nominally the same scene as that of the captured digital image.

35. A method according to claim 34, wherein said at least two images comprise said captured image and a reference image taken outside the exposure period of said captured image.

36. A method according to claim 35, wherein said reference image is taken before said exposure period of said captured image.

37. A method according to claim 35, wherein the determining comprises matching pixel resolutions of the reference and captured images by at least one of up-sampling a reference image and sub-sampling the captured image.

38. A method according to claim 35, wherein the determining comprises aligning at least portions of the captured image and a reference image.

39. A method according to claim 35, further comprising identifying at least one characteristic of the reference image which is relatively less blurred than the corresponding characteristic of the captured image, and calculating a motion blur function in respect of said characteristic.

40. A method according to claim 35, further comprising calculating a trajectory of at least one characteristic in the reference image, extrapolating such characteristic onto the captured image, and calculating a motion blur function in respect of said characteristic.

41. A method according to claim 34, further comprising determining a camera motion blur function based on said comparison, and wherein the camera motion blur function comprises a point spread function (PSF).

42. A method according to claim 41, further including de-blurring the captured image using said determined camera motion blur function.

43. A method according to claim 42, wherein the de-blurring comprises de-convolving the captured image using the blur function.

44. A method according to claim 34, further including visually informing the photographer when the captured image is determined to be blurred.

45. A method according to claim 34, further including informing the photographer by audio signal when the captured image is determined to be blurred.

46. A method according to claim 34, further including transferring the at least two images from the digital image capturing apparatus to an external processing device for determining a motion blur function.

47. A method according to claim 34, wherein the motion blur determining is performed in real time relative to the capturing of the digital image.

48. A method according to claim 34, further comprising acquiring images corresponding to said at least two image data sets with a same image sensor component.

49. A method according to claim 47, wherein said capturing of said images occurs along a same optical path.

50. One or more storage devices having digital code embedded therein for programming one or more processors to perform a camera motion blur determining method that comprises capturing digital images, and notifying the photographer upon determining that a threshold camera motion blur has occurred in a captured digital image based on a comparison of at least two sets of image data each acquired within a temporal range that includes an exposure period of the captured digital image, and times proximately before and after said exposure period, and of nominally the same scene as that of the captured digital image.

51. The one or more storage devices of claim 50, wherein said at least two images comprise said captured image and a reference image taken outside the exposure period of said captured image.

52. The one or more storage devices of claim 51, wherein said reference image is taken before said exposure period of said captured image.

53. The one or more storages devices of claim 51, wherein the determining comprises matching pixel resolutions of the reference and captured images by at least one of up-sampling a reference image and sub-sampling the captured image.

54. The one or more storage devices of claim 51, wherein the determining comprises aligning at least portions of the captured image and a reference image.

55. The one or more storage devices of claim 51, the method further comprising identifying at least one characteristic of the reference image which is relatively less blurred than the corresponding characteristic of the captured image, and calculating a camera motion blur function in respect of said characteristic.

56. The one or more storage devices of claim 51, the method further comprising calculating a trajectory of at least one characteristic in the reference image, extrapolating such characteristic onto the captured image, and calculating a camera motion blur function in respect of said characteristic.

57. The one or more storage devices of claim 50, the method further comprising determining a camera motion blur function, and wherein the camera motion blur function comprises a point spread function (PSF).

58. The one or more storage devices of claim 57, further including de-blurring the captured image using said determined camera motion blur function.

59. The one or more storage devices of claim 58, wherein the de-blurring comprises de-convolving the captured image using the blur function.

60. The one or more storage devices of claim 50, the method further including visually informing the photographer when the captured image is determined to be blurred.

61. The one or more storage devices of claim 50, the method further including informing the photographer by audio signal when the captured image is determined to be blurred.

62. The one or more storage devices of claim 50, the method further including transferring the at least two images from the digital image capturing apparatus to an external processing device for determining a motion blur function.

63. The one or more storage devices of claim 50, wherein the motion blur determining is performed in real time relative to the capturing of the digital image.

64. The one or more storage devices of claim 50, the method further comprising acquiring images corresponding to said at least two image data sets with a same image sensor component.

65. The one or more storage devices of claim 64, wherein said capturing of said images occurs along a same optical path.

66. A digital image acquisition system, comprising an apparatus for capturing digital images and a digital processing component for notifying the photographer upon determining that a threshold camera motion blur has occurred in a captured digital image based on a real time analysis of the captured digital image.

67. A system according to claim 66, wherein the determining that a threshold camera motion blur has occurred is based on qualitatively determining a motion blur.

68. A system according to claim 67, wherein the camera motion blur is mathematically defined as a point spread function (PSF).

69. A system according to claim 67, wherein said digital processing component further de-blurs the captured image using said determined camera motion blur function.

70. A system according to claim 67, wherein said qualitatively determining a motion blur is performed using an auto-focus mechanism of the camera as the digital processing component.

71. A system according to claim 66, wherein said digital processing component is configured to inform the photographer if it determines that the captured image is blurred.

72. A system of according to claim 71, wherein the digital image capturing apparatus is signalled to capture another image if the digital processing component determines that the captured image is blurred.

73. A system of according to claim 66, wherein the digital image capturing apparatus is signalled to capture another image if the digital processing component determines that the captured image is blurred.

74. A system according to claim 66, wherein said digital image acquisition system comprises a portable digital camera.

75. A system according to claim 74, wherein said digital processing component is located in said portable digital camera.

76. A system according to claim 66, wherein said determining that a threshold camera motion blur has occurred in the captured digital image is further based on a comparison of at least two sets of image data each acquired within a temporal range that includes an exposure period of the captured digital image, and times proximately before and after said exposure period, and of nominally the same scene as that of the captured digital image.