Method of determining PSF using multiple instances of a nominally similar scene

• US 8,243,996 B2
• Filed: 11/30/2010
• Issued: 08/14/2012
• Est. Priority Date: 11/10/2004
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Corcoran, Peter, Prilutsky, Yury
• Primary Examiner(s): Bhatnagar, Anand
• Assistant Examiner(s): BLOOM, NATHAN J
• Application Number:
  US12/956,904
• Publication Number:
  US 20110069207A1
• Time in Patent Office: 623 Days
• Field of Search: 382/100, 382103-104, 382/107, 382/141, 382152-154, 382/165, 382/170, 382/173, 382/181, 382/190, 382/195, 348/61, 348/113, 348/135, 348143-144, 348147-149, 348152-155, 348/158
• US Class Current: 382/107
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction

Abstract

A digital image acquisition system includes a portable apparatus for capturing digital images and a digital processing component for detecting, analyzing and informing the photographer regarding motion blur, and for reducing camera motion blur in an image captured by the apparatus. The digital processing component operates by comparing the image with at least one other image, for example a preview image, of nominally the same scene taken outside the exposure period of the main image. In one embodiment the digital processing component identifies at least one feature in a single preview image which is relatively less blurred than the corresponding feature in the main image, calculates a point spread function (PSF) in respect of such feature, and de-convolves the main image using the PSF. In another embodiment, the digital processing component calculates a trajectory of at least one feature in a plurality of preview images, extrapolates such feature on to the main image, calculates a PSF in respect of the feature, and de-convolves the main image using the PSF. In another embodiment the digital processing unit after determining the degree of blur notifies the photographer of the existing blur or automatically invokes consecutive captures.

Claims

1. A method of extrapolating camera movement using multiple preview images, comprising:

capturing multiple preview images and a full resolution image, each with a same sensor, such that at least a same portion of said same sensor captures at least a portion of each of said full resolution image and said multiple preview images, within a temporal range that includes an exposure period of the full resolution image, and times proximately before and after said exposure period, and of approximately the same scene as that of the full resolution image;

recording said images with a time stamp;

determining time intervals between images;

selecting one or more distinctive regions within at least one of the preview images;

matching corresponding regions within the preview images;

recording coordinates of each of the corresponding regions; and

extrapolating camera movement as a function of time based on the time intervals between images and the coordinates of the corresponding regions within the preview images.

2. The method of claim 1, wherein the time intervals comprise the time interval between capture of the full resolution image and the last preview image.

3. The method of claim 1, wherein the extrapolating of camera movement is further based on a duration of exposure of the full resolution image.

4. The method of claim 1, wherein the extrapolating is based on tracking of camera movement before said capturing of the full resolution image.

5. The method of claim 4, wherein the extrapolating is further based on the time interval between capture of the full resolution image and the last preview image.

6. The method of claim 5, wherein the extrapolating is further based on a duration of exposure of the full resolution image.

7. The method of claim 1, wherein the extrapolating of said camera movement comprises extrapolating movement of single points.

8. The method of claim 1, wherein the extrapolating of camera movement comprises extrapolating movement of high contrast image features.

9. The method of claim 1, wherein intervals between preview images are equally separated, in a manner of several images per second.

10. The method of claim 1, wherein the selecting of distinctive regions comprises isolating a distinction region from background.

11. The method of claim 10, wherein the distinction region comprises a noticeable difference in contrast or brightness, or both compared with the background.

12. The method of claim 1, wherein the matching of corresponding regions within preview images comprises filtering images with intolerable motion blurring or object obscuration defects, or where a corresponding region has moved outside the field of the preview image, or combinations thereof.

13. The method of claim 1, wherein the recording of coordinates comprises recording coordinates for preview images and for the full resolution image.

14. The method of claim 1, further comprising determining distance of a subject using a focus component.

15. The method of claim 1, further comprising determining whether the full resolution image comprises a threshold amount of motion blur based on the extrapolated camera movement.

16. The method of claim 1, further comprising warning a photographer, capturing a new image or de-blurring the full resolution image based on the extrapolated camera movement.

17. An image acquisition device configured to extrapolate image movement using multiple preview images, comprising:

a processor;

focus and image capture components configured to capture preview images and a full resolution image each with a same sensor, said components being configured such that at least a same portion of said same sensor captures at least a portion of each of said full resolution image and said multiple preview images, within a temporal range that includes an exposure period of the full resolution image, and times proximately before and after said exposure period, and of approximately the same scene as that of the full resolution image;

a memory having code embedded therein for programming the processor to perform a method to extrapolate image movement using the preview images, wherein the method comprises;

recording the multiple preview images and the full resolution image with a time stamp;

determining time intervals between images;

selecting one or more distinctive regions within at least one of the preview images;

matching corresponding regions within the preview images;

recording coordinates of each of the corresponding regions; and

extrapolating camera movement as a function of time based on the time intervals between images and on the coordinates of the corresponding regions within the preview images.

18. The device of claim 17, wherein the time intervals comprise the time interval between capture of the full resolution image and the last preview image.

19. The device of claim 17, wherein the extrapolating of camera movement is further based on a duration of exposure of the full resolution image.

20. The device of claim 17, wherein the extrapolating is based on tracking of camera movement before said capturing of the full resolution image.

21. The device of claim 20, wherein the extrapolating is further based on the time interval between capture of the full resolution image and the last preview image.

22. The device of claim 21, wherein the extrapolating is further based on a duration of exposure of the full resolution image.

23. The device of claim 17, wherein the extrapolating of said camera movement comprises extrapolating movement of single points.

24. The device of claim 17, wherein the extrapolating of camera movement comprises extrapolating movement of high contrast image features.

25. The device of claim 17, wherein intervals between preview images are equally separated, in a manner of several images per second.

26. The device of claim 17, wherein the selecting of distinctive regions comprises isolating a distinction region from background.

27. The device of claim 26, wherein the distinction region comprises a noticeable difference in contrast or brightness, or both compared with the background.

28. The device of claim 17, wherein the matching of corresponding regions within preview images comprises filtering images with intolerable motion blurring or object obscuration defects, or where a corresponding region has moved outside the field of the preview image, or combinations thereof.

29. The device of claim 17, wherein the recording of coordinates comprises recording coordinates for preview images and for the full resolution image.

30. The device of claim 17, wherein the focus component is configured to determine a distance of a subject within one or more of the preview images and the full resolution image.

31. The device of claim 17, wherein the method further comprises determining whether the full resolution image comprises a threshold amount of motion blur based on the extrapolated camera movement.

32. The device of claim 17, wherein the method further comprises warning a photographer, capturing a new image or de-blurring the full resolution image based on the extrapolated camera movement.

33. A non-transitory memory having code embedded therein for programming a processor to perform a method to extrapolate image movement using preview images, wherein the method comprises:

recording multiple preview images and a full resolution image with a time stamp, said multiple preview images and said full resolution images having been captured with a same digital image sensor, such that at least a same portion of said same sensor captures at least a portion of each of said full resolution image and said multiple preview images, within a temporal range that includes an exposure period of the full resolution image, and times proximately before and after said exposure period, and of approximately the same scene as that of the full resolution image;

determining time intervals between images;

selecting one or more distinctive regions within at least one of the preview images;

matching corresponding regions within the preview images;

recording coordinates of each of the corresponding regions; and

extrapolating camera movement as a function of time based on the time intervals between images and on the coordinates of the corresponding regions within the preview images.

34. The memory of claim 33, wherein the time intervals comprise the time interval between capture of the full resolution image and the last preview image.

35. The memory of claim 33, wherein the extrapolating of camera movement is further based on a duration of exposure of the full resolution image.

36. The memory of claim 33, wherein the extrapolating is based on tracking of camera movement before said capturing of the full resolution image.

37. The memory of claim 36, wherein the extrapolating is further based on the time interval between capture of the full resolution image and the last preview image.

38. The memory of claim 37, wherein the extrapolating is further based on a duration of exposure of the full resolution image.

39. The memory of claim 33, wherein the extrapolating of said camera movement comprises extrapolating movement of single points.

40. The memory of claim 33, wherein the extrapolating of camera movement comprises extrapolating movement of high contrast image features.

41. The memory of claim 33, wherein intervals between preview images are equally separated, in a manner of several images per second.

42. The memory of claim 33, wherein the selecting of distinctive regions comprises isolating a distinction region from background.

43. The memory of claim 42, wherein the distinction region comprises a noticeable difference in contrast or brightness, or both compared with the background.

44. The memory of claim 33, wherein the matching of corresponding regions within preview images comprises filtering images with intolerable motion blurring or object obscuration defects, or where a corresponding region has moved outside the field of the preview image, or combinations thereof.

45. The memory of claim 33, wherein the recording of coordinates comprises recording coordinates for preview images and for the full resolution image.

46. The memory of claim 33, wherein the focus component is configured to determine a distance of a subject within one or more of the preview images and the full resolution image.

47. The memory of claim 33, wherein the method further comprises determining whether the full resolution image comprises a threshold amount of motion blur based on the extrapolated camera movement.

48. The memory of claim 33, wherein the method further comprises warning a photographer, capturing a new image or de-blurring the full resolution image based on the extrapolated camera movement.