Method and apparatus to correct digital image blur due to motion of subject or imaging device

• US 8,630,484 B2
• Filed: 11/19/2008
• Issued: 01/14/2014
• Est. Priority Date: 03/25/2004
• Status: Active Grant

• Current Assignee: Clear Imaging Research, LLC
• Original Assignee: Fatih M. Ozluturk
• Inventors: Ozluturk, Fatih M.
• Primary Examiner(s): TSAI, TSUNG YIN
• Application Number:
  US12/274,057
• Publication Number:
  US 20090135272A1
• Time in Patent Office: 1,882 Days
• Field of Search: 382/162
• US Class Current: 382/162
• CPC Class Codes: G02B 27/646 compensating for small deviations, e.g. due to vibration or shake movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/681 Motion detection
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/615 involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF]
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Signal processing techniques are applied to digital image data to remove the distortion caused by motion of the camera, or the movement of the subject being photographed, or defective optics, or optical distortion from other sources. When the image is captured, the effect of relative motion between the camera and the subject is that it transforms the true image into a blurred image according to a 2-dimensional transfer function. The 2-dimensional transfer function representing the motion is derived using blind estimation techniques or by using information from sensors that detect the motion. The transfer function is inverted and used to define a corrective filter. The filter is applied to the image and the blur due to the motion is removed, restoring the correct image. Another embodiment uses the transfer function to avoid blur by combining multiple consecutive images taken at a fast shutter speed.

Claims

1. A method, comprising:

capturing, at a recording medium, a plurality of sequential images, wherein the images are two-dimensional photographic images;

detecting, by a processor, a main subject in each of the images in the plurality of images, wherein the main subject is the same in each of the images;

shifting, by the processor, each of the images vertically and horizontally such that the main subject is aligned at a same location in each of the shifted images; and

combining, by the processor, the shifted images to obtain a corrected image,

wherein the corrected image is a two-dimensional photographic image, and

wherein the combining includes determining, for each image point in the corrected image, a pixel value for the image point based on pixel values in the shifted images at the image point.

2. The method of claim 1 wherein the plurality of sequential images is captured using a highest available shutter speed.

3. The method of claim 1 wherein the detecting the main subject is performed using a pattern recognition technique.

4. The method of claim 1 wherein the detecting the main subject is performed using an image segmentation technique.

5. The method of claim 1 wherein the detecting the main subject includes tracking a signal that is transmitted from the main subject.

6. The method of claim 1 wherein the detecting the location of the main subject is performed based on input from a user.

7. The method of claim 1, wherein the determining the pixel value for each image point in the corrected image includes adding pixel values of the shifted images at the image point or averaging pixel values of the shifted images at the image point.

8. An apparatus, comprising:

a recording medium configured to capture a plurality of sequential images, wherein the images are two-dimensional photographic images;

a memory, configured to store the images;

a processor configured;

to detect a main subject in each of the images in the plurality of images, wherein the main subject is the same in each of the images;

to shift each of the images vertically and horizontally such that the main subject is aligned at a same location in each of the shifted images; and

to combine the shifted images to obtain a corrected image,

wherein the corrected image is a two-dimensional photographic image, and

wherein, for each image point in the corrected image, a pixel value for the image point is based on pixel values from the shifted images at the image point.

9. The apparatus of claim 8 wherein the images are captured using a highest available shutter speed.

10. The apparatus of claim 8 wherein the processor is configured to detect the main subject using a pattern recognition technique.

11. The apparatus of claim 8 wherein the processor is configured to detect the main subject using an image segmentation technique.

12. The apparatus of claim 8 wherein the processor is configured to detect the main subject based on a tracking a signal that is transmitted from the main subject.

13. The apparatus of claim 8, wherein the processor is configured to detect the main subject based on input from a user.

14. The camera of claim 8, wherein the determining the pixel value for each image point in the corrected image includes adding pixel values of the shifted images at the image point or averaging pixel values of the shifted images at the image point.

15. A computing device, comprising:

a memory, configured to store a plurality of sequential images, wherein the images are two-dimensional photographic images;

a processor configured;

to detect a main subject for each of the images in the plurality of images, wherein the main subject is the same in each of the images;

to shift each of the images vertically and horizontally such that the main subject is aligned at a same location in each of the shifted images; and

to combine the shifted images to obtain a corrected image,

wherein the corrected image is a two-dimensional photographic image, and

wherein, for each image point in the corrected image, a pixel value for the image point is based on pixel values from the shifted images at the image point.

16. The computing device of claim 15 wherein the images are captured using a highest available shutter speed.

17. The computing device of claim 15 wherein the processor is configured to detect the main subject using a pattern recognition technique.

18. The computing device of claim 15 wherein the processor is configured to detect the main subject using an image segmentation technique.

19. The computing device of claim 15 wherein the processor is configured to detect the main subject by tracking a signal that is transmitted from the main subject.

20. The computing device of claim 15 wherein the processor is configured to detect the main subject based on input from a user.

21. The computing device of claim 15, wherein the determining the pixel value for each image point in the corrected image includes adding pixel values of the shifted images at the image point or averaging pixel values of the shifted images at the image point.

22. A method comprising:

recording, at a recording medium, a plurality of images, wherein the images are two-dimensional photographic images;

detecting, by a processor, a main subject in each of the images in the plurality of images;

shifting, by the processor, the images such that the main subject is placed at a same location in each of the shifted images; and

combining, by the processor, the shifted images to obtain a corrected image,

wherein the corrected image is a two-dimensional photographic image,

wherein the combining includes determining, for each image point in the corrected image, a pixel value for the image point based on pixel values in the shifted images at the image point.

23. An apparatus, comprising:

a recording medium configured to record a plurality of images, wherein the images are two-dimensional photographic images;

a memory, configured to store the images;

a processor configured;

to detect a main subject in each of the images in the plurality of images;

to shift the images such that the main subject is placed at a same location in each of the shifted images; and

to combine the shifted images to obtain a corrected image,

wherein the corrected image is a two-dimensional photographic image,

wherein, for each image point in the corrected image, a pixel value for the image point is based on pixel values from the shifted images at the image point.

24. A computing device, comprising:

a memory, configured to store a plurality of images, wherein the images are two-dimensional photographic images;

a processor configured;

to detect a main subject in each of the images in the plurality of images;

to shift the images such that the main subject is placed at a same location in each of the shifted images; and

to combine the shifted images to obtain a corrected image,

wherein the corrected image is a two-dimensional photographic image,

wherein, for each image point in the corrected image, a pixel value for the image point is based on pixel values from the shifted images at the image point.

25. A method, comprising:

capturing, at a recording medium, a plurality of sequential images, wherein the images are two-dimensional photographic images;

receiving, from a subject of the images, a tracking signal that is transmitted from the subject and indicates a location of the subject;

shifting, by a processor, each of the plurality of sequential images vertically and horizontally such that the subject is aligned at a same location in each of the shifted images; and

combining, by the processor, the shifted images to obtain a corrected image,

wherein the corrected image is a two-dimensional photographic image, and

wherein the combining includes determining, for each image point in the corrected image, a pixel value for the image point based on pixel values in the shifted images at the image point.

26. An apparatus, comprising:

a recording medium configured to capture a plurality of images, wherein the images are two-dimensional photographic images;

a receiver configured to receive, from a subject of the images, a tracking signal that is transmitted from the subject and indicates a location of the subject; and

a processor configured to;

shift each of the plurality of images vertically and horizontally such that the subject is aligned at a same location in each of the shifted images; and

combine the shifted images to obtain a corrected image,

wherein the corrected image is a two-dimensional photographic image, and

wherein, for each image point in the corrected image, a pixel value for the image point is based on pixel values in the shifted images at the image point.

27. A method for use in a device that includes a viewfinder, a recording medium, and a processor, the method comprising:

displaying, at the viewfinder, an image;

receiving, at the processor, user input data that designates a subject in the image;

capturing, at the recording medium, a plurality of images, wherein the images are two-dimensional photographic images and include the designated subject;

shifting, by the processor, each of the plurality of images vertically and horizontally such that the designated subject is aligned at a same location in each of the shifted images; and

combining, by the processor, the shifted images to obtain a corrected image, wherein the corrected image is a two-dimensional photographic image.

28. An apparatus, comprising:

a viewfinder configured to display an image;

a processor configured to receive user input data that designates a subject in the image; and

a recording medium configured to capture a plurality of images, wherein the images are two-dimensional photographic images and include the designated subject;

wherein the processor is further configured to;

shift each of the plurality of images vertically and horizontally such that the designated subject is aligned at a same location in each of the shifted images; and

combine the shifted images to obtain a corrected image, wherein the corrected image is a two-dimensional photographic image.

29. A method, comprising:

capturing, at a recording medium, a plurality of images,

wherein the images are two-dimensional photographic images,

wherein the images include a main subject that is the same in each of the images, and

wherein the images include a first image and a plurality of subsequent images;

detecting, by a processor, a location of the main subject in each of the images;

generating, by the processor, a final image based on the images, wherein the generating the final image includes, for each of the plurality of subsequent images;

shifting the subsequent image such that the location of the main subject in the shifted subsequent image is the same as the location of the main subject in the first image; and

combining the shifted subsequent image with the first image.

30. An apparatus, comprising:

a recording medium configured to capture a plurality of images,

wherein the images are two-dimensional photographic images,

wherein the images include a main subject that is the same in each of the images, and

wherein the images include a first image and a plurality of subsequent images; and

a processor configured to;

detect a location of the main subject in each of the images;

shift each of the subsequent images such that, for each shifted subsequent image, the location of the main subject in the shifted subsequent image is the same as the location of the main subject in the first image; and

generate a final image based on the images by, for each of the plurality of subsequent images;

shifting the subsequent image such that the location of the main subject in the shifted subsequent image is the same as the location of the main subject in the first image; and

combining the shifted subsequent image with the first image.