Method and apparatus for using motion information and image data to correct blurred images

• US 9,800,788 B2
• Filed: 07/11/2016
• Issued: 10/24/2017
• Est. Priority Date: 03/25/2004
• Status: Active Grant

• Current Assignee: Clear Imaging Research, LLC
• Original Assignee: Clear Imaging Research, LLC
• Inventors: Ozluturk, Fatih M.
• Primary Examiner(s): HO, TUAN V
• Application Number:
  US15/207,057
• Publication Number:
  US 20160323512A1
• Time in Patent Office: 470 Days
• CPC Class Codes: G02B 27/646 compensating for small deviations, e.g. due to vibration or shake movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/681 Motion detection
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/615 involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF]
• CPC Class Codes: H04N 5/265 Mixing

Abstract

A method and apparatus for use in a digital imaging device for correcting image blur in digital images by combining plurality of images. The plurality of images that are combined include a main subject that can be selected by user input or automatically by the digital imaging device. Blur correction can be performed to make the main subject blur-free while the rest of the image is blurred. All of the image may be made blur-free or the main subject can be made blur-free at the expense of the rest of the image. Result is a blur corrected image that is recorded in a memory.

Claims

1. A method for capturing stabilized video, for use in a device that includes a lens, an image sensor, a display, a memory, and a processor, the device having a field of view, the method comprising:

displaying, in the display of the device, a preview of a subject within the field of view of the device;

capturing a video of the subject, with the lens and the image sensor, wherein the video is a sequence of images;

detecting, by the processor, the subject in one or more images of the sequence of images and determining a location of the subject within the images;

shifting, by the processor, the one or more images vertically and horizontally by an integer number of pixels to obtain corrected images, wherein the amount of vertical and horizontal shift for each of the one or more images is determined at least in part based on the location of the subject in the image;

combining the corrected images to obtain a stabilized video; and

displaying the stabilized video in the display of the device.

2. The method of claim 1, wherein shifting by the processor the one or more images vertically and horizontally by the integer number of pixels is done such that pixels representing an image point of the subject in the one or more images are aligned in a same location in the corrected images.

3. The method of claim 1, further comprising receiving a user input in the display of the device, and combining the corrected images to obtain the stabilized video is based in part on the user input.

4. The method of claim 1, wherein the processor detects the subject in each of the images of the sequence of images.

5. The method of claim 1, wherein the processor detects a first subject and a second subject in the one or more images of the sequence of images and the shifting of the images vertically and horizontally to obtain corrected images is based in part on the location of the first subject in a first set of images of the one or more images and the location of the second subject in a second set of images of the one or more images.

6. The method of claim 1, wherein the shifting of the one or more images vertically by a vertical shift amount and horizontally by a horizontal shift amount by the processor is performed by shifting a reference point in the images in a vertical direction by a vertical shift amount and in a horizontal direction by a horizontal shift amount.

7. The method of claim 6, wherein the stabilized video is a sequence of corrected images, and wherein pixels representing an image point of the subject in the sequence of corrected images are aligned in a same location in the stabilized video.

8. A device for capturing stabilized video, comprising:

a display configured to display a preview of a subject within field of view of the device;

a lens and an image sensor configured to capture a video of the subject, wherein the video is a sequence of images; and

a processor configured to;

detect the subject in one or more images of the sequence of images and determine a location of the subject within the images;

shift the one or more images vertically and horizontally by an integer number of pixels to obtain corrected images, wherein the amount of vertical and horizontal shift for each of the one or more images is determined at least in part based on the location of the subject in the image; and

combine the corrected images to obtain a stabilized video; and

the display further configured to display the stabilized video.

9. The device of claim 8, wherein the processor is configured to shift the one or more images vertically and horizontally by the integer number of pixels such that pixels representing an image point of the subject in the one or more images are aligned in a same location in the corrected images.

10. The device of claim 8, wherein the display of the device is further configured to receive a user input, and the processor is further configured to combine the corrected images to obtain the stabilized video based in part on the user input.

11. The device of claim 8, wherein the processor is configured to detect the subject in each image of the sequence of images.

12. The device of claim 8, wherein the processor is configured to detect a first subject and a second subject in the one or more images of the sequence of images and to shift the images to obtain corrected images based in part on the location of the first subject in a first set of images of the one or more images and the location of the second subject in a second set of images of the one or more images.

13. The device of claim 8, wherein the processor is configured to shift the one or more images vertically by a vertical shift amount and horizontally by a horizontal shift amount by shifting a reference point in the images in a vertical direction by a vertical shift amount and in a horizontal direction by a horizontal shift amount.

14. The device of claim 13, wherein the processor is configured to combine the corrected images to obtain a stabilized video such that an image point of the subject in each of the corrected images is in a same location in the stabilized video.

15. A method for use in a computing device for stabilizing video comprising:

receiving by the device a video wherein the video is a sequence of images and wherein one or more of the images of the sequence of images include a subject;

detecting, by a processor of the device, the subject in one or more images of the sequence of images and determining the location of the subject within the images;

determining, by the processor of the device, a vertical shift amount and a horizontal shift amount for the one or more images, wherein the vertical and horizontal shift amounts are determined at least in part based on the location of the subject in the images, and wherein the vertical and horizontal shift amounts are an integer number of pixels;

modifying, by the processor of the device, the one or more images according to the vertical and horizontal shift amounts to obtain corrected images;

combining, by the processor of the device, the corrected images to obtain a stabilized video; and

storing the stabilized video in a memory of the device.

16. The method of claim 15, wherein the modifying by the processor of the device the one or more images according to the vertical and horizontal shift amounts is performed such that pixels representing an image point of the subject in the one or more images are aligned in a same location in the corrected images.

17. The method of claim 15, wherein the subject is detected by the processor of the device in each of the images of the sequence of images.

18. The method of claim 15, wherein the processor detects a first subject and a second subject in the one or more images of the sequence of images and the processor determines the vertical and horizontal shift amounts based in part on the location of the first subject in a first set of images of the one or more images and the location of the second subject in a second set of images of the one or more images.

19. The method of claim 15, wherein the modifying of the one or more images by the processor of the device is performed by shifting a reference point in the images by the vertical shift amount in the vertical direction and by the horizontal shift amount in the horizontal direction.

20. The method of claim 19, wherein the stabilized video is a sequence of corrected images, and wherein pixels representing an image point of the subject in the sequence of corrected images are aligned in a same location in the stabilized video.

21. A computing device for stabilizing video comprising:

a processor configured to;

receive a video wherein the video is a sequence of images and wherein one or more of the images of the sequence of images include a subject;

detect the subject in one or more images of the sequence of images and determine a location of the subject within the images;

determine a vertical shift amount and a horizontal shift amount for the one or more images, wherein the amount of vertical and horizontal shifts are determined at least in part based on the location of the subject in the images, and wherein the vertical and horizontal shift amounts are an integer number of pixels;

modify the one or more images according to the vertical and horizontal shift amounts for each image to obtain corrected images; and

combine the corrected images to obtain a stabilized video; and

a memory configured to store the stabilized video.

22. The device of claim 21, wherein the processor is configured to modify the one or more images according to the vertical and horizontal shift amounts for each image such that pixels representing an image point of the subject in the one or more images are aligned in a same location in the corrected images.

23. The device of claim 21, wherein the processor is configured to detect the subject in each image of the sequence of images.

24. The device of claim 21, wherein the processor is configured to detect a first subject and a second subject in the one or more images of the sequence of images and to determine a vertical shift amount and a horizontal shift amount for the one or more images based in part on the location of the first subject in a first set of images of the one or more images and the location of the second subject in a second set of images of the one or more images.

25. The device of claim 21, wherein the processor is configured to modify the one or more images by shifting a reference point in the images by the vertical shift amount in the vertical direction and by the horizontal shift amount in the horizontal direction.

26. The device of claim 25, wherein the processor is configured to combine the corrected images to obtain a stabilized video such that an image point of the subject in each of the corrected images is in a same location in the stabilized video.

27. A method for stabilizing video, the method when implemented in a processor causes the processor to execute the steps of:

receiving a video wherein the video is a sequence of images and wherein one or more of the images of the sequence of images include a subject;

detecting the subject in one or more images of the sequence of images and determining a location of the subject within the images;

determining a vertical shift amount and a horizontal shift amount for the one or more images, wherein the amount of vertical and horizontal shifts are determined at least in part based on the location of the subject in the images, and wherein the vertical and horizontal shift amounts are integer number of pixels;

modifying the one or more images according to the vertical and horizontal shift amounts to obtain corrected images;

combining the corrected images to obtain a stabilized video; and

storing the stabilized video in a memory of the device.

28. The method of claim 27, wherein the modifying the one or more images according to the vertical and horizontal shift amounts is performed such that pixels representing an image point of the subject in the one or more images are aligned in a same location in the corrected images.

29. The method of claim 27, wherein the processor further receives a user input and the combining of the corrected images to obtain a stabilized video is based in part on the user input.

30. The method of claim 27, wherein the processor detects the subject in each of the images of the sequence of images.

31. The method of claim 27, wherein the processor detects a first subject and a second subject in the one or more images of the sequence of images and the processor determines the vertical and horizontal shift amounts based in part on the location of the first subject in a first set of images of the one or more images and the location of the second subject in a second set of images of the one or more images.

32. The method of claim 27, wherein the modifying of the one or more images by the processor is performed by shifting a reference point in the images by the vertical shift amount in the vertical direction and by the horizontal shift amount in the horizontal direction.

33. The method of claim 32, wherein the stabilized video is a sequence of corrected images, and wherein pixels representing an image point of the subject in the sequence of corrected images are aligned in a same location in the stabilized video.