Method and apparatus to correct digital video to counteract effect of camera shake

• US 9,338,356 B2
• Filed: 04/20/2015
• Issued: 05/10/2016
• Est. Priority Date: 03/25/2004
• Status: Active Grant

• Current Assignee: Clear Imaging Research, LLC
• Original Assignee: Clear Imaging Research, LLC
• Inventors: Ozluturk, Fatih M.
• Primary Examiner(s): HO, TUAN V
• Application Number:
  US14/690,818
• Publication Number:
  US 20150237258A1
• Time in Patent Office: 386 Days
• Field of Search: 34820899-20813
• US Class Current: 1/1
• CPC Class Codes: G02B 27/646 compensating for small deviations, e.g. due to vibration or shake movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/681 Motion detection
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/615 involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF]
• CPC Class Codes: H04N 5/265 Mixing

Abstract

The effect of camera shake in digital video is corrected using signal processing techniques. The digital video is a sequence of digital images. When the sequence of digital images are being captured, movement of the imaging device causes the images to shift on the image sensor of the imaging device and affects the quality of the eventual video. Movement of the imaging device is detected while the video is being captured, and a motion information representing the motion is recorded. A processor determines a correcting filter based on the motion information and user input. The processor modifies the sequence of images captured according to the correcting filter and obtains a final corrected video. Corrected video is displayed in a viewfinder.

Claims

1. A method for use in a device that includes a lens, an image sensor, a viewfinder, a memory, one or more motion sensors, and a processor, comprising:

capturing by the image sensor a video and storing the video in the memory;

while capturing the video, detecting by one or more motion sensors a motion information representing the motion of the device, and storing the motion information in the memory;

calculating, by the processor, a correcting filter to counteract the motion of the device based in part on the motion information;

modifying the video, by the processor, according to the correcting filter to obtain a final video; and

displaying the final video in the viewfinder.

5. The method of claim 1, wherein the video captured by the image sensor is a plurality of sequential videos.

6. The method of claim 1, wherein the motion information is stored synchronously with the storing of the video.

7. The method of claim 1, wherein the calculating a correcting filter to counteract the motion of the device is based in part on the focal distance of the lens or the zoom level.

8. The method of claim 1, wherein the device receives user input at the viewfinder, and the modifying the video to obtain a final video is based in part on the user input.

2. A computing device, comprising:

a camera including a lens and an image sensor configured to capture a video;

one or more motion sensors configured to record a motion information representing motion of the device while the video is being captured;

a memory configured to store the video and to store the motion information;

a processor configured;

to calculate a correcting filter to counteract motion of the device based in part on the motion information; and

to modify the video according to the correcting filter to generate a corrected video such that motion of the device is counteracted in the corrected video; and

a viewfinder configured to display the corrected video.

9. The computing device of claim 2, wherein the video captured by the camera is a plurality of sequential videos.

10. The computing device of claim 2, wherein the motion information is stored synchronously with the storing of the video.

11. The computing device of claim 2, wherein the calculating a correcting filter to counteract the motion of the device is based in part on the focal distance of the lens or the zoom level.

12. The computing device of claim 2, wherein the device receives user input at the viewfinder, and the modifying the video to obtain a final video is based in part on the user input.

3. A method for use in a device that includes a lens, an image sensor, a viewfinder, a memory, one or more motion sensors, and a processor, comprising:

capturing by the image sensor, a plurality of sequential images, wherein the images form part of a video, and storing the sequential images in the memory;

while capturing the images, detecting by one or more motion sensors a motion information representing the motion of the device, and storing the motion information in the memory;

calculating, by the processor, a correcting filter for each of the images to counteract the motion of the device in sequential images;

modifying the sequential images, by the processor, according to the correcting filter to obtain a final video; and

displaying the final video in the viewfinder.

13. The method of claim 3, wherein the sequential images captured by the image sensor form part of a plurality of sequential videos.

14. The method of claim 3, wherein the motion information is stored synchronously with the storing of the sequential images.

15. The method of claim 3, wherein the calculating a correcting filter to counteract the motion of the device is based in part on the focal distance of the lens or the zoom level.

16. The method of claim 3, wherein the device receives user input at the viewfinder, and the modifying the sequential images to obtain a final video is based in part on the user input.

4. A computing device, comprising:

a camera including a lens and an image sensor configured to capture a plurality of sequential images, wherein the sequential images form part of a video;

one or more motion sensors configured to record a motion information representing the motion of the device while the images are being captured;

a memory configured to store the images and to store the motion information; and

a processor configured;

to calculate a correcting filter to counteract motion of the device in sequential images based in part on the motion information;

to modify each of the images to generate a corrected image according to the correcting filter to counteract the motion of the device in each of the sequential images;

to combine the corrected images to obtain a final video; and

a viewfinder configured to display the final video.

17. The computing device of claim 4, wherein the sequential images captured by the camera form part of a plurality of sequential videos.

18. The computing device of claim 4, wherein the motion information is stored synchronously with the storing of the sequential images.

19. The computing device of claim 4, wherein the calculating a correcting filter to counteract the motion of the device is based in part on the focal distance of the lens or the zoom level.

20. The computing device of claim 4, wherein the device receives user input at a viewfinder, and the modifying the sequential images to obtain a final video is based in part on the user input.