Method and apparatus to correct digital video to counteract effect of camera shake

• US 9,800,787 B2
• Filed: 05/09/2016
• Issued: 10/24/2017
• Est. Priority Date: 03/25/2004
• Status: Active Grant

• Current Assignee: Clear Imaging Research, LLC
• Original Assignee: Clear Imaging Research, LLC
• Inventors: Ozluturk, Fatih M.
• Primary Examiner(s): HO, TUAN V
• Application Number:
  US15/149,481
• Publication Number:
  US 20160255275A1
• Time in Patent Office: 533 Days
• CPC Class Codes: G02B 27/646 compensating for small deviations, e.g. due to vibration or shake movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/681 Motion detection
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/615 involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF]
• CPC Class Codes: H04N 5/265 Mixing

Abstract

The effect of camera shake in digital video is corrected using signal processing techniques. The digital video is a sequence of digital images. When the sequence of digital images are being captured, movement of the imaging device causes the images to shift on the image sensor of the imaging device and affects the quality of the eventual video. Movement of the imaging device is detected while the video is being captured, and a motion information representing the motion is recorded. A processor determines a correcting filter based on the motion information and user input. The processor modifies the sequence of images captured according to the correcting filter and obtains a final corrected video. Corrected video is displayed in a viewfinder.

Claims

1. A method for use in a device that includes a lens, an image sensor, a display, a memory, one or more motion sensors, and a processor, comprising:

displaying, in the display of the device, a preview of a subject within a field of view of the device;

capturing a video of the subject, with the lens and the image sensor, wherein the video is a sequence of images, and storing the video in the memory;

while capturing the video, detecting by one or more motion sensors a motion information, wherein the motion information indicates motion of the device during capturing of one or more images of the sequence of images;

calculating, by the processor, a correcting filter for the sequence of images to counteract the motion of the device based in part on the motion information;

modifying the sequence of images, by the processor, according to the correcting filter to obtain a final video;

and displaying the final video in the display of the device.

2. The method of claim 1, wherein the video captured by the image sensor is a plurality of sequential videos.

3. The method of claim 1, wherein the motion information is stored synchronously with the storing of the video.

4. The method of claim 1, wherein the calculating a correcting filter to counteract the motion of the device is based in part on the focal distance of the lens or the zoom level associated with the lens.

5. The method of claim 1, wherein the device receives user input in the display of the device, and the modifying the video to obtain a final video is based in part on the user input.

6. A device, comprising:

a display configured to display a preview of a subject within a field of view of the device;

a lens and an image sensor configured to capture a video of the subject, wherein the video is a sequence of images;

a memory configured to store the video;

one or more motion sensors configured to record a motion information, wherein the motion information indicates motion of the device while capturing of one or more images of the sequence of images;

a processor configured;

to calculate a correcting filter for the sequence of images to counteract motion of the device based in part on the motion information;

and to modify the sequence of images according to the correcting filter to generate a corrected video such that motion of the device is counteracted in the corrected video;

and the display further configured to display the corrected video.

7. The device of claim 6, wherein the video captured by the camera is a plurality of sequential videos.

8. The device of claim 6, wherein the motion information is stored synchronously with the storing of the video.

9. The device of claim 6, wherein the calculating a correcting filter to counteract the motion of the device is based in part on the focal distance of the lens or the zoom level associated with the lens.

10. The device of claim 6, wherein the device receives user input in the display of the device, and the modifying the video to obtain a final video is based in part on the user input.

11. A method for use in a device that includes a lens, an image sensor, a display, a memory, one or more motion sensors, and a processor, comprising:

displaying, in the display of the device, a preview of a subject within a field of view of the device;

capturing a plurality of sequential images of the subject, with the lens and the image sensor, wherein the images form part of a video, and storing the sequential images in the memory;

while capturing the images, detecting by one or more motion sensors a motion information, wherein the motion information indicates motion of the device during capturing of one or more images of the sequential images;

calculating, by the processor, a correcting filter for one or more images of the sequential images to counteract the motion of the device based in part on the motion information;

modifying one or more of the sequential images, by the processor, according to the correcting filter to obtain a final video;

and displaying the final video in the display of the device.

12. The method of claim 11, wherein the sequential images captured by the image sensor form part of a plurality of sequential videos.

13. The method of claim 11, wherein the motion information is stored synchronously with the storing of the sequential images.

14. The method of claim 11, wherein the calculating a correcting filter to counteract the motion of the device is based in part on the focal distance of the lens or the zoom level.

15. The method of claim 11, wherein the device receives user input in the display of the device, and the modifying the sequential images to obtain a final video is based in part on the user input.

16. A device, comprising:

a display configured to display a preview of a subject within a field of view of the device;

a lens and an image sensor configured to capture a plurality of sequential images, wherein the sequential images form part of a video of the subject;

a memory configured to store the images;

one or more motion sensors configured to record a motion information, wherein the motion information indicates motion of the device during capturing one or more images of the sequential images;

a processor configured;

to calculate a correcting filter for one or more images of the sequential images to counteract the motion of the device based in part on the motion information;

to modify one or more images of the sequential images according to the correcting filter to generate corrected sequential images;

and to combine the corrected images to obtain a final video;

and the display further configured to display the final video.

17. The device of claim 16, wherein the sequential images captured form part of a plurality of sequential videos.

18. The device of claim 16, wherein the motion information is stored synchronously with the storing of the sequential images.

19. The device of claim 16, wherein the calculating a correcting filter to counteract the motion of the device is based in part on the focal distance of the lens or the zoom level.

20. The device of claim 16, wherein the device receives user input in the display of the device, and the modifying the sequential images to obtain a final video is based in part on the user input.

21. A method for use in a computing device that includes a lens, an image sensor, a display, a memory, one or more motion sensors, and a processor, the method when performed by a software running on the processor causes the processor to execute the steps of:

receiving a video of the subject captured with the lens and the image sensor, wherein the video is a sequence of images;

receiving a motion information detected by one or more motion sensors, wherein the motion information indicates motion of the device during capturing of one of more images of the sequence of images;

calculating a correcting filter for the sequence of images to counteract the motion of the device based in part on the motion information;

modifying the sequence of images according to the correcting filter to obtain a final video;

and sending the final video to the display of the device for displaying.

22. The method of claim 21, wherein the video received is a plurality of sequential videos.

23. The method of claim 21, wherein the calculating a correcting filter to counteract the motion of the device is based in part on the focal distance of the lens or the zoom level associated with the lens.

24. The method of claim 21, wherein the processor receives indication of a user input from the display of the device, and the modifying the video to obtain a final video is based in part on the user input.