Method and apparatus to correct digital video to counteract effect of camera shake

• US 9,860,450 B2
• Filed: 02/13/2017
• Issued: 01/02/2018
• Est. Priority Date: 03/25/2004
• Status: Active Grant

• Current Assignee: Clear Imaging Research, LLC
• Original Assignee: Clear Imaging Research, LLC
• Inventors: Ozluturk, Fatih M.
• Primary Examiner(s): HO, TUAN V
• Application Number:
  US15/431,332
• Publication Number:
  US 20170163897A1
• Time in Patent Office: 323 Days
• CPC Class Codes: G02B 27/646 compensating for small deviations, e.g. due to vibration or shake movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/681 Motion detection
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/615 involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF]
• CPC Class Codes: H04N 5/265 Mixing

Abstract

The effect of camera shake in digital video is corrected using signal processing techniques. The digital video is a sequence of digital images. When the sequence of digital images are being captured, movement of the imaging device causes the images to shift on the image sensor of the imaging device and affects the quality of the eventual video. Movement of the imaging device is detected while the video is being captured, and a motion information representing the motion is recorded. A processor determines a correcting filter based on the motion information and user input. The processor modifies the sequence of images captured according to the correcting filter and obtains a final corrected video. Corrected video is displayed in a viewfinder.

Claims

1. A method for use in an imaging device comprising an image sensor, a processor, a memory, and one or more motion sensors, the method comprising:

capturing a sequence of images with the image sensor, wherein the sequence of images comprise a video, and storing the images in the memory;

detecting, by the one or more motion sensors, motion information for one or more images of the sequence of images, wherein the motion information represents motion of the device during capturing of the one or more images of the sequence of images, and storing the motion information in the memory synchronously with the storing of the one or more images;

determining, by the processor, a vertical shift value and a horizontal shift value for one or more images of the sequence of images based at least in part on the motion information;

modifying, by the processor, one or more images of the sequence of images based at least in part on the vertical and the horizontal shift values;

combining, by the processor, the modified images to obtain a final video; and

storing the final video in the memory.

2. The method of claim 1, wherein the processor determines the vertical and horizontal shift values for one or more images for which the motion information is detected.

3. The method of claim 1, wherein the processor modifies one or more images for which the vertical and horizontal shift values are determined.

4. The method of claim 1, wherein the processor modifies one or more images of the sequence of images such that effect of motion of the device during capturing of the one or more images of the sequence of images is reduced in the final video.

5. The method of claim 1, wherein the processor determines a vertical shift value and a horizontal shift value for each image of the sequence of images.

6. The method of claim 1, wherein the motion information represents motion of the device at time of capturing of one or more images of the sequence of images.

7. The method of claim 1, wherein the one or more images of the sequence of images is at least two images, and wherein the motion information represents motion of the device between capturing of consecutive images.

8. The method of claim 1, wherein the vertical and horizontal shift values for an image indicate how much the image is displaced due to motion of the device during capturing of the image.

9. The method of claim 1, wherein the modifying by the processor of the one or more images of the sequence of images comprises shifting a reference point in each image according to the vertical shift value and the horizontal shift value for the image in a direction that reduces the effect of motion of the device in the final video.

10. The method of claim 1, wherein the method further comprises displaying the final video in a user interface.

11. The method of claim 1, wherein the method further comprises modifying the sequence of images using a video compression technique.

12. The method of claim 1, wherein determining a vertical shift value and a horizontal shift value for one or more images of the sequence of images is based at least in part on the focal distance of a lens of the imaging device.

13. The method of claim 1, wherein the method further comprises receiving user input in a user interface, and at least one of modifying one or more images of the sequential images or combining the modified images to obtain a final video is based at least in part on the user input.

14. An imaging device, comprising:

an image sensor configured to capture a sequence of images, wherein the sequence of images comprise a video, and store the images in a memory;

one or more motion sensors configured to detect motion information for one or more images of the sequence of images, wherein the motion information represents motion of the imaging device during capturing of the one or more images of the sequence of images, and store the motion information in the memory synchronously with the storing of the one or more images; and

a processor configured to;

determine a vertical shift value and a horizontal shift value for one or more images of the sequence of images based at least in part on the motion information;

modify one or more images of the sequence of images based at least in part on the vertical and the horizontal shift values; and

combine the modified images to obtain a final video; and

wherein the memory is further configured to store the final video.

15. The imaging device of claim 14, wherein the processor is configured to determine the vertical and horizontal shift values for one or more images of the sequence of images for which the motion information is detected.

16. The imaging device of claim 14, wherein the processor is configured to modify one or more images of the sequence of images for which the vertical and horizontal shift values are determined.

17. The imaging device of claim 14, wherein the processor is configured to modify one or more images of the sequence of images such that effect of motion of the device during capturing of the one or more images of the sequence of images is reduced in the final video.

18. The imaging device of claim 14, wherein the processor is configured to determine a vertical shift value and a horizontal shift value for each of the images of the sequence of images.

19. The imaging device of claim 14, wherein the motion information detected by the one or more motion sensors represents motion of the device at time of capturing of one or more images of the sequence of images.

20. The imaging device of claim 14, wherein the one or more images of the sequence of images is at least two images, and wherein the motion information detected by the one or more motion sensors represents motion of the device between capturing of consecutive images.

21. The imaging device of claim 14, wherein the processor is configured to determine a vertical shift value and a horizontal shift value for an image such that the vertical shift value and the horizontal shift value indicate how much the image is displaced due to motion of the device during capturing of the image.

22. The imaging device of claim 14, wherein the processor is configured to modify one or more images of the sequence of images by shifting a reference point in each image according to the vertical shift value and the horizontal shift value for the image in a direction that reduces the effect of motion of the device in the final video.

23. The imaging device of claim 14, wherein the device further comprises a display configured to display the final video.

24. The imaging device of claim 14, wherein the processor is further configured to modify the sequence of images using a video compression technique.

25. The imaging device of claim 14, wherein the processor is configured to determine a vertical shift value and a horizontal shift value for one or more images of the sequence of images based at least in part on the focal distance of a lens of the imaging device.

26. The imaging device of claim 14, wherein the device further comprises a display configured to receive user input, and the device is configured to modify one or more images of the sequential images and to obtain a final video based at least in part on the user input.

27. The imaging device of claim 14, wherein the processor is two or more processors.

28. A method for use in an imaging device comprising an image sensor, a processor, a memory, and one or more motion sensors, the method comprising:

capturing a sequence of images with the image sensor, wherein the sequence of images comprise a video;

detecting, by the one or more motion sensors, motion information for one or more images of the sequence of images, wherein the motion information represents motion of the device during capturing of the one or more images of the sequence of images;

determining, by the processor, a vertical shift value and a horizontal shift value for one or more images of the sequence of images based at least in part on the motion information;

modifying, by the processor, one or more images of the sequence of images based at least in part on the vertical and the horizontal shift values;

combining, by the processor, the modified images, and applying a video compression technique to obtain a final video; and

storing the final video in the memory.

29. An imaging device, comprising:

an image sensor configured to capture a sequence of images, wherein the sequence of images comprise a video;

one or more motion sensors configured to detect motion information for one or more images of the sequence of images, wherein the motion information represents motion of the imaging device during capturing of the one or more images of the sequence of images;

a processor configured to;

determine a vertical shift value and a horizontal shift value for one or more images of the sequence of images based at least in part on the motion information;

modify one or more images of the sequence of images based at least in part on the vertical and the horizontal shift values; and

combine the modified images, and apply a video compression technique to obtain a final video; and

a memory configured to store the final video.

30. The imaging device of claim 29, wherein the one or more images of the sequence of images is at least two images, and wherein the motion information detected by the one or more motion sensors represents motion of the device between capturing of consecutive images.

31. The imaging device of claim 29, wherein the processor is configured to determine a vertical shift value and a horizontal shift value for one or more images of the sequence of images based at least in part on the focal distance of a lens of the imaging device.

32. The imaging device of claim 29, wherein the device further comprises a display configured to receive user input, and the device is configured to modify one or more images of the sequential images and to obtain a final video based at least in part on the user input.