Method and apparatus for capturing stabilized video in an imaging device

• US 10,382,689 B2
• Filed: 12/27/2018
• Issued: 08/13/2019
• Est. Priority Date: 03/25/2004
• Status: Active Grant

• Current Assignee: Clear Imaging Research, LLC
• Original Assignee: Clear Imaging Research, LLC
• Inventors: Ozluturk, Fatih M.
• Primary Examiner(s): Monk, Mark T
• Application Number:
  US16/233,914
• Publication Number:
  US 20190132519A1
• Time in Patent Office: 229 Days
• Field of Search: None
• CPC Class Codes: G02B 27/646 compensating for small deviations, e.g. due to vibration or shake movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/681 Motion detection
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/615 involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF]
• CPC Class Codes: H04N 5/265 Mixing

Abstract

An imaging apparatus for capturing stabilized digital video, includes a user interface for displaying a preview of the field of view of the imaging apparatus and to receive user input, an image sensor for capturing digital video data for a video, a lens comprising one or more lens elements, wherein the one or more lens elements include at least one movable lens element that is movable relative to the imaging apparatus, and a video image processor to receive the video data and the motion data. The processor calculates shift amounts for the sequence of images based at least in part on the motion data generated by the motion sensors, modifies the sequence of images using the calculated shift amounts, combines the modified sequence of images to obtain a stabilized video. A memory connected to the video image processor stores the stabilized video.

Claims

1. An imaging apparatus for capturing stabilized digital video, comprising:

a user interface for displaying a preview of the field of view of the imaging apparatus and to receive user input;

an image sensor for capturing digital video data for a video, wherein the video comprises a sequence of images;

one or more motion sensors for detecting motion of the apparatus during capturing of the video data and generating motion data representing the motion of the apparatus;

a lens comprising one or more lens elements, wherein the one or more lens elements include at least one movable lens element that is movable relative to the imaging apparatus;

a video image processor communicatively connected to the image sensor and the motion sensors to receive the video data and the motion data, the processor configured to;

calculate shift amounts for the sequence of images based at least in part on the motion data generated by the motion sensors;

modify the sequence of images using the calculated shift amounts; and

combine the modified sequence of images to obtain a stabilized video; and

a memory connected to the video image processor for storing the stabilized video.

2. The imaging apparatus of claim 1, wherein the video image processor is further configured to cause the at least one moveable lens element of the lens to move by an amount calculated using the motion data generated by the motion sensors.

3. The imaging apparatus of claim 2, wherein the video image processor causes the at least one movable lens element to move such that the moving of the lens element at least partially compensates for the motion of the apparatus detected by the motion sensors in the stabilized video.

4. The imaging apparatus of claim 1, wherein the shift amounts for the sequence of images include a vertical shift amount in a vertical direction and a horizontal shift amount in the horizontal direction for each image.

5. The imaging apparatus of claim 2, wherein the shift amounts for the sequence of images include a vertical shift amount in a vertical direction and a horizontal shift amount in the horizontal direction for each image.

6. The imaging apparatus of claim 4, wherein the processor is configured to modify the sequence of images by shifting the images in the vertical direction by the vertical shift amount and in the horizontal direction by the horizontal shift amount.

7. The imaging apparatus of claim 5, wherein the processor is configured to modify the sequence of images by shifting the images in the vertical direction by the vertical shift amount and in the horizontal direction by the horizontal shift amount.

8. The imaging apparatus of claim 1, wherein the shift amounts for the sequence of images represent an amount of shift for a reference point in each image.

9. The imaging apparatus of claim 2, wherein the shift amounts for the sequence of images represent an amount of shift for a reference point in each image.

10. The imaging apparatus of claim 8, wherein the processor is configured to modify the sequence of images by shifting the reference point by the shift amount in each image.

11. The imaging apparatus of claim 9, wherein the processor is configured to modify the sequence of images by shifting the reference point by the shift amount in each image.

12. The imaging apparatus of claim 1, wherein the processor is connected to the user interface for receiving user input from the user interface and for displaying the stabilized video in the user interface.

13. The imaging apparatus of claim 1, wherein the processor is configured to calculate shift amounts for the sequence of images based at least in part on a focal length of the lens.

14. The imaging apparatus of claim 2, wherein the processor is configured to calculate shift amounts for the sequence of images based at least in part on a focal length of the lens.

15. The imaging apparatus of claim 1, wherein the processor is configured to obtain a stabilized video such that the effect of the motion of the imaging apparatus in the stabilized video is reduced.

16. A method of capturing stabilized digital video in an imaging device, the method comprising:

displaying a preview of the field of view of the imaging device in a user interface;

capturing a digital video with an image sensor and a lens, wherein the video is a sequence of images, and wherein the lens includes one or more lens elements and at least one of the lens elements is movable relative to the imaging device;

detecting motion information with one or more motion sensors, wherein the motion information represents motion of the imaging device during capturing of the video;

determining shift amounts for the sequence of images of the video by a processor, wherein the shift amounts are determined based at least in part on the motion information;

modifying the sequence of images of the video by the processor, wherein modifying the images is based at least in part on the shift amounts determined;

combining modified images by the processor to obtain a stabilized video; and

storing the stabilized video in a memory.

17. The method of claim 16, wherein the method further comprises moving the at least one moveable lens element of the lens by an amount calculated using the motion data generated by the motion sensors.

18. The method of claim 17, wherein the moving the at least one moveable lens element is performed such that the moving of the lens element at least partially compensates for the motion of the apparatus detected by the motion sensors in the stabilized video.

19. The method of claim 16, wherein the shift amounts for the sequence of images include a vertical shift amount in a vertical direction and a horizontal shift amount in the horizontal direction for each image.

20. The method of claim 17, wherein the shift amounts for the sequence of images include a vertical shift amount in a vertical direction and a horizontal shift amount in the horizontal direction for each image.

21. The method of claim 19, wherein the modifying the sequence of images is performed by shifting the images in the vertical direction by the vertical shift amount and in the horizontal direction by the horizontal shift amount.

22. The method of claim 20, wherein the modifying the sequence of images is performed by shifting the images in the vertical direction by the vertical shift amount and in the horizontal direction by the horizontal shift amount.

23. The method of claim 16, wherein the shift amounts for the sequence of images represent an amount of shift for a reference point in each image.

24. The method of claim 17, wherein the shift amounts for the sequence of images represent an amount of shift for a reference point in each image.

25. The method of claim 23, wherein the modifying the sequence of images is performed by shifting the reference point by the shift amount in each image.

26. The method of claim 24, wherein the modifying the sequence of images is performed by shifting the reference point by the shift amount in each image.

27. The method of claim 16, wherein the method further comprises receiving user input from the user interface, or displaying the stabilized video in the user interface, or both.

28. The method of claim 16, wherein the shift amounts for the sequence of images are determined based at least in part on a focal length of the lens.

29. The method of claim 17, wherein the shift amounts for the sequence of images are determined based at least in part on a focal length of the lens.

30. The method of claim 16, wherein the modified images are combined to obtain the stabilized video such that the effect of the motion of the imaging apparatus in the stabilized video is reduced.