Combined Optical And Electronic Image Stabilization

• US 20160360111A1
• Filed: 09/24/2015
• Published: 12/08/2016
• Est. Priority Date: 08/25/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Thivent, Damien J., Williams, George E., Zhou, Jianping, Baer, Richard L., Toft, Rolf, Beysserie, Sebastien X.
• Granted Patent: US 9,596,411 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/685 performed by mechanical compensation
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position

Abstract

Techniques to improve a digital image capture device'"'"'s ability to stabilize a video stream are presented. According to some embodiments, improved stabilization of captured video frames is provided by intelligently harnessing the complementary effects of both optical image stabilization (OIS) and electronic image stabilization (EIS). In particular, OIS may be used to remove intra-frame motion blur that is typically lower in amplitude and dominates with longer integration times, while EIS may be used to remove residual unwanted frame-to-frame motion that is typically larger in amplitude. The techniques disclosed herein may also leverage information provided from the image capture device'"'"'s OIS system to perform improved motion blur-aware video stabilization strength modulation, which permits better video stabilization performance in low light conditions, where integration times tend to be longer, thus leading to a greater amount of motion blurring in the output stabilized video.

Claims

1. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:

capture a first time-sequenced set of images with an image capture device comprising a lens assembly mechanically coupled to a camera module, wherein each image in the first time-sequenced set of images is associated with one or more capture times, one or more camera motion information samples, and one or more optical image stabilization (OIS) lens motion information samples;

perform a high-pass filtering operation on the one or more camera motion information samples of the first time-sequenced set of images;

apply a first amount of OIS to one or more parts of the lens assembly of the image capture device based, at least in part, on only the portions of the one or more camera motion information samples that passed the high-pass filtering operation;

select a current image from the first time-sequenced set of images;

identify a first electronic image stabilization (EIS) transform matrix for a first portion of the current image based, at least in part, on the one or more camera motion information samples associated with the first portion of the current image;

adjust the first EIS transform matrix based, at least in part, on only the portions of the one or more OIS lens motion information samples associated with the first portion of the current image that passed the high-pass filter operation;

apply the adjusted first EIS transform matrix to the first portion of the current image to stabilize the first portion of the current image; and

store the stabilized first portion of the current image in a memory.

2. The non-transitory program storage device of claim 1, wherein the camera motion information samples for each image in the first time-sequenced set of images comprise time stamped gyroscope data points, and wherein the OIS lens motion information samples for each image in the first time-sequenced set of images comprise time stamped Hall sensor data points.

3. The non-transitory program storage device of claim 1, wherein the instructions to adjust the first EIS transform matrix further comprise instructions to cause one or more processors to:

estimate an amount of motion blur in the first portion of the current image, wherein the estimated amount of motion blur in the first portion of the current image is based, at least in part, on one or more of the one or more camera and OIS lens motion information samples associated with the first portion of the current image.

4. The non-transitory program storage device of claim 3, wherein the instructions to estimate the amount of motion blur in the first portion of the current image based, at least in part, on one or more of the one or more camera and OIS lens motion information samples associated with the current image further comprise instructions to:

calculate an image row blur metric for one or more rows of pixels in the current image.

5. The non-transitory program storage device of claim 1, wherein the instructions to adjust the first EIS transform matrix further comprise instructions to:

subtract out the portions of the one or more OIS lens motion information samples associated with the first portion of the current image that passed the high-pass filtering operation from the EIS transform matrix.

6. The non-transitory program storage device of claim 1, wherein the high-pass filtering operation comprises a first order infinite impulse response (IIR) high-pass filter with a movable cut-off frequency.

7. The non-transitory program storage device of claim 6, wherein the movable cut-off frequency is based, at least in part, on an amplitude of at least one of the one or camera motion information samples associated with at least one of the images in the first time-sequenced set of images.

8. A video image stabilization method, comprising:

capturing a first time-sequenced set of images with an image capture device comprising a lens assembly mechanically coupled to a camera module, wherein each image in the first time-sequenced set of images is associated with one or more capture times, one or more camera motion information samples, and one or more optical image stabilization (OIS) lens motion information samples;

performing a high-pass filtering operation on the one or more camera motion information samples of the first time-sequenced set of images;

applying a first amount of OIS to one or more parts of the lens assembly of the image capture device based, at least in part, on only the portions of the one or more camera motion information samples that passed the high-pass filtering operation;

selecting a current image from the first time-sequenced set of images;

identifying a first electronic image stabilization (EIS) transform matrix for a first portion of the current image based, at least in part, on the one or more camera motion information samples associated with the first portion of the current image;

adjusting the first EIS transform matrix based, at least in part, on only the portions of the one or more OIS lens motion information samples associated with the first portion of the current image that passed the high-pass filter operation;

applying the first EIS transform matrix to the first portion of the current image to stabilize the first portion of the current image; and

storing the stabilized first portion of the current image in a memory.

9. The method of claim 8, wherein the camera motion information samples for each image in the first time-sequenced set of images comprise time stamped gyroscope data points, and wherein the OIS lens motion information samples for each image in the first time-sequenced set of images comprise time stamped Hall sensor data points.

10. The method of claim 8, wherein the act of adjusting the first EIS transform matrix comprises estimating an amount of motion blur in the first portion of the current image, wherein the estimated amount of motion blur in the first portion of the current image is based, at least in part, on one or more of the one or more camera and OIS lens motion information samples associated with the first portion of the current image.

11. The method of claim 10, wherein the act of estimating the amount of motion blur in the first portion of the current image based, at least in part, on one or more of the one or more camera and OIS lens motion information samples associated with the current image further comprises:

calculating an image row blur metric for one or more rows of pixels in the current image.

12. The method of claim 8, wherein the act of adjusting the first EIS transform matrix further comprises:

subtracting out the portions of the one or more OIS lens motion information samples associated with the first portion of the current image that passed the high-pass filtering operation from the EIS transform matrix.

13. The method of claim 8, wherein the high-pass filtering operation comprises a first order infinite impulse response (IIR) high-pass filter with a movable cut-off frequency, wherein the movable cut-off frequency is based, at least in part, on an amplitude of at least one of the one or more camera motion information samples associated with at least one of the images in the first time-sequenced set of images.

14. A video image capture device, comprising:

a camera module comprising an image sensor;

memory communicatively coupled to the image sensor;

one or more motion sensors;

a lens assembly mechanically coupled to the camera module, wherein the lens assembly comprises one or more parts subject to optical image stabilization (OIS);

one or more processors operatively coupled to the image sensor, the memory, and the one or more motion sensors configured to perform instructions to cause the one or more processors to;

capture a first time-sequenced set of images at the image sensor, wherein each image in the first time-sequenced set of images is associated with one or more capture times, one or more camera motion information samples measured by the one or more motion sensors, and one or more OIS lens motion information samples measured by the one or more motion sensors;

perform a high-pass filtering operation on the one or more camera motion information samples of the first time-sequenced set of images;

apply a first amount of OIS to one or more parts of the lens assembly based, at least in part, on only the portions of the one or more camera motion information samples that passed the high-pass filtering operation;

select a current image from the first time-sequenced set of images;

identify a first electronic image stabilization (EIS) transform matrix for a first portion of the current image based, at least in part, on the one or more camera motion information samples associated with the first portion of the current image;

adjust the first EIS transform matrix based, at least in part, on only the portions of the one or more OIS lens motion information samples associated with the first portion of the current image that passed the high-pass filter operation;

apply the adjusted first EIS transform matrix to the first portion of the current image to stabilize the first portion of the current image; and

store the stabilized first portion of the current image in a memory.

15. The video image capture device of claim 14, wherein the camera motion information samples for each image in the first time-sequenced set of images comprise time stamped gyroscope data points, and wherein the OIS lens motion information samples for each image in the first time-sequenced set of images comprise time stamped Hall sensor data points.

16. The video image capture device of claim 14, wherein the instructions to adjust the first EIS transform matrix further comprise instructions to cause the one or more processors to:

estimate an amount of motion blur in the first portion of the current image, wherein the estimated amount of motion blur in the first portion of the current image is based, at least in part, on one or more of the one or more camera and OIS lens motion information samples associated with the first portion of the current image.

17. The video image capture device of claim 16, wherein the instructions to estimate the amount of motion blur in the first portion of the current image based, at least in part, on one or more of the one or more camera and OIS lens motion information samples associated with the current image further comprise instructions to cause the one or more processors to:

calculate an image row blur metric for one or more rows of pixels in the current image.

18. The video image capture device of claim 14, wherein the instructions to adjust the first EIS transform matrix further comprise instructions to cause the one or more processors to:

subtract out the portions of the one or more OIS lens motion information samples associated with the first portion of the current image that passed the high-pass filtering operation from the EIS transform matrix.

19. The video image capture device of claim 14, wherein the high-pass filtering operation comprises a first order infinite impulse response (IIR) high-pass filter with a movable cut-off frequency.

20. The video image capture device of claim 19, wherein the movable cut-off frequency is based, at least in part, on an amplitude of at least one of the one or more camera motion information samples associated with at least one of the images in the first time-sequenced set of images.