Low Light Video Image Stabilization Strength Modulation

• US 20160006935A1
• Filed: 08/25/2014
• Published: 01/07/2016
• Est. Priority Date: 07/06/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Zhou, Jianping, Beysserie, Sebastien X.
• Granted Patent: US 9,674,438 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

Techniques to improve a digital image capture device'"'"'s ability to stabilize a video stream in low light capture conditions are presented. In general, techniques are disclosed for stabilizing video images using an overscan region and a look-ahead technique enabled by buffering a number of video input frames before generating a first stabilized video output frame. More particularly, techniques are disclosed for buffering an initial number of input frames so that a “current” frame can use time stamped image capture device motion data from both “past” and “future” frames to adjust the strength of a stabilization metric value so as to keep the current frame within its overscan. The calculated adjustments to the strength of the stabilization metric value for a particular frame may be limited to a target max strength value that is determined based, at least in part, on an estimated amount of motion blur in the current frame.

Claims

1. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:

capture a first time-sequenced set of images, wherein each image in the first time-sequenced set of images is associated with a capture time and motion information;

select a current image from the first time-sequenced set of images;

identify a first stabilization metric value for the current image;

adjust the first stabilization metric value based, at least in part, on an estimated amount of motion blur in the current image, wherein the estimated amount of motion blur in the current image is based, at least in part, on at least part of the motion information associated with the current image;

apply the adjusted first stabilization metric value to the current image to stabilize the current image; and

store the stabilized current image in a memory.

2. The non-transitory program storage device of claim 1, wherein the motion information for each image comprises gyroscope information.

3. The non-transitory program storage device of claim 2, wherein the gyroscope information comprises one or more time stamped gyroscope data points.

4. The non-transitory program storage device of claim 1, wherein the instructions to estimate the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprise instructions to calculate an image row blur metric for one or more rows of pixels in the current image.

5. The non-transitory program storage device of claim 4, wherein the instructions to estimate the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprise instructions to calculate an overall image blur metric based, at least in part, on the one or more calculated image row blur metrics for the current image.

6. The non-transitory program storage device of claim 3, wherein the instructions to estimate the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprise instructions to calculate the beginning and ending integration time of one or more rows of pixels in the current image based, at least in part, on one or more of:

one or more time stamped gyroscope data points for the current image, a rolling shutter readout time, and an exposure time.

7. The non-transitory program storage device of claim 1, wherein the instructions to adjust the first stabilization metric value based, at least in part, on an estimated amount of motion blur in the current image further comprise instructions to limit the first stabilization metric value to a calculated target maximum value.

8. A video image stabilization method, comprising:

capturing a first time-sequenced set of images, wherein each image in the first time-sequenced set of images is associated with a capture time and motion information;

selecting a current image from the first time-sequenced set of images;

identifying a first stabilization metric value for the current image;

adjusting the first stabilization metric value based, at least in part, on an estimated amount of motion blur in the current image, wherein the estimated amount of motion blur in the current image is based, at least in part, on at least part of the motion information associated with the current image;

applying the adjusted first stabilization metric value to the current image to stabilize the current image; and

storing the stabilized current image in a memory.

9. The method of claim 8, wherein the motion information for each image comprises one or more time stamped gyroscope data points.

10. The method of claim 8, wherein the act of estimating the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprises calculating an image row blur metric for one or more rows of pixels in the current image.

11. The method of claim 10, wherein the act of estimating the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprises calculating an overall image blur metric based, at least in part, on the one or more calculated image row blur metrics for the current image.

12. The method of claim 9, wherein the act of estimating the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprises calculating the beginning and ending integration time of one or more rows of pixels in the current image based, at least in part, on one or more of:

one or more time stamped gyroscope data points for the current image, a rolling shutter readout time, and an exposure time.

13. The method of claim 8, wherein the act of adjusting the first stabilization metric value based, at least in part, on an estimated amount of motion blur in the current image further comprises limiting the first stabilization metric value to a calculated target maximum value.

14. A video image capture device, comprising:

an image sensor;

memory communicatively coupled to the image sensor;

one or more motion sensors;

one or more processors operatively coupled to the image sensor, the memory and the one or more motion sensors configured to;

capture a first time-sequenced set of images, wherein each image in the first time-sequenced set of images is associated with a capture time and motion information;

select a current image from the first time-sequenced set of images;

identify a first stabilization metric value for the current image;

adjust the first stabilization metric value based, at least in part, on an estimated amount of motion blur in the current image, wherein the estimated amount of motion blur in the current image is based, at least in part, on at least part of the motion information associated with the current image;

apply the adjusted first stabilization metric value to the current image to stabilize the current image; and

store the stabilized current image in a memory.

15. The video image capture device of claim 14, wherein the motion information for each image comprises gyroscope information.

16. The video image capture device of claim 15, wherein the gyroscope information comprises one or more time stamped gyroscope data points.

17. The video image capture device of claim 14, wherein the instructions to estimate the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprise instructions to calculate an image row blur metric for one or more rows of pixels in the current image.

18. The video image capture device of claim 17, wherein the instructions to estimate the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprise instructions to calculate an overall image blur metric based, at least in part, on the one or more calculated image row blur metrics for the current image.

19. The video image capture device of claim 16, wherein the instructions to estimate the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprise instructions to calculate the beginning and ending integration time of one or more rows of pixels in the current image based, at least in part, on one or more of:

one or more time stamped gyroscope data points for the current image, a rolling shutter readout time, and an exposure time.

20. The video image capture device of claim 14, wherein the instructions to adjust the first stabilization metric value based, at least in part, on an estimated amount of motion blur in the current image further comprise instructions to limit the first stabilization metric value to a calculated target maximum value.