Video Image Stabilization

• US 20160337589A1
• Filed: 07/25/2016
• Published: 11/17/2016
• Est. Priority Date: 05/30/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Beysserie, Sebastien X., Thivent, Damien J., Zhou, Jianping, van der Merwe, Rudolph, Klivington, Jason, Li, Xiaoxing, Holtsberg, Anders, Lindberg, Martin, Williams, George E.
• Granted Patent: US 9,706,123 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

Techniques to permit a digital image capture device to stabilize a video stream in real-time (during video capture operations) are presented. In general, techniques are disclosed for stabilizing video images using an overscan region and a look-ahead technique enabled by buffering a number of video input frames before generating a first stabilized video output frame. (Capturing a larger image than is displayed creates a buffer of pixels around the edge of an image; overscan is the term given to this buffer of pixels.) More particularly, techniques are disclosed for buffering an initial number of input frames so that a “current” frame can use motion data from both “past” and “future” frames to adjust the strength of a stabilization metric value so as to keep the current frame within its overscan. This look-ahead and look-behind capability permits a smoother stabilizing regime with fewer abrupt adjustments.

Claims

1. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:

capture a first time-sequenced set of images, wherein each image in the first time-sequenced set of images has a capture time;

select a current image from the first time-sequenced set of images, wherein the current image has a capture time that is before the capture time of each image in a second time-sequenced set of images, wherein the second time-sequenced set of images is a subset of the first time-sequenced set of images;

identify a first stabilization metric value for the current image;

determine a second stabilization metric value based on an analysis of one or more captured images of the second time-sequenced set of images;

adjust the first stabilization metric value based on a difference between the first stabilization metric value and the second stabilization metric value;

apply the adjusted first stabilization metric value to the current image; and

store the stabilized current image in a memory.

2. The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more processors to capture a first time-sequenced set of images comprise instructions to cause the one or more processors to:

capture and buffer the current image; and

capture and buffer each image in the second time-sequenced set of images.

3. The non-transitory program storage device of claim 1, further comprising instructions to cause the one or more processors to:

identify a first image in the second time-sequenced set of images, wherein the first image'"'"'s capture time is immediately after the current image'"'"'s capture time and before every other image in the second time-sequenced set of images;

identify a last image in the second time-sequenced set of images, wherein the last image'"'"'s capture time is after every other image'"'"'s capture time in the second time-sequenced set of images;

remove the first image from the second time-sequenced set of images and identify it as a new current image;

capture a new image, the new image having a capture time, wherein the captured new image'"'"'s capture time is immediately after the last image'"'"'s capture time;

include the captured new image in the second time-sequenced set of images;

identify the adjusted first stabilization metric value as a third stabilization metric value for the new current image;

determine a fourth stabilization metric value based on an analysis of one or more captured images of the second time-sequenced set of images;

adjust the third stabilization metric value based on a difference between the third stabilization metric value and the fourth stabilization metric value;

apply the adjusted third stabilization metric value to the new current image; and

store the stabilized new current image in the memory.

4. The non-transitory program storage device of claim 1, further comprising instructions to cause the one or more processors to adjust a stabilization metric value for at least one image in the second time-sequenced set of images based on the adjusted first stabilization metric value and the second stabilization metric value.

5. The non-transitory program storage device of claim 4, wherein the instructions to cause the one or more processors to adjust a stabilization metric value for at least one image comprise instructions to cause the one or more processors to adjust the stabilization metric value of two or more images in the second time-sequenced set of images so as to create a smooth transition between the adjusted first stabilization metric value and the second stabilization metric value.

6. The non-transitory program storage device of claim 1, wherein each image in the first time-sequenced set of images has motion information, and the instructions to cause the one or more processors to adjust the first stabilization metric value are further based on the motion information of the one or more captured images of the second time-sequenced set of images and the current image'"'"'s motion information.

7. The non-transitory program storage device of claim 6, wherein the instructions to cause the one or more processors to adjust the first stabilization metric value are further based on the motion information from one or more images in the first time-sequenced set of images.

8. A video image stabilization method, comprising:

capturing a first time-sequenced set of images, each image in the first time-sequenced set of images having a capture time;

selecting a current image from the first time-sequenced set of images, the current image having a capture time that is before the capture time of each image in a second time-sequenced set of images, wherein the second time-sequenced set of images is a subset of the first time-sequenced set of images;

identifying a first stabilization metric value for the current image;

determining a second stabilization metric value based on an analysis of one or more later captured images of the second time-sequenced set of images;

adjusting the first stabilization metric value based on a difference between the first stabilization metric value and the second stabilization metric value;

applying the adjusted first stabilization metric value to the current image; and

storing the stabilized current image in a memory.

9. The method of claim 8, further comprising:

capturing and buffering the current image; and

capturing and buffering each image in the second time-sequenced set of images.

10. The method of claim 8, further comprising:

identifying a first image in the second time-sequenced set of images, wherein the first image'"'"'s capture time is immediately after the current image'"'"'s capture time and before every other image in the second time-sequenced set of images;

identifying a last image in the second time-sequenced set of images, wherein the last image'"'"'s capture time is after every other image'"'"'s capture time in the second time-sequenced set of images;

removing the first image from the second time-sequenced set of images and identify it as a new current image;

capturing a new image, the new image having a capture, wherein the captured new image'"'"'s capture time is immediately after the last image'"'"'s capture time;

including the captured new image in the second time-sequenced set of images;

identifying the adjusted first stabilization metric value as a third stabilization metric value for the new current image;

determining a fourth stabilization metric value based on an analysis of one or more captured images of the second time-sequenced set of images;

adjusting the third stabilization metric value based on a difference between the third stabilization metric value and the fourth stabilization metric value;

applying the adjusted third stabilization metric value to the new current image; and

storing the stabilized new current image in the memory.

11. The method of claim 8, further comprising adjusting a stabilization metric value for at least one image in the second time-sequenced set of images based on the adjusted first stabilization metric value and the second stabilization metric value.

12. The method of claim 11, wherein adjusting a stabilization metric value for at least one image comprises adjusting the stabilization metric value of two or more images in the second time-sequenced set of images to create a smooth transition between the adjusted first stabilization metric value and the second stabilization metric value.

13. The method of claim 8, wherein each image in the first time-sequenced set of images has motion information, and adjusting the first stabilization metric value is further based on the motion information of the one or more captured images of the second time-sequenced set of images and the current image'"'"'s motion information.

14. The method of claim 13, further comprising adjusting the first stabilization metric value based on the motion information from one or more images in the first time-sequenced set of images.

15. A video image capture device, comprising:

image sensor;

memory communicatively coupled to the image sensor;

one or more motion sensors;

one or more processors operatively coupled to the image sensor, the memory and the one or more motion sensors configured to;

capture a first time-sequenced set of images with the image sensor and, contemporaneously for each image in the first time-sequenced set of images;

select a current image from the first time-sequenced set of images, wherein the current image has a capture time that is before the capture time of each image in a second time-sequenced set of images, wherein the second time-sequenced set of images is a subset of the first time-sequenced set of images;

identify a first stabilization metric value for the current image;

determine a second stabilization metric value based on an analysis of one or more captured images of the second time-sequenced set of images;

adjust the first stabilization metric value based on a difference between the first stabilization metric value and the second stabilization metric value;

apply the adjusted first stabilization metric value to the current image; and

store the stabilized current image in the memory.

16. The video image capture device of claim 15, wherein the one or more processors are further configured to:

capture and buffer the current image in the memory; and

capture and buffer each image in the second time-sequenced set of images in the memory.

17. The video image capture device of claim 15, wherein the one or more processors are further configured to:

identify a first image in the second time-sequenced set of images, wherein the first image'"'"'s capture time is immediately after the current image'"'"'s capture time and before every other image in the second time-sequenced set of images;

identify a last image in the second time-sequenced set of images, wherein the last image'"'"'s capture time is after every other image'"'"'s capture time in the second time-sequenced set of images;

remove the first image from the second time-sequenced set of images and identify it as a new current image;

capture a new image with the motion sensor, the new image having a capture time wherein the captured new image'"'"'s capture time is immediately after the last image'"'"'s capture time;

include the captured new image in the second time-sequenced set of images;

identify the adjusted first stabilization metric value as a third stabilization metric value for the new current image;

determine a fourth stabilization metric value based on an analysis of one or more captured images of the second time-sequenced set of images;

adjust the third stabilization metric value based on a difference between the third stabilization metric value and the fourth stabilization metric value;

apply the adjusted third stabilization metric value to the new current image and store the stabilized new current image in the memory.

18. The video image capture device of claim 15, wherein the one or more processors are further configured to adjust a stabilization metric value for at least one image in the second time-sequenced set of images based on the adjusted first stabilization metric value and the second stabilization metric value.

19. The video image capture device of claim 18, wherein the configuration to cause the one or more processors to adjust a stabilization metric value, comprises a configuration to cause the one or more processors to adjust the stabilization metric value of two or more images in the second time-sequenced set of images so as to create a smooth transition between the adjusted first stabilization metric value and the second stabilization metric value.

20. The video image capture device of claim 15, wherein each image in the first time-sequenced set of images has motion information, and adjusting the first stabilization metric value is further based on the motion information of the one or more captured images of the second time-sequenced set of images and the current image'"'"'s motion information.