Video image stabilization with enforced stabilization constraints

• US 9,787,902 B1
• Filed: 06/10/2016
• Issued: 10/10/2017
• Est. Priority Date: 06/10/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Beysserie, Sebastien X., Zhou, Jianping, Ben Soussan, Stephane S.
• Primary Examiner(s): Diep, Trung
• Application Number:
  US15/179,680
• Time in Patent Office: 487 Days
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

Techniques to improve a digital image capture device'"'"'s ability to stabilize a video stream—while enforcing desired stabilization constraints on particular images in the video stream—are presented that utilize an overscan region and a look-ahead technique enabled by buffering a number of video input frames before generating a first stabilized video output frame. More particularly, techniques are disclosed for buffering an initial number of input frames so that a “current” frame can use motion data from both “past” and “future” frames to adjust the value of a stabilization strength parameter and/or the weighted contribution of particular frames from the buffer in the determination of stabilization motion values for the current frame. Such techniques keep the current frame within its overscan and ensure that the stabilization constraints are enforced, while maintaining desired smoothness in the video stream. In some embodiments, the stabilization constraint may comprise a maximum allowed frame displacement.

Claims

1. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:

obtain a first time-sequenced set of images, wherein;

each image in the first time-sequenced set of images has a capture time and motion data;

one image in the first time-sequenced set of images is designated as a current image;

a first one or more images within the first time-sequenced set of images having capture times before the current image are designated past images; and

a second one or more images within the first time-sequenced set of images having capture times after the current image are designated future images;

select a first image from the first time-sequenced set of images to meet a first stabilization constraint;

identify a weighting value for each image in the first time-sequenced set of images based, at least in part, on the proximity of the respective image'"'"'s capture time to the capture time of the current image;

adjust the identified weighting value for the selected first image according to a first weighting adjustment value;

identify a first stabilization motion value for the current image in the first time-sequenced set of images based, at least in part, on the identified weighting values and the adjusted weighting value for the selected first image;

apply the first stabilization motion value to the current image to stabilize the current image; and

store the stabilized current image in a memory.

2. The non-transitory program storage device of claim 1, wherein the motion data for each image comprises gyroscope information.

3. The non-transitory program storage device of claim 1, wherein the identified weighting value for each image in the first time-sequenced set of images is further based, at least in part, on a stabilization strength parameter value for the current image.

4. The non-transitory program storage device of claim 1, wherein the first weighting adjustment value is further determined based, at least in part, on the proximity of the selected first image'"'"'s capture time to the capture time of the current image.

5. The non-transitory program storage device of claim 1, wherein the first weighting adjustment value is further determined based, at least in part, on whether the selected first image is presently a past image or a future image.

6. The non-transitory program storage device of claim 1, wherein the first stabilization constraint comprises a maximum allowed displacement for the stabilization of the selected first image.

7. The non-transitory program storage device of claim 1, wherein the instructions to apply the first stabilization motion value to the current image to stabilize the current image further comprise instructions to aggregate one or more pixels with the current image, so as to maintain a consistent Field of View (FOV) between the current image and the other images in the first time-sequenced set of images.

8. A video image stabilization method, comprising:

obtaining a first time-sequenced set of images, wherein;

each image in the first time-sequenced set of images has a capture time and motion data;

one image in the first time-sequenced set of images is designated as a current image;

a first one or more images within the first time-sequenced set of images having capture times before the current image are designated past images; and

a second one or more images within the first time-sequenced set of images having capture times after the current image are designated future images;

selecting a first image from the first time-sequenced set of images to meet a first stabilization constraint;

identifying a weighting value for each image in the first time-sequenced set of images based, at least in part, on the proximity of the respective image'"'"'s capture time to the capture time of the current image;

adjusting the identified weighting value for the selected first image according to a first weighting adjustment value;

identifying a first stabilization motion value for the current image in the first time-sequenced set of images based, at least in part, on the identified weighting values and the adjusted weighting value for the selected first image;

applying the first stabilization motion value to the current image to stabilize the current image; and

storing the stabilized current image in a memory.

9. The method of claim 8, wherein the motion data for each image comprises one or more time stamped gyroscope data points.

10. The method of claim 8, wherein the identified weighting value for each image in the first time-sequenced set of images is further based, at least in part, on a stabilization strength parameter value for the current image.

11. The method of claim 8, wherein the first weighting adjustment value is further determined based, at least in part, on the proximity of the selected first image'"'"'s capture time to the capture time of the current image.

12. The method of claim 8, wherein the first stabilization constraint comprises a maximum allowed displacement for the stabilization of the selected first image.

13. The method of claim 8, wherein the act of applying the first stabilization motion value to the current image to stabilize the current image further comprises aggregating one or more pixels with the current image, so as to maintain a consistent FOV between the current image and the other images in the first time-sequenced set of images.

14. A video image capture device, comprising:

an image sensor;

memory communicatively coupled to the image sensor;

one or more motion sensors;

one or more processors operatively coupled to the image sensor, the memory and the one or more motion sensors configured to execute instructions causing the one or more processors to;

obtain a first time-sequenced set of images, wherein;

each image in the first time-sequenced set of images has a capture time and motion data;

one image in the first time-sequenced set of images is designated as a current image;

a first one or more images within the first time-sequenced set of images having capture times before the current image are designated past images; and

a second one or more images within the first time-sequenced set of images having capture times after the current image are designated future images;

select a first image from the first time-sequenced set of images to meet a first stabilization constraint;

identify a weighting value for each image in the first time-sequenced set of images based, at least in part, on the proximity of the respective image'"'"'s capture time to the capture time of the current image;

adjust the identified weighting value for the selected first image according to a first weighting adjustment value;

identify a first stabilization motion value for the current image in the first time-sequenced set of images based, at least in part, on the identified weighting values and the adjusted weighting value for the selected first image;

apply the first stabilization motion value to the current image to stabilize the current image; and

store the stabilized current image in a memory.

15. The video image capture device of claim 14, wherein the motion data for each image comprises gyroscope information.

16. The video image capture device of claim 14, wherein the identified weighting value for each image in the first time-sequenced set of images is further based, at least in part, on a stabilization strength parameter value for the current image.

17. The video image capture device of claim 14, wherein the first weighting adjustment value is further determined based, at least in part, on the proximity of the selected first image'"'"'s capture time to the capture time of the current image.

18. The video image capture device of claim 14, wherein the first weighting adjustment value is further determined based, at least in part, on whether the selected first image is presently a past image or a future image.

19. The video image capture device of claim 14, wherein the first stabilization constraint comprises a maximum allowed displacement for the stabilization of the selected first image.

20. The video image capture device of claim 14, wherein the instructions to cause the one or more processors to apply the first stabilization motion value to the current image to stabilize the current image further comprise instructions to aggregate one or more pixels with the current image, so as to maintain a consistent Field of View (FOV) between the current image and the other images in the first time-sequenced set of images.