Depth-based image stabilization

• US 9,538,081 B1
• Filed: 03/14/2013
• Issued: 01/03/2017
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Zhou, Dong
• Primary Examiner(s): Anyikire, Chikaodili E
• Assistant Examiner(s): Kalapodas, Dramos I
• Application Number:
  US13/829,614
• Time in Patent Office: 1,391 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Depth information can be used to assist with image processing functionality, such as image stabilization and blur reduction. In at least some embodiments, depth information obtained from stereo imaging or distance sensing, for example, can be used to determine a foreground object and background object(s) for an image or frame of video. The foreground object then can be located in later frames of video or subsequent images. Small offsets of the foreground object can be determined, and the offset accounted for by adjusting the subsequent frames or images. Such an approach provides image stabilization for at least a foreground object, while providing simplified processing and reduce power consumption. Similarly processes can be used to reduce blur for an identified foreground object in a series of images, where the blur of the identified object is analyzed.

Claims

1. A computer-implemented method of performing stabilization for image capture, comprising:

receiving a first image from a first camera of a computing device;

receiving a second image from a second camera of the computing device;

determining a first representation of a first object of at least two objects represented in the first image;

determining a second representation of the first object of at least two objects represented in the second image;

determining a third representation of a second object of the at least two objects represented in the first image;

determining a fourth representation of the second object of the at least two objects represented in the second image;

determining first distance information for the first object based upon disparity differences between the first representation and the second representation;

determining second distance information for the second object based upon disparity differences between the third representation and the fourth representation;

determining a foreground object from the first object and the second object based upon the first distance information and the second distance information, the foreground object being less than a threshold distance from the computing device;

determining a background object from the first object and the second object based upon the first distance information and the second distance information, the background object being more than the threshold distance from the computing device;

displaying at least the first image received from the first camera on a display element of the computing device;

receiving a subsequent image from the first camera;

locating the foreground object represented in the subsequent image;

determining a fifth representation of the foreground object presented in the first image;

determining a sixth representation of the foreground object represented in the subsequent image;

determining an offset between the fifth representation and the sixth representation based upon the foreground object represented in the first image, the foreground object represented in the subsequent image, and distance information of the foreground object;

determining, that the offset is less than a threshold offset value;

determining a location of the foreground object represented in the subsequent image, as displayed on a display element of the computing device, to remove the offset of the foreground object; and

displaying the subsequent image on the display element of the computing device, the foreground object represented in the subsequent image being shifted by an equal amount of the offset in an opposite direction.

2. The computer-implemented method of claim 1, wherein the first image and the subsequent image are adjacent frames of a video file.

3. The computer-implemented method of claim 1, further comprising:

determining at least one of (a) the first representation of the first object in the first image being within a central portion of the first image, or (b) the disparity difference for the first object indicating that the first object is positioned within a specified distance range from the computing device, before designating the first object as the foreground object.

4. The computer-implemented method of claim 1, further comprising:

determining that a difference between the disparity difference for the first object and the disparity difference for the second object exceeds a minimum threshold before designating the first object as the foreground object and the second object as the background object.

5. The computer-implemented method of claim 1, wherein displaying the subsequent image on the display element such that the location of the foreground object, as displayed on the display element, corresponds substantially to a first position of the foreground object in the first image includes displaying only a portion of the subsequent image as captured, the portion being selected so that the location of the foreground object as displayed in the subsequent image corresponds substantially to the first position of the foreground object in the first image as previously displayed.

6. A computing device, comprising:

at least one processor;

a display element;

a camera; and

non-transitory computer-readable storage medium including instructions that, when executed by the at least one processor, cause the computing device to;

receive a first image from the camera of the computing device;

determine a first representations of a first object of at least two objects in the first image;

determine a second representation of a second object of the at least two objects represented in the first image;

determine first distance information for the first object;

determine second distance information for the second object;

determine a foreground object from the first object and the second object based upon the first distance information and the second distance information, the foreground object being less than a threshold distance from the computing device;

determine a background object from the first object and the second object based upon the first distance information and the second distance information, the background object being more than the threshold distance from the computing device;

receive a subsequent image from the camera;

determine a third representation of the foreground object presented in the first image;

determine a fourth representation of the foreground object represented in the subsequent image;

determine an offset between the third representation and the fourth representation based upon the foreground object represented in the first image, the foreground object represented in the subsequent image, and distance information of the foreground object;

determine, by analyzing the foreground object represented in the subsequent image and the foreground object represented in the first image, that the offset is less than a threshold offset value;

determine a location of the representation of the foreground object in the subsequent image, as displayed on a display element of the computing device; and

display the subsequent image on the display element, the representation in the subsequent image being shifted based at least in part upon the offset in an opposite direction.

7. The computing device of claim 6, wherein the camera is one of a pair of cameras configured for stereoscopic imaging, and wherein obtaining the distance information for the at least two objects includes determining an amount of disparity for each representation of at least a portion of the at least two objects as represented in at least one stereoscopic image received from the pair of cameras.

8. The computing device of claim 6, wherein the representation of the foreground object in the subsequent image is located at least in part by locating an object with a similar amount of disparity to the foreground object in the subsequent image.

9. The computing device of claim 6, further comprising:

at least one distance sensor configured to obtain the distance information for the at least two objects.

10. The computing device of claim 6, wherein displaying the subsequent image on the display element such that the location of the foreground object, as displayed on the display element, corresponds substantially to a previous location of the representation of the foreground object in the first image as previously displayed includes displaying only a portion of the subsequent image as received, the portion being selected so that the location of the foreground object as displayed in the subsequent image corresponds substantially to a first position of the first representation of the foreground object in the first image as previously displayed.

11. The computing device of claim 6, further comprising:

an infrared (IR) emitter configured to emit IR radiation; and

an IR sensor configured to capture first IR radiation reflected from the first object and second IR radiation reflected from the second object, wherein the distance information is obtained by determining a difference in intensity between the first IR radiation and the second IR radiation as captured by the IR sensor.

12. The computing device of claim 11, wherein the IR sensor is configured to capture a first image of the at least two objects when the IR emitter is emitting IR radiation capable of being reflected by the at least two objects, and a second image of the at least two objects when the IR emitter is not emitting IR radiation, the difference in intensity being determined by comparing intensity data for the first image with intensity data for the second image.

13. The computing device of claim 11, wherein the instructions when executed further cause the computing device to:

display a central portion of the subsequent image if a change between the location of the representation of the foreground object in the subsequent image and a previous location of the representation of the foreground object in the first image exceeds a stabilization threshold.

14. The computing device of claim 6, wherein displaying the subsequent image includes displaying a corresponding portion of the subsequent image.

15. The computing device of claim 6, wherein the instructions when executed further cause the computing device to:

cause the camera to capture one or more additional images as part of a series of images automatically captured in response to an instruction to capture the image;

use at least one feature of the foreground object to determine a relative position of the foreground object in each image of the one or more additional images; and

determine an amount of blur of the foreground object in at least one of the series of images.

16. The computing device of claim 15, wherein the instructions when executed further cause the computing device to:

determine the amount of blur of the foreground object represented in each image of the series of images; and

select an image, from the series of images, that is determined to have a lowest amount of blur of the foreground object.

17. The computing device of claim 16, wherein the instructions when executed further cause the computing device to:

merge two or more images, from the series of images, that have at most a maximum blur amount.

18. The computing device of claim 6, wherein at least one feature of the foreground object includes at least one of a center point, a feature point, a contour, or a shape.

19. A computer-implemented method, comprising:

receiving an image captured using a camera of a computing device;

determining a first representation of a first object of at least two objects in the image;

determining a second representation of a second object of the at least two objects represented in the image;

determining first distance information for the first object;

determining second distance information for the second object;

determining a foreground object from the first object and the second object based upon the first distance information and the second distance information, the foreground object being less than a threshold distance from the computing device;

determining a background object from the first object and the second object based upon the first distance information and the second distance information, the background object being more than the threshold distance from the computing device;

receiving a subsequent image captured using the camera;

determining an offset between a third representation and a fourth representation based upon the foreground object represented in the image, the foreground object represented in the subsequent image, and distance information of the foreground object;

determining, by analyzing the foreground object represented in the subsequent image and the foreground object represented in the image, that the offset is less than a threshold offset value;

determining a location of the representation of the foreground object in the subsequent image, as displayed on a display element of the computing device; and

displaying the subsequent image on the display element, the representation in the subsequent image being shifted based at least in part upon the offset in an opposite direction.

20. The computer-implemented method of claim 19, wherein the distance information is obtained by at least one of:

determining at least a first amount of disparity for the foreground object and a second amount of disparity for the background object using a pair of images concurrently captured by a pair of cameras, the first distance information determined based at least in part upon a difference between the first amount of disparity and the second amount of disparity;

obtaining the first distance information using at least one distance sensor configured to obtain the distance information for the at least two objects;

or

capturing first IR radiation reflected from the foreground object and second IR radiation reflected from the background object, wherein the first distance information is obtained by determining a difference in intensity between the first IR radiation and the second IR radiation as captured by an IR sensor.

21. The computer-implemented method of claim 19, further comprising:

causing the camera to capture one or more additional images as part of a series of images automatically captured in response to an instruction to capture the image;

using at least one feature of the foreground object to determine a relative position of the foreground object in each image of the one or more additional images; and

determining an amount of blur of the foreground object in at least one of the series of images.

22. The computer-implemented method of claim 21, further comprising:

determining the amount of blur of the foreground object represented in each image of the series of images and rejecting an image from the series of images that is determined to have more than a determined amount of blur of the foreground object;

or

merge image data for images of the series having at most the determined amount of blur to produce an improved image.