Separating foreground and background objects in captured images

• US 9,223,404 B1
• Filed: 01/27/2012
• Issued: 12/29/2015
• Est. Priority Date: 01/27/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Ivanchenko, Volodymyr V.
• Primary Examiner(s): Simpson, Lixi C
• Application Number:
  US13/359,777
• Time in Patent Office: 1,432 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Locating features in image information captured by multiple cameras of a computing device can be facilitated by removing background objects from the image information. A distance past which objects should be considered to be background objects is determined, and an amount of disparity for that distance calculated. Any objects having less than that amount of disparity in the image information can be removed, leaving objects that should be treated as foreground objects. An advantage of such an approach is that objects can be determined to be foreground objects even when those objects are only viewable by one of the cameras and disparity information cannot be calculated for those objects from the image information. Such an approach can be combined with other approaches as well to improve accuracy.

Claims

1. A computer-implemented method of enabling gesture input to a computing device, comprising:

setting a threshold number of pixels for a processor of the computing device to differentiate background objects from foreground objects;

capturing image information using a pair of cameras of the computing device configured for stereoscopic image capture, a first camera of the pair of cameras capturing a first set of the image information and a second camera of the pair of cameras capturing a second set of the image information;

analyzing the image information, using the processor of the computing device and the threshold number of pixels set prior to capturing image information, to determine that a first object is a background object where a difference in position between a first location of a reference point on the first object in the first set of image information and a second location of a corresponding reference point on the first object in the second set of image information represents less than the threshold number of pixels;

determining that a second object in the image information is one or more foreground objects where a difference in position between a first location of a reference point on the second object in the first set of image information and a second location of a corresponding reference point on the second object in the second set of image information is at least the threshold number of pixels; and

providing the image information that includes the second object to at least one gesture recognition process, the second object including one or more foreground objects.

2. The computer-implemented method of claim 1, further comprising:

adjusting at least one color value of a portion of the image corresponding to the background object to a predetermined background value.

3. The computer-implemented method of claim 1, further comprising:

analyzing the image information that includes the second object to locate a feature of a user of the device capable of providing gesture input.

4. The computer-implemented method of claim 3, further comprising:

locating the feature of the user in image information that includes the second object captured over a period of time in order to track movement of the feature while performing a gesture.

5. The computer-implemented method of claim 1, further comprising:

multiplying a focal length of one of the first camera or the second camera by a baseline of one of the first camera or the second camera to generate a first result; and

dividing the first result by the distance past which objects are considered background objects to generate the threshold number of pixels.

6. A computer-implemented method of identifying foreground objects in captured image information, comprising:

setting a threshold number of pixels for a processor of a computing device to differentiate background objects from foreground objects;

capturing image information using a pair of cameras of the computing device, the pair of cameras including a first camera for capturing a first set of image information and a second camera for capturing a second set of image information;

analyzing the image information, using the threshold number of pixels set prior to capturing the image information, to determine that a first object is a first type of object where a difference in position between a location of a reference point on the first object in the first set of image information and a second location of a corresponding reference point on the first object in the second set of image information represents less than the threshold number of pixels;

determining that a second object in the image information is a second type of object where a difference in position between a first location of a reference point on the second object in the first set of image information and a second location of a corresponding reference point on the second object in the second set of image information is at least the threshold number of pixels; and

providing the image information that includes the second object, the second object in the image information corresponding to one or more foreground objects.

7. The computer-implemented method of claim 6, wherein the portion of the image information includes at least one feature of a user operable to provide motion or gesture input to the computing device.

8. The computer-implemented method of claim 6, further comprising:

processing the captured image information to remove at least one of noise or artifacts from the captured image information.

9. The computer-implemented method of claim 6, wherein the image information that includes the second object has a disparity greater than the threshold amount of disparity or has a disparity that is undeterminable from the captured stereoscopic image information.

10. The computer-implemented method of claim 6, further comprising:

determining a distance from the computing device past which objects are to be classified as background objects.

11. The computer-implemented method of claim 10, further comprising:

calculating the specified amount of disparity using at least the determined distance and configuration information for the pair of cameras.

12. The computer-implemented method of claim 6, wherein the computing device includes at least three cameras, and wherein the pair of cameras is selected from the at least three cameras based on at least one environmental condition.

13. The computer-implemented method of claim 12, wherein the at least one environmental condition includes at least one of an orientation of the computing device, a relative orientation of a user, an amount of light, or an amount of motion of the device.

14. The computer-implemented method of claim 6, further comprising:

detecting an amount of light; and

selectively activating an illumination element during stereoscopic image capture based at least in part upon the detected amount of light.

15. The computer-implemented method of claim 6, further comprising:

determining at least one environmental condition, a value of the environmental condition determining whether to activate infrared imaging on the computing device.

16. The computer-implemented method of claim 15, wherein the infrared imaging is activated instead of, or in addition to, the capturing of stereoscopic image information, the infrared imaging capable of selectively providing infrared illumination to separate one or more foreground objects captured by the infrared imaging.

17. The computer-implemented method of claim 16, further comprising:

determining an amount of motion between infrared image capture instances using at least one sensor of the computing device, the amount of motion capable of being used to offset positions of one or more objects in captured infrared images for the infrared image capture instances.

18. A computing device, comprising:

a device processor;

a pair of cameras; and

a memory device including instructions operable to be executed by the processor to perform a set of actions, enabling the computing device to;

set a threshold number of pixels for the device processor of the computing device to differentiate background objects from foreground objects;

capture stereoscopic image information using the pair of cameras, the image information comprising a first set of image information captured by a first camera of the pair of cameras and a second set of image information captured by a second camera of the pair of cameras;

analyzing the stereoscopic image information, using the threshold number of pixels set prior to the capture of the stereoscopic image information, to determine that a first object is a background object in the stereoscopic image information where a difference in position between a first location of a reference point on the first object in the first set of image information and a second location of a corresponding reference point on the first object in the second set of image information represents less than the threshold number of pixels;

determine that a second object in the image information is a foreground object where a difference in position between a first location of a reference point on the second object in the first set of image information and a second location of a corresponding reference point on the second object in the second set of image information is at least the threshold number of pixels; and

adjust color values for the background object region to a specified background color value,

wherein the captured image information after adjusting including the second object is provided for analysis.

19. The computing device of claim 18, wherein the instructions when executed further cause the computing device to:

determine a distance from the computing device past which objects are to be classified as background objects;

calculate the threshold amount of disparity using at least the determined distance and configuration information for the pair of cameras.

20. The computing device of claim 18, wherein the instructions when executed further cause the computing device to:

detect an amount of light; and

selectively activate an illumination element during image capture based at least in part upon the detected amount of light.

21. The computing device of claim 18, wherein the instructions when executed further cause the computing device to:

activate infrared imaging on the computing device, the infrared imaging being activated instead of, or in addition to, the capturing of image information, the infrared imaging capable of selectively providing infrared illumination to separate one or more foreground objects captured by the infrared imaging.

22. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor of a computing device, cause the computing device to:

set a threshold number of pixels for the at least one processor of the computing device to differentiate background objects from foreground objects;

capture image information using a pair of cameras of the computing device, the image information comprising a first set of image information captured by a first camera of the pair of cameras and a second set of image information captured by a second camera of the pair of cameras;

analyze the image information, using the at least one processor of the computing device and the threshold number of pixels set prior to the capture of the image information, to locate background objects in the image information that correspond to less than the threshold number of pixels in difference between a first location of a reference point on a first object in the first set of image information and a second location of a reference point on the first object in the second set of the image information;

adjust portions of the image information corresponding to the background objects to correspond to a background setting to provide adjusted image information; and

provide the adjusted image information after the adjustment to a feature recognition process, the adjusted image information including a remaining region determined to correspond to foreground objects to be processed using the feature recognition process, the remaining region having one or more of the foreground objects;

the one or more foreground objects determined to have at least the threshold number of pixels between the first set of image information and the second set of image information.

23. The non-transitory computer-readable storage medium of claim 22, wherein the adjusted image information has greater than the threshold number of pixels or an indeterminable corresponding disparity value from the captured image information.

24. The non-transitory computer-readable storage medium of claim 22, wherein the instructions when executed further cause the computing device to:

determine a distance from the computing device past which objects are to be classified as background objects;

calculate the threshold amount of disparity using at least the determined distance and configuration information for the pair of cameras.