DEPTH CAMERA BASED ON STRUCTURED LIGHT AND STEREO VISION

• US 20120056982A1
• Filed: 09/08/2010
• Published: 03/08/2012
• Est. Priority Date: 09/08/2010
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Katz, Sagi, Adler, Avishai
• Application Number:
  US12/877,595
• Publication Number:
  US 20120056982A1
• Time in Patent Office: Days
• US Class Current: 348/43
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

A depth camera system uses a structured light illuminator and multiple sensors such as infrared light detectors, such as in a system which tracks the Motion of a user in a field of view. One sensor can be optimized for shorter range detection while another sensor is optimized for longer range detection. The sensors can have a different baseline distance from the illuminator, as well as a different spatial resolution, exposure time and sensitivity. In one approach, depth values are obtained from each sensor by matching to the structured light pattern, and the depth values are merged to obtain a final depth map which is provided as an input to an application. The merging can involve unweighted averaging, weighted averaging, accuracy measures and/or confidence measures. In another approach, additional depth values which are included in the merging are obtained using stereoscopic matching among pixel data of the sensors.

Claims

1. A depth camera system, comprising:

an illuminator which illuminates an object in a field of view with a pattern of structured light;

a first sensor which senses reflected light from the object to obtain a first frame of pixel data, the first sensor is optimized for shorter range imaging;

a second sensor which senses reflected light from the object to obtain a second frame of pixel data, the second sensor is optimized for longer range imaging; and

at least one control circuit, the at least one control circuit derives a first structured light depth map of the object by comparing the first frame of pixel data to the pattern of the structured light, derives a second structured light depth map of the object by comparing the second frame of pixel data to the pattern of the structured light, and derives a merged depth map which is based on the first and second structured light depth maps.

2. The depth camera system of claim 1, wherein:

a baseline distance between the first sensor and the illuminator is less than a baseline distance between the second sensor and the illuminator.

3. The depth camera system of claim 2, wherein:

an exposure time of the first sensor is shorter than an exposure time of the second sensor.

4. The depth camera system of claim 2, wherein:

a sensitivity of the first sensor is less than a sensitivity of the second sensor.

5. The depth camera system of claim 2, wherein:

a spatial resolution of the first sensor is less than a resolution of the second sensor.

6. The depth camera system of claim 1, wherein:

the second structured light depth map includes depth values; and

in deriving the merged depth map, the depth values in the second structured light depth map are weighted more heavily than depth values in the first structured light depth map.

7. A depth camera system, comprising:

an illuminator which illuminates an object in a field of view with a pattern of structured light;

a first sensor which senses reflected light from the object to obtain a first frame of pixel data;

a second sensor which senses reflected light from the object to obtain a second frame of pixel data; and

at least one control circuit, the at least one control circuit derives a merged depth map which is based on first and second structured light depth maps of the object, and at least a first stereoscopic depth map of the object, where the at least one control circuit derives the first structured light depth map of the object by comparing the first frame of pixel data to the pattern of the structured light, derives the second structured light depth map of the object by comparing the second frame of pixel data to the pattern of the structured light, and derives the at least a first stereoscopic depth map by stereoscopic matching of the first frame of pixel data to the second frame of pixel data.

8. The depth camera system of claim 7, wherein:

the at least one control circuit derives the merged depth map based on a second stereoscopic depth map of the object, where the second stereoscopic depth map is derived by stereoscopic matching of the second frame of pixel data to the first frame of pixel data.

9. The depth camera system of claim 7, wherein:

the first and second structured light depth maps and the first stereoscopic depth map include depth values; and

the at least one control circuit assigns a first set of weights to the depth values in the first structured light depth map of the object, a second set of weights to the depth values in the second structured light depth map of the object, and a third set of weights to the depth values in the first stereoscopic depth map of the object, and derives the merged depth map based on the first, second and third sets of weights.

10. The depth camera system of claim 9, wherein:

the first set of weights is assigned based on a baseline distance between the first sensor and the illuminator;

the second set of weights is assigned based on a baseline distance between the second sensor and the illuminator; and

the third set of weights is assigned based on a baseline distance between the first and second sensors.

11. The depth camera system of claim 9, wherein:

the first and third sets of weights are assigned based on a spatial resolution of the first sensor, which is different than a spatial resolution of the second sensor; and

the second set of weights is assigned based on the spatial resolution of the second sensor.

12. The depth camera system of claim 9, wherein:

the first, second and third sets of weights are assigned based on at least one of confidence measures and accuracy measures associated with the first structured light depth map, the second structured light depth map and the first stereoscopic depth map, respectively.

13. A method for processing image data in a depth camera system, comprising:

illuminating an object in a field of view with a pattern of structured light;

at a first sensor, sensing reflected light from the object to obtain a first frame of pixel data;

at a second sensor, sensing reflected light from the object to obtain a second frame of pixel data;

deriving a first structured light depth map of the object by comparing the first frame of pixel data to the pattern of the structured light, the first structured light depth map includes depth values for pixels of the first frame of pixel data;

deriving a second structured light depth map of the object by comparing the second frame of pixel data to the pattern of the structured light, the second structured light depth map includes depth values for pixels of the second frame of pixel data;

determining whether refinement of the depth values of one or more pixels of the first frame of pixel data map is desired; and

if the refinement is desired, performing stereoscopic matching of the one or more pixels of the first frame of pixel data to one or more pixels of the second frame of pixel data.

14. The method of claim 13, wherein:

the refinement is desired when the one or more pixels of the first frame of Pixel data were not successfully matched to the pattern of structured light in the comparing of the first frame of pixel data to the pattern of the structured light.

15. The method of claim 13, wherein:

the refinement is desired when the one or more pixels of the first frame of pixel data were not successfully matched to the pattern of structured light with a sufficiently high level of at least one of confidence and accuracy, in the comparing of the first frame of pixel data to the pattern of the structured light.

16. The method of claim 13, wherein:

the refinement is desired when the depth values exceed a threshold distance.

17. The method of claim 13, wherein:

a baseline distance between the first and second sensors is greater than a baseline distance between the first sensor and the illuminator, and is greater than a baseline distance between the second sensor and the illuminator.

18. The method of claim 13, wherein:

the stereoscopic matching is performed for the one or more pixels of the first frame of pixel data for which the refinement is desired, but not for one or more other pixels of the first frame of pixel data for which the refinement is not desired.

19. The method of claim 13, wherein:

if the refinement is desired, providing a merged depth map based on the stereoscopic matching and the first and second structured light depth maps.

20. The method of claim 19, further comprising:

using the merged depth map as an input to an application in a motion capture system, where the object is a human which is tracked by the motion capture system, and where the application changes a display of the motion capture system in response to a gesture or movement by the human.