DEPTH SENSING USING AN RGB CAMERA

• US 20150248765A1
• Filed: 02/28/2014
• Published: 09/03/2015
• Est. Priority Date: 02/28/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Criminisi, Antonio, Robertson, Duncan Paul, Kontschieder, Peter, Kohli, Pushmeet, Dumitras, Adriana, Shotton, Jamie Daniel Joseph, Turbell, Henrik, Munasinghe, Indeera
• Granted Patent: US 9,626,766 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20072 Graph-based image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/50 Depth or shape recovery

Abstract

A method of sensing depth using an RGB camera. In an example method, a color image of a scene is received from an RGB camera. The color image is applied to a trained machine learning component which uses features of the image elements to assign all or some of the image elements a depth value which represents the distance between the surface depicted by the image element and the RGB camera. In various examples, the machine learning component comprises one or more entangled geodesic random decision forests.

Claims

1. A computer-implemented method of image processing comprising:

receiving, at a processor, at least one color image from an RGB camera, the image being of a scene comprising a plurality of surfaces;

applying the at least one color image to a trained machine learning component; and

receiving as output from the trained machine learning component, a depth map comprising, for individual image elements of the color image, distances from the RGB camera to the surfaces in the scene depicted by the individual image elements.

2. The method of claim 1, wherein the machine learning component was trained using pairs of color images and corresponding ground truth depth maps.

3. The method of claim 1, wherein the machine learning component has been trained to calculate a depth map from two successive RGB images.

4. The method of claim 1, wherein the machine learning component has been trained using pairs of color images and corresponding depth maps, the depth maps being empirically observed using a depth camera.

5. The method of claim 1, wherein the machine learning component has been trained using pairs of synthetically generated color images and corresponding synthetically generated depth maps.

6. The method of claim 1, further comprising, receiving, at the processor, a stream of color images from the RGB camera at a frame rate of the RGB camera;

and using the machine learning component to calculate a stream of depth maps from the stream of color images, at least at the frame rate of the RGB camera.

7. The method of claim 1, further comprising applying the at least one color image to the machine learning component by sending the image to the machine learning component at a remote entity.

8. The method of claim 1, further comprising applying the at least one color image to the machine learning component at a hand held computing device comprising the RGB camera.

9. The method of claim 1, wherein the depth map is computed without the need for a depth camera.

10. The method of claim 1, wherein the trained machine learning component comprises at least one random decision forest.

11. The method of claim 10, further comprising computing features at split nodes of the random decision forest, the features comprising comparisons between reference image elements and probe image elements.

12. The method of claim 1, further comprising reducing the size of the machine learning component by using at least one random decision forest with one or more of:

merged nodes, auto-context, entanglement, and a classification layer used to direct input between a plurality of expert forests.

13. The method of claim 1, further comprising storing the machine learning component in a hand held computing device.

14. The method of claim 1, wherein the trained machine learning component is arranged to classify the surfaces depicted in the color image into one of a plurality of depth bins, a depth bin being a range of distances from the RGB camera.

15. The method of claim 1, further comprising using the depth map in at least one of:

video compression, segmentation for background removal, gesture recognition, natural user interface, surface reconstruction and object scanning.

16. The method of claim 1, wherein the machine learning component is stored in memory in one of a smartphone, a tablet computer and a laptop computer.

17. The method of claim 1, wherein the machine learning component uses raw image data from a particular color image and temporal derivative information derived from the particular color image and a subsequent color image to generate the depth map for the particular color image.

18. A system to process an image, the system comprising a computing-based device configured to:

receive at least one color image from an RGB camera, the image being of a scene comprising a plurality of surfaces;

apply the at least one color image to a trained machine learning component; and

receive as output from the trained machine learning component, a depth map comprising, for individual image elements of the color image, distances from the RGB camera to the surfaces in the scene depicted by the individual image elements.

19. The system according to claim 18, the computing-based device being at least partially implemented using hardware logic selected from any one of more of:

a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device.

20. A computer-implemented method of image processing comprising:

receiving, at a processor, at least one color image from an RGB camera, the image being of a scene comprising a plurality of surfaces;

applying the at least one color image to a trained machine learning component, the trained machine learning component comprising an entangled geodesic random decision forest; and

receiving as output from the trained machine learning component, a depth map comprising, for individual image elements of the color image, distances from the RGB camera to the surfaces in the scene depicted by the individual image elements.