Depth From Time of Flight Camera

• US 20170372126A1
• Filed: 09/11/2017
• Published: 12/28/2017
• Est. Priority Date: 10/14/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: SHOTTON, Jamie Daniel Joseph, KESKIN, Cem, RHEMANN, Christoph, SHARP, Toby, ROBERTSON, Duncan Paul, KOHLI, Pushmeet, FITZGIBBON, Andrew William, IZADI, Shahram
• Granted Patent: US 10,311,282 B2
• Time in Patent Office: Days
• CPC Class Codes: G01S 17/10 using transmission of interrupted, pulse-modulated waves determination of distance by phase measurements G01S17/32
• CPC Class Codes: G01S 17/36 with phase comparison between the received signal and the contemporaneously transmitted signal
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G01S 7/4911 Transmitters
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

Region of interest detection in raw time of flight images is described. For example, a computing device receives at least one raw image captured for a single frame by a time of flight camera. The raw image depicts one or more objects in an environment of the time of flight camera (such as human hands, bodies or any other objects). The raw image is input to a trained region detector and in response one or more regions of interest in the raw image are received. A received region of interest comprises image elements of the raw image which are predicted to depict at least part of one of the objects. A depth computation logic computes depth from the one or more regions of interest of the raw image.

Claims

1. A method comprising:

receiving at least one raw image captured for a single frame by a time of flight camera, the raw image depicting one or more objects in an environment of the time of flight camera;

inputting the raw image to a trained region detector and receiving in response one or more regions of interest in the raw image, a region of interest comprising image elements which are predicted to depict at least part of one of the objects; and

computing depth from the one or more regions of interest of the raw image.

2. The method as claimed in claim 1 the region detector having been trained to learn associations between image elements of raw time of flight images and regions of interest.

3. The method as claimed in claim 1 comprising computing depth only from the one or more regions of interest of the raw image.

4. The method as claimed in claim 1 wherein the raw image from the time of flight camera is a plurality of infra-red intensity values.

5. The method as claimed in claim 1 comprising receiving a plurality of raw images captured for a single frame by the time of flight camera;

aggregating the plurality of raw images to produce an aggregated raw image; and

inputting the aggregated raw image to the trained region detector.

6. The method as claimed in claim 1 comprising receiving a plurality of raw images captured for a single frame by the time of flight camera;

and inputting each of the raw images to the trained region detector.

7. The method as claimed in claim 1 comprising using the trained region detector to classify image elements of the raw image as being in a region of interest or not.

8. The method as claimed in claim 7 comprising clustering the classified image elements.

9. The method as claimed in claim 7 comprising applying a sliding window detector to the classified image elements.

10. The method as claimed in claim 1 comprising training the region detector using training data comprising raw time of flight images depicting examples of the object, the raw time of flight images having labeled regions of interest.

11. The method as claimed in claim 1 wherein the trained region detector is a random decision forest.

12. A computing system comprising:

an input/output controller arranged to receive at least one raw image captured for a single frame by a time of flight camera, the raw image depicting one or more objects in an environment of the time of flight camera;

a region detector arranged to compute from the raw image one or more regions of interest in the raw image, a region of interest comprising image elements which are predicted to depict at least part of one of the objects; and

a depth map computation logic arranged to compute depth from the one or more regions of interest of the raw image.

13. The computing system of claim 12 the region detector comprising a classifier having been trained to learn associations between image elements of raw time of flight images and regions of interest.

14. The computing system of claim 13 the region detector comprising a clustering component arranged to cluster labeled image elements output by the classifier.

15. The computing system of claim 13 the region detector comprising a sliding window component arranged to compute one or more regions from labeled image elements output by the classifier.

16. The computing system of claim 13 integrated in a time of flight camera.

17. A computer readable medium storing instructions which when executed by a computing device control the device to:

receive at least one raw image captured for a single frame by a time of flight camera, the raw image depicting one or more objects in an environment of the time of flight camera;

send the raw image to a region detector, having been trained to learn associations between image elements of raw time of flight images and regions of interest, and receive in response one or more regions of interest in the raw image, a region of interest comprising image elements which are predicted to depict at least part of one of the objects; and

compute depth from the one or more regions of interest of the raw image.

18. The computer readable medium of claim 17 storing instructions which when executed by a computing device control the device to:

receive a plurality of raw images captured for a single frame by the time of flight camera;

aggregate the plurality of raw images to produce an aggregated raw image; and

input the aggregated raw image to the trained region detector.

19. The computer readable medium of claim 17 storing instructions which when executed by a computing device control the device to receive a plurality of raw images captured for a single frame by the time of flight camera;

and input each of the raw images to the trained region detector.

20. The computer readable medium of claim 17 storing instructions which when executed by a computing device control the device to compute depth only from the one or more regions of interest.