Depth from time of flight camera

• US 10,311,282 B2
• Filed: 09/11/2017
• Issued: 06/04/2019
• Est. Priority Date: 10/14/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Shotton, Jamie Daniel Joseph, Keskin, Cem, Rhemann, Christoph, Sharp, Toby, Robertson, Duncan Paul, Kohli, Pushmeet, Fitzgibbon, Andrew William, Izadi, Shahram
• Primary Examiner(s): Thomas, Mia M
• Application Number:
  US15/701,170
• Publication Number:
  US 20170372126A1
• Time in Patent Office: 631 Days
• CPC Class Codes: G01S 17/10 using transmission of interrupted, pulse-modulated waves determination of distance by phase measurements G01S17/32
• CPC Class Codes: G01S 17/36 with phase comparison between the received signal and the contemporaneously transmitted signal
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G01S 7/4911 Transmitters
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

Region of interest detection in raw time of flight images is described. For example, a computing device receives at least one raw image captured for a single frame by a time of flight camera. The raw image depicts one or more objects in an environment of the time of flight camera (such as human hands, bodies or any other objects). The raw image is input to a trained region detector and in response one or more regions of interest in the raw image are received. A received region of interest comprises image elements of the raw image which are predicted to depict at least part of one of the objects. A depth computation logic computes depth from the one or more regions of interest of the raw image.

Claims

1. A method comprising:

receiving at least one image captured for a single frame by a time of flight camera, the image depicting one or more objects in an environment of the time of flight camera;

inputting the at least one image into a trained region detector;

determining, by the trained region detector, one or more regions of interest in the at least one image based on associations between pixels of time of flight images and regions that depict an object of interest in the time of flight images,

wherein a region of interest comprises image elements which are predicted to depict at least part of one of the objects; and

computing depth from the one or more regions of interest of the at least one image.

2. The method as claimed in claim 1, wherein the single frame comprises a plurality of sensor data images.

3. The method as claimed in claim 1 comprising computing depth only from the one or more regions of interest of the at least one image.

4. The method as claimed in claim 1 wherein the at least one image from the time of flight camera is a plurality of infra-red intensity values.

5. The method as claimed in claim 1 comprising:

receiving a plurality of images captured for a single frame by the time of flight camera;

aggregating the plurality of images to produce an aggregated image; and

inputting the aggregated image to the trained region detector.

6. The method as claimed in claim 1 comprising:

receiving a plurality of images captured for a single frame by the time of flight camera; and

inputting each of the images to the trained region detector.

7. The method as claimed in claim 1 comprising using the trained region detector to classify image elements of the at least one image as being in a region of interest or not.

8. The method as claimed in claim 7 comprising clustering the classified image elements.

9. The method as claimed in claim 7 comprising applying a sliding window detector to the classified image elements.

10. The method as claimed in claim 1 comprising training the region detector using training data comprising time of flight images depicting examples of the object, the time of flight images having labeled regions of interest.

11. The method as claimed in claim 1 wherein the trained region detector is a random decision forest.

12. A computing system comprising:

an input/output controller arranged to receive at least one image captured for a single frame by a time of flight camera, the at least one image depicting one or more objects in an environment of the time of flight camera;

a region detector arranged to compute from the at least one image one or more regions of interest in the at least one image based on associations between pixels of time of flight images and regions that depict an object of interest in the time of flight images, a region of interest comprising image elements which are predicted to depict at least part of one of the objects; and

a depth map computation logic arranged to compute depth from the one or more regions of interest of the at least one image.

13. The computing system of claim 12, wherein the single frame comprises a plurality of sensor data images.

14. The computing system of claim 13 the region detector comprising a clustering component arranged to cluster labeled image elements output by the classifier.

15. The computing system of claim 13 the region detector comprising a sliding window component arranged to compute one or more regions from labeled image elements output by the classifier.

16. The computing system of claim 13 integrated in a time of flight camera.

17. A computer storage media storing instructions which when executed by a computing device control the device to:

receive at least one image captured for a single frame by a time of flight camera, the at least one image depicting one or more objects in an environment of the time of flight camera;

send the at least one image to a region detector, having been trained to learn associations between image elements of time of flight images and regions of interest;

determine, by the trained region detector, one or more regions of interest in the at least one image based on associations between pixels of time of flight images and regions that depict an object of interest in the time of flight images, a region of interest comprising image elements which are predicted to depict at least part of one of the objects; and

computer depth from the one or more regions of interest of the at least one image.

18. The computer readable medium of claim 17 storing instructions which when executed by a computing device control the device to:

receive a plurality of images captured for a single frame by the time of flight camera;

aggregate the plurality of images to produce an aggregated image; and

input the aggregated image to the region detector.

19. The computer readable medium of claim 17 storing instructions which when executed by a computing device control the device to receive a plurality of images captured for a single frame by the time of flight camera;

and

input each of the plurality of images to the region detector.

20. The computer readable medium of claim 17 storing instructions which when executed by a computing device control the device to compute depth only from the one or more regions of interest.