Temporal time-of-flight

• US 10,229,502 B2
• Filed: 02/03/2016
• Issued: 03/12/2019
• Est. Priority Date: 02/03/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Adam, Amit, Nowozin, Sebastian, Yair, Omer, Mazor, Shai, Schober, Michael
• Primary Examiner(s): Luo, Kate H
• Application Number:
  US15/015,065
• Publication Number:
  US 20170221212A1
• Time in Patent Office: 1,133 Days
• Field of Search: None
• CPC Class Codes: G01S 17/50 Systems of measurement based on relative movement of target
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

A depth detection apparatus is described which has a memory and a computation logic. The memory stores frames of raw time-of-flight sensor data received from a time-of-flight sensor, the frames having been captured by a time-of-flight camera in the presence of motion such that different ones of the frames were captured using different locations of the camera and/or with different locations of an object in a scene depicted in the frames. The computation logic has functionality to compute a plurality of depth maps from the stream of frames, whereby each frame of raw time-of-flight sensor data contributes to more than one depth map.

Claims

1. A depth detection apparatus comprising:

a memory storing frames of raw time-of-flight sensor data received from a time-of-flight camera, the frames having been captured by the time-of-flight camera in the presence of motion such that different ones of the frames were captured using different locations of the camera or with different locations of an object in a scene depicted in the frames or with both different locations of the camera and different locations of an object in a scene depicted in the frames; and

a hardware processor configured to perform computation logic to compute a plurality of depth maps from the frames of raw time-of-flight sensor data, whereby each frame of raw time-of-flight sensor data contributes to the computation of more than one depth map by using a block of the frames of raw time-of-flight sensor data to compute the more than one depth map, the hardware processor further configured to provide the plurality of depth maps to a system that uses the plurality of depth maps to determine depth information, wherein the computation logic has functionality to carry out inference with respect to a model of temporal time-of-flight, the model of temporal time-of-flight comprising a weighted combination of a static model and a temporal model, the temporal model describing how imaging conditions at individual pixels evolve over time and the static model omitting data about evolution over time.

2. The apparatus of claim 1 wherein the model of temporal time-of-flight describes how raw time-of-flight data is generated by the camera under imaging conditions comprising albedo, illumination and surface depth from the camera.

3. The apparatus of claim 2 where the model is a probabilistic model.

4. The apparatus of claim 3 wherein the model of temporal time-of-flight comprises a prior term expressing knowledge about a trajectory of the time-of-flight camera and about scene geometry.

5. The apparatus of claim 4 wherein the prior term has been empirically determined.

6. The apparatus of claim 2 wherein the functionality to carry out inference comprises a look-up table or regressor trained to be a practical working equivalent of the model of temporal time-of-flight.

7. The apparatus of claim 6 where the look up-table or regressor has been trained using training data generated by the model.

8. The apparatus of claim 1 where the memory stores the frames such that different ones of the frames have been captured using different measurement patterns of the time-of-flight camera, and where a sequence of measurement patterns used by the time-of-flight camera has been specified taking into account the model.

9. The apparatus of claim 1 wherein the computation logic is configured to operate in real time whereby a frame rate of the frames captured by the time-of-flight camera is matched or bettered by a rate at which the depth maps are output by the computation logic.

10. The apparatus of claim 1 wherein the computation logic is configured to output the depth maps at a rate of 30 per second or more.

11. An apparatus comprising:

a memory storing frames of raw time-of-flight sensor data received from a time-of-flight camera, the frames having been captured by the time-of-flight camera in the presence of motion such that different ones of the frames were captured using different locations of the camera and/or with different locations of an object in a scene depicted in the frames; and

a hardware processor configured to perform computation logic to compute a plurality of depth maps from the frames of raw time-of-flight sensor data by carrying out inference with respect to a model of temporal time-of-flight that uses each frame of raw time-of-flight sensor data to compute more than one depth map by using a block of the frames of raw time-of-flight sensor data, wherein the model of temporal time-of-flight comprises a weighted combination of a static model and a temporal model, the temporal model describing how imaging conditions at individual pixels evolve over time and the static model omitting data about evolution over time, where the model of temporal time-of-flight describes how raw time-of-flight data is generated by the camera under imaging conditions comprising albedo, illumination and surface depth from the camera, the processor further configured to provide the plurality of depth maps to a system that uses the plurality of depth maps to determine depth information.

12. The apparatus of claim 11 integral with a time-of-flight camera.

13. The apparatus of claim 11 integral with a mobile computing device.

14. A computer-implemented method comprising:

storing, at a memory, frames of raw time-of-flight sensor data received from a time-of-flight camera, the frames having been captured by the time-of-flight camera in the presence of motion such that different ones of the frames were captured using different locations of the camera or with different locations of an object in a scene depicted in the frames or with both different locations of the camera and different locations of an object in a scene depicted in the frames;

computing a plurality of depth maps from the frames of raw time-of-flight sensor data, whereby each frame of raw time-of-flight sensor data contributes to the computation of more than one depth map by using a block of the frames of raw time-of-flight sensor data to compute the more than one depth map, wherein computing the depth maps comprises carrying out inference with respect to a model of temporal time-of-flight and making a weighted combination of a static model and a temporal model, the temporal model describing how imaging conditions at individual pixels evolve over time and the static model omitting data about evolution over time; and

providing the plurality of depth maps to a system that uses the plurality of depth maps to determine depth information.

15. The method of claim 14 wherein the model describes how raw time-of-flight data is generated by the camera under imaging conditions comprising albedo, illumination and surface depth from the camera.

16. The method of claim 15 where the model is a probabilistic model.

17. The method of claim 14 wherein computing the depth maps comprises using a look-up table or regressor trained to be a practical working equivalent of a model of temporal time-of-flight.

18. The method of claim 17 comprising filling the look up-table or training the regressor using data generated by the model.

19. The method of claim 14 wherein the model of temporal time-of-flight comprises an empirical prior term expressing knowledge about a trajectory of the time-of-flight camera and about scene geometry.

20. The method of claim 14 comprising computing a sequence of measurement patterns to be used by the time-of-flight camera taking into account the model.