Fast general multipath correction in time-of-flight imaging

• US 9,542,749 B2
• Filed: 01/06/2014
• Issued: 01/10/2017
• Est. Priority Date: 01/06/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Freedman, Daniel, Krupka, Eyal, Smolin, Yoni, Leichter, Ido, Schmidt, Mirko
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US14/148,184
• Publication Number:
  US 20150193938A1
• Time in Patent Office: 1,100 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01S 17/36 with phase comparison between the received signal and the contemporaneously transmitted signal
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

Fast general multipath correction in time of flight imaging is described, for example, to obtain accurate depth maps at frame rate from a time of flight camera. In various embodiments accurate depth maps are calculated by looking up corrected depth values stored in a look up table. In various embodiments the corrected depth values are highly accurate as they take into account three or more possible light ray paths between the camera and a surface in a scene being imaged. In an example accurate depth maps are computed at a frame rate of a time of flight camera. In an example accurate depth maps are computed in less than 30 milliseconds for an image having over 200,000 pixels using a standard CPU.

Claims

1. A computer-implemented method comprising:

receiving, at a processor, time of flight depth camera amplitude and phase measurements depicting surfaces in a scene;

computing, via the processor, an index of a look up table from the measurements;

using the index to read, via the processor, a corrected depth value from the look up table, the corrected value being a depth of a surface from the camera having been calculated by taking into account at least three possible light ray paths between the camera and a surface in the scene and searching for values of a backscattering vector, wherein the backscattering vector values comprise weights indicating how much contribution is made to the amplitude and phase values observed at a pixel from light associated with a plurality of different distance bins, and the distance bins comprise a distance range from the time of flight camera divided into intervals.

2. A method as claimed in claim 1 comprising repeating the process of claim 1 for a plurality of pixels of an image captured by the time of flight depth camera to obtain a multipath corrected depth map.

3. A method as claimed in claim 2 wherein the depth value that is read from the lookup table is stored in a depth map at a location corresponding to the location of the pixel in the image.

4. A method as claimed in claim 1 wherein the look up table has a number of dimensions which is lower than a number of dimensions of the received time of flight depth camera amplitude and phase measurements.

5. A method as claimed in claim 1 comprising computing the index from the measurements at least by allocating the measurements to one or more of a plurality of bins, where a bin is a specified range of values.

6. A method as claimed in claim 1 comprising computing the index from the measurements by mapping the measurements to a lower dimensional form.

7. A method as claimed in claim 1, the corrected value having been calculated by searching for values of the backscattering vector which sum to a smallest absolute amount.

8. A method as claimed in claim 1, the corrected value having been calculated by taking into account noise in the amplitude and phase values.

9. A method as claimed in claim 1, wherein the search is carried out using a greedy algorithm which allocates a value of the backscattering vector without taking into account future entries of the vector which may influence the allocated value.

10. A method as claimed in claim 1, wherein the search is carried out using an optimization engine.

11. A method as claimed in claim 1, the corrected value having been calculated by selecting one of the values of the backscattering vector on the basis of a noise model.

12. A method as claimed in claim 1, the corrected value having been calculated by selecting one of the values of the backscattering vector on the basis of the position of the values in the backscattering vector.

13. A method as claimed in claim 1 at least partially carried out using hardware logic.

14. A depth map computation apparatus comprising:

a processor arranged to receive time of flight depth camera amplitude and phase measurements depicting a surface in a scene;

a depth map computation logic configured to compute via the processor an index of a look up table from the measurements;

the depth map computation logic configured to read via the processor a corrected depth value from the look up table using the index, the corrected value being a depth of the surface from the camera having been calculated by taking into account at least three possible light ray paths between the camera and the surface in the scene and searching for values of a backscattering vector, wherein the backscattering vector values comprise weights indicating how much contribution is made to the amplitude and phase values observed at a pixel from light associated with a plurality of different distance bins, and the distance bins comprise a distance range from the time of flight camera divided into intervals.

15. An apparatus as claimed in claim 14 comprising a memory storing the look up table.

16. An apparatus as claimed in claim 14 the depth map computation logic being at least partially implemented using hardware logic selected from any one or more of:

a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device, a graphics processing unit.

17. A method comprising:

receiving, at a processor, time of flight depth camera amplitude and phase measurements depicting surfaces in a scene, the measurements being received at a frame rate of the time of flight camera;

computing, via the processor, indices of a look up table from the measurements;

using the indices to read, via the processor, corrected depth values from the look up table, the corrected values being depths of surfaces from the camera having been calculated so as to reduce the effects of general multipath interference by taking into account at least three possible light ray paths between the camera and a surface in the scene and searching for values of a backscattering vector; and

using the corrected depth values to form, via the processor, a depth map in a time at or below the frame rate of the time of flight camera, wherein the backscattering vector values comprise weights indicating how much contribution is made to the amplitude and phase values observed at a pixel from light associated with a plurality of different distance bins, and the distance bins comprise a distance range from the time of flight camera divided into intervals.

18. A method as claimed in claim 17 wherein general multipath interference comprises interference from two or more light ray paths in addition to a light ray path having no interference.

19. A method as claimed in claim 17 wherein the depth map is formed at a frame rate of a time of flight camera.