Time-of-flight depth imaging

• US 8,983,233 B2
• Filed: 08/30/2013
• Issued: 03/17/2015
• Est. Priority Date: 10/04/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Katz, Sagi, Adler, Avishai
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US14/015,887
• Publication Number:
  US 20140002611A1
• Time in Patent Office: 564 Days
• Field of Search: 382/106, 382/154, 382/260, 382/264, 382/285, 382/286, 348/46
• US Class Current: 382/286
• CPC Class Codes: G01S 17/18 wherein range gates are used
• CPC Class Codes: G01S 17/32 using transmission of continuous waves, whether amplitude-, frequency-, or phase-modulated, or unmodulated
• CPC Class Codes: G01S 17/50 Systems of measurement based on relative movement of target
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

Techniques are provided for determining depth to objects. A depth image may be determined based on two light intensity images. This technique may compensate for differences in reflectivity of objects in the field of view. However, there may be some misalignment between pixels in the two light intensity images. An iterative process may be used to relax a requirement for an exact match between the light intensity images. The iterative process may involve modifying one of the light intensity images based on a smoothed version of a depth image that is generated from the two light intensity images. Then, new values may be determined for the depth image based on the modified image and the other light intensity image. Thus, pixel misalignment between the two light intensity images may be compensated.

Claims

1. A method, comprising:

calculating, by a depth camera system, a depth image based on a first light intensity image and a second light intensity image, the first and second light intensity images contain pixel values for the same scene;

smoothing the depth image by the depth camera system;

modifying the first light intensity image by the depth camera system;

calculating new values for the depth image based on the modified first light intensity image and the second light intensity image by the depth camera system;

repeating, by the depth camera system, the smoothing the depth image, the modifying the first light intensity image, and the calculating new values for the depth image until it is determined that processing the depth image is complete.

2. The method of claim 1, wherein the modifying the first light intensity image includes modifying the first light intensity image based on either the smoothed depth image or the second light intensity image.

3. The method of claim 1, wherein the modifying the first light intensity image includes modifying the first light intensity image based on the smoothed depth image.

4. The method of claim 1, wherein the modifying the first light intensity image includes modifying the first light intensity image based on the second light intensity image.

5. The method of claim 1, wherein the modifying the first light intensity image includes keeping pixel values of the first light intensity image within boundary values for pixels in the first light intensity image.

6. The method of claim 5, further comprising determining, by the depth camera system, the boundary values for pixels in the first light intensity image based on one or more neighbor pixels in the first light intensity image.

7. The method of claim 1, wherein the calculating a depth image includes using a mathematical function, the modifying the first light intensity image includes using an inverse of the mathematical function.

8. The method of claim 1, wherein the modifying the first light intensity image includes performing an inverse of the calculation to determine the depth image.

9. An apparatus, comprising:

a processor configured to access a first light intensity image and a second light intensity image, the first and second light intensity images contain pixel values for the same scene, calculate a depth image based on the first light intensity image and the second light intensity image, smooth the depth image, modify the first light intensity image, calculate new values for the depth image based on the modified first light intensity image and the second light intensity image, repeat the smoothing the depth image, the modifying the first light intensity image, and the calculating new values for the depth image until it is determined that processing the depth image is complete.

10. The apparatus of claim 9, wherein the processor configured to modify the first light intensity image includes the processor configured to modify the first light intensity image based on either the smoothed depth image or the second light intensity image.

11. The apparatus of claim 9, wherein the processor configured to modify the first light intensity image includes the processor configured to modify the first light intensity image based on the smoothed depth image.

12. The apparatus of claim 9, wherein the processor configured to calculate a depth image includes the processor applying a mathematical function, the processor configured to modify the first light intensity image includes the processor applying an inverse of the mathematical function.

13. The apparatus of claim 9, wherein the processor is configured to determine boundary values based on possible pixel misalignment between the first and second light intensity images, wherein the processor configured to modify the first light intensity image includes the processor configured to keep pixel values in the first light intensity image within the boundary values.

14. The apparatus of claim 13, further comprising:

a first sensor; and

a second sensor, wherein the processor being configured to determine boundary values include the processor being configured to determine an upper boundary value and a lower boundary value for a first pixel in the first light intensity image based on the value of the first pixel in the first light intensity image and the value of one or more neighbors to the first pixel, wherein the one or more neighbors are determined based on geometry of the first sensor and the second sensor.

15. The apparatus of claim 9, further comprising:

a first sensor, the processor is configured to determine the first light intensity image by gating the first sensor; and

a second sensor, the processor is configured to determine the second light intensity image without gating the second sensor.

16. A computer readable storage medium having instructions stored thereon which, when executed by a processor, cause the processor to:

calculate a depth image based on a first light intensity image and a second light intensity image, the first and second light intensity images contain pixel values for the same scene;

smooth the depth image;

modify the first light intensity image after smoothing the depth image;

calculate new values for the depth image based on the modified first light intensity image and the second light intensity image;

repeat the smoothing the depth image, the modifying the first light intensity image, and the calculating new values for the depth image until it is determined that processing the depth image is complete.

17. The computer readable storage medium of claim 16, wherein the instructions which cause the processor to modify the first light intensity image include instructions which cause the processor to modify the first light intensity image based on either the smoothed depth image or the second light intensity image.

18. The computer readable storage medium of claim 16, wherein the instructions which cause the processor to modify the first light intensity image include instructions which cause the processor to modify the first light intensity image based on the smoothed depth image.

19. The computer readable storage medium of claim 16, further comprising instructions which when executed on the processor cause the processor to determine boundary values for pixels in the first light intensity image based on a possible pixel misalignment between the first light intensity image and the second light intensity image, the instructions which cause the processor to modify the first light intensity image include instructions which when executed on the processor cause the processor to keep pixel values in the first light intensity image within the boundary values.

20. The computer readable storage medium of claim 16, wherein the instructions which cause the processor to calculate a depth image include instructions which when executed on the processor cause the processor to apply a mathematical function to the first light intensity image and the second light intensity image, the instructions which cause the processor to modify the first light intensity image include instructions which when executed on the processor cause the processor to apply an inverse of the mathematical function to one or more of the smoothed depth image or the second light intensity image.