Time-of-flight depth imaging

• US 8,548,270 B2
• Filed: 10/04/2010
• Issued: 10/01/2013
• Est. Priority Date: 10/04/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Katz, Sagi, Adler, Avishai
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US12/897,145
• Publication Number:
  US 20120082346A1
• Time in Patent Office: 1,093 Days
• Field of Search: 382/286, 382/285, 382/260, 382/264, 382/106, 382/154
• US Class Current: 382/286
• CPC Class Codes: G01S 17/18 wherein range gates are used
• CPC Class Codes: G01S 17/32 using transmission of continuous waves, whether amplitude-, frequency-, or phase-modulated, or unmodulated
• CPC Class Codes: G01S 17/50 Systems of measurement based on relative movement of target
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

Techniques are provided for determining depth to objects. A depth image may be determined based on two light intensity images. This technique may compensate for differences in reflectivity of objects in the field of view. However, there may be some misalignment between pixels in the two light intensity images. An iterative process may be used to relax a requirement for an exact match between the light intensity images. The iterative process may involve modifying one of the light intensity images based on a smoothed version of a depth image that is generated from the two light intensity images. Then, new values may be determined for the depth image based on the modified image and the other light intensity image. Thus, pixel misalignment between the two light intensity images may be compensated.

Claims

1. A method of determining depth, comprising:

calculating a depth image based on a first light intensity image and a second light intensity image, the first and second light intensity images contain pixel values for the same scene;

smoothing the depth image;

modifying the first light intensity image based on the smoothed depth image and the second light intensity image;

calculating new values for the depth image based on the modified first light intensity image and the second light intensity image;

repeating the smoothing the depth image, the modifying the first light intensity image, and the calculating new values for the depth image until it is determined that processing the depth image is complete, wherein the calculating a depth image, the smoothing, the modifying, and the calculating new values are performed by a processor.

2. The method of claim 1, wherein the modifying the first light intensity image based on the smoothed depth image and the second light intensity image includes performing an inverse of calculating the depth image based on the first light intensity image and the second light intensity image.

3. The method of claim 2, wherein the calculating a depth image based on the first light intensity image and the second light intensity image includes dividing, on a pixel-by-pixel basis, the first light intensity image by the second light intensity image,

wherein the modifying the first light intensity image based on the smoothed depth image and the second light intensity image includes multiplying, on a pixel-by-pixel basis, the smoothed depth image by the second light intensity image.

4. The method of claim 2, further comprising determining boundary values for pixels in the first light intensity image based on one or more neighbor pixels in the first light intensity image and a possible pixel misalignment between the first and second light intensity images, wherein the modifying the first light intensity image based on the smoothed depth image and the second light intensity image includes keeping pixel values in the first light intensity image within the boundary values.

5. The method of claim 4, wherein the modifying the first light intensity image includes modifying a value of a pixel in the second light intensity image if a value of a corresponding pixel in the first light intensity image would otherwise have gone outside of its boundary value as a result of performing the inverse of calculating the depth image.

6. The method of claim 1, further comprising determining the first light intensity image and the second light intensity image using the same sensor but at different times.

7. The method of claim 1, further comprising determining the first light intensity image using a first sensor and determining the second light intensity image using a second sensor at the same time.

8. The method of claim 1, further comprising:

determining the first light intensity image by gating a first sensor; and

determining the second light intensity image without gating a second sensor.

9. The method of claim 1, further comprising:

transmitting a modulated light beam;

collecting data at a first sensor when the modulated light beam is within a first range of phases to generate the first light intensity image; and

collecting data at a second sensor when the modulated light beam is within a second range of phases to generate the second light intensity image, the first and second ranges of phases do not overlap.

10. An apparatus, comprising:

a processor; and

a computer readable storage medium coupled to the processor, the computer readable storage medium having instructions stored thereon which, when executed on the processor cause the processor to;

access a first light intensity image and a second light intensity image, the first and second light intensity images contain pixel values for the same scene;

determine boundary values for pixels in the first light intensity image based on one or more neighbor pixels in the first light intensity image;

calculate a depth image based on the first light intensity image and the second light intensity image;

smooth the depth image;

determine whether to modify the first light intensity image based on results of smoothing the depth image;

modify the first light intensity image based on the smoothed depth image and the second light intensity image if it is determined to modify the first light intensity image, the modifying includes keeping values for the pixels in the first light intensity image within the boundary values;

calculate new values for the depth image based on the modified first light intensity image and the second light intensity image if the first light intensity image was modified; and

continue to modify the first light intensity image and calculate new values for the depth image until it is determined not to modify the first light intensity image.

11. The apparatus of claim 10, wherein the instructions that cause the processor to modify the first light intensity image based on the smoothed depth image and the second light intensity image include instructions that cause the processor to perform an inverse of calculating a depth image based on the first light intensity image and the second light intensity image.

12. The apparatus of claim 10, wherein the instructions that cause the processor to calculate a depth image based on the first light intensity image and the second light intensity image include instructions that cause the processor to divide the first light intensity image by the second light intensity image,

wherein the instructions that cause the processor to modify the first light intensity image based on the smoothed depth image and the second light intensity image include instructions that cause the processor to multiply the smoothed depth image by the second light intensity image.

13. The apparatus of claim 10, wherein one of the first and second light intensity images is a gated light intensity image and the other is an un-gated light intensity image.

14. The apparatus of claim 10, wherein the first and the second light intensity images are based on data that is different in phase of a transmitted light beam.

15. The apparatus of claim 10, wherein instructions that cause the processor to determine boundary values for each pixel in the first light intensity image based on one or more neighbor pixels in the first light intensity image include instructions that cause the processor to determine an upper boundary and a lower boundary based on the value of a first pixel in the first light intensity image and the value of a neighbor to the first pixel, wherein the neighbor is determined based on geometry of a first sensor used to collect the first light intensity image and a second sensor used to collect the second light intensity image.

16. A method of determining depth, comprising:

accessing a first light intensity image and a second light intensity image, the first and second light intensity images contain pixel values of light intensity for the same field of view;

determining boundary values for pixels in the first light intensity image based on a possible pixel misalignment between the first light intensity image and the second light intensity image;

calculating a depth image based on the first light intensity image and the second light intensity image;

smoothing the depth image;

determining whether to modify the first light intensity image based on results of smoothing the depth image;

modifying the first light intensity image based on the smoothed depth image and the second light intensity image, the modifying includes performing an inverse of the calculating the depth image, the modifying includes keeping pixel values in the first light intensity image within the boundary values;

calculating new values for the depth image based on the modified first light intensity image and the second light intensity image; and

repeating the smoothing, the modifying the first light intensity image, and the calculating new values for the depth image until it is determined not to modify the first light intensity image, wherein the determining boundary values, the calculating a depth image, the smoothing, the modifying, and the calculating new values are performed by a processor.

17. The method of claim 16, wherein the modifying the first light intensity image includes modifying a value of a pixel in the second light intensity image if a value of a corresponding pixel in the first light intensity image would otherwise have gone outside of its boundary value as a result of performing the inverse of calculating the depth image.

18. The method of claim 16, further comprising:

transmitting a light beam having a plurality of pulses;

collecting first light intensity data at a first sensor during first gating periods;

collecting second light intensity data at a second sensor during second gating periods that are different from the first gating periods; and

determining the first and second light intensity images based on the first and second light intensity data.

19. The method of claim 18, wherein the determining boundary values includes determining an upper boundary value and a lower boundary value for a first pixel in the first light intensity image based on the value of the first pixel in the first light intensity image and the value of one or more neighbors to the first pixel, wherein the one or more neighbors are determined based on geometry of the first sensor and the second sensor.

20. The method of claim 16, further comprising:

transmitting a modulated light beam;

collecting data at a first sensor when the modulated light beam is within a first range of phases;

collecting data at a second sensor when the modulated light beam is within a second range of phases, the first and second ranges of phases do not overlap; and

determining the first and second light intensity images based on the data collected at the first sensor and the data collected at the second sensor.