Active brightness-based strategy for invalidating pixels in time-of-flight depth-sensing

• US 10,557,921 B2
• Filed: 01/23/2017
• Issued: 02/11/2020
• Est. Priority Date: 01/23/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bleyer, Michael, Price, Raymond Kirk, Zhao, Jian, Demandolx, Denis Claude Pierre
• Primary Examiner(s): Hellner, Mark
• Application Number:
  US15/412,485
• Publication Number:
  US 20180210070A1
• Time in Patent Office: 1,114 Days
• Field of Search: 356 501
• CPC Class Codes: G01S 17/36 with phase comparison between the received signal and the contemporaneously transmitted signal
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating

Abstract

A computer-implemented technique is described herein for invalidating pixels in a time-of-flight depth-sensing device based on active brightness (AB) measurements. In one implementation, the technique involves, for each sensing element of a sensor: generating frequency-specific sensor readings in response to receiving instances of radiation having plural frequencies (e.g., frequencies f₁, f₂, and f₃); generating a set of active brightness measurements (AB_f1, AB_f2, and AB_f3) associated with the respective frequencies; generating a variation measure that reflects an extent of variation within the set of active brightness measurements; and invalidating a pixel associated with the particular sensing element if the variation measure satisfies a prescribed invalidation condition.

Claims

1. A method for sensing an environment using a time-of-flight depth-sensing device, comprising, for a particular sensing element of a sensor of the time-of-flight depth-sensing device:

receiving instances of radiation having plural respective frequencies, each instance of radiation having a given frequency, originating from an illumination source, and being reflected from the environment;

generating frequency-specific sensor readings in response to receiving the instances of radiation;

generating a set of active brightness measurements for the plural frequencies, each active brightness measurement being based on frequency-specific sensor readings made by the particular sensing element that are associated with a particular frequency;

generating a variation measure that reflects an extent of variation within the set of active brightness measurements; and

invalidating a pixel associated with the particular sensing element if the variation measure satisfies a prescribed invalidation condition.

2. The method of claim 1, where the method generates each active brightness measurement for a particular frequency by:

representing the frequency-specific sensor readings associated with the particular frequency as a vector within a complex domain having real and imaginary axes; and

generating the active brightness measurement for the particular frequency by computing a magnitude of the vector.

3. The method of claim 1, wherein the method generates the variation measure by:

identifying a minimum active brightness measurement in the set of active brightness measurements;

identifying a maximum active brightness measurement in the set of active brightness measurements; and

generating the variation measure as a normalized difference between the minimum active brightness measurement and the maximum active brightness measurement.

4. The method of claim 3, wherein the method invalidates the pixel when the normalized difference fails to satisfy a prescribed threshold.

5. The method of claim 1, further comprising, for the particular sensing element:

generating a set of phase measurements for the plural frequencies, each phase measurement being based on the frequency-specific sensor readings made by the particular sensing element that are associated with a particular frequency; and

determining whether the set of phase measurements corresponds to an anomalous depth measurement, to provide a phase-based anomaly assessment,

wherein said invalidating is based on the variance measure in conjunction with the anomaly assessment.

6. The method of claim 5, wherein said determining whether the set of phase measurements corresponds to an anomalous condition comprises mapping the set of phase measurements into the phase-based anomaly assessment using a lookup table.

7. The method of claim 5, wherein, when the frequency-specific sensor readings from the particular sensing element have not been invalidated:

generating, via a de-aliasing operation, a depth value based on the set of phase measurements, the depth value reflecting a distance between a reference point and a point in the environment that has been imaged by the particular sensing element.

8. The method of claim 1, wherein the pixel associated with the particular sensing element is invalidated due to motion blur that affects the particular pixel.

9. The method of claim 1, wherein the pixel associated with the particular sensing element is invalidated due to multipath interference that affects the particular pixel.

10. The method of claim 1, further comprising correcting the pixel when the pixel has been invalidated, by generating a correct depth value for the pixel.

11. The method of claim 10, wherein said correcting comprises:

computing optical flow information based on a plurality of successive active brightness images, each active brightness image containing a collection of active brightness measurements associated with a particular frequency;

using the optical flow information to identify movement of a scene point, associated with the pixel that has been invalidated, across the successive active brightness images;

generating a set of phase measurements based on positions of the scene point across the successive active brightness images; and

generating, via a de-aliasing operation, the correct depth value based on the set of phase measurements.

12. A time-of-flight depth-sensing device, comprising:

a sensor having a plurality of sensing elements, a particular sensing element operating to;

receive instances of radiation having plural respective frequencies, each instance of radiation having a given frequency, originating from an illumination source, and being reflected from an environment; and

generate frequency-specific sensor readings in response to receiving the instances of radiation; and

a depth-generating engine configured to;

generate a set of active brightness measurements for the plural frequencies, each active brightness measurement being based on frequency-specific sensor readings made by the particular sensing element that are associated with a particular frequency;

generate a variation measure that reflects an extent of variation within the set of active brightness measurements; and

invalidate a pixel associated with the particular sensing element if the variation measure satisfies a prescribed invalidation condition.

13. The time-of-flight depth-sensing device of claim 12, where the depth-generating engine is configured to generate each active brightness measurement for a particular frequency by:

representing the frequency-specific sensor readings associated with the particular frequency as a vector within a complex domain having real and imaginary axes; and

generating the active brightness measurement for the particular frequency by computing a magnitude of the vector.

14. The time-of-flight depth-sensing device of claim 12, wherein the depth-generating engine is configured to generate the variation measure by:

identifying a minimum active brightness measurement in the set of active brightness measurements;

identifying a maximum active brightness measurement in the set of active brightness measurements; and

generating the variation measure as a normalized difference between the minimum active brightness measurement and the maximum active brightness measurement.

15. The time-of-flight depth-sensing device of claim 14, wherein the depth-generating engine is configured to invalidate the pixel when the normalized difference fails to satisfy a prescribed threshold.

16. The time-of-flight depth-sensing device of claim 12, wherein the depth-generating engine is further configured to, for the particular sensing element:

generate a set of phase measurements for the plural frequencies, each phase measurement being based on the frequency-specific sensor readings made by the particular sensing element that are associated with a particular frequency; and

determine whether the set of phase measurements corresponds to an anomalous depth measurement, to provide a phase-based anomaly assessment,

wherein the depth-generating engine is configured to invalidate the pixel based on the variance measure in conjunction with the anomaly assessment.

17. The time-of-flight depth-sensing device of claim 16, wherein the depth-generating engine includes a de-aliasing component that is configured to, when the frequency-specific sensor readings from the particular sensing element have not been invalidated:

generate a depth value based on the set of phase measurements, the depth value reflecting a distance between a reference point and a point in the environment that has been imaged by the particular sensing element.

18. The time-of-flight depth-sensing device of claim 12, further including a pixel-correcting component configured to correct the pixel when the pixel has been invalidated, by generating a correct depth value for the pixel.

19. A computer-readable storage medium for storing computer-readable instructions, the computer-readable instructions, when executed by one or more processor devices, performing a method that comprises:

receiving frequency-specific sensor readings from a particular sensing element of a sensor of a time-of-flight depth-sensing device,

the frequency-specific sensor readings being received, in turn, in response to receiving instances of radiation having plural respective frequencies by the particular sensor element,

each instance of radiation having a given frequency, originating from an illumination source, and being reflected from an environment;

representing the frequency-specific sensor readings associated with each frequency as a vector within a complex domain having real and imaginary axes, the vector having a magnitude, corresponding to active brightness, and a phase, corresponding to a candidate depth value,

overall, said representing producing;

a set active brightness measurements for the plural frequencies for the particular sensing element; and

a set of phase measurements for the plural frequencies for the particular sensing element;

generating a variation measure that reflects an extent of variation within the set of active brightness measurements;

when the variation measure satisfies a prescribed invalidation condition, invalidating a pixel associated with the particular sensing element; and

when the variation measure does not satisfy the prescribed invalidation condition, providing a depth value for the particular sensing element based on the set of phase measurements.

20. The computer-readable storage medium of claim 19, wherein the method further includes correcting the pixel when the pixel has been invalidated, by generating a correct depth value for the pixel.