Interference mitigation via adaptive depth imaging

• US 10,712,561 B2
• Filed: 11/04/2016
• Issued: 07/14/2020
• Est. Priority Date: 11/04/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Price, Raymond Kirk, Zhao, Jian, Bleyer, Michael, Demandolx, Denis
• Primary Examiner(s): Dang, Philip P.
• Application Number:
  US15/344,093
• Publication Number:
  US 20180130209A1
• Time in Patent Office: 1,348 Days
• CPC Class Codes: G01S 17/10 using transmission of interrupted, pulse-modulated waves determination of distance by phase measurements G01S17/32
• CPC Class Codes: G01S 17/48 Active triangulation systems, i.e. using the transmission and reflection of electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/88 Lidar systems specially adapted for specific applications
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/4876 by removing unwanted signals G01S7/495 takes precedence
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06V 10/141 Control of illumination
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 2201/121 using special illumination

Abstract

An illumination source is operated to illuminate an operating environment and an optical sensor is periodically operated for a detection period to detect illumination reflected from one or more subjects within the operating environment. Upon recognizing a source of interfering pulsed illumination within the operating environment, the timing of a subsequent detection period may be varied. In this way, sensing of the interfering pulsed illumination may be averted.

Claims

1. A method for operating an imaging device, comprising:

operating an illumination source to illuminate an operating environment;

periodically operating an optical sensor for a detection period to detect illumination reflected from one or more subjects within the operating environment;

recognizing, via an evaluation of an input at the optical sensor, a source of interfering pulsed illumination within the operating environment, the recognized source of the interfering pulsed illumination controlled independently from the imaging device; and

varying, based on the recognized source of the interfering pulsed illumination, a timing of a subsequent detection period by introducing a first random delay period prior to a subsequent operation of the optical sensor.

2. The method of claim 1, wherein recognizing, via the evaluation of the input at the optical sensor, the source of the interfering pulsed illumination includes receiving data from a device comprising the source of the interfering pulsed illumination.

3. The method of claim 2, wherein the received data comprises information regarding a timing of the interfering pulsed illumination.

4. The method of claim 1, wherein recognizing, via the evaluation of the input at the optical sensor, the source of the interfering pulsed illumination includes receiving the interfering pulsed illumination at a photodetector, the interfering pulsed illumination comprising a wavelength within a threshold of the wavelength of light output by the illumination source.

5. The method of claim 1, further comprising:

generating a depth map of the one or more subjects based on the detected illumination, and wherein recognizing, via the evaluation of the input at the optical sensor, the source of the interfering pulsed illumination includes recognizing invalidated pixels of the depth map.

6. The method of claim 5, wherein recognizing the invalidated pixels of the depth map includes recognizing a number of invalidated pixels greater than a threshold.

7. The method of claim 6, wherein recognizing the invalidated pixels of the depth map includes recognizing a predetermined pattern of invalidated pixels.

8. The method of claim 1, wherein introducing the first random delay period does not include adjusting a frame capture rate of the imaging device.

9. The method of claim 1, wherein introducing the first random delay period includes introducing a delay having a duration less than 5 times the detection period.

10. The method of claim 1, further comprising:

responsive to detecting the interfering pulsed illumination during the subsequent detection period, introducing a second random delay period prior to further operation of the optical sensor, the second random delay period being longer in duration than the first random delay period.

11. The method of claim 10, further comprising:

responsive to detecting additional interfering pulsed illumination during further operation of the optical sensor, adjusting a frame capture rate of the imaging device.

12. The method of claim 1, wherein operating the illumination source to illuminate the operating environment comprises periodically operating the illumination source to illuminate the operating environment for an illumination period, and wherein the method further comprises varying, based on the recognized source of the interfering pulsed illumination, a timing of a subsequent illumination period.

19. The method of claim 5, wherein recognizing invalidated pixels of the depth map includes recognizing pixels that demonstrate a total saturation.

20. The method of claim 7, wherein recognizing the predetermined pattern of invalidated pixels includes recognizing an increased correction in a vertical direction of the depth map.

13. A depth imaging device, comprising:

an illumination source;

an optical sensor;

a storage machine holding instructions executable by a logic machine to;

periodically operate the illumination source to illuminate an operating environment for an illumination period;

periodically operate the optical sensor for a detection period to detect illumination reflected from one or more subjects within the operating environment;

generating a depth map of the one or more subjects based on the detected illumination;

recognize, via an evaluation of an input at the optical sensor, a source of interfering pulsed illumination based on invalidated pixels of the depth map, the recognized source of the interfering pulsed illumination controlled independently from the depth imaging device; and

vary a timing of a subsequent illumination period and a subsequent detection period based on the recognized source of the interfering pulsed illumination by introducing a random delay to the timing of the subsequent illumination period and the subsequent detection period.

14. The depth imaging device of claim 13, wherein recognizing, via the evaluation of the input at the optical sensor, the source of the interfering pulsed illumination based on the invalidated pixels of the depth map includes recognizing a number of invalidated pixels greater than a threshold.

15. The depth imaging device of claim 13, wherein recognizing, via the evaluation of the input at the optical sensor, the source of the interfering pulsed illumination based on the invalidated pixels of the depth map includes recognizing a predetermined pattern of invalidated pixels.

16. The depth imaging device of claim 13, wherein recognizing, via the evaluation of the input at the optical sensor, the source of the interfering pulsed illumination based on the invalidated pixels of the depth map includes a comparison with previously generated depth maps.

17. A method for operating a depth camera, comprising:

determining a frame capture rate for generating depth maps of an environment;

operating an illumination source to illuminate an operating environment for an illumination period during each frame allotment period;

operating an optical sensor for a detection period during each frame allotment period to detect illumination reflected from one or more subjects within the operating environment;

recognizing, via an evaluation of input at the optical sensor, a source of interfering pulsed illumination within the operating environment, the recognized source of the interfering pulsed illumination controlled independently from the imaging device; and

varying, based on the recognized source of interfering pulsed illumination, a timing of a subsequent illumination period and a subsequent detection period relative to a subsequent frame allotment period by introducing a random delay to the timing of the subsequent illumination period and the subsequent detection period.

18. The method of claim 17, wherein delaying the timing of the subsequent illumination period and the subsequent detection period relative to the frame allotment period does not comprise adjusting the frame capture rate.