Single-Frequency Time-of-Flight Depth Computation using Stereoscopic Disambiguation

• US 20180247424A1
• Filed: 02/27/2017
• Published: 08/30/2018
• Est. Priority Date: 02/27/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bleyer, Michael, Price, Raymond Kirk, Demandolx, Denis
• Granted Patent: US 10,810,753 B2
• Time in Patent Office: Days
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G01S 17/26 wherein the transmitted pulses use a frequency-modulated or phase-modulated carrier wave, e.g. for pulse compression of received signals
• CPC Class Codes: G01S 17/36 with phase comparison between the received signal and the contemporaneously transmitted signal
• CPC Class Codes: G01S 17/48 Active triangulation systems, i.e. using the transmission and reflection of electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/88 Lidar systems specially adapted for specific applications
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/4813 Housing arrangements
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

Disclosed is a technique for 3D reconstruction through a combination of time-of-flight (ToF) and stereoscopy. Use of a single modulation frequency in ToF computation provides a set of ambiguous distances. To determine a single accurate distance out of this candidate set, a stereoscopic comparison of an image pair provides a disambiguating distance. The stereoscopic comparison uses a stored virtual image of at least part of the emitted light, and the detected image of light reflected from an object. The stereoscopic distance is used to determine which of the multiple accurate distances is correct based on proximity. The closest of the multiple distances to the disambiguating distance is taken as the actual distance.

Claims

1. A method comprising:

emitting electromagnetic (EM) radiation from an emitter of a device;

capturing, by a ToF sensor on the device, a first image based on returned radiation, wherein the returned radiation is at least a portion of the emitted EM radiation after said at least a portion of the emitted EM radiation has been reflected by an object located a distance from the device; and

computing the distance of the object from the device by using a stereoscopic image pair that includes the first image.

2. The method of claim 1, wherein the stereoscopic image pair further includes a second image, and wherein the first and second images record different variations of the EM radiation.

3. The method of claim 2, wherein the EM radiation is emitted from the emitter in a specified structured-light pattern.

4. The method of claim 3, wherein the second image represents at least a portion of the structured-light pattern exactly as emitted by the emitter, the method further comprising:

storing the second image in a memory of the device prior to said emitting.

5. The method of claim 4, wherein the second image is a virtual image corresponding to a virtual image sensor collocated with the emitter, and wherein a baseline distance for the stereoscopic image pair is a distance between the emitter and the ToF sensor.

6. The method of claim 1, wherein the distance of the object is a first distance measured by stereoscopy, the method further comprising:

computing a preliminary distance to the object based on a phase of the returned radiation; and

computing a disambiguated distance to the object based on the preliminary distance and the first distance.

7. The method of claim 1, wherein the EM radiation is infrared light.

8. A method comprising:

transmitting electromagnetic (EM) radiation from an emitter of a device configured to determine a distance from the device to an object, over a wireless medium;

capturing, by a ToF sensor on the device, a first image based on returned radiation, returned radiation is at least a portion of the emitted EM radiation after said at least a portion of the emitted EM radiation has been reflected by an object located a distance from the device; and

computing an ambiguous distance to the object based on a phase of the returned radiation; and

computing a de-aliased distance from the device to the object from the preliminary distance using stereoscopy based on a stereoscopic image pair that includes the first image and a second image, the second image stored in a memory of the device and representing at least a portion of the EM radiation exactly as emitted by the emitter.

9. The method of claim 8, wherein the EM radiation is infrared light.

10. The method of claim 8, wherein the EM radiation is emitted by the emitter in a structured-light pattern.

11. The method of claim 10, wherein the returned radiation includes a spatially offset structured-light pattern and said computing a de-aliased distance further comprises:

computing an estimated distance based upon a stereoscopic comparison of the first image and the second image;

based on the phase of the returned radiation and a frequency of the EM radiation determining a number of phase wraps of the returned radiation that are the closest to the estimated distance; and

outputting a de-aliased distance based on the number of phase wraps and the phase of the returned radiation.

12. The method of claim 10, wherein the second image represents at least a portion of the structured-light pattern exactly as emitted by the emitter, the method further comprising:

storing the second image in a memory of the device prior to said emitting.

13. The method of claim 8, wherein the second image is a virtual image corresponding to a virtual image sensor collocated with the emitter, and wherein a baseline distance for the stereoscopic image pair is a distance between the emitter and the ToF sensor.

14. The method of claim 8, wherein said de-aliasing further comprises:

determining an active brightness of the returned radiation

comparing an illumination power of the EM radiation to the active brightness of the returned radiation.

15. The method of claim 8, wherein said transmitting EM radiation comprises transmitting the EM radiation at only one frequency.

16. The method of claim 8, wherein the device is a near-eye display device.

17. An apparatus comprising:

an electromagnetic (EM) emitter configured to emit EM radiation to determine a distance from the apparatus to an object, over a wireless medium at one frequency;

a time-of-flight (ToF) sensor configured to capture a first frame of returned radiation that is at least a portion of emitted EM radiation after said at least a portion of emitted EM radiation has been reflected by an object located a distance from the apparatus; and

a processor configured to compute a first distance to the object using a stereoscopic image pair that includes the first frame.

18. The apparatus of claim 17, wherein the first frame further includes a phase of the returned radiation and wherein the processor is further configured to compute an ambiguous distance based on the phase of the returned radiation and de-alias the ambiguous distance using first distance.

19. The apparatus of claim 17, further comprising:

a memory storing a second frame and a mapped stereographic position, the second frame is a virtual image corresponding to a virtual image sensor located at the mapped stereographic position.

20. The apparatus of claim 17, further comprising:

a housing shaped to allow the apparatus to be worn on a user'"'"'s head as at least a portion of a head-mounted display device.