Multi-Spectrum Illumination-and-Sensor Module for Head Tracking, Gesture Recognition and Spatial Mapping

• US 20180253856A1
• Filed: 03/01/2017
• Published: 09/06/2018
• Est. Priority Date: 03/01/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Price, Raymond Kirk, Bleyer, Michael, Demandolx, Denis
• Granted Patent: US 10,628,950 B2
• Time in Patent Office: Days
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G01S 17/08 for measuring distance only indirect measurement G01S17/46; active triangulation systems G01S17/48
• CPC Class Codes: G01S 17/46 Indirect determination of position data
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G01S 7/4813 Housing arrangements
• CPC Class Codes: G01S 7/4815 using multiple transmitters
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 5/208 for use with infrared or ultraviolet radiation, e.g. for separating visible light from infrared and/or ultraviolet radiation
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: H04N 13/366 : using viewer tracking
• CPC Class Codes: H04N 13/383 : for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 23/11 : for generating image signals from visible and infrared light wavelengths

Abstract

A device and method use multiple light emitters with a single, multi-spectrum imaging sensor to perform multi-modal infrared light based depth sensing and visible light based Simultaneous Localization and Mapping (SLAM). The multi-modal infrared based depth sensing may include, for example, any combination of infrared-based spatial mapping, infrared based hand tracking and/or infrared based semantic labeling. The visible light based SLAM may include head tracking, for example.

Claims

1. A near-eye display device comprising:

an illumination module that, when in operation, emits first light of a first frequency range toward an environment of the near-eye display device;

an imaging sensor that, when in operation, receives the first light of the first frequency range reflected by the environment and second light of a second frequency range reflected by the environment; and

a processor that, when in operation, determines a depth map based on the received first light and that tracks a position and/or orientation of at least a part of the user based on the received second light.

2. The near-eye display device of claim 1, wherein the imaging sensor includes a first plurality of pixels configured to receive light of the first frequency range and a second plurality of pixels configured to receive light of the second frequency range, and wherein the second plurality of pixels are interspersed with the first plurality of pixels.

3. The near-eye display device of claim 2, further comprising:

an optical bandpass filter that reduces ambient light of the first frequency range incident on the first plurality of pixels.

4. The near-eye display device of claim 1, wherein the first frequency range is within an infrared range and the second frequency range is within a visible range.

5. The near-eye display device of claim 1, wherein the illumination module is a first illumination module, the near-eye display device further comprising a second illumination module that, when in operation, emits third light of the first frequency range toward the environment of the near-eye display device;

wherein the imaging sensor is further configured to receive at least a portion of the third light of the first frequency range.

6. The near-eye display device of claim 5, wherein the processor is further configured to generate or update a depth map based on the received portion of the third light of the first frequency range.

7. The near-eye display device of claim 5, wherein the first illumination module has an effective illumination range that is greater than an effective illumination range of the second illumination module.

8. The near-eye display device of claim 5, wherein the first illumination module has a field of illumination that is narrower than a field of illumination of the second illumination module.

9. The near-eye display device of claim 5, wherein the first illumination module and the second illumination module emit light at different points in time, and the imaging sensor receives the first light of the first frequency range and the second light of the first frequency range at different points in time.

10. The near-eye display device of claim 5, wherein the second illumination module emits at least one of the first light or the third light with a structured illumination pattern.

11. A near-eye display device comprising:

a first illumination module that, when in operation, emits structured infrared light toward an environment in which the near-eye display device is located, the environment including an object other than a user of the near-eye display device;

a second illumination module that, when in operation, emits second infrared light toward the environment in which the near-eye display device is located;

an imaging sensor that, when in operation, receives a portion of the structured infrared light reflected by the object, a portion of the second infrared light reflected by a body part of the user of the near-eye display device, and ambient visible light reflected by at least a portion of the environment; and

a processor that, when in operation, determines a first set of depth values based on the received portion of the structured infrared light, determines a second set of depth values based on the received portion of the second infrared light, and performs a Simultaneous Localization and Mapping (SLAM) function based on the received portion of the visible light.

12. The near-eye display device of claim 11, wherein the imaging sensor converts the received portion of the first infrared light and the received portion of the second infrared light into depth information based on time of flight and phase shifts from the first and second illumination modules to the imaging sensor.

13. The near-eye display device of claim 11, wherein the first infrared light has a structured illumination pattern.

14. The near-eye display device of claim 11, wherein the processor applies a time-of-flight calculation to the received portion of the structured light.

15. The near-eye display device of claim 11, wherein the processor, when in operation, uses the received portion of the first infrared light to determine depth values for spatial mapping and/or semantic labeling, uses the received portion of the second infrared light to determine depth values for hand tracking, and uses the received portion of the visible light to perform head tracking.

16. The near-eye display device of claim 11, wherein the first illumination module has an effective illumination range that is greater than an illumination range of the second illumination module.

17. The near-eye display device of claim 16, wherein the first illumination module has a field of illumination that is narrower than a field of illumination of the second illumination module.

18. The near-eye display device of claim 11, wherein the first illumination module and the second illumination module emit light at different times, and the imaging sensor receives the first infrared light and the second infrared light range at different times.

20. The method of claim 18, wherein:

the structured light has an illumination range that is greater than an illumination range of the second light; and

the structured light has a field of illumination that is narrower than a field of illumination of the second light.

19. A method comprising:

emitting, by a first illumination module, structured light of a first frequency range toward an environment in which the near-eye display device is located, the environment including an object other than a user of the near-eye display device;

emitting, by a second illumination module, second light of the first frequency range toward the environment;

receiving, by an imaging sensor of the near-eye display device, at least a portion of the structured light reflected by the object, at least a portion of the second light reflected by a body part of the user of the near-eye display device, and ambient light of a second frequency range reflected by at least a portion of the environment;

tracking a position and/or orientation of the head of the user based on the received ambient light of the second frequency range; and

generating a depth map of the environment based on the received portion of the structured light and detecting a gesture made by the body part of the user based on the received at least a portion of the second light.