Augmented reality systems having a virtual image overlaying an infrared portion of a live scene

• US 9,995,936 B1
• Filed: 04/29/2016
• Issued: 06/12/2018
• Est. Priority Date: 04/29/2016
• Status: Active Grant

• Current Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Original Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Inventors: Macannuco, David J., Beard, Wayne A.
• Primary Examiner(s): Drennan, Barry
• Assistant Examiner(s): Pringle-Parker, Jason
• Application Number:
  US15/142,380
• Time in Patent Office: 774 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0118 comprising devices for improving the contrast of the display / brillance control visibility
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 9/307 by helmet-mounted projector or display G09B9/36 takes precedence; optical parts of head-up displays G02B27/01
• CPC Class Codes: H04N 23/20 for generating image signals from infrared radiation only
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A head-mounted display (HMD) for an augmented reality system allows a user to view an augmented scene comprising a real-world portion of a live scene combined with virtual images overlaying an infrared (IR) portion of the live scene. The HMD includes a head-wearable frame, a lens defining a user field of view (FOV), and a camera. The lens permits the user to view a live scene corresponding to the user FOV, and the camera is configured to capture a captured image of the live scene containing data indicative of an infrared (IR) portion of the live scene. The IR portion of the live scene includes reflected IR light above a predetermined threshold, such as an IR reflective background surface in a simulation environment. Based on the IR portion, a virtual image is displayed on an interior surface of the lens so that the virtual image overlays the IR portion.

Claims

1. A head-mounted display (HMD) comprising:

a head-wearable frame configured to be worn by a user;

a lens coupled to the frame, the lens defining a user field of view (FOV), the lens comprising an interior surface configured to face the user;

a camera coupled to the frame, the camera having a camera FOV registered to the user FOV, the camera configured to capture a captured image of a live scene corresponding to the user FOV, the captured image comprising data indicative of an IR portion of the live scene, the IR portion comprising an infrared (IR) light in a predetermined IR wavelength band reflected above a predetermined threshold;

a processing device configured to;

detect the IR portion of the live scene based on the captured image;

determine the virtual image based on the IR portion by;

determining a relative position and orientation of the user FOV;

determining a virtual FOV within a virtual environment model based on the relative position and orientation of the user FOV, the virtual FOV defining a virtual scene; and

registering a portion of the virtual scene to the IR portion of the live scene to determine the virtual image; and

a display device configured to display the virtual image on the interior surface of the lens, the virtual image registered to the IR portion of the live scene based on the captured image, the virtual image configured to overlay the IR portion of the live scene in the user FOV.

2. The HMD of claim 1, wherein the predetermined threshold is an IR brightness threshold, the processing device further configured to:

determine, for a plurality of predefined sections of the captured image, whether an IR brightness of each section in the predetermined IR wavelength band is above the IR brightness threshold; and

register each section of the captured image having an IR brightness above the IR brightness threshold to the IR portion of the live scene.

3. The HMD of claim 2, wherein the plurality of predefined sections of the captured image is a plurality of pixels, each pixel mapped to a predetermined location of the interior surface of the lens.

4. The HMD of claim 1, wherein the interior surface of the lens comprises a reflective surface, and

wherein the display device is configured to project the virtual image on the reflective surface of the lens, the virtual image configured to overlay the IR portion of the live scene.

11. The augmented reality system of claim 1, wherein the interior surface of the lens comprises a reflective surface, and

wherein the display device is configured to project the virtual image on the reflective surface of the lens, the virtual image configured to overlay the IR portion of the live scene.

5. An augmented reality system comprising:

a head-mounted display (HMD) comprising;

a head-wearable frame configured to be worn by a user;

a lens coupled to the frame, the lens defining a user field of view (FOV), the lens comprising an interior surface configured to face the user;

a camera coupled to the frame, the camera having a camera FOV registered to the user FOV, the camera configured to capture a captured image of a live scene corresponding to the user FOV; and

a display device configured to display a virtual image on the interior surface of the lens;

a simulation environment comprising an infrared (IR) reflective surface disposed in the live scene, the IR reflective surface configured to reflect IR light to define an IR portion of the live scene; and

a processing device configured to;

detect the IR portion of the live scene based on the captured image by;

determining, for a plurality of predefined sections of the captured image, whether an IR brightness of each section in a predetermined IR wavelength band is above an IR brightness threshold; and

registering each section of the captured image having an IR brightness above the IR brightness threshold to the IR portion of the live scene; and

determine the virtual image based on the IR portion;

wherein the virtual image is registered to the IR portion of the live scene on the interior surface of the lens, the virtual image configured to overlay the IR portion of the live scene in the user FOV.

6. The augmented reality system of claim 5, wherein the IR reflective surface in the live scene comprises a window region of the simulation environment.

7. The augmented reality system of claim 5, further comprising an object disposed in the live scene in a line of sight between a portion of the IR reflective surface in the live scene and the HMD.

8. The augmented reality system of claim 5, wherein the IR reflective surface in the live scene is configured to reduce a translucency of the virtual image.

9. The augmented reality system of claim 8, wherein the IR reflective surface in the live scene is configured to absorb visible light to reduce the translucency of the virtual image.

10. The augmented reality system of claim 5, wherein the plurality of predefined sections of the captured image is a plurality of pixels, each pixel mapped to a predetermined location of the interior surface of the lens.

12. A method of providing a virtual image to a head-mounted display (HMD), the method comprising:

determining an infrared (IR) portion of a live scene based on a captured image of the live scene, the live scene corresponding to a user field of view (FOV) defined by a lens of the HMD, the IR portion comprising IR light in a predetermined IR wavelength band reflected above a predetermined threshold;

determining a relative position and orientation of the user FOV;

determining a virtual FOV within a virtual environment model based on the relative position and orientation of the user FOV, the virtual FOV defining a virtual scene;

registering a portion of the virtual scene to the IR portion of the live scene to determine the virtual image; and

displaying, via a display device of the HMD, the virtual image on an interior surface of the lens, the virtual image registered to the IR portion of the live scene based on the captured image, the virtual image configured to overlay the IR portion of the live scene in the user FOV.

13. The method of claim 12, wherein the IR portion corresponds to an IR reflective surface disposed in the live scene, the IR reflective surface configured to reflect IR light in the predetermined IR wavelength band.

14. The method of claim 13, further comprising an object disposed in the live scene in a line of sight between a portion of the IR reflective surface in the live scene and the HMD.

15. The method of claim 13, wherein the IR reflective surface in the live scene is configured to reduce a translucency of the virtual image overlaying the IR portion of the live scene in the user FOV.

16. The method of claim 15, wherein the IR reflective surface in the live scene is configured to absorb visible light to reduce the translucency of the virtual image overlaying the IR portion of the live scene in the user FOV.

17. A head-mounted display (HMD) comprising:

a head-wearable frame configured to be worn by a user;

a lens coupled to the frame, the lens defining a user field of view (FOV), the lens comprising an interior surface configured to face the user;

a camera coupled to the frame, the camera having a camera FOV registered to the user FOV, the camera configured to capture a captured image of a live scene corresponding to the user FOV, the captured image comprising data indicative of an IR portion of the live scene, the IR portion comprising an infrared (IR) light in a predetermined IR wavelength band reflected above an IR brightness threshold;

a processing device configured to;

determine, for a plurality of predefined sections of the captured image, whether an IR brightness of each section in the predetermined IR wavelength band is above the IR brightness threshold;

register each section of the captured image having an IR brightness above the IR brightness threshold to the IR portion of the live scene; and

determine the virtual image based on the IR portion; and

a display device configured to display the virtual image on the interior surface of the lens, the virtual image registered to the IR portion of the live scene based on the captured image, the virtual image configured to overlay the IR portion of the live scene in the user FOV.