See-through augmented reality system

• US 9,581,819 B1
• Filed: 12/17/2014
• Issued: 02/28/2017
• Est. Priority Date: 12/17/2014
• Status: Active Grant

• Current Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Original Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Inventors: Boggs, Richard P., Goergen, Patrick J., Harrison, Gregory A., Sorokowsky, Eric T., Grant, Edward T.
• Primary Examiner(s): Sasinowski, Andrew
• Assistant Examiner(s): WILLS-BURNS, CHINEYERE D
• Application Number:
  US14/573,664
• Time in Patent Office: 804 Days
• Field of Search: 345/694, 345/8
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features

Abstract

A see-through augmented reality system is disclosed. The system includes a head-wearable frame and a lens. The lens has at least one field-of-view (FOV). A sensor is configured to be coupled with respect to a head of a user. The sensor generates sensor data that changes responsive to movements of the head. A display apparatus is mapped to an interior surface of the lens. A controller is coupled to the sensor and to the display apparatus. The controller is configured to determine a current scene within the FOV based at least in part on the sensor data. The controller determines an out-the-window (OTW) scene portion of the current scene and a non-OTW scene portion of the current scene and generates OTW imagery that depicts the OTW scene portion. The controller causes the display apparatus to emit light in accordance with the OTW imagery toward the interior surface of the lens.

Claims

1. A system, comprising:

a head-wearable frame;

a lens coupled to the head-wearable frame, the lens being transmissive and reflective and having at least one field-of-view (FOV) and an interior surface;

a sensor configured to be coupled with respect to a head of a user and to generate sensor data that changes responsive to movements of the head;

a display apparatus that is mapped to the interior surface of the lens; and

a controller coupled to the sensor and the display apparatus, the controller being configured to;

determine a current scene of a real world environment within the FOV based at least in part on the sensor data, the real world environment comprising a window region and a non-window region;

determine an out-the-window (OTW) scene portion comprising a portion of the window region that is in the current scene and a non-OTW scene portion comprising a portion of the non-window region that is in the current scene;

generate OTW imagery configured to overlay the OTW scene portion without overlaying the non-OTW scene portion; and

cause the display apparatus to emit light in accordance with the OTW imagery toward the interior surface of the lens, wherein the interior surface of the lens is configured to reflect the light toward an eye of the user to provide the OTW imagery in conjunction with the non-OTW scene portion.

2. The system of claim 1, wherein the display apparatus comprises a plurality of pixels, each pixel being mapped to a particular surface location of the interior surface of the lens.

3. The system of claim 2, wherein the controller causes the display apparatus to turn off each pixel of the plurality of pixels other than pixels that display the OTW imagery.

4. The system of claim 1, wherein the controller is further configured to:

generate FOV imagery that includes the OTW imagery and non-OTW imagery; and

modify the FOV imagery to mask the non-OTW imagery to generate the OTW imagery.

5. The system of claim 1, wherein the lens is configured to transmit light received from an exterior surface of the lens through the lens and out the interior surface of the lens, and to reflect the light received from the display apparatus toward an eye of the user.

6. The system of claim 1, wherein:

the lens comprises a right eye lens and a left eye lens;

the at least one FOV comprises a right eye FOV and a left eye FOV;

the display apparatus comprises a right eye display apparatus that is mapped to an interior surface of the right eye lens and comprises a left eye display apparatus that is mapped to an interior surface of the left eye lens; and

the controller is further configured to;

determine a current right eye scene of the real world environment within the right eye FOV and a current left eye scene of the real world environment within the left eye FOV;

determine an OTW right eye scene portion comprising a portion of the window region that is in the current right eye scene and a non-OTW right eye scene portion comprising a portion of the non-window region that is in the current right eye scene;

determine an OTW left eye scene portion comprising a portion of the window region that is in the current left eye scene and a non-OTW left eye scene portion comprising a portion of the non-window region that is in the current left eye scene;

generate right eye OTW imagery, configured to overlay the OTW right eye scene portion without overlaying the non-OTW right eye scene portion and left eye OTW imagery, configured to overlay the OTW left eye scene portion without overlaying the non-OTW left eye scene portion; and

cause the right eye display apparatus to emit light in accordance with the right eye OTW imagery toward the interior surface of the right eye lens to provide the right eye OTW imagery in conjunction with the non-OTW right eye scene portion, and cause the left eye display apparatus to emit light in accordance with the left eye OTW imagery toward the interior surface of the left eye lens to provide the left eye OTW imagery in conjunction with the non-OTW left eye scene portion.

7. The system of claim 1, further including a platform comprising a cockpit comprising an instrument panel, the instrument panel forming at least part of the non-window region of the real world environment.

8. The system of claim 7, further comprising a structure positioned over the cockpit, the structure forming at least part of the window region of the real world environment, and a darkening material configured to be positioned over the structure, the darkening material corresponding to the OTW scene portion.

9. The system of claim 1, further comprising:

a simulation environment comprising the window region and the non-window region.

10. The system of claim 9, wherein the non-window region comprises a vehicle interior.

11. A method, comprising:

receiving sensor data from a sensor coupled with respect to a head of a user, the sensor data identifying a location of the head and an orientation of the head;

determining, by a controller comprising a processor, a current scene of a real world environment based at least in part on the sensor data and a field-of-view (FOV) of a lens coupled to a head-wearable frame, the lens being transmissive and reflective and having an interior surface, the real world environment comprising a window region and a non-window region;

determining an out-the-window (OTW) scene portion comprising a portion of the window region that is in the current scene and a non-OTW scene portion comprising a portion of the non-window region that is in the current scene;

generating OTW imagery configured to overlay the OTW scene portion; and

causing a display apparatus to emit light in accordance with the OTW imagery toward the interior surface of the lens to provide the OTW imagery in conjunction with the non-OTW scene portion.

12. The method of claim 11, wherein the display apparatus is mapped to the interior surface of the lens.

13. The method of claim 12, wherein the display apparatus comprises a plurality of pixels, each pixel being mapped to a particular surface location of the interior surface of the lens.

14. The method of claim 13, further comprising causing the display apparatus to turn off each pixel of the plurality of pixels other than pixels that display the OTW imagery.

15. The method of claim 11, wherein the controller is further configured to:

generate FOV imagery that includes the OTW imagery and non-OTW imagery; and

modify the FOV imagery to mask the non-OTW imagery to generate the OTW imagery.

16. The method of claim 11, wherein the lens is configured to transmit light received from an exterior surface of the lens through the lens and out the interior surface of the lens, and to reflect the light received from the display apparatus toward an eye of the user.

17. The method of claim 11, wherein the sensor comprises an inertial measurement unit.

18. The method of claim 11, wherein:

the lens comprises a right eye lens and a left eye lens;

the FOV comprises a right eye FOV and a left eye FOV;

the display apparatus comprises a right eye display apparatus that is mapped to an interior surface of the right eye lens and comprises a left eye display apparatus that is mapped to an interior surface of the left eye lens; and

further comprising;

determining a current right eye scene of the real world environment within the right eye FOV and a current left eye scene of the real world environment within the left eye FOV;

determining an OTW right eye scene portion comprising a portion of the window region that is in the current right eye scene and a non-OTW right eye scene portion comprising a portion of the non-window region that is in the current right eye scene;

determining an OTW left eye scene portion comprising a portion of the window region that is in the current left eye scene and a non-OTW left eye scene portion comprising a portion of the non-window region that is in the current left eye scene;

generating right eye OTW imagery, configured to overlay the OTW right eye scene portion without overlaying the non-OTW right eye scene portion and left eye OTW imagery, configured to overlay the OTW left eye scene portion without overlaying the non-OTW left eye scene portion; and

causing the right eye display apparatus to emit light in accordance with the right eye OTW imagery toward the interior surface of the right eye lens to provide the right eye OTW imagery in conjunction with the non-OTW right eye scene portion, and causing the left eye display apparatus to emit light in accordance with the left eye OTW imagery toward the interior surface of the left eye lens to provide the left eye OTW imagery in conjunction with the non-OTW left eye scene portion.

19. The method of claim 11, wherein a first portion of the interior surface of the lens corresponds to the OTW scene portion, and a second portion of the interior surface of the lens corresponds to the non-OTW scene portion;

and further comprising;

determining a subset of pixels of a plurality of pixels of the display apparatus that correspond to the first portion of the interior surface; and

causing the subset of pixels in the display apparatus to emit light in accordance with the OTW imagery toward the interior surface of the lens.

20. A method, comprising:

receiving sensor data from a sensor coupled with respect to a head of a user, the sensor data identifying a location of the head and an orientation of the head;

determining, by a controller comprising a processor, a current scene of a real world environment based at least in part on the sensor data and a field-of-view (FOV) of a lens coupled to a head-wearable frame, the lens being transmissive and reflective and having an interior surface, the real world environment comprising a window region and a non-window region;

determining an out-the-window (OTW) scene portion comprising a portion of the window region that is in the current scene and a non-OTW scene portion comprising a portion of the non-window region that is in the current scene, wherein a first portion of the interior surface of the lens corresponds to the OTW scene portion and a second portion of the interior surface of the lens corresponds to the non-OTW scene portion;

generating OTW imagery that depicts the OTW scene portion;

determining a subset of pixels of a plurality of pixels of a display apparatus that corresponds to the first portion of the interior surface; and

causing the subset of pixels in the display apparatus to emit light in accordance with the OTW imagery toward the interior surface of the lens.