Apparatus and method for selectively displaying an operational environment

• US 10,096,166 B2
• Filed: 11/10/2015
• Issued: 10/09/2018
• Est. Priority Date: 11/19/2014
• Status: Active Grant

• Current Assignee: BAE Systems Plc
• Original Assignee: BAE Systems Plc
• Inventors: Wright, Julian David, Colosimo, Nicholas Giacomo Robert, Whiteford, Christopher James
• Primary Examiner(s): CRADDOCK, ROBERT J
• Application Number:
  US15/527,988
• Publication Number:
  US 20170330381A1
• Time in Patent Office: 1,064 Days
• CPC Class Codes: B64D 2011/0061 Windows displaying outside view, artificially generated
• CPC Class Codes: B64D 43/00 Arrangements or adaptations of instruments arrangements of cameras B64D47/08; aeronautical measuring instruments per se G01C
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Apparatus for selectively displaying an operational area comprising an internal and an external environment separated from another by at least one physical obstruction, the apparatus comprising a headset including a screen for placing over a user'"'"'s eyes, the system further comprising a processor configured to generate a three-dimensional virtual environment, and an image capture device for capturing images of the internal environment, said processor being configured to blend image data representative thereof into said three dimensional virtual environment to create a mixed reality environment including a representation of said at least one physical obstruction, the processor configured to receive image data representative of said external environment and to remove at least a portion of said physical obstruction from said mixed reality environment displayed on said screen, and blend said image data of said external environment into said mixed reality environment wherein said physical obstruction appears to be transparent.

Claims

1. A system for selectively displaying an operational area, in a real world environment, to a user, said operational area comprising:

an internal environment and an external environment, wherein said external environment is separated from said internal environment by at least one physical obstruction, the system comprising;

a headset for placing over a user'"'"'s eyes, in use, the headset including a screen, the system further comprising;

a processor configured to generate a three-dimensional virtual environment; and

an image capture device for capturing images of the real world environment within said internal environment;

said processor being configured to blend image data representative thereof into said three dimensional virtual environment to create a mixed reality environment, including a representation of said at least one physical obstruction, and display said mixed reality environment on said screen;

the processor including an input for receiving image data representative of images captured by an image capture device having its field of view covering a section of said external environment;

the processor being further configured to remove at least a portion of said at least one obstruction from said mixed reality environment displayed on said screen;

and blending said image data representative of a corresponding portion of said external environment beyond said obstruction into said mixed reality environment wherein said physical obstruction appears to be transparent or translucent relative to said external environment; and

wherein said three dimensional virtual environment is identified in said processor as a first image layer;

said image data representative of said internal environment is identified in said processor as a second image layer; and

said image data representative of said external environment, beyond said physical obstruction, is identified in said processor as a third image layer;

wherein, in respect of a selected portion of said physical obstruction, the processor is configured to;

remove from the mixed reality environment a corresponding portion of said second image layer including image data representative of said physical obstruction; and

blend image data from a corresponding portion, relative to the user'"'"'s field of view, of said third image layer into said mixed reality environment.

2. The system according to claim 1, configured to allow a user, in use, to select said at least a portion of said physical obstruction to be removed from said mixed reality environment.

3. The system according to claim 2, configured to allow a user, in use, to select said at least a portion of said physical obstruction to be removed from said mixed reality environment by means of one or more hand gestures.

4. The system according to claim 3, wherein said one or more hand gestures comprise predefined hand gestures associated with specific control commands.

5. The system according to claim 4, wherein said processor is configured to identify a user hand gesture and its location within the images of said internal environment, determine therefrom the control command with which it is associated, and generate said control command.

6. The system according to claim 3, further comprising a physical control panel including one or more physical control devices which are manually actuatable by the user, wherein said processor is configured to identify, within said captured images, user hand gestures indicative of actuation of said one or more physical control devices and generate a respective control signal for removing a selected portion of said physical obstruction from said mixed reality environment.

7. The system according to claim 1, comprising a pair of spatially separated image capture devices for capturing respective images of the real world environment in the user'"'"'s field of view, said processor being configured to define a depth map using respective image frame pairs to produce three dimensional image data.

8. The system according to claim 7, wherein said image capture devices are mounted on said headset.

9. The system according to claim 8, wherein said image capture devices are mounted on said headset so as to be substantially aligned with a user'"'"'s eyes, in use.

10. The apparatus according to claim 1, wherein the processor is configured to generate information symbols or messages in relation to said external environment, and selectively superimpose image data representative thereof onto said third image layer.

11. A method for selectively displaying an operational area to a user, said operational area comprising:

an internal environment and an external environment;

wherein said external environment is separated from said internal environment by at least one physical obstruction, the method comprising;

providing a system for selectively displaying said operational area, in a real world environment, to said user, said operational area comprising;

a headset for placing over said user'"'"'s eyes, in use, the headset including a screen, the system further comprising;

a processor configured to generate a three-dimensional virtual environment; and

an image capture device for capturing images of the real world environment within said internal environment;

said processor being configured to blend image data representative thereof into said three dimensional virtual environment to create a mixed reality environment, including a representation of said at least one physical obstruction, and display said mixed reality environment on said screen;

the processor including an input for receiving image data representative of images captured by an image capture device having its field of view covering a section of said external environment;

the processor being further configured to remove at least a portion of said at least one obstruction from said mixed reality environment displayed on said screen;

and

receiving, from the image capture device, said images of the real world environment within said internal environment, blending said image data representative thereof into said three dimensional virtual environment to create said mixed reality environment, including said representation of said at least one physical obstruction; and

displaying said mixed reality environment on said screen;

receiving an input representative of a selected portion of said physical obstruction;

removing said selected portion of said at least one obstruction from said mixed reality environment displayed on said screen;

and blending said image data representative of a corresponding portion of said external environment beyond said obstruction into said mixed reality environment wherein said physical obstruction appears to be transparent or translucent relative to said external environment.

12. The method according to claim 11, wherein said internal environment comprises an aircraft cockpit, and said physical obstruction comprises one or more of a control panel, an aircraft side wall and an aircraft floor, the method including the step of providing at least one image capture device in or on the external wall of the aircraft for capturing images representative of said external environment.