Interactive control station

• US 10,262,465 B2
• Filed: 11/09/2015
• Issued: 04/16/2019
• Est. Priority Date: 11/19/2014
• Status: Active Grant

• Current Assignee: BAE Systems Plc
• Original Assignee: BAE Systems Plc
• Inventors: Wright, Julian David, Colosimo, Nicholas Giacomo Robert, Whiteford, Christopher James, Page, Heather Jean, Goodall, Mark Robert
• Primary Examiner(s): Craddock, Robert J
• Application Number:
  US15/527,817
• Publication Number:
  US 20170316613A1
• Time in Patent Office: 1,254 Days
• Field of Search: 345633
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A mixed reality system for creating a terminal control station enabling communication with and/or control of remote functions and applications, the system comprising a headset (100) for placing over a user'"'"'s eyes, in use, said headset including a screen (102), the system further comprising a processor (104) configured to receive data from multiple sources and display said data on said screen within a three-dimensional virtual environment, and an input for receiving control information data from an external data source within the real world environment, said processor (104) being further configured to receive image data representative of said external data source and blend said image data into said three-dimensional virtual environment to create a mixed reality environment, including a representation of said external data source and said control information data, to be displayed on said screen (102), the system being configured to allow a user, in use, to manipulate data displayed within said mixed reality environment.

Claims

1. A mixed reality system for creating a terminal control station enabling communication with and/or control of remote functions and applications, the system comprising:

a headset for placing over a user'"'"'s eyes, in use, said headset including a screen, the system further comprising;

a processor configured to receive data from multiple sources and display said data on said screen within a three-dimensional virtual environment;

a pair of spatially separated image capture devices for capturing respective images of a real world environment in a vicinity of the user; and

an input for receiving control information data from an external data source within the real world environment;

applying a threshold function to extract said image data from a background image; and

whereby all computational steps require minimal processing and time and are performed in at least near real time whereby selected moving object image data within said virtual environment is updated in at least near real time;

said processor being further configured to receive image data representative of said external data source and blend said image data into said three-dimensional virtual environment to create a mixed reality environment, including a representation of said external data source and said control information data, to be displayed on said screen, the system being configured to allow a user, in use, to manipulate data displayed within said mixed reality environment.

2. The system according to claim 1, wherein the terminal control station comprises physical components associated therewith, and said processor is preconfigured to display at least one physical component of said terminal control station, and data associated therewith, on said screen within said three-dimensional virtual environment.

3. The system according to claim 2, wherein said preconfigured display of said at least one physical component of said terminal control station comprises a critical function physical component of said terminal control station.

4. The system according to claim 2, wherein said preconfigured display of said at least one physical component of said terminal control station comprises a critical safety function physical component of said terminal control station.

5. The system according to claim 2, wherein said preconfigured display of said at least one physical component of said terminal control station comprises a critical safety function physical component of said terminal control station required to be manipulated in physical form.

6. The system according to claim 1, configured to allow a user, in use, to manipulate data displayed within said mixed reality environment by means of hand gestures.

7. The system according to claim 6, further comprising a control panel or keyboard in respect of which a user can, in use, manually indicate data selection and manipulation commands, wherein said processor is configured to identify from said captured images said data selection and manipulation commands and generate control signals representative thereof.

8. The system according to claim 1, wherein said processor is preconfigured to identify, from said captured images, at least one physical component of said terminal control station within said real world environment, and blend image data representative of said physical component into said three-dimensional virtual environment.

9. The system according to claim 1, said processor being further configured to define a depth map using respective image frame pairs to produce three dimensional image data.

10. The system according to claim 9, wherein said image capture devices are mounted on said headset.

11. The system according to claim 10, wherein said image capture devices are mounted on said headset so as to be substantially aligned with a user'"'"'s eyes, in use.

12. The system according to claim 1, configured to allow a user, in use, to select objects, portions or people from their real world environment to be included within said mixed reality environment by means of one or more predefined hand gestures.

13. A method of providing a terminal control station enabling communication with and/or control of remote functions and applications, the method comprising:

providing a mixed reality system according to claim 1;

configuring the processor to receive data from multiple sources and display said data on said screen within a three-dimensional virtual environment;

receiving control information data from an external data source within the real world environment;

receiving image data representative of said external data source; and

blending said image data into said three-dimensional virtual environment to create a mixed reality environment, including a representation of said external data source and said control information data, to be displayed on said screen.

14. The method according to claim 13, wherein said terminal control station is an aircraft or Unmanned Aerial Vehicle ground station, and the method includes the step of configuring the processor to receive image data representative of the real world environment, identify, from said image data, at least one physical component of said terminal control station within said real world environment, and blend image data representative of said physical component into said three-dimensional virtual environment.

15. The system according to claim 1, wherein said blending of said image data comprises:

marker data whereby relative location and orientation are extracted and preserved; and

whereby all computational steps require minimal processing and time and are performed in at least near real time whereby selected moving object image data within said virtual environment is updated in at least near real time.

16. The system according to claim 1, wherein said blending of said image data comprises:

marker data whereby relative location and orientation are extracted and preserved;

adaptive thresholding whereby said image data and said marker data is converted to a binary image; and

transforming said marker data and said binary image into a set of coordinates which match a location within said three-dimensional virtual environment;

whereby all computational steps require minimal processing and time and are performed in at least near real time whereby selected moving object image data within said virtual environment is updated in at least near real time.

17. The system according to claim 1, wherein said blending of said image data comprises:

using black and white image data; and

wherein colour data sampled from said source image is backward warped using homography, to each pixel in a resultant virtual scene;

whereby all computational steps require minimal processing and time and are performed in at least near real time whereby selected moving object image data within said virtual environment is updated in at least near real time.

18. The system according to claim 1, wherein said terminal control station is preconfigured to constrain said user from configuring said terminal control station for at least one function.

19. A mixed reality Unmanned Aerial Vehicle (UAV) terminal control ground station enabling communication with and/or control of remote functions and applications, the station comprising:

a headset for placing over a user'"'"'s eyes, in use, said headset including a screen, the system further comprising;

a processor configured to receive data from multiple sources and display said data on said screen within a three-dimensional virtual environment;

a pair of spatially separated image capture devices for capturing respective images of a real world environment in a vicinity of the user; and

an input for receiving control information data from an external data source within the real world environment;

said processor being further configured to receive image data representative of said external data source and blend said image data into said three-dimensional virtual environment to create a mixed reality environment, including a representation of said external data source and said control information data, to be displayed on said screen, the system being configured to allow a user, in use, to manipulate data displayed within said mixed reality environment;

wherein said blending of said image data comprises;

applying a threshold function to extract said image data from a background image;

marker data whereby relative location and orientation are extracted and preserved;

adaptive thresholding whereby said image data and said marker data is converted to a binary image;

transforming said marker data and said binary image into a set of coordinates which match a location within the virtual environment;

using black and white image data; and

wherein colour data sampled from said source image is backward warped using homography, to each pixel in a resultant virtual scene;

whereby all computational steps require minimal processing and time and are performed in at least near real time whereby selected moving object image data within said virtual environment is updated in at least near real time;

wherein said preconfigured display of said at least one physical component of said terminal control station comprises a critical safety function physical component of said terminal control station required to be manipulated in physical form;

wherein preconfiguration of said preconfigured display of said critical safety function physical component of said terminal control station constrains said user from configuring said preconfigured display of said critical safety function physical component;

whereby said UAV is remotely controlled from said ground station.