INTERACTIVE VEHICLE CONTROL SYSTEM

• US 20180218631A1
• Filed: 11/11/2015
• Published: 08/02/2018
• Est. Priority Date: 11/19/2014
• Status: Abandoned Application

• Current Assignee: BAE Systems Plc
• Original Assignee: BAE Systems Plc
• Inventors: Wright, Julian David, Colosimo, Nicholas Giacomo Robert, Whiteford, Christopher James
• Application Number:
  US15/527,892
• Publication Number:
  US 20180218631A1
• Time in Patent Office: Days
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 9/307 by helmet-mounted projector or display G09B9/36 takes precedence; optical parts of head-up displays G02B27/01

Abstract

A mixed reality vehicle control system comprising a headset (100) including a screen (102), the system further comprising a processor (104) configured to display images representing virtual control elements (30) within a three dimensional virtual environment on said screen, wherein the system is configured to allow a user to interact with said virtual control elements (30) to control respective vehicle functions or operations, the system further comprising an image capture device for capturing images of the real world environment in the vicinity of the user within the user'"'"'s field of view, including image data representative of physical control elements (70, 80) therein, and blend image data representative of at least portions of said user'"'"'s field of view, including at least one of said physical control elements, into said three dimensional virtual environment to create a mixed reality vehicle control environment.

Claims

1. A mixed reality vehicle control system for enabling monitoring and/or control within a vehicle of functions and/or operations thereof by a user, the system comprising:

a headset including a screen, the system further comprising;

a processor configured to receive data from one or more sources within said vehicle and display images representing virtual control elements in respect of said vehicle, together with said data, within a three dimensional virtual environment on said screen;

wherein the system is configured to allow said user, in use, to interact with and/or manipulate said virtual control elements;

the processor being further configured, in response to such user interaction or manipulation, transmit control data to respective vehicle functions or operations for control thereof;

the system further comprising an image capture device for capturing images of the real world environment in the vicinity of the user within the user'"'"'s field of view, including image data representative of physical control elements therein, and blend image data representative of at least portions of said user'"'"'s field of view, including at least one of said physical control elements, into said three dimensional virtual environment to create a mixed reality vehicle control environment.

3. The system according to claim 1, wherein said processor is preconfigured to identify within said captured images at least one predefined physical control element in the real world within said vehicle, and blend image data representative thereof into said three dimensional virtual environment.

4. The system according to claim 1, configured to allow the user, in use, to manipulate data and/or interact with said virtual control elements by means of hand gestures.

5. The system according to claim 4, further comprising a physical control panel including one or more physical control devices which are manually actuatable by a user, wherein said processor is configured to identify, within said captured images, user hand gestures indicative of actuation of said one or more physical control devices and generate a respective control signal for controlling a function and/or operation of said vehicle.

6. The system according to claim 1, comprising a pair of spatially separated image capture devices for capturing respective images of the real world environment in the user'"'"'s field of view, said processor being configured to define a depth map using respective image frame pairs to produce three dimensional image data.

7. The system according to claim 6, wherein said image capture devices are mounted on said headset.

8. The system according to claim 7, wherein said image capture devices are mounted on said headset so as to be substantially aligned with a user'"'"'s eyes, in use.

9. The system according to claim 1, wherein the processor is configured to generate information symbols or messages in relation to real world objects identified within said captured images, and blend image data representative thereof into said three dimensional virtual environment at an associated location therein.

10. A method of providing a vehicle control station enabling monitoring and/or control within a vehicle of functions and/or operations thereof, the method comprising:

providing a mixed reality system according to claim 1, and configuring the processor to

receive data from one or more sources within said vehicle and display images representing virtual control elements in respect of said vehicle, together with said data, within a three dimensional virtual environment on said screen, wherein the system is configured to allow said user, in use, to interact with and/or manipulate said virtual control elements;

in response to such user interaction or manipulation, transmit control data to respective vehicle functions or operations for control thereof, the system further comprising an image capture device for capturing images of the real world environment in the vicinity of the user within the user'"'"'s field of view, including image data representative of physical control elements therein; and

blend image data representative of at least portions of said user'"'"'s field of view, including at least one of said physical control elements, into said three dimensional virtual environment to create a mixed reality vehicle control environment.

12. The method according to claim 10, wherein the vehicle control station is an aircraft cockpit comprising a plurality of control elements.

13. The method according to claim 12, including the steps of providing a vehicle cab or cockpit structure including only a selected number of said control elements as physical control elements, providing the remaining control elements as virtual control elements within said three dimensional virtual environment, and configuring the processor to blend image data representative of a user'"'"'s field of view, from said captured images, into said three dimensional virtual environment to create a mixed reality environment.

2. (canceled)

11. (canceled)