Integrated real and virtual environment display system

• US 5,394,517 A
• Filed: 10/14/1992
• Issued: 02/28/1995
• Est. Priority Date: 10/12/1991
• Status: Expired due to Fees

• Current Assignee: British Aerospace, Inc. (BAE Systems Plc)
• Original Assignee: British Aerospace PLC (BAE Systems Plc)
• Inventors: Kalawsky, Roy S.
• Primary Examiner(s): Powell, Mark R.
• Assistant Examiner(s): HUYNH, BA
• Application Number:
  US07/959,919
• Time in Patent Office: 867 Days
• Field of Search: 395/129, 395/123, 395/125, 395/126, 395/131, 395/135, 395/155-161
• US Class Current: 345/632
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/0118 comprising devices for improving the contrast of the display / brillance control visibility
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 2027/0198 System for aligning or maintaining alignment of an image in a predetermined direction
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G09B 5/02 with visual presentation of the material to be studied, e.g. using film strip
• CPC Class Codes: G09B 9/302 the image being transformed by computer processing, e.g. updating the image to correspond to the changing point of view
• CPC Class Codes: G09B 9/307 by helmet-mounted projector or display G09B9/36 takes precedence; optical parts of head-up displays G02B27/01
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/286 having separate monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/324 Colour aspects
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 9/75 Chroma key

Abstract

This invention relates to improvements in so-called "virtual reality" computer systems. The display system described enables the effective integration of computer generated images and real, naturally occurring images in the visual display units which provide the user with his entire visual stimulation. Areas of the user'"'"'s notional field of view where computer-generated imagery is required (for example the cockpit canopy in an aircraft flight simulator) are coloured with a predetermined colour or intensity. Two video cameras are provided, each of which is mounted so that its field of view corresponds to a respective one of the notional fields of view of the user'"'"'s eyes. The signals from the video cameras are analysed to determine the presence of the predetermined colour or intensity, thereby giving an indication that a computer generated image is required for that part of the video camera'"'"'s field of view. If the predetermined colour or intensity is not present, the signal from the video camera is relayed directly to the appropriate one of the visual display units.

Claims

1. A display system for presenting a composite image to a user'"'"'s eyes, said image comprised of a real portion provided by a video camera means and a complementary computer-generated synthetic portion said system including stored reference data indicative of the real portion and the synthetic portion, said user having a field of view in a direction, the system comprising:

visual display means for presenting the composite image;

direction sensor means for producing a direction signal indicative of the direction of the user'"'"'s field of view;

video camera means for producing a sequence of video signals which in combination are representative of a field of view thereof;

camera mounting means for mounting said video camera means such that the field of view of the video camera means is substantially coincident with and follows movement of the field of view of the user;

image generating means responsive to said direction signal for producing synthetic signals representing a computer-generated portion of the composite image; and

,

image selecting and combining means, responsive to said video camera means, to said image generating means and to said direction sensor means and providing an output to said visual display means, for analyzing each of said video signals and determining, by comparison with said stored reference data, whether each video signal is indicative of an area of the field of view of the video camera means which said computer-generated portion is to occupy and for correspondingly selecting one of said video signals and said synthetic signals, and for combining said video and synthetic signals to form a composite video signal representing the composite image and for applying said composite video to said visual display means for presentation of the composite image.

2. A display system according to claim 1, wherein said camera mounting means comprises means for mounting said video camera means on the user'"'"'s head for movement therewith.

3. A display system according to claim 1, wherein said camera mounting means comprises moveable mounting means for mounting said video camera for movement in response to said direction signal.

4. A display system according to claim 1 wherein the stored reference data relates to a predetermined colour and said image selecting and combining means comprises means for analyzing each of said video signals by determining the colour thereof and comparing the colour with the predetermined colour and for selecting said synthetic signals whenever the colours match.

5. A display system according to claim 2 wherein the stored reference data relates to a predetermined colour and said image selecting and combining means comprises means for analyzing each of said video signals by determining the colour thereof and comparing the colour with the predetermined colour and for selecting said synthetic signals whenever the colours match.

6. A display system according to claim 3 wherein the stored reference data relates to a predetermined colour and said image selecting and combining means comprises means for analyzing each of said video signals by determining the colour thereof and comparing the colour with the predetermined colour and for selecting said synthetic signals whenever the colours match.

7. A display system according to claim 1, wherein said user has two eyes, each eye having a separate field of view, said visual display means comprises two visual displays which present different composite images to respective ones of the user'"'"'s eyes;

said video camera means comprises two video cameras each of which have a field of view which is substantially coincident with the fields of view of respective ones of the user'"'"'s eyes;

said image generating means produces two sets of synthetic signals representing the computer-generated portions; and

, said image selecting and combining means includes means for analyzing the video signals from respective ones of said two video cameras and produces two different composite images which, when presented to respective ones of the user'"'"'s eyes, are perceived stereoscopically by said user.

8. A display system according to claim 1 wherein the stored reference data relates to a predetermined intensity and said image selecting and combining means includes means for analyzing each of said video signals by determining the intensity thereof and comparing the intensity with a predetermined intensity and selecting said synthetic signal whenever one of the following conditions apply (1) the intensities match and (2) one intensity exceeds the other intensity according to a predetermined rule.

9. A display system according to claim 2 wherein the stored reference data relates to a predetermined intensity and said image selecting and combining means includes means for analyzing each of said video signals by determining the intensity thereof and comparing the intensity with a predetermined intensity and selecting said synthetic signals whenever one of the following conditions apply (1) the intensities match and (2) one intensity exceeds the other intensity according to a predetermined rule.

10. A display system according to claim 3 wherein the stored reference data relates to a predetermined intensity and said image selecting and combining means includes means for analyzing each of said video signals by determining the intensity thereof and comparing the intensity with a predetermined intensity and selecting said synthetic signals whenever one of the following conditions apply (1) the intensities match and (2) one intensity exceeds the other intensity according to a predetermined rule.

11. A display system according to claim 1, wherein said visual display unit means provides the only visual stimulation for the user.