Simulated head mounted display system and method

• US 8,704,882 B2
• Filed: 11/18/2011
• Issued: 04/22/2014
• Est. Priority Date: 11/18/2011
• Status: Expired due to Fees

• Current Assignee: L-3 Communications Corporation (L3Harris Technologies, Inc.)
• Original Assignee: L-3 Communications Corporation (L3Harris Technologies, Inc.)
• Inventors: Turner, James A.
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Owens, Tsion B
• Application Number:
  US13/300,272
• Publication Number:
  US 20130128012A1
• Time in Patent Office: 886 Days
• Field of Search: 348/53, 345/8, 434/69
• US Class Current: 348/53
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/363 using image projection screens volumetric display H04N13/388
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

To overcome problems with vergence, a binocular head mounted display (HMD) is used in a simulator in which an out-the-window scene is displayed in real time on a screen arrangement. Imagery for the left and right eyes of the HMD is derived by generating a starting HMD image for a Cyclops viewpoint between the user'"'"'s eves, and then rendering respective views for each eve from the position of the eye in a virtual 3D model of the screen arrangement, wherein the starting HMD image is frustum projected against the screen arrangement of the 3D model.

Claims

1. A simulator for training in simulation to a user, said simulator comprising:

a display screen arrangement supported so as to be viewed by the user;

a simulation computer system having computer accessible data storage storing scene data defining a virtual environment of the simulation and virtual objects therein, said simulation computer system generating out-the-window (OTW) scenery video from the scene data, and causing the OTW scenery video to be displayed on the display screen arrangement in real time as OTW scenery visible to the user;

a tracking apparatus continually sensing a position of the head of the user or a portion thereof in real time and generating from said sensing a tracking signal containing tracking data from which current time-varying locations of the eyes of the user can be determined, and transmitting said tracking signal to the simulation computer system;

a head-mounted display (HMD) configured to be supported on the head of the user, said HMD having an eye display over one of the eyes of the user configured to display to said eye of the user HMD imagery video so that an eye of the user sees the OTW scenery video on the screen arrangement there through with the HMD imagery video superimposed thereon;

said simulation computer system supporting a three-dimensional model of the simulator including data defining a virtual screen arrangement corresponding to the display screen arrangement in the simulator;

the simulator computer system generating said HMD imagery video as a sequence of HMD images each corresponding to a respective point in time by rendering each frame by first generating a starting HMD image corresponding to HMD imagery for a predetermined projection point that is fixed relative to the position of the head of the user based on the data of the tracking signal, and then rendering the HMD image as a view from the location of said eye in the three-dimensional model of the virtual screen arrangement with said starting HMD image projected there against from the projection point.

2. The simulator of claim 1, wherein the HMD has a second eye display over the other of the eyes of the user configured to display to said other eye of the user second HMD imagery video so that the other eye of the user sees the OTW scenery video on the screen arrangement therethrough with the second HMD imagery video superimposed thereon;

the simulator computer system generating said second HMD imagery video as a second sequence of HMD images each corresponding to a respective point in time by rendering each frame by rendering the HMD image as a view from the location of said other eye in the three-dimensional model of the virtual screen arrangement with said starting HMD image projected thereagainst from the projection point.

3. The simulator of claim 2, wherein the eye displays of the HMD are part of a visor supported on a helmet configured to be worn by a user, said eye displays each including

a respective mirror member supported in front of the associated eye of the user and through which the user can see the display screen arrangement, and

a respective image display device that is reflected in said mirror member so as to be seen by the associated eye of the user, each of said image display devices comprising an LCD or LED producing a visible image in real time that is seen reflected in the associated mirror member in focus with and superimposed over the out-the-window scene.

4. The simulator of claim 1, wherein the display screen arrangement comprises a rear-projection screen.

5. The simulator of claim 1, wherein the starting HMD images are defined by data that include a plurality of straight segments in the starting HMD image that are each defined by a respective pair of endpoints in a two-dimensional reference frame of the starting HMD image, said rendering including projecting said endpoints onto the virtual display screens in the model and then drawing a straight line therebetween.

6. The simulator of claim 5, wherein the display screen arrangement comprises a plurality of substantially planar display screens supported contiguous to and angulated with respect to one another, and the three-dimensional model of the simulator includes data defining the virtual screen arrangement corresponding to positions of said screens, and

said rendering includes identifying one of the straight segments that has endpoints on different display screens, and projecting said straight segment onto the virtual screen arrangement by dividing the segment into a segment portion on each of the display screens involved defined by two endpoints both of which are on the same display screen in the model.

7. The simulator of claim 1, wherein the starting HMD images are rasterized images that are projected as texture onto the virtual display screen arrangement of the model as a frustum projection with an apex at the predetermined projection point.

8. The simulator of claim 7, wherein the starting HMD images further include data defining straight-line symbology segments by data values defining endpoints thereof in the rasterized images.

9. The simulator of claim 7, wherein the HMD video imagery includes simulated FLIR imagery.

10. The simulator of claim 1, wherein the display screen arrangement has a non-planar three-dimensional shape and the three-dimensional model of the simulator includes data defining the virtual screen arrangement as a virtual surface corresponding to the non-planar three-dimensional shape.

11. The simulator of claim 1, wherein the tracking apparatus includes an apparatus supported on the head of a user detecting the position of the head of the user and comprising magnetic sensors, sonic sensors, or optical sensors operatively associated therewith, or a mechanical coupling with the apparatus on the head of the user.

12. The simulator of claim 1, wherein the tracking apparatus generates a signal containing data including a position of the head of the user and an angular orientation thereof.

13. The simulator of claim 1, wherein the predetermined projection point is a Cyclops point midway between the locations of the eyes of the user as determined from the tracking signal.

14. A method for simulating a vehicle for a user, said method comprising:

providing a simulation station simulating the vehicle and in which the user is accommodated during simulation, said simulation station including a display screen supported fixedly so as to be viewable to the user in the simulation and an HMD supported on the head of the user and having left and right eye displays through which the user can view the display screen over the left and right eyes of the user;

displaying OTW imagery video in real time on the display screen, said OTW imagery video including rendered images of virtual objects in a virtual environment in which the simulation takes place as defined by scene data stored in a computer accessible memory and managed by a computer system administering the simulation;

continually detecting positional information indicative of positions of the eyes of the user and transmitting a data signal containing said positional information to the computer system;

generating in real time with said computer system left- and right-eye HMD imagery video comprising a series of left- and right-eye frames that are displayed on respective eye displays so that the user sees the HMD imagery superimposed over the OTW scenery;

said generating of each of said frames comprising

deriving a starting HMD image as seen from a predetermined projection point fixedly located relative to the head of the user, and

deriving left- and right-eye HMD images that corresponds to a rendered view of a computer-supported three-dimensional model of the display screen as viewed from a viewpoint in the model corresponding to a current determined position of the left or the right eye respectively, wherein the starting image is projected onto the model; and

displaying the left- and right-eye HMD imagery video on the left and right eye displays respectively.

15. The method of claim 14, wherein a head tracker transmits information for the user'"'"'s right and left eye positions to a computer system.

16. The method of claim 15, wherein the predetermined projection point is located midway between the right and left eyes of the user.

17. The method of claim 14, wherein said computer system calculates multiple frames per second so that a user viewing images in the vehicle simulator does not experience any hesitation or visual distortions while viewing the images.

18. The method of claim 14, wherein the starting HMD image comprises computer-stored data corresponding to a raster image field.

19. The method of claim 14, wherein the starting HMD image comprises one or more data sets each defining a respective pair of endpoints of a segment of the starting HMD image, and wherein the deriving of the left and right eye HMD images includes locating the endpoints on the model of the display screen and rendering the left and right eye HMD images with a line therebetween.

20. The method of claim 14, wherein the display screen comprises at least two screen facets supported angled with respect to each other,

wherein the starting HMD image comprises one or more data sets each defining a respective pair of endpoints of a segment of the starting HMD image, and

wherein the deriving of the left and right eye HMD images includes locating the endpoints on respective screen facets of the model of the display screen,

determining a midpoint where the screen facets meet that is coplanar with the endpoints and the left or right eyepoint, and

rendering the left and right eye HMD images with a line segment between one of the endpoints and the midpoint and a line segment between the midpoint and the other of the endpoints.